Curb detection method and device, equipment and storage medium

Abstract

The invention relates to the field of image processing, and provides a curb detection method and device, equipment and a storage medium. The method comprises the following steps: acquiring target environment image data acquired by a target camera in an automatic driving vehicle, wherein the target camera is one of cameras on the vehicle; extracting intersection points of view rays in the target environment image data based on the view rays of the target camera; based on a preset shielding identification rule, identifying whether the extracted view ray corresponding to each intersection point is shielded or not; extracting a curb contour based on all intersection points in the environment image data of other cameras and the target environment image data by using a first curb detection algorithm when the target environment image data is shielded; and extracting a curb contour from the environment image data by using a second curb detection algorithm when the environment image data is not shielded. According to the method, different curb detection algorithms are set for curb recognition of ray shielding and ray non-shielding scenes, so that the problem that curb detection cannot be realized in a shielding scene in an existing curb detection scheme is solved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to a curb detection method, device, equipment and storage medium. Background Art

[0002] Autonomous vehicles can provide higher safety, productivity and traffic rate, and will play an important role in the future urban transportation system. In most autonomous driving or assisted driving scenarios, the perception of the surrounding environment is a crucial task. For example, curb detection is an indispensable function in realizing autonomous driving.

[0003] Traditional curb detection methods usually rely on complex image processing algorithms, such as deep learning and image segmentation. Although these methods can achieve high accuracy, they cannot detect the initial curb in complex scenes or in scenes with obstacles and occlusions. Summary of the invention

[0004] In view of this, the purpose of the present disclosure is to provide a curb detection method, device, equipment and storage medium to solve the problem that existing curb detection solutions cannot achieve curb detection in the presence of occlusion scenarios.

[0005] In a first aspect, an embodiment of the present disclosure provides a curb detection method, which is applied to an autonomous driving vehicle, wherein the autonomous driving vehicle is provided with at least two cameras; the method comprises: obtaining target environment image data collected by a target camera in the autonomous driving vehicle, wherein the target camera is one of the at least two cameras; based on the field of view rays of the target camera, extracting the intersection points of each field of view rays in the target environment image data; based on a preset occlusion recognition rule, identifying whether the field of view rays corresponding to each extracted intersection point are blocked; if blocked, using a first curb detection algorithm to extract a curb contour based on the environment image data of other cameras and all intersection points in the target environment image data; if not blocked, using a second curb detection algorithm to extract the curb contour from the target environment image data.

[0006] Optionally, the use of the first curb detection algorithm to generate a curb contour based on environmental image data from other cameras and all intersections in the target environmental image data includes: using the first curb detection algorithm to extract target intersections generated by blocking rays from all intersections, and determining the position and direction of the target intersection in the target environmental image data; calculating the direction of the blocked curb based on the position information and the position and direction of the target camera; reading environmental image data from other cameras and extracting curb information based on the curb direction; and generating a curb contour based on the curb information and intersections other than the target intersection in the target environmental image data.

[0007] Optionally, the use of a second curb detection algorithm to extract a curb contour from the target environmental image data includes: determining a type of the environmental image data, and determining a target second curb detection algorithm based on the type; using the target second curb detection algorithm to extract a potential curb area in the environmental image data; using an edge detection algorithm to extract edge information in the potential curb area, and filtering out curb information that meets curb determination conditions based on the position information of the autonomous driving vehicle to obtain a curb contour.

[0008] Optionally, the target second curb detection algorithm is a color segmentation method; the method of using the target second curb detection algorithm to extract the potential curb area in the environmental image data includes: performing color space conversion on the environmental image data to obtain an image to be processed in an HSV color space; in the HSV color space, separating the curb from the background in the image to be processed according to a preset threshold range, and extracting the potential curb area from the separated image to be processed, wherein the threshold range includes: a first range of hue H, a second range of saturation S and a third range of brightness V.

[0009] Optionally, the extracting of potential curb areas from the separated image to be processed includes: performing edge enhancement on the separated image to be processed using a Canny edge detection algorithm; extracting potential curb line segments from the edge-enhanced image to be processed using a Hough transform detection method; determining specific positions of the potential curb line segments using a geometric analysis method, and calculating distances and combination forms between potential curb lines to obtain potential curb areas.

[0010] Optionally, after extracting the curb profile, the method further includes: fitting each curve segment in the curb profile using a polynomial curve fitting method, and outputting a final curb profile.

[0011] Optionally, after extracting the curb contour, it also includes: calculating the spatial coordinate information of each curb point in the curb contour; calling the point cloud map, and extracting the point cloud image located at the spatial coordinate information in the point cloud map; optimizing and completing the curb contour based on the point cloud image to obtain a complete curb contour.

[0012] In a second aspect, an embodiment of the present disclosure provides a curb detection device, which is applied to an autonomous driving vehicle, wherein the autonomous driving vehicle is provided with at least two cameras; the device comprises: an acquisition module, for acquiring target environment image data collected by a target camera in the autonomous driving vehicle, wherein the target camera is one of the at least two cameras; an extraction module, for extracting the intersection points of each field of view ray in the target environment image data based on the field of view ray of the target camera; an identification module, for identifying whether the field of view ray corresponding to each extracted intersection point is blocked based on a preset occlusion identification rule; a detection module, for extracting a curb contour based on the environment image data of other cameras and all intersection points in the target environment image data using a first curb detection algorithm when identifying occlusion; and for extracting a curb contour from the target environment image data using a second curb detection algorithm when identifying no occlusion.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the curb detection method provided above.

[0014] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the curb detection method provided above.

[0015] The embodiments of the present disclosure bring the following beneficial effects:

[0016] The above-mentioned curb detection method, device, equipment and storage medium successfully solve the problem of the prior art that there is no curb detection in the blocked scene by setting the first curb detection algorithm and the second curb detection algorithm and selecting the corresponding algorithm according to whether the ray is blocked. In addition, the two algorithms set in this application can simplify the complexity of the detection algorithm of the prior art to a certain extent and provide real-time performance. In particular, when the ray is not blocked, the first curb detection algorithm is used for curb detection, which also improves the efficiency of the system and ensures the accuracy of the detection results. It is suitable for different autonomous driving scenarios and has broad application prospects.

[0017] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or understood by practicing the present disclosure. The purpose and other advantages of the present disclosure are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of an embodiment of a curb detection method provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of another embodiment of the curb detection method provided by the embodiment of the present disclosure;

[0022] Figure 3 The perspective effect diagram of the four cameras provided in the embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of a first embodiment of a curb detection device provided in an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a second embodiment of a curb detection device provided in an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of this embodiment clearer, the technical solution of this disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of this disclosure, rather than all the embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this disclosure.

[0027] This embodiment provides a curb detection method, device, equipment and storage medium, which can be applied to video retrieval scenarios in any field, especially obstacle-prone autonomous driving scenarios in the field of autonomous driving.

[0028] The curb detection method in one embodiment of the present disclosure may be run on a terminal device or a server. The terminal device may be a local terminal device. When the curb detection method is run on a server, the method may be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device (vehicle-mounted terminal).

[0029] For ease of understanding, the specific process of this embodiment is described below. Figure 1 In this embodiment, a method for detecting a curb is provided. The method is applied to an autonomous driving vehicle. The vehicle is provided with at least two cameras, which are respectively distributed around the vehicle to achieve a full field of view. The method specifically includes the following steps:

[0030] 101, obtaining target environment image data collected by a target camera in the autonomous driving vehicle, where the target camera is one of at least two cameras;

[0031] As an example but not a limitation, the target camera may be a main camera configured in an autonomous vehicle, such as a camera set on the left and right sides of the vehicle, and then the target camera captures the image data in front of it in real time during the vehicle's driving. In actual applications, the target camera may also be set to multiple, and the camera may be set in which position range of the vehicle according to the vehicle's driving position.

[0032] In another embodiment, when acquiring the target environment image data, the target camera is first determined, and the process of determining the target camera includes: based on the camera that first recognizes the curb after the vehicle is started, the position of the target camera in the vehicle is determined, and based on the position, other cameras located in the same direction are selected as target cameras, and each target camera is controlled to continuously collect the corresponding target environment image data.

[0033] 102, extracting the intersection points of each field of view ray in the target environment image data based on the field of view ray of the target camera;

[0034] It should be noted that the field of view ray refers to the light reflected to the target camera by objects such as obstacles and curbs within the maximum shooting field of view of the target camera.

[0035] Image processing technology is used to extract the intersection of light and objects in the target environment image data. Specifically, based on the light, the target camera is used as the emission point to simulate the ray and map it to the target environment image data; then a geometric model is built based on the ray, the trajectory of the ray is calculated and collision detection is performed with obstacles in the target environment image data; the intersection is determined based on the result of the collision detection.

[0036] 103, based on a preset occlusion recognition rule, identifying whether the visual field ray corresponding to each extracted intersection point is blocked;

[0037] As an example but not a limitation, whether the intersection is blocked can be determined by identifying the positional relationship between the intersections. The positional relationship can be a depth of field relationship in the image or the type of obstacle corresponding to the intersection.

[0038] When the depth of field relationship is used to implement the method, a three-dimensional coordinate system is constructed based on the image in the target environment image data; the intersection of the ray and the obstacle in the target environment image data is determined, and the depth of field of the intersection is determined based on the three-dimensional coordinate system; the depth of field of each intersection is compared to obtain the recognition result of whether the intersection is blocked. For example, the intersection with the largest depth of field is first determined as the intersection of the curb to obtain the standard depth of field, and then the depth of field of other intersections is compared with the standard depth of field to determine the intersection caused by the ray being blocked.

[0039] When the obstacle type corresponding to the intersection is used for implementation, the intersections are connected to form an obstacle outline; the obstacle outline is segmented into multiple small obstacle blocks using the adjacency segmentation method; each obstacle block is matched with the preset outline features of each obstacle, such as similarity matching, to obtain the obstacle type, and whether the field of view ray corresponding to the intersection is blocked is determined based on the obstacle type.

[0040] 104, if blocked, using a first curb detection algorithm to extract a curb contour based on all intersections in the environment image data of other cameras and the target environment image data;

[0041] It should be noted that the detection using the first curb detection algorithm here can be implemented using existing curb detection algorithms, such as deep learning, detection models, etc.; it can also be implemented using a camera collaborative detection method. For the camera collaborative detection method, specifically, the first curb detection algorithm is used to extract the target intersection point generated by the blocked ray from all intersection points, and determine the position direction of the target intersection point in the target environmental image data; based on the position information and the position direction of the target camera, the direction of the blocked curb is calculated; the environmental image data in other cameras is read, and the curb information is extracted based on the curb direction; based on the curb information and the intersection points other than the target intersection point in the target environmental image data, a curb contour is generated.

[0042] It can be understood that the target obstacle is determined based on the obstacle type corresponding to the intersection, and then the position information of the target intersection in the target environmental image data is calculated, and the obstructed line of sight direction is determined in combination with the position of the target camera relative to the vehicle; then the environmental image data is obtained from other target cameras, and the line of sight direction is determined to determine whether it is also obstructed in the environmental image data. If it is not obstructed, the information therein is extracted and merged into the intersection of the curb extracted from the target environmental image data, thereby obtaining a complete curb outline.

[0043] 105. If it is not blocked, a second curb detection algorithm is used to extract a curb contour from the target environment image data.

[0044] In this embodiment, the second curb detection algorithm can be set in multiple forms, corresponding to image data, point cloud data, etc., and the corresponding target curb detection algorithm is matched according to the source of the image, and then the features of the target environment image data are extracted; the extracted features are matched with the preset curb features to obtain the matched feature positions, and the matched feature positions are merged and concatenated to form a curb contour.

[0045] The above curb detection method obtains the target environment image data collected by the target camera in the autonomous driving vehicle, and the target camera is one of the cameras on the vehicle; based on the field of view rays of the target camera, the intersection points of each field of view ray in the target environment image data are extracted; based on the preset occlusion recognition rules, whether the field of view rays corresponding to each extracted intersection point are blocked is identified; when blocked, the first curb detection algorithm is used to extract the curb contour based on the environment image data of other cameras and all the intersection points in the target environment image data; when not blocked, the second curb detection algorithm is used to extract the curb contour from the environment image data. By identifying occlusion and non-occlusion to select the corresponding curb detection algorithm for curb detection, the curb detection of the blocked scene is realized, and at the same time, the appropriate curb detection algorithm is selected, which reduces the computational complexity and improves the system efficiency.

[0046] See also Figure 2 Another embodiment of the curb detection method in this embodiment is described by taking radar as a camera as an example. The complexity of the first curb detection algorithm in the method is greater than that of the second curb detection algorithm. Specifically, the following steps are included:

[0047] 201. Obtain target environment image data collected by a target camera in the autonomous driving vehicle, where the target camera is one of at least two cameras;

[0048] In this embodiment, the autonomous driving vehicle is equipped with at least two cameras. These cameras are used to collect environmental image data around the vehicle, and at least one of the cameras is selected as a target camera. In the process of acquiring target environmental image data, each camera has an independent field of view and can provide target environmental image data in real time according to the driving state of the vehicle.

[0049] Target camera: Among at least two cameras, one is selected as the target camera to obtain the target environment image data under the camera's field of view. The data of the target camera is used for subsequent curb detection processing.

[0050] Other cameras: Data from other cameras will be used to supplement the target camera's curb detection in the case of occlusion or incomplete viewing angle, ensuring the integrity and accuracy of the final curb detection results.

[0051] It is understandable that the target camera collects environmental image data of the current road section. The environmental image data is usually image data obtained by an RGB camera or a stereo vision camera, and the data will contain road, curb, obstacles and other environmental information related to curb detection.

[0052] 202. Extracting the intersection points of each field of view ray in the target environment image data based on the field of view ray of the target camera;

[0053] Through the field of view ray of the target camera, all intersection points that intersect with the field of view ray of the target camera are identified in the environment image. The starting point of each ray is located at the optical center of the target camera (i.e., the camera position), and the direction is defined according to the field of view direction of the camera.

[0054] The intersection point is determined by a geometric calculation method. In this embodiment, the intersection point of the ray and the obstacle outline on the image plane can be calculated. The specific algorithm may include a ray-polygon intersection algorithm or a calculation method based on image projection, so as to identify potential obstacle intersection points in the image.

[0055] 203. Based on a preset occlusion recognition rule, identify whether the visual field rays corresponding to each extracted intersection point are blocked;

[0056] It can be understood that a radar in the vehicle is selected as a target camera to collect target environment image data in the direction of the radar.

[0057] Furthermore, after obtaining the target environment image data, the target environment image data is preprocessed to facilitate subsequent intersection extraction. Specifically, rays are set in the target environment image data in a simulated manner, and the rays are arranged in order according to pixels, such as Figure 3 As shown by the dashed lines, each ray intersects with an object in the target environment image data.

[0058] In another feasible implementation, obtaining the target environment image data and extracting the intersection point of the field of view ray in the target environment image data include the following steps:

[0059] Step 1: The image data acquired by the radar is used to simulate and generate a number of rays, with the starting point of the ray located at the front end of the radar and the direction extending outward from one side of the vehicle where the radar is located;

[0060] Step 2: According to the direction of the ray, a geometric model of each ray is constructed, the trajectory of the ray is calculated, and collision detection is performed with objects (such as obstacles) in the target environment image data. The obstacle can be a vehicle, a fence, etc.;

[0061] Step 3: Using the geometric relationship between the ray and the obstacle, the intersection point between each ray and the obstacle is calculated by an intersection algorithm. The obstacle may be a contour area obtained by processing the image data.

[0062] Step 4: For each ray, determine whether the ray is blocked by the obstacle by detecting the intersection with the obstacle. If the intersection exists, the ray is considered blocked, otherwise it is not blocked.

[0063] Furthermore, when detecting whether the ray intersects with an obstacle in the target environment image data, the bounding box algorithm can be used to roughly locate the obstacle area; the intersection position of the ray and the obstacle is calculated based on the geometric relationship between the obstacle's bounding box and the ray, and if the intersection is within the boundary of the obstacle, it is considered that the ray intersects with the obstacle; using the ray-polygon intersection detection algorithm, for complex obstacle shapes, the intersection position between the ray and the obstacle is further accurately calculated.

[0064] Furthermore, the obstacle contour in the image data is extracted, and the obstacle contour is compared with the direction of the ray. A straight line and polygon intersection calculation algorithm is used to calculate the intersection of the ray and the obstacle contour boundary to determine whether the ray collides with the obstacle. Based on the distance relationship between the ray and the obstacle intersection, it is further calculated whether the ray is completely blocked or only partially blocked, and the ray detection result is updated according to the relative position of the intersection.

[0065] In addition, the starting point and direction of the ray can be converted into a three-dimensional world coordinate system according to the spatial position of the obstacle, and the intersection of the ray and the obstacle model can be accurately calculated; the three-dimensional scene can be gridded through spatial gridding technology, and the intersection of the obstacle and the ray can be mapped to the grid to obtain the intersection.

[0066] 204. If blocked, using a first curb detection algorithm, extract the curb contour based on all intersections in the environment image data of other cameras and the target environment image data;

[0067] Through the first curb detection algorithm, the target intersections are extracted from all the intersections, and the positions and directions of these target intersections in the target environment image data are determined. These target intersections are the intersections generated by the blocked rays and cannot be directly obtained by the target camera, so they need to be supplemented by data from other cameras.

[0068] The direction of the blocked curb is calculated based on the position information of the target camera and the direction of the intersection. The curb information of the corresponding area is extracted based on the image data of other cameras.

[0069] Based on the extracted curb information and non-target intersection points in the target environment image data, a complete curb contour is generated. This process uses image-based contour extraction techniques such as edge detection, Hough transform, or curve fitting.

[0070] 205. If not blocked, determine the type of the environment image data, and determine the target second curb detection algorithm based on the type;

[0071] Determine the type of the collected environmental image data, and select an appropriate second curb detection algorithm according to the type of the image. Common second curb detection algorithms include color segmentation method, edge detection method, etc.

[0072] 206. Extracting a potential curb area in the environment image data using a target second curb detection algorithm;

[0073] If the target second curb detection algorithm is a color segmentation method, the environmental image data is converted into a color space to obtain an image to be processed in an HSV color space; in the HSV color space, the curb and the background in the image to be processed are separated according to a preset threshold range, and a potential curb area is extracted from the separated image to be processed, wherein the threshold range includes: a first range of hue H, specifically set to [0°, 50°], a second range of saturation S, specifically set to [0.1, 1.0] and a third range of brightness V, specifically set to [0.3, 1.0].

[0074] It can be understood that the color segmentation method is used to convert the image from the RGB color space to the HSV color space; then in the HSV color space, an appropriate threshold range (such as the range of hue H, saturation S, and lightness V) is set to extract the potential curb area.

[0075] In another embodiment, the potential curb area is extracted from the separated image to be processed, specifically by using the Canny edge detection algorithm to perform edge enhancement on the separated image to be processed; the potential curb line segments are extracted from the edge-enhanced image to be processed by the Hough transform detection method; the specific positions of the potential curb line segments are determined by a geometric analysis method (such as straight line fitting, curve fitting, etc.), and the distances and combination forms between the potential curb lines are calculated to obtain the potential curb area.

[0076] Among them, edge enhancement specifically includes: graying the original image and converting it into a single-channel grayscale image to simplify the subsequent edge detection operation; using the Canny edge detection algorithm to perform edge enhancement. Furthermore, the Canny algorithm includes three main steps: first, smoothing the image through a Gaussian filter to remove noise; then, calculating the gradient of each pixel in the image to extract the edge in the image; finally, through non-maximum suppression and double thresholding, further extracting refined edge segments; post-processing the image after edge detection, such as corrosion and dilation operations, to remove small noise points and enhance edge connectivity.

[0077] 207. The edge detection algorithm is used to extract edge information in the potential curb area, and the curb information that meets the curb determination conditions is screened out based on the position information of the autonomous driving vehicle to obtain the curb contour.

[0078] Specifically, the edge of the potential curb area is enhanced by using an edge detection algorithm, wherein the edge detection algorithm is at least one of a Canny edge detection algorithm, a Sobel operator, a Prewitt operator, or a Laplace operator, and the algorithm identifies edge features in the potential curb area by calculating gradient values ​​of pixels in an image;

[0079] After obtaining the edge information, the contours in the potential curb area are preliminarily identified using the edge information. Potential curb boundaries are identified by extracting continuous edge segments, which will serve as the basis for subsequent curb identification.

[0080] According to the position information of the autonomous vehicle, the edge information is converted into position data in the vehicle coordinate system. The current coordinates of the vehicle are obtained through the vehicle's positioning system (such as GPS, IMU sensor or vehicle positioning system), and the edge information in the image is matched with the actual position of the vehicle, thereby converting the edge information into curb information in the real world.

[0081] The predicted position of the curb is further adjusted based on the vehicle's motion state (such as the vehicle's current speed, acceleration, etc.) to improve the accuracy of curb detection. Combined with the vehicle's motion trajectory, edge segments and areas that meet the curb characteristics are screened out. Specifically, if the detected edge segment meets the specific pattern of the vehicle's driving direction and road surface structure, it is identified as curb information that meets the curb determination conditions.

[0082] The selected curb information is optimized to eliminate misjudgments caused by image noise, lighting changes, etc. Through geometric analysis methods such as straight line fitting or polynomial fitting, the curb line segments are smoothed and possible deviations are corrected to ensure the continuity and accuracy of the curb contour.

[0083] Based on the optimized edge information and curb judgment conditions, the accurate curb profile is finally extracted. The curb profile includes the curb information on the left and right sides of the vehicle, which can be used for subsequent path planning and obstacle avoidance decisions.

[0084] It should be noted that the curb determination condition specifically includes at least one of the following:

[0085] The curb information should satisfy the relative position relationship with the current driving path of the vehicle, that is, the detected curb information is located on both sides of the vehicle's scheduled driving path and is basically parallel to the vehicle's driving direction;

[0086] The extracted curb information should have certain continuity and stability;

[0087] By analyzing the geometric shape of the curb line segment, it is determined whether it conforms to common curb shapes, such as straight segments or smooth curve segments, and abnormal edges that do not conform to the actual road structure are eliminated.

[0088] In another feasible implementation, after the curb profile is extracted, the method further includes: fitting each curve segment in the curb profile using a polynomial curve fitting method, and outputting a final curb profile.

[0089] It is understandable that the fitting process includes:

[0090] According to the discrete data points of each curve segment in the extracted curb profile, select a suitable polynomial model for fitting. Commonly used polynomial models include quadratic polynomials, cubic polynomials or high-order polynomials. The specific selection depends on the complexity of the curve and the fitting accuracy requirements;

[0091] The least squares method is applied to the discrete points of each curve segment for curve fitting. The least squares method determines the polynomial coefficients by minimizing the sum of squares of the errors between the fitted curve and the data points.

[0092] During the fitting process of each curve segment, check the continuity and smoothness of the curve. For areas with discontinuities or drastic changes, use appropriate smoothing algorithms (such as B-spline, spline curves, etc.) to further optimize the smoothness of the curve;

[0093] If multiple curve segments together form a complete curb profile, the fitted curve segments are connected to ensure the overall continuity of the curb profile. To this end, a smooth connection algorithm (for example, using polynomial splicing and constraints) is used to ensure that the transition between different fitted curve segments is natural and smooth, avoiding sudden changes;

[0094] For each fitted curve segment, calculate its geometric features such as direction, curvature and radius to further verify whether it conforms to the actual shape of the curb and exclude invalid fitting due to image noise or error;

[0095] The quality of the fitting result is further evaluated by calculating the geometric properties of the final curb profile (such as the total length of the curb, the curvature of the curve, etc.), and the order or parameters of the polynomial fitting are adjusted according to the quality evaluation to ensure that the output curb profile has the best fitting effect;

[0096] Output the final curb profile. At this point, the curb profile has been optimized by polynomial curve fitting to obtain a high-precision, smooth and continuous curb trajectory, which can be used in modules such as path planning and obstacle avoidance decision-making in the autonomous driving system.

[0097] Specifically, the polynomial curve fitting can be performed by dynamically adjusting the order of the polynomial to select the minimum order suitable for fitting. If the order is too low, complex curves may not be accurately fitted, and if the order is too high, overfitting may occur. The optimal order is selected by cross-validation or information criteria (such as AIC, BIC); if large errors are detected in certain curve segments in the curb profile (such as image acquisition errors, influence of obstructions, etc.), robust regression methods (such as Huber regression or RANSAC algorithm) are used to correct them to ensure the accuracy and robustness of the fitting.

[0098] The output of the final curb profile can be specifically achieved by converting the fitted curb curve into coordinate data in the vehicle coordinate system to ensure that the fitting result can be connected with the actual position of the vehicle; based on the map or point cloud data, the final curb profile is corrected to ensure that it is consistent with the curb position in the existing road network or point cloud map; and the final curb profile is transmitted to the autonomous driving system for use by subsequent path planning, control system and decision modules.

[0099] In another feasible implementation, after extracting the curb contour, it also includes: calculating the spatial coordinate information of each curb point in the curb contour; calling the point cloud map, and extracting the point cloud image located at the spatial coordinate information in the point cloud map; optimizing and completing the curb contour based on the point cloud image to obtain a complete curb contour.

[0100] The above-mentioned curb detection method is based on the curb detection method of multiple cameras. By extracting intersections in environmental image data, judging occlusion, and selecting two curb detection algorithms with different complexities, it ensures that the curb contour can be accurately and efficiently identified in different scenarios. The method includes technical means such as color space conversion, edge detection, geometric analysis, and curve fitting, and combines the optimization and completion of point cloud maps to further improve the accuracy and robustness of curb detection, which is suitable for complex road conditions in autonomous driving environments.

[0101] Corresponding to the above method embodiment, see Figure 4 A schematic diagram of a curb detection device is shown, the device is applied to an autonomous driving vehicle, the autonomous driving vehicle is provided with at least two cameras, and specifically comprises:

[0102] An acquisition module 410 is used to acquire target environment image data collected by a target camera in the autonomous driving vehicle, wherein the target camera is one of the at least two cameras;

[0103] An extraction module 420, configured to extract the intersection points of each of the field of view rays in the target environment image data based on the field of view rays of the target camera;

[0104] The identification module 430 is used to identify whether the visual rays corresponding to each extracted intersection point are blocked based on a preset blockage identification rule;

[0105] The detection module 440 is used to extract the curb contour based on all intersections in the environmental image data of other cameras and the target environmental image data using a first curb detection algorithm when identifying occlusion; and to extract the curb contour from the target environmental image data using a second curb detection algorithm when identifying non-occlusion.

[0106] The above-mentioned curb detection device obtains the target environment image data collected by the target camera in the autonomous driving vehicle, and the target camera is one of the cameras on the vehicle; based on the field of view rays of the target camera, the intersection points of each field of view ray in the target environment image data are extracted; based on the preset occlusion recognition rules, whether the field of view rays corresponding to each extracted intersection point are blocked is identified; when blocked, the first curb detection algorithm is used to extract the curb contour based on the environment image data of other cameras and all the intersection points in the target environment image data; when not blocked, the second curb detection algorithm is used to extract the curb contour from the environment image data. The present application solves the problem that the existing curb detection scheme cannot realize curb detection in the presence of occlusion scenes by setting different curb detection algorithms for curb recognition in scenes with blocked rays and unblocked rays.

[0107] See also Figure 5 Another embodiment of the curb detection device in the embodiment of the present application includes:

[0108] An acquisition module 410 is used to acquire target environment image data collected by a target camera in the autonomous driving vehicle, wherein the target camera is one of the at least two cameras;

[0109] An extraction module 420, configured to extract the intersection points of each of the field of view rays in the target environment image data based on the field of view rays of the target camera;

[0110] The identification module 430 is used to identify whether the visual rays corresponding to each extracted intersection point are blocked based on a preset blockage identification rule;

[0111] The detection module 440 is used to extract the curb contour based on all intersections in the environmental image data of other cameras and the target environmental image data using a first curb detection algorithm when identifying occlusion; and to extract the curb contour from the target environmental image data using a second curb detection algorithm when identifying non-occlusion.

[0112] Optionally, the detection module 440 includes a first detection unit 441, which is used to:

[0113] Extracting a target intersection point generated by blocking rays from all intersection points using a first curb detection algorithm, and determining a position direction of the target intersection point in the target environment image data;

[0114] Calculating the direction of the blocked curb based on the position information and the position direction of the target camera;

[0115] Reading environmental image data from other cameras and extracting curb information based on the curb direction;

[0116] A curb contour is generated based on the curb information and intersection points other than target intersection points in the target environment image data.

[0117] Optionally, the detection module 440 includes a second detection unit 442, which is used to:

[0118] determining a type of the environmental image data, and determining a target second curb detection algorithm based on the type;

[0119] Extracting a potential curb area in the environment image data using the target second curb detection algorithm;

[0120] The edge information in the potential curb area is extracted using an edge detection algorithm, and the curb information that meets the curb determination conditions is screened out based on the position information of the autonomous driving vehicle to obtain a curb contour.

[0121] Optionally, the first detection unit 441 is specifically configured to:

[0122] When the target second curb detection algorithm is a color segmentation method, the environment image data is converted into a color space to obtain an image to be processed in an HSV color space;

[0123] In the HSV color space, the curb and the background in the image to be processed are separated according to a preset threshold range, and a potential curb area is extracted from the separated image to be processed, wherein the threshold range includes: a first range of hue H, a second range of saturation S and a third range of brightness V.

[0124] Optionally, the first detection unit 441 is specifically configured to:

[0125] The Canny edge detection algorithm is used to enhance the edges of the separated image to be processed;

[0126] By using the Hough transform detection method, potential road edge segments are extracted from the edge-enhanced image to be processed;

[0127] The specific position of the potential curb line segment is determined by a geometric analysis method, and the distance and combination form between the potential curb lines are calculated to obtain the potential curb area.

[0128] Optionally, the curb detection device further includes a fitting module 450, which is used to:

[0129] The polynomial curve fitting method is used to fit each curve segment in the curb profile, and the final curb profile is output.

[0130] Optionally, the curb detection device further includes an optimization module 460, which is used to:

[0131] Calculating spatial coordinate information of each curb point in the curb profile;

[0132] Calling a point cloud map, and extracting a point cloud image located at the spatial coordinate information in the point cloud map;

[0133] The curb contour is optimized and completed based on the point cloud image to obtain a complete curb contour.

[0134] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above curb detection method. The electronic device can be a server or a terminal device.

[0135] See also Figure 6 As shown, the electronic device includes a processor 600 and a memory 601 , wherein the memory 601 stores machine executable instructions that can be executed by the processor 600 , and the processor 600 executes the machine executable instructions to implement the above-mentioned curb detection method.

[0136] Further, Figure 6 The electronic device shown further includes a bus 602 and a communication interface 603 , and the processor 600 , the communication interface 603 and the memory 601 are connected via the bus 602 .

[0137] The memory 601 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 603 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 602 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0138] The processor 600 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 600. The above processor 600 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in this embodiment can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with this embodiment can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601, and the processor 600 reads the information in the memory 601 and completes the steps of the curb detection method in combination with its hardware.

[0139] This embodiment also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned curb detection method.

[0140] The computer program product of the curb detection method, device, electronic device and storage medium provided in this embodiment includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments, which will not be repeated here.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0142] In addition, in the description of this embodiment, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in this disclosure can be understood according to specific circumstances.

[0143] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0144] In the description of the present disclosure, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present disclosure. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0145] Finally, it should be noted that the above embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art who is familiar with the technical field can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the present embodiments, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A curb detection method, applied to an autonomous driving vehicle, characterized in that: The autonomous driving vehicle is provided with at least two cameras; the method comprises: Acquire target environment image data captured by a target camera in the autonomous driving vehicle, wherein the target camera is one of the at least two cameras; Based on the field of view rays of the target camera, extracting the intersection points of each field of view ray in the target environment image data; Based on the preset occlusion recognition rules, identify whether the view rays corresponding to each extracted intersection point are blocked; If blocked, using a first curb detection algorithm to extract a curb contour based on all intersections in the environment image data of other cameras and the target environment image data; If it is not blocked, a second curb detection algorithm is used to extract the curb contour from the target environment image data.

2. The method for detecting a road edge according to claim 1, characterized in that: The step of generating a curb contour based on the environment image data of other cameras and all intersection points in the target environment image data by using a first curb detection algorithm comprises: Extracting a target intersection point generated by blocking rays from all intersection points using a first curb detection algorithm, and determining a position direction of the target intersection point in the target environment image data; Calculating the direction of the blocked curb based on the position information and the position direction of the target camera; Reading environmental image data from other cameras and extracting curb information based on the curb direction; A curb contour is generated based on the curb information and intersection points other than target intersection points in the target environment image data.

3. The method for detecting a road edge according to claim 1, characterized in that: The step of extracting a curb contour from the target environment image data using a second curb detection algorithm includes: determining a type of the environmental image data, and determining a target second curb detection algorithm based on the type; Extracting a potential curb area in the environment image data using the target second curb detection algorithm; The edge information in the potential curb area is extracted using an edge detection algorithm, and the curb information that meets the curb determination conditions is screened out based on the position information of the autonomous driving vehicle to obtain a curb contour.

4. The method for detecting a road edge according to claim 3, characterized in that: The target second curb detection algorithm is a color segmentation method; The step of extracting a potential curb area in the environment image data by using the target second curb detection algorithm includes: Performing color space conversion on the environment image data to obtain an image to be processed in an HSV color space; In the HSV color space, the curb and the background in the image to be processed are separated according to a preset threshold range, and a potential curb area is extracted from the separated image to be processed, wherein the threshold range includes: a first range of hue H, a second range of saturation S and a third range of brightness V.

5. The method for detecting a road edge according to claim 4, characterized in that: The step of extracting a potential curb area from the separated image to be processed comprises: The Canny edge detection algorithm is used to enhance the edges of the separated image to be processed; By using the Hough transform detection method, potential road edge segments are extracted from the edge-enhanced image to be processed; The specific position of the potential curb line segment is determined by a geometric analysis method, and the distance and combination form between the potential curb lines are calculated to obtain the potential curb area.

6. The method for detecting a curb according to any one of claims 1 to 5, characterized in that: After extracting the curb contour, it also includes: The polynomial curve fitting method is used to fit each curve segment in the curb profile, and the final curb profile is output.

7. The method for detecting a curb according to any one of claims 1 to 5, characterized in that: After extracting the curb contour, it also includes: Calculating spatial coordinate information of each curb point in the curb profile; Calling a point cloud map, and extracting a point cloud image located at the spatial coordinate information in the point cloud map; The curb contour is optimized and completed based on the point cloud image to obtain a complete curb contour.

8. A curb detection device, applied to an autonomous driving vehicle, characterized in that: The autonomous driving vehicle is provided with at least two cameras; the device comprises: an acquisition module, configured to acquire target environment image data captured by a target camera in the autonomous driving vehicle, wherein the target camera is one of the at least two cameras; An extraction module, used for extracting the intersection points of each of the field of view rays in the target environment image data based on the field of view rays of the target camera; An identification module, used to identify whether the visual field rays corresponding to each extracted intersection point are blocked based on a preset blockage identification rule; The detection module is used to extract the curb contour based on all intersections in the environmental image data of other cameras and the target environmental image data using a first curb detection algorithm when identifying occlusion; and to extract the curb contour from the target environmental image data using a second curb detection algorithm when identifying non-occlusion.

9. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the curb detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the curb detection method according to any one of claims 1 to 7.

Images