Obstacle detection method and system and storage medium

Abstract

The invention provides an obstacle detection method and system and a storage medium. The method comprises the following steps: segmenting a to-be-processed image into a passable area and a first obstacle; screening out a second obstacle of a preset type from the first obstacles; obtaining the outer contour of the second obstacle, and segmenting the outer contour of the second obstacle into a plurality of edges; determining a distorted edge in the plurality of edges of the outer contour; and marking the distorted edge as a passable area so as to re-determine the second obstacle. According to the embodiment of the invention, the distortion removal processing is only carried out on the distorted edge of the obstacle which can be seen under the view angle of the current vehicle, and the non-distorted edge and the edge of the obstacle which cannot be seen under the view angle of the current vehicle do not need to be processed, so that the computing power is saved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to an obstacle detection method, system and storage medium. Background Art

[0002] In traditional technology, when a vehicle is driving, reversing, or parking, it often uses a camera to collect images around the vehicle to detect the surrounding driving environment. The images captured by the camera will have perspective effects. For example, originally parallel lane lines will appear to intersect in the image. Therefore, the images captured by the camera need to be processed by inverse perspective mapping (IPM) to eliminate the perspective effect of the image.

[0003] Inverse Perspective Mapping (IPM) is a method that eliminates the perspective effect and converts lane lines that are not parallel due to perspective transformation into parallel lines in a bird's-eye view, thereby helping us understand the scene in the image more accurately. However, in the process of generating the Inverse Perspective Mapping (IPM) image, due to the complexity of perspective transformation and the accuracy of image processing, obstacle distortion effects may sometimes occur.

[0004] The obstacle distortion effect means that in the image processed by Inverse Perspective Mapping (IPM), the originally flat ground or obstacles that are not actually prominent or obstacles of a certain category are mistakenly identified as obstacles. This effect may be caused by factors such as errors in the image processing algorithm, inaccurate camera parameters, or noise in the image acquisition process. When these factors affect the accuracy of the inverse perspective transformation (IPM), pseudo obstacles may appear in the generated inverse perspective transformation (IPM) image.

[0005] In the subsequent image processing, if the image with the obstacle distortion effect is used for subsequent processing, adverse consequences will occur. For example, if the image with the pseudo obstacle effect is used to extract the parking space line, the parking space line detection will be inaccurate, and the free space (FS) detection will also be inaccurate. Therefore, it is urgent to propose an obstacle detection method to solve the above problems. Summary of the invention

[0006] In view of at least one of the above technical problems existing in the prior art, the present application is proposed. According to one aspect of the present application, an obstacle detection method is provided, the method comprising: Segmenting the image to be processed into a traversable area and a first obstacle; Screening out a second obstacle of a preset type from the first obstacles; Acquire an outer contour of the second obstacle, and divide the outer contour of the second obstacle into a plurality of edges; determining a distorted edge among the plurality of edges of the outer contour; The distorted edge is marked as a traversable area to redefine the second obstacle.

[0007] In some embodiments, before segmenting the image to be processed, the method further includes: Fisheye images of the current vehicle surroundings captured by at least two fisheye cameras are obtained, and the fisheye images are spliced ​​into a panoramic bird's-eye view to obtain the image to be processed.

[0008] In some embodiments, determining the distorted edges among the plurality of edges of the outer contour comprises: Determine a fisheye camera corresponding to the second obstacle; Determine whether there is a first edge among the multiple edges of the second obstacle that has an intersection with the edge of the vehicle body where the fisheye camera is located, and the distance between the intersection and the corresponding fisheye camera is within a second preset range; When the intersection point exists and the distance between the intersection point and the corresponding fisheye camera is within a second preset range, the first edge is determined to be a distorted edge.

[0009] In some embodiments, dividing the outer contour of the second obstacle into a plurality of edges comprises: calculating an inflection point of the outer contour of the second obstacle; The plurality of edges of the outer contour are determined according to adjacent inflection points.

[0010] In some embodiments, determining the field of view of the fisheye camera corresponding to the second obstacle includes: When the second obstacle is located in the overlapping field of view of two adjacent fisheye cameras and the second obstacle occupies different ranges in each shooting field of view, the shooting field of view of the fisheye camera corresponding to the second obstacle is determined by the shooting field of view occupying the larger range.

[0011] In some embodiments, segmenting the image to be processed into a traversable area and a first obstacle includes: Dividing the panoramic bird's-eye view into a pre-processing area, a post-processing area, a left-side processing area, and a right-side processing area; The passable area and the first obstacle in the left processing area and the right processing area are divided.

[0012] In some embodiments, the current vehicle has four vehicle side edges; after selecting a second obstacle of a preset type from the first obstacle, the method further includes: When the second obstacle is a square column, determining the vehicle side edge and the distance to the second obstacle that are the shortest; Determine the relationship between the distance and a preset distance threshold range; If the distance is less than the minimum value of the preset distance threshold range, adjusting the internal parameters and distortion parameters of the fisheye camera arranged at the edge of the vehicle body at the minimum distance from the second obstacle to increase the shooting field of view of the fisheye camera arranged at the edge of the vehicle body at the minimum distance from the second obstacle; If the distance is greater than the maximum value of the preset distance threshold range, the internal parameters and distortion parameters of the fisheye camera set at the edge of the vehicle body with the smallest distance from the second obstacle are adjusted to reduce the shooting field of view of the fisheye camera set at the edge of the vehicle body with the smallest distance from the second obstacle.

[0013] In some embodiments, marking the distorted edge as a traversable area to redefine the second obstacle includes: Calculating a first distance between any point on the distorted edge and the corresponding vehicle body edge; Calculating the distortion component of the arbitrary point according to the first distance and a preset distortion coefficient; When the distortion score of any point exceeds a preset distortion score threshold, the any point is added to the set of points to be dedistorted until all points of the distorted edge are calculated; All points in the point set to be dedistorted are uniformly marked as passable areas.

[0014] Another aspect of the present application provides an obstacle detection system, the system comprising: A segmentation module, used for segmenting the image to be processed into a passable area and a first obstacle; A screening module, used to screen out second obstacles of a preset type from the first obstacles; an acquisition module, configured to acquire an outer contour of the second obstacle, and divide the outer contour of the second obstacle into a plurality of edges; A determination module, configured to determine the distorted edge among the plurality of edges of the external contour; A labeling module is used to label the distorted edge as a passable area to redefine the second obstacle.

[0015] Another aspect of the embodiments of the present application provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the obstacle detection method as described above.

[0016] The obstacle detection method of the embodiment of the present application divides the image to be processed into a passable area and a first obstacle within a first preset range, and screens out a second obstacle of a preset type from the first obstacle, and then processes the outer contour of the second obstacle to determine the distorted edge, and marks the distorted edge as a passable area to redetermine the second obstacle; the embodiment of the present application only performs dedistortion processing on the distorted edge of the preset type of obstacle, and does not need to process the undistorted edge, thereby saving computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic flow chart of an obstacle detection method according to an embodiment of the present application is shown; Figure 2 A schematic flow chart showing a process 200 of obtaining an image to be processed according to an embodiment of the present application; Figure 3 A schematic flowchart of dividing the outer contour of the second obstacle into multiple edges according to an embodiment of the present application is shown; Figure 4 A schematic flowchart of step S301 according to an embodiment of the present application is shown; Figure 5 A schematic flow chart of S104 according to an embodiment of the present application is shown; Figure 6 A schematic flowchart of step S501 according to an embodiment of the present application is shown; Figure 7 A schematic flow chart showing how to adjust the shooting field of view of a fisheye camera when detecting a special obstacle according to an embodiment of the present application; Figure 8 A schematic diagram showing the position relationship between the current vehicle and the second obstacle according to an embodiment of the present application; FIG9( a ) is a schematic diagram showing the actual situation of obstacles around the current vehicle according to an embodiment of the present application; FIG9( b ) is a schematic diagram showing a distortion condition of obstacles around the current vehicle according to an embodiment of the present application; FIG. 10 ( a ) is a schematic diagram showing a method of completing the initial bird's-eye view stitching process in a manner of maximally retaining the rear fisheye image according to an embodiment of the present application; FIG10( b ) is a schematic diagram showing a method of completing the initial bird's-eye view stitching process in a manner of maximally retaining the left and right fisheye images according to an embodiment of the present application; Fig.11 A schematic diagram showing an obstacle detection method 1100 according to yet another embodiment of the present application is shown; Fig.12 A schematic diagram showing a panoramic bird's-eye view with segmentation lines formed after re-determining the traversable area and the obstacle area according to an embodiment of the present application; Fig.13 A schematic block diagram of an obstacle detection system according to an embodiment of the present application is shown; Fig.14 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] There are currently two main algorithms for obstacle detection on panoramic images: one is to convert the three-dimensional space coordinates into a two-dimensional image plane coordinate system, and then match them with the segmentation results output by the segmentation model (the segmentation results include marked obstacles) to obtain dynamic obstacles and static obstacles, and then filter the distorted areas of the dynamic obstacles and static obstacles output by the segmentation model to obtain accurate passable space; the other is to use the occupation network model to determine the obstacles according to the space occupancy results output by the model, remove the space except for the obstacles in the occupied space, and obtain the passable space.

[0021] In the first solution, although the detection results of two-dimensional image (2D) obstacles are used for matching, the obstacle position information detected by this purely visual two-dimensional image (2D) obstacle model is not accurate. Even if the three-dimensional spatial coordinates of the three-dimensional (3D) obstacles can be detected again by laser radar, the position of the obstacles in the two-dimensional image plane coordinate system can be calculated. However, if there are three vehicles arranged continuously in the panoramic image, due to the distortion effect of the panoramic image, the vehicles shown in the image are connected together, and the segmentation model will treat these three vehicles as a semantic segmentation block of a vehicle. The result is that although the obstacles in the three-dimensional space can be matched when converted to the obstacles in the image coordinate system, another set of processing strategies needs to be added, that is, to filter out the distorted space generated by each obstacle according to the direction of the car and the position of the parking space entrance line. It can be seen that the first type of solution is strongly dependent on the three-dimensional (3D) obstacle detection method, and the amount of calculation is very large.

[0022] In the second solution, the occupancy network model is a technology that can describe whether each position in a three-dimensional or two-dimensional space is occupied. It does not have too many geometric assumptions about objects, so it can model objects of any shape and any form of object movement. The model trained by this technology can predict the occupancy of each position in a three-dimensional or two-dimensional space. Using the occupied space information predicted by the occupancy network model, obstacles are obtained based on the occupied space, and the passable space is obtained from the space other than the occupied space. Since there is little open source data for training the occupancy network model, the standard occupancy data that needs to be obtained when training the occupancy network model is often in the order of 10w+ to obtain a good enough effect. Moreover, the iteration and optimization of the occupancy network model also takes a lot of time. Therefore, using the occupancy network model to predict the passable space will increase computing power, which is not a suitable solution for low-performance chips.

[0023] Therefore, in view of the above problems and considering chip performance, this application proposes an obstacle detection method to solve the distortion problem based on the fisheye perspective. This method performs de-distortion processing on the basis of the segmentation results obtained after semantic segmentation of the panorama. And because the segmentation results are often used for the extraction of passable areas (FS), in order to reduce computing power, the extracted passable areas are no longer processed, but only the distorted obstacles in the segmentation results are de-distorted to optimize computing power.

[0024] The obstacle detection method of the embodiment of the present application divides the processed image into a passable area and a first obstacle, and screens out a second obstacle of a preset type from the first obstacle, and then processes the outer contour of the second obstacle to determine the distorted edge, and marks the distorted edge as a passable area to redetermine the second obstacle; the embodiment of the present application only performs dedistortion processing on the distorted edge of the preset type of obstacle, and does not need to process the undistorted edge, thereby saving computing power.

[0025] The embodiments of the present application are not limited to application scenarios applicable to intelligent driving, but are also applicable to other related visual scenarios involving fisheye distortion.

[0026] Figure 1 A schematic flow chart of an obstacle detection method according to an embodiment of the present application is shown; Figure 1 As shown, the obstacle detection method 100 according to the embodiment of the present application may include the following steps S101, S102, S103, S104 and S105: In step S101, the image to be processed is segmented into a passable area and a first obstacle.

[0027] The image to be processed may include a panoramic bird's-eye view or one or more fisheye images. In most application scenarios, fisheye images are usually concatenated into a panoramic bird's-eye view to accurately detect obstacles in the surrounding environment.

[0028] like Figure 2 FIG. 2 is a flow chart of a process 200 for obtaining an image to be processed. In one embodiment of the present application, before segmenting the image to be processed, the method further includes step S201: In step S201, fisheye images of the surrounding environment of the current vehicle collected by at least two fisheye cameras are obtained, and the fisheye images are spliced ​​into a panoramic bird's-eye view from the perspective of the current vehicle to obtain the image to be processed.

[0029] For example, four fisheye cameras can be installed on the current vehicle, located at the front, rear, left and right sides of the vehicle, for example, they are installed at the center between the front cover opening and the front license plate, at the center of the upper frame of the rear license plate and below the two rearview mirrors. The shooting range of the four fisheye cameras can cover the area around the vehicle body. Specifically, by acquiring multiple fisheye images of different angles collected by each fisheye camera, each fisheye image can be subjected to inverse perspective transformation to generate a corresponding initial bird's-eye view, and then each of the initial bird's-eye views can be spliced ​​to generate a panoramic bird's-eye view from the perspective of the current vehicle.

[0030] Generally, because the pictures taken by the fisheye camera will have perspective effects, for example, as shown in Figure 9 (a), it is a schematic diagram of the actual situation of obstacles around the current vehicle. Figure 9 (b) is a schematic diagram of the distortion of the surrounding obstacles. As can be seen from Figures 9 (a) and 9 (b), the front vehicle, the right vehicle and the square column are all deformed. Therefore, when converting the fisheye image into a panoramic bird's-eye view, it is necessary to perform inverse perspective mapping (IPM) on the fisheye image.

[0031] In an embodiment of the present application, when stitching together a panoramic bird's-eye view from fisheye images taken by fisheye cameras at four installation positions, inverse perspective mapping (IPM) processing can be performed on each fisheye image first to restore the image to its true state and obtain an initial bird's-eye view.

[0032] In addition, since the factory configuration and characteristics of each fisheye camera are different, the method of stitching the initial bird's-eye view is also different. For example, as shown in Figures 10 (a) and 10 (b), it is a schematic diagram of a panoramic bird's-eye view formed after stitching the initial bird's-eye view. In Figure 10 (a), the fisheye image taken by the rear fisheye camera is retained to the maximum extent, and in Figure 10 (b), the fisheye images taken by the left and right fisheye cameras are retained to the maximum extent.

[0033] The embodiment of the present application converts the fisheye image from the original fisheye view to a more intuitive bird's-eye view by performing inverse perspective mapping (IPM) processing on the fisheye image. The bird's-eye view provides a complete view of the current vehicle's surroundings, which enables the vehicle to more accurately judge the space and obstacles around it, thereby improving the accuracy of subsequent analysis work.

[0034] In one embodiment of the present application, the panoramic bird's-eye view image may be input into a preset segmentation model for segmentation to obtain a passable area (FS) and a first obstacle within a first preset range.

[0035] The preset segmentation model may include a segmentation model in traditional technologies such as a semantic segmentation model. The segmentation model is not limited here, as long as the panoramic bird's-eye view image can be segmented into a passable area (FS) and a first obstacle. When the panoramic bird's-eye view image is segmented, all pixel points in the panoramic bird's-eye view image are labeled, for example, each pixel is labeled as a passable area (FS) or a first obstacle.

[0036] In one example, the passable area (FS) and the first obstacle within the first preset range may refer to dividing the panoramic bird's-eye view into a front processing area, a post-processing area, a left processing area, and a right processing area, and determining the first obstacle in the left processing area and the right processing area. For example, the panoramic bird's-eye view may be divided into processing areas according to the shooting field of view of the fisheye camera (excluding overlapping parts) according to the division method as shown in Figure 10 (a) or 10 (b). For another example, the front, rear, left, and right processing areas may be divided into rectangles according to the developer's processing convenience, and the rectangles of the front, rear, left, and right processing areas may form a panoramic bird's-eye view.

[0037] In actual application scenarios, since the vehicle itself is currently in the passable area (FS) and the driver can see obstacles in the pre-processing area from his perspective, the computing power allocated to the pre-processing area and the post-processing area can be reduced, and the computing power allocated to the left and right processing areas can be increased. This regional processing method makes it easier to allocate computing power according to actual conditions to save computing power.

[0038] In another example, the passable area (FS) and the first obstacle within the first preset range refer to the passable area (FS) and the obstacle area within the preset range (e.g., closer) from the current vehicle. For example, the area within a radius of 4m with the current vehicle as the center. Since the current vehicle will not temporarily drive to the distant passable area (FS) when parking, there is no need to detect the distant first obstacle, and there is no need to segment the distant passable area (FS) and the first obstacle. Only the nearby passable area (FS) and the first obstacle are processed to save computing power.

[0039] In step S102, second obstacles of a preset type are screened out from the first obstacles.

[0040] Among them, the first obstacle is all obstacles segmented from the image to be processed, for example, it can include any obstacles such as pedestrians, vehicles, lane lines, ground locks, houses, building columns, cones and walls. Due to the perspective effect of fisheye images, before the fisheye images are spliced ​​into a panoramic bird's-eye view, the fisheye images need to be processed by inverse perspective mapping (IPM). However, the inverse perspective mapping (IPM) processing can eliminate the perspective effect of ground elements, but it cannot eliminate the distortion of obstacles such as columns, pedestrians, vehicles, ground locks, cones and walls, that is, the obstacles have obstacle distortion. Therefore, the embodiment of the present application needs to screen out the second obstacle with obstacle distortion from the first obstacle for de-distortion processing.

[0041] The preset types of second obstacles in the present application may include square columns, cone barrels, pedestrians, ground locks, vehicles, and walls, etc. Since the first obstacles in the output results of the segmentation model are all marked with their types, the preset types of second obstacles can be directly filtered out from the first obstacles. Since these types of obstacles have obvious obstacle distortion, the obstacle detection method of the embodiment of the present application is used to perform de-distortion processing on certain types of second obstacles, which can achieve a better de-distortion effect and make the identified obstacles more accurate.

[0042] In step S103, the outer contour of the second obstacle is obtained, and the outer contour of the second obstacle is divided into a plurality of edges.

[0043] Wherein, the outer contour of the second obstacle includes a plurality of edges.

[0044] The outer contour of the second obstacle is the boundary point of the obstacle area. The outer contour information of the second obstacle may include position information or coordinate information of each point constituting the outer contour.

[0045] Furthermore, if Figure 3 As shown, step S103 of dividing the outer contour of the second obstacle into multiple edges includes step S301 and step S302: In step S301, calculating the inflection point of the outer contour of the second obstacle; In step S302, the plurality of edges of the outer contour are determined according to adjacent inflection points.

[0046] When calculating the inflection point of the outer contour of the second obstacle, all boundary points on the outer contour of the obstacle area can be traversed. For example, the inflection point can be found by calculating the slope of the outer contour, and after traversing all boundary points of the second obstacle twice, all inflection points of the second obstacle can be determined. Here, traversing all boundary points of the second obstacle twice means traversing all boundary points of the second obstacle from the starting point to the end point, and then traversing all boundary points of the second obstacle from the end point to the starting point to determine that all boundary points of the second obstacle are detected. The embodiment of the present application adopts an algorithm with a computing power, that is, it can traverse all boundary points on the second obstacle from the beginning to the end, and mark the boundary points that meet the conditions as inflection points. Since there is no need to re-determine the obstacle and its boundary points during the second traversal, computing power is saved.

[0047] The line connecting two adjacent inflection points forms an edge, so the edge of the outer contour of the second obstacle that can be seen from the current vehicle perspective can be determined by determining the inflection point.

[0048] In step S104, the distorted edges among the multiple edges of the outer contour are determined.

[0049] In one embodiment of the present application, Figure 4 As shown, step S104 of determining the edges of the plurality of edges of the outer contour that have fisheye distortion includes step S401, step S402 and step S403: In step S401, the fisheye camera corresponding to the second obstacle is determined.

[0050] In step S402, it is determined whether there is a first edge among the multiple edges that has an intersection with the edge of the vehicle body where the fisheye camera is located, and the distance between the intersection and the corresponding fisheye camera is within a second preset range.

[0051] In step S403, when the intersection point exists and the distance between the intersection point and the corresponding fisheye camera is within a second preset range, the first edge is determined to be a distorted edge.

[0052] The field of view of a camera refers to the maximum range that the camera can observe, usually expressed in degrees. This range determines the breadth of the scene that the camera can capture. In camera technology, field of view is an important parameter that affects the camera's monitoring capabilities and application range. For example, some cameras have a small field of view and can only observe a limited area, while some cameras have a very large field of view and can capture a wider scene.

[0053] For a fisheye camera, its shooting field of view is usually up to 180 degrees, which enables it to capture a very wide scene in one photo. Therefore, the embodiment of the present application uses a fisheye camera to collect the current vehicle surroundings, in order to capture more information in the vehicle surroundings.

[0054] like Figure 8As shown, it is a schematic diagram of the positional relationship between the current vehicle and the second obstacle. Continuing from the above example, fisheye cameras are respectively set on the body edges AB, BC, CD, and DA of the current vehicle. The square column is located within the shooting field of view of the fisheye camera on the BC edge (that is, the camera corresponding to the second obstacle is the camera on the BC edge), and the intersection point of one of the sides of the square column with the body edge BC of the vehicle (that is, the body edge where the camera corresponding to the second obstacle is located) is E, and the distance between the intersection point E and the fisheye camera on the BC edge is within the second preset range (for example, the distance between E and the fisheye camera on the BC edge is less than 2 meters), so one of the above sides of the square column is a distorted side. The intersection point of the other side of the square column with the body edge BC of the vehicle is F, but because the distance between F and the fisheye camera on the BC edge is greater than the preset distance range (for example, the distance between F and the fisheye camera on the BC edge is greater than 2 meters), it does not meet the condition for judging the distorted side, and the other side of the square column is not a distorted side.

[0055] In one embodiment of the present application, Figure 5 As shown, in step S401, determining the fisheye camera shooting field of view corresponding to the second obstacle includes step S501: In step S501, when the second obstacle is located in the overlapping field of view of two adjacent fisheye cameras and the second obstacle occupies different ranges in each shooting field of view, the shooting field of view of the fisheye camera corresponding to the second obstacle is determined by the shooting field of view occupying the larger range.

[0056] Continue to combine Figure 8 Since the shooting angle of a fisheye camera can reach 180 degrees and can capture a very wide scene, the shooting fields of two adjacent fisheye cameras must overlap. If the fisheye images taken by each fisheye camera are processed, obstacles may be processed repeatedly, which wastes computing power. In order to avoid repeated processing of obstacle areas, it is necessary to determine the ownership of the obstacle areas that fall into the overlapping area.

[0057] For example, when both cameras can capture the same obstacle ( Figure 8Only the case where the obstacle can be photographed by only one camera is shown). In the two-dimensional plane coordinate system, the area of ​​the obstacle falling into the images taken by two adjacent fisheye cameras can be calculated respectively. For the convenience of description, the two adjacent fisheye cameras can be called the first fisheye camera and the second fisheye camera. If the area of ​​the obstacle in the image taken by the first fisheye camera is larger than the area in the image taken by the second fisheye camera, it means that the fisheye camera corresponding to the obstacle is the first fisheye camera, otherwise, the fisheye camera corresponding to the obstacle is the second fisheye camera; if the area of ​​the obstacle in the image taken by the first fisheye camera is equal to the area of ​​the obstacle in the image taken by the second fisheye camera, one of the fisheye cameras can be designated as the corresponding fisheye camera.

[0058] In one embodiment of the present application, determining the distorted edges among the multiple edges of the external contour includes: Determine an edge where the outer contour of the second obstacle is distorted under the current vehicle viewing angle.

[0059] Continuing with the above example, based on the installation positions of the four fisheye cameras currently installed on the vehicle, the fisheye cameras can only capture images of the environment within the field of view, while the sides of the second obstacle facing away from the vehicle cannot be captured by the fisheye cameras. For example, when the vehicle is about to pass one of the corners of a square column, only the two sides of the square column that constitute the above-mentioned corner can be captured, while the other two sides facing away from the vehicle cannot be captured. Therefore, it is only necessary to determine the distorted edges among the edges captured from the perspective of the current vehicle. The embodiment of the present application only determines the distorted edges and performs de-distortion processing from the perspective of the current vehicle, while the edges that are not in the perspective of the current vehicle and the edges that are not distorted do not need to be processed, thereby saving computing power.

[0060] In step S105 , the distorted edge is marked as a passable region (FS) to redefine the second obstacle.

[0061] Since the distorted edge of the second obstacle does not belong to the second obstacle, the distorted edge should be marked as a passable area (FS) to re-determine the second obstacle. In one embodiment, marking the distorted edge as a passable area (FS) means marking each point on the distorted edge as a passable area (FS).

[0062] Specifically, the distortion score can be calculated based on the distance between the point on the distorted edge (i.e., the distorted edge) and the corresponding vehicle side edge, the length of the distorted edge, and the preset distortion coefficient, wherein the preset distortion coefficient can be set by the user according to the need for detection accuracy. A set of points to be dedistorted can be constructed, and when traversing the distorted edge, the distortion score corresponding to the point on the distorted edge is calculated. When the distortion score exceeds the preset distortion score threshold, the point is added to the set of points to be dedistorted. When all points on the distorted edge are calculated, the points in the set of points to be dedistorted are uniformly marked as passable areas or distorted points.

[0063] like Figure 6 As shown, the step S105 of marking the distorted edge as a passable area (FS) to redefine the second obstacle includes steps S601, S602, S603 and S604: In step S601, a first distance between any point on the distorted edge and the corresponding vehicle body edge is calculated; In step S602, the distortion score of the arbitrary point is calculated according to the first distance, the length of the distorted side and a preset distortion coefficient: In step S603, when the distortion score of the arbitrary point exceeds a preset distortion score threshold, the arbitrary point is added to the set of points to be dedistorted until all points of the distorted edge are calculated; In step S604, all points in the set of points to be dedistorted are uniformly marked as passable areas.

[0064] In the embodiment of the present application, the purpose of calculating the distortion score is to avoid erroneous distortion processing and to avoid unnecessary distortion processing, which can also save some computing power.

[0065] When determining the corresponding vehicle side edge, the center of the obstacle (or the obstacle is regarded as a point) may be used to calculate the distances between the four vehicle side edges and the obstacle respectively, and the vehicle side edge closest to the obstacle may be selected as the corresponding vehicle side edge.

[0066] Specifically, the distortion coefficients may include a first distortion coefficient, a second distortion coefficient, a third distortion coefficient, a fourth distortion coefficient, and a fifth distortion coefficient.

[0067] In one embodiment, the distortion score may be calculated as follows: The first distortion coefficient is multiplied by the first distance to obtain the first product, the second distortion coefficient is subtracted from the first product to obtain the first result, and the third distortion coefficient is multiplied by the first result to obtain the first distortion factor; the fourth distortion coefficient, the fifth distortion coefficient and the length of the distortion side where the point to be marked are located are multiplied to obtain the second distortion factor; the first distortion factor and the second distortion factor are then added to obtain the distortion score. For example, the first distortion coefficient is 0.02, the second distortion coefficient is 8.96, the third distortion coefficient is 0.7, the fourth distortion coefficient is 0.02, and the fifth distortion coefficient is 0.3.

[0068] The formula for calculating the distortion score is as follows: ……(1) in, Represents the distortion score of the current point to be labeled (that is, any point on the edge where the distortion occurs); Indicates the distance from the current point to be marked to the corresponding vehicle side edge (i.e. the first distance); Indicates the length of the side of the obstacle where the current point to be marked is located.

[0069] After traversing all points on the distorted edge, the points that meet the conditions can be determined, and the point set to be dedistorted can be obtained. The label information (label) of all points in the point set to be dedistorted can be modified, and it can be updated to the pass area (Road) or the dedistorted point type (Deformation).

[0070] In the embodiment of the present application, in the distortion score formula, the reason why the distorted edge on which the point to be marked (any point on the distorted edge) is located needs to be used as one of the calculation factors is that when the length of the distorted edge is very small, the impact is not significant, so when the distorted edge is very short (for example, less than 1 meter), it can be ignored. In other words, when the distorted edge is very short, the points on the distorted edge do not need to be dedistorted.

[0071] In the application scenario of parking, in the spatial design of underground parking lots, columns of various shapes are used to support the building structure and provide necessary space separation, among which the most important are square columns. These square columns are usually located on both sides or around the parking spaces to support the weight of the above-ground buildings and maintain the load in emergencies. Therefore, when parking, drivers may often encounter square columns as one of the structural elements of the parking lot. In order to more accurately identify the position of the square columns, the internal parameters and distortion parameters of the fisheye camera can be adjusted according to the distance between the square columns and the vehicle body to facilitate parking for users. Figure 7As shown, after selecting the second obstacle of the preset type from the first obstacle, the method may include the following steps S701, S702, S703, S704 and S705: In step S701, detecting whether the second obstacle is a square column; In step S702, when the second obstacle is a square pillar, determine the vehicle side edge and distance with the second obstacle at the shortest distance; In step S703, determining the relationship between the distance and a preset distance threshold range; In step S704, if the distance is less than the minimum value of the preset distance threshold range, adjusting the internal parameters and distortion parameters of the fisheye camera arranged at the edge of the vehicle body with the smallest distance from the second obstacle, so as to increase the shooting field of view of the fisheye camera arranged at the edge of the vehicle body with the smallest distance from the second obstacle; In step S705, if the distance is greater than the maximum value of the preset distance threshold range, the internal parameters and distortion parameters of the fisheye camera set at the edge of the vehicle body with the smallest distance to the second obstacle are adjusted to reduce the shooting field of view of the fisheye camera set at the edge of the vehicle body with the smallest distance to the second obstacle.

[0072] As described above, when the panoramic bird's-eye view image is input into the preset segmentation model for segmentation, the first obstacle is output, that is, the type of the first obstacle is marked. Since the second obstacle is selected from the first obstacle, that is, the type of the second obstacle is also marked, the obstacle marked as "square column" can be selected from these types.

[0073] Among them, the internal parameters of the fisheye camera may include: focal length fu, fv and optical center cu, cv, etc. The external parameters of the fisheye camera may include: pitch angle Pitch, yaw angle Yaw and height h of the center of the fisheye camera from the ground, etc. The distortion parameters (i.e., inverse perspective transformation parameters) of the fisheye camera may include: the size of the inverse perspective image, the area of ​​inverse perspective transformation and the distance from the forward projection of the camera, etc.

[0074] Continue to combine Figure 8, the current vehicle has four edges: AB, BC, CD, and DA. As shown in the figure, BC is an edge facing the second obstacle column, and is also the edge of the vehicle body with the shortest distance from the column. Draw a vertical line segment from the center point of the column (or regard the column as a point) to the BC edge, and the length of the vertical line segment is the distance from the column to the BC edge. If the distance is less than the minimum value of the preset distance threshold range, the field of view of the fisheye camera on the edge of the vehicle body (i.e., the camera on the BC edge) is increased, and the field of view of the adjacent fisheye camera can be reduced accordingly; if the closest distance is greater than the maximum value of the preset distance threshold range, the field of view of the fisheye camera on the edge of the vehicle body is reduced, and the field of view of the adjacent fisheye camera can be increased accordingly. For example, the preset distance threshold range is 3 to 4 meters. When the distance from the square column to the BC side is less than 3 meters, the shooting field of view of the fisheye camera set on the side edge BC of the car body is increased; when the distance from the square column to the BC side is greater than 4 meters, the shooting field of view of the fisheye camera set on the side edge BC of the car body is decreased.

[0075] The embodiment of the present application proposes an obstacle detection method based on obstacle dedistortion of a panoramic image segmentation map. After extracting the passable area (FS) and the first obstacle, the selected second obstacle is dedistorted, thereby eliminating the step of calculating the distortion of obstacles other than the second obstacle and saving computing power.

[0076] Moreover, the technical solution of the embodiment of the present application does not rely on the results of other deep learning models, and does not require the introduction of other 3D models or occupancy network models (OCC) for processing.

[0077] Moreover, in the technical solution of the embodiment of the present application, the re-determined passable area (FS) can be used for the raster map fusion of the subsequent fusion module, can also be used for the fusion calculation of the passable area (FS) based only on the inverse perspective transformation (IPM), and can also be used for occupancy detection of parking space.

[0078] In addition, in the embodiment of the present application, the re-determined second obstacle is beneficial to the subsequent fusion calculation. For example, the distorted edge of the second obstacle is updated to a point of the dedistorted category, and more accurate information calculation can be performed for these dedistorted category points in the fusion calculation or in the occupancy calculation.

[0079] like Fig.11 FIG. 1 is a schematic diagram of an obstacle detection method 1100 according to another embodiment of the present application. Fig.11It can be seen that four fisheye cameras are respectively arranged at the front, rear, left and right of the current vehicle to collect fisheye images within the corresponding shooting range. The obstacle detection method 1100 of the embodiment of the present application includes the following steps S1101, S1102, S1103, S1104, S1105 and S1106: In step S1101, fisheye images within a corresponding shooting range are acquired; and the fisheye images are subjected to inverse perspective transformation processing, wherein the fisheye images are images directly captured by a fisheye camera, including a front fisheye image, a rear fisheye image, a left fisheye image, and a right fisheye image.

[0080] In step S1102, the four fisheye images after the inverse perspective transformation are spliced ​​to obtain a panoramic bird's-eye view.

[0081] In step S1103, the panoramic bird's-eye view image is segmented using a segmentation model to obtain a passable area (FS) and a second obstacle. The second obstacle here can be an obstacle of a preset type, for example, the second obstacle can at least include obstacles such as square columns, cones, pedestrians, ground locks, vehicles, and walls.

[0082] In step S1104 , boundary points of the passable area (FS) are extracted.

[0083] The outer contour of the second obstacle includes a plurality of edges, wherein the plurality of edges include distorted edges and undistorted edges.

[0084] In step S1105, the distorted edge is subjected to dedistortion processing.

[0085] In step S1106 , the boundary points of the traversable area (FS) are converted from the segmentation map coordinate system to the vehicle coordinate system (VCS).

[0086] like Fig.12 Shown is a schematic diagram of the panoramic bird's-eye view formed after the passable area (FS) and the second obstacle are re-determined. Fig.12 The green dotted segments in the upper left dotted box and the lower right dotted box show the distorted edge processing results. It can be seen from the figure that the drivable area is more in line with the actual situation. Compared with the drivable area directly segmented by the segmentation model, the range of the drivable area is expanded by using the method of the present application, and the results show that the processing is successful.

[0087] The obstacle detection method of the embodiment of the present application divides the image to be processed into a passable area and a first obstacle within a first preset range, and screens out a second obstacle of a preset type from the first obstacle, and then processes the outer contour of the second obstacle to determine the distorted edge, and marks the distorted edge as a passable area to redetermine the second obstacle. The embodiment of the present application only performs dedistortion processing on the distorted edge of the preset type of obstacle, and does not need to process the undistorted edge, thereby saving computing power.

[0088] Combine the following Fig.13 The obstacle detection system of the present application is described, wherein: Fig.13 A schematic block diagram of an obstacle detection system 1300 according to an embodiment of the present application is shown.

[0089] Wherein, the obstacle detection system 1300 includes: A segmentation module 1301 is used to segment the image to be processed into a passable area and a first obstacle; A screening module 1302, configured to screen out second obstacles of a preset type from the first obstacles; An acquisition module 1303 is used to acquire an outer contour of the second obstacle and divide the outer contour of the second obstacle into multiple edges; A determination module 1304 is used to determine the distorted edges among the multiple edges of the external contour; The marking module 1305 is used to mark the distorted edge as a passable area to redefine the second obstacle.

[0090] Combine the following Fig.14 The electronic device of the present application is described, wherein: Fig.14 A schematic block diagram of an electronic device according to an embodiment of the present application is shown.

[0091] like Fig.14 As shown, the electronic device 1400 includes: one or more memories 1401 and one or more processors 1402, wherein the memory 1401 stores a computer program executed by the processor 1402, and when the computer program is executed by the processor 1402, the processor 1402 executes the obstacle detection method described above.

[0092] The electronic device 1400 may be part or all of a computer device that can implement the obstacle detection method through software, hardware, or a combination of software and hardware.

[0093] like Fig.14As shown, the electronic device 1400 includes one or more memories 1401, one or more processors 1402, a display (not shown), and a communication interface, etc. These components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). It should be noted that Fig.14 The components and structures of the electronic device 1400 shown are merely exemplary and non-limiting. The electronic device 1400 may also have other components and structures as required.

[0094] The memory 1401 is used to store various data and executable program instructions generated during the operation of the relevant program, such as for storing various application programs or algorithms for implementing various specific functions. It may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc.

[0095] The processor 1402 may be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may be other components in the electronic device 1400 to perform desired functions.

[0096] In one example, the electronic device 1400 also includes an output device that can output various information (eg, images or sounds) to the outside (eg, a user), and can include one or more of a display device, a speaker, and the like.

[0097] The communication interface can be an interface of any currently known communication protocol, such as a wired interface or a wireless interface, wherein the communication interface may include one or more serial ports, USB interfaces, Ethernet ports, WiFi, wired networks, DVI interfaces, device integrated interconnect modules or other suitable ports, interfaces, or connections.

[0098] In addition, according to an embodiment of the present application, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the corresponding steps of the obstacle detection method of the embodiment of the present application are executed. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0099] The obstacle detection system, electronic device, and storage medium of the embodiments of the present application can implement the aforementioned obstacle detection method, and therefore have the same advantages as the aforementioned obstacle detection method.

[0100] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application to this. Those of ordinary skill in the art may make various changes and modifications therein without departing from the scope and spirit of the present application. All these changes and modifications are intended to be included within the scope of the present application as required by the appended claims.

[0101] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0103] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0104] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present application should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with features less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0105] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0106] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0107] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all functions of some modules according to the embodiments of the present application. The present application can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0108] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0109] The above is only a specific implementation or description of a specific implementation of the present application, and the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An obstacle detection method, characterized in that: The method comprises: Segmenting the image to be processed into a traversable area and a first obstacle; Screening out a second obstacle of a preset type from the first obstacles; Acquire an outer contour of the second obstacle, and divide the outer contour of the second obstacle into a plurality of edges; determining a distorted edge among the plurality of edges of the outer contour; The distorted edge is marked as a traversable area to redefine the second obstacle.

2. The method according to claim 1, characterized in that: Before segmenting the image to be processed, the method further includes: Fisheye images of the current vehicle surroundings captured by at least two fisheye cameras are obtained, and the fisheye images are spliced ​​into a panoramic bird's-eye view to obtain the image to be processed.

3. The method according to claim 1, characterized in that Determining the distorted edges among the plurality of edges of the outer contour comprises: Determining a camera corresponding to the second obstacle; Determine whether there is a first edge among the multiple edges of the second obstacle that has an intersection with the edge of the vehicle body where the camera is located, and the distance between the intersection and the corresponding camera is within a second preset range; When the intersection point exists and the distance between the intersection point and the corresponding camera is within a second preset range, the first edge is determined to be a distorted edge.

4. The method according to claim 3, characterized in that The outer contour of the second obstacle is divided into a plurality of edges, including: calculating an inflection point of the outer contour of the second obstacle; The plurality of edges of the outer contour are determined according to adjacent inflection points.

5. The method according to claim 3, characterized in that: Determining the field of view of the fisheye camera corresponding to the second obstacle includes: When the second obstacle is located in the overlapping field of view of two adjacent fisheye cameras and the second obstacle occupies different ranges in each shooting field of view, the shooting field of view of the fisheye camera corresponding to the second obstacle is determined by the shooting field of view occupying the larger range.

6. The method according to claim 2, characterized in that The image to be processed is divided into a traversable area and a first obstacle, including: Dividing the panoramic bird's-eye view into a pre-processing area, a post-processing area, a left-side processing area, and a right-side processing area; The passable area and the first obstacle in the left processing area and the right processing area are divided.

7. The method according to claim 6, characterized in that The current vehicle has four vehicle side edges; after selecting a second obstacle of a preset type from the first obstacle, the method further includes: When the second obstacle is a square column, determining the vehicle side edge and the distance to the second obstacle that are the shortest; Determine the relationship between the distance and a preset distance threshold range; If the distance is less than the minimum value of the preset distance threshold range, adjusting the internal parameters and distortion parameters of the fisheye camera arranged at the edge of the vehicle body at the minimum distance from the second obstacle, so as to increase the shooting field of view of the fisheye camera arranged at the edge of the vehicle body at the minimum distance from the second obstacle; If the distance is greater than the maximum value of the preset distance threshold range, the internal parameters and distortion parameters of the fisheye camera set at the edge of the vehicle body with the smallest distance from the second obstacle are adjusted to reduce the shooting field of view of the fisheye camera set at the edge of the vehicle body with the smallest distance from the second obstacle.

8. The method according to claim 1, characterized in that Marking the distorted edge as a passable area to redefine the second obstacle includes: Calculating a first distance between any point on the distorted edge and the corresponding vehicle body edge; Calculating the distortion component of the arbitrary point according to the first distance and a preset distortion coefficient; When the distortion score of any point exceeds a preset distortion score threshold, the any point is added to the set of points to be dedistorted until all points of the distorted edge are calculated; All points in the point set to be dedistorted are uniformly marked as passable areas.

9. An obstacle detection system, characterized in that: The system comprises: A segmentation module, used for segmenting the image to be processed into a passable area and a first obstacle; A screening module, used to screen out second obstacles of a preset type from the first obstacles; an acquisition module, configured to acquire an outer contour of the second obstacle, and divide the outer contour of the second obstacle into a plurality of edges; A determination module, configured to determine the distorted edge among the plurality of edges of the external contour; A labeling module is used to label the distorted edge as a passable area to redefine the second obstacle.

10. A storage medium, characterized in that: The storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the obstacle detection method according to any one of claims 1 to 8.

Images