An obstacle detection method, system and storage medium

Abstract

The present application provides an obstacle detection method, system, and storage medium. The method includes: segmenting an image to be processed into a passable area and a first obstacle; screening out a second obstacle of a preset type from the first obstacle; obtaining an external contour of the second obstacle, and dividing the external contour of the second obstacle into multiple edges; determining the edges that are distorted among the multiple edges of the external contour; and labeling the distorted edges as passable areas to re-determine the second obstacle. In the embodiments of the present application, only the distorted edges of the obstacles visible from the perspective of the current vehicle are de-distorted, while the non-distorted edges and the edges of the obstacles that cannot be seen from the perspective of the current vehicle are not processed, thus saving computing power.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and particularly to an obstacle detection method, system and storage medium. Background Art

[0002] In traditional technologies, when a vehicle is driving, reversing, or parking, images around the vehicle are often collected by a camera to detect the surrounding driving environment. Perspective effects exist in the images captured by the camera. For example, lane lines that are originally parallel will tend to intersect in the picture. Therefore, inverse perspective mapping (IPM) processing needs to be performed on the images captured by the camera to eliminate the perspective effects of the images.

[0003] Inverse Perspective Mapping (IPM) refers to converting lane lines that are not parallel due to perspective transformation into parallel lines in a bird's-eye view by eliminating perspective effects, thereby helping us understand the scene in the image more accurately. However, during the generation of inverse perspective mapping (IPM) images, due to the complexity of perspective transformation and the precision of image processing, sometimes an obstacle distortion effect may occur.

[0004] The obstacle distortion effect means that in the image processed by inverse perspective mapping (IPM), a flat ground, an obstacle that is not actually prominent, or a specific type of obstacle is misidentified as an obstacle. This effect may be caused by factors such as errors in the image processing algorithm, inaccurate camera parameters, or noise during image acquisition. When these factors affect the precision of the inverse perspective mapping (IPM) transformation, pseudo-obstacles may appear in the generated inverse perspective mapping (IPM) image.

[0005] In subsequent image processing, if an image with an obstacle distortion effect is used for subsequent processing, adverse consequences will occur. For example, when using an image with a pseudo-obstacle effect to extract parking space lines, the detection of parking space lines will be inaccurate, and at the same time, the detection of the free space (FS) will also be inaccurate. Therefore, there is an urgent need to propose an obstacle detection method to solve the above problems. Summary of the Invention

[0006] In view of at least one of the above technical problems existing in the prior art, this application is proposed. According to one aspect of this application, an obstacle detection method is provided, and the method includes:

[0007] Segment the image to be processed into a passable area and a first obstacle;

[0008] Screen out a second obstacle of a preset type from the first obstacles;

[0009] Obtain the outer contour of the second obstacle, and divide the outer contour of the second obstacle into multiple edges;

[0010] Determine the distorted edges among the multiple edges of the outer contour;

[0011] Label the distorted edges as passable areas to re-determine the second obstacle.

[0012] In some embodiments, before segmenting the image to be processed, the method further includes:

[0013] Obtain fisheye images of the surrounding environment of the current vehicle collected by at least two fisheye cameras, and splice the fisheye images into a panoramic bird's-eye view image to obtain the image to be processed.

[0014] In some embodiments, determining the distorted edges among the multiple edges of the outer contour includes:

[0015] Determine the fisheye camera corresponding to the second obstacle;

[0016] Judge whether there is a first edge among the multiple edges of the second obstacle that intersects with the vehicle body edge where the fisheye camera is located, and the distance between the intersection point and the corresponding fisheye camera is within a second preset range;

[0017] When there is such an intersection point and the distance between the intersection point and the corresponding fisheye camera is within the second preset range, determine the first edge as the distorted edge.

[0018] In some embodiments, dividing the outer contour of the second obstacle into multiple edges includes:

[0019] Calculate the inflection points of the outer contour of the second obstacle;

[0020] Determine the multiple edges of the outer contour according to adjacent inflection points.

[0021] In some embodiments, determining the shooting field of view of the fisheye camera corresponding to the second obstacle includes:

[0022] When the second obstacle is within the overlapping field of view range of two adjacent fisheye cameras and the occupied ranges of the second obstacle in each shooting field of view are different, determine the shooting field of view of the fisheye camera corresponding to the second obstacle as the shooting field of view with the larger occupied range.

[0023] In some embodiments, segmenting the image to be processed into a passable area and a first obstacle includes:

[0024] Dividing the panoramic bird's-eye view into a pre-processing area, a post-processing area, a left processing area, and a right processing area;

[0025] Segmenting the passable area and the first obstacle in the left processing area and the right processing area.

[0026] In some embodiments, the current vehicle has four vehicle body edges; after screening out a second obstacle of a preset type from the first obstacle, the method further includes:

[0027] When the second obstacle is a square column, determining the vehicle body edge with the smallest distance from the second obstacle and the distance;

[0028] Judging the magnitude relationship between the distance and a preset distance threshold range;

[0029] If the distance is less than the minimum value of the preset distance threshold range, adjusting the internal parameters and distortion parameters of the fisheye camera arranged at the vehicle body edge with the smallest distance from the second obstacle to increase the shooting field of view of the fisheye camera arranged at the vehicle body edge with the smallest distance from the second obstacle;

[0030] If the distance is greater than the maximum value of the preset distance threshold range, adjusting the internal parameters and distortion parameters of the fisheye camera arranged at the vehicle body edge with the smallest distance from the second obstacle to reduce the shooting field of view of the fisheye camera arranged at the vehicle body edge with the smallest distance from the second obstacle.

[0031] In some embodiments, labeling the distorted edge as a passable area to re-determine the second obstacle includes:

[0032] Calculating a first distance between any point on the distorted edge and the corresponding vehicle body edge;

[0033] Calculating a distortion score of the any point according to the first distance and a preset distortion coefficient;

[0034] When the distortion score of the any point exceeds a preset distortion score threshold, adding the any point to a point set to be de-distorted until all points on the distorted edge are calculated;

[0035] Uniformly labeling all points in the point set to be de-distorted as passable areas.

[0036] Another aspect of the embodiments of the present application provides an obstacle detection system, and the system includes:

[0037] A segmentation module, configured to segment an image to be processed into a passable area and a first obstacle;

[0038] A screening module, configured to screen out a second obstacle of a preset type from the first obstacles;

[0039] An obtaining module, configured to obtain an external contour of the second obstacle and divide the external contour of the second obstacle into multiple edges;

[0040] A determining module, configured to determine a distorted edge among the multiple edges of the external contour;

[0041] A labeling module, configured to label the distorted edge as a passable area to re-determine the second obstacle.

[0042] Another aspect of the embodiments of the present application provides a storage medium, on which a computer program is stored, and when the computer program is run by a processor, the processor is caused to execute the obstacle detection method as described above.

[0043] In the obstacle detection method of the embodiments of the present application, by segmenting an image to be processed into a passable area within a first preset range and a first obstacle, screening out a second obstacle of a preset type from the first obstacles, then processing the external contour of the second obstacle to determine a distorted edge, and labeling the distorted edge as a passable area to re-determine the second obstacle; the embodiments of the present application only perform distortion removal processing on the distorted edges of obstacles of a preset type, and do not need to process the edges that are not distorted, thus saving computing power. Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0045] Figure 1 A schematic flowchart showing the obstacle detection method according to an embodiment of the present application;

[0046] Figure 2 A schematic flowchart showing the process 200 of obtaining an image to be processed according to an embodiment of the present application;

[0047] Figure 3 A schematic flowchart showing dividing the external contour of a second obstacle into multiple edges according to an embodiment of the present application;

[0048] Figure 4Shows a schematic flowchart of step S301 according to an embodiment of the present application;

[0049] Figure 5 Shows a schematic flowchart of S104 according to an embodiment of the present application;

[0050] Figure 6 Shows a schematic flowchart of step S501 according to an embodiment of the present application;

[0051] Figure 7 Shows a schematic flowchart of adjusting the shooting field of view of a fish-eye camera when detecting special obstacles according to an embodiment of the present application;

[0052] Figure 8 Shows a schematic diagram of the positional relationship between the current vehicle and the second obstacle according to an embodiment of the present application;

[0053] Figure 9(a) shows a schematic diagram of the real situation of obstacles around the current vehicle according to an embodiment of the present application;

[0054] Figure 9(b) shows a schematic diagram of the distortion situation of obstacles around the current vehicle according to an embodiment of the present application;

[0055] Figure 10(a) shows a schematic diagram of completing the initial bird's-eye view stitching process in a way that maximally retains the rear fish-eye image according to an embodiment of the present application;

[0056] Figure 10(b) shows a schematic diagram of completing the initial bird's-eye view stitching process in a way that maximally retains the left and right fish-eye images according to an embodiment of the present application;

[0057] Figure 11 Shows a schematic diagram of the obstacle detection method 1100 according to another embodiment of the present application;

[0058] Figure 12 Shows a schematic diagram of a panoramic bird's-eye view with segmentation lines formed after re-determining the passable area and the obstacle area according to an embodiment of the present application;

[0059] Figure 13 Shows a schematic block diagram of an obstacle detection system according to an embodiment of the present application;

[0060] Figure 14 Shows a schematic block diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0061] To enable those skilled in the art to better understand the technical solutions of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts shall fall within the protection scope of the present application.

[0062] Regarding the problem of obstacle detection on the current panoramic image, there are currently two main algorithms: One is to convert the three-dimensional space coordinates into the two-dimensional image plane coordinate system, and then match them with the segmentation results output by the segmentation model (the segmentation results include the labeled obstacles) to obtain dynamic obstacles and static obstacles, and then filter the distorted areas of the dynamic and static obstacles output by the segmentation model to obtain an accurate passable space; the other is to use an occupancy network model to determine obstacles based on the space occupancy results output by the model, and remove the space except for obstacles in the occupied space to obtain a passable space.

[0063] In the first solution, although the detection results of two-dimensional (2D) obstacles in an image are used for matching, the position information of the obstacles detected by this pure-vision two-dimensional (2D) obstacle model in the image is not accurate. Even if lidar can be used to detect the three-dimensional space coordinates of three-dimensional (3D) obstacles again to calculate the positions of the obstacles in the two-dimensional image plane coordinate system. However, if there are three cars arranged continuously on the panoramic image, due to the distortion effect of the panoramic image, the cars shown in the image are connected together, and the segmentation model will regard these three cars as a semantic segmentation block of one car. The result is that although the obstacles in the three-dimensional space converted into the obstacles in the image coordinate system can all be matched, an additional processing strategy is still needed, that is, to filter out the distorted space generated by each obstacle according to the orientation of the car and the position of the parking space entrance line. It can be seen that the first type of solution strongly depends on the three-dimensional (3D) obstacle detection method, and the computational complexity is very high.

[0064] In the second solution, the occupancy network model is a technique that can describe whether each position in three-dimensional or two-dimensional space is occupied. It can make few geometric assumptions about objects, so it can model objects of any shape and any form of object movement. The model trained by this technique can predict the occupancy of each position in three-dimensional or two-dimensional space. Using the occupancy space information predicted by the occupancy network model, obstacles are obtained based on the occupied space, and the passable space is obtained from the space other than the occupied space. Since there is little open-source data for training the occupancy network model, and when training the occupancy network model, the order of magnitude of the occupancy standard data that needs to be obtained is usually above 100,000 to achieve good enough results. Moreover, the iteration and optimization of the occupancy network model also require a lot of time. Therefore, using the occupancy network model to predict the passable space will increase the computing power, which is not a suitable solution for low-performance chips.

[0065] Therefore, in view of the above problems and considering the chip performance, the present application proposes an obstacle detection method to solve the distortion problem based on the fish-eye view. This method performs undistortion processing on the basis of the segmentation result obtained after semantic segmentation of the panoramic image. And since the segmentation result is often used for the extraction of the passable area (FS), in order to reduce the computing power, no further processing is performed on the already extracted passable area, but only the obstacles with distortion in the segmentation result are undistorted to optimize the computing power.

[0066] The obstacle detection method of the embodiment of the present application divides the image to be processed into a passable area and a first obstacle, and screens out a second obstacle of a preset type from the first obstacle, and then processes the outer contour of the second obstacle to determine the distorted edge, and marks the distorted edge as the passable area to re-determine the second obstacle; the embodiment of the present application only performs undistortion processing on the distorted edges of the obstacles of the preset type, and there is no need to process the undistorted edges, thus saving computing power.

[0067] The embodiment of the present application is not limited to the application scenario of intelligent driving, but also applicable to other related visual scenarios involving fish-eye distortion.

[0068] Figure 1 Shows a schematic flowchart of the obstacle detection method according to an embodiment of the present application; as Figure 1 shown, the obstacle detection method 100 according to an embodiment of the present application may include the following steps S101, step S102, step S103, step S104 and step S105:

[0069] In step S101, the image to be processed is divided into a passable area and a first obstacle.

[0070] Among them, the image to be processed may include a panoramic bird's-eye view or one or more fisheye images. In most application scenarios, fisheye images are usually stitched into a panoramic bird's-eye view to facilitate accurate detection of obstacles in the surrounding environment.

[0071] As Figure 2 shown, it is a schematic flowchart of a process 200 for obtaining an image to be processed. In an embodiment of the present application, before segmenting the image to be processed, the method further includes step S201:

[0072] In step S201, fisheye images of the surrounding environment of the current vehicle collected by at least two fisheye cameras are obtained, and the fisheye images are stitched into a panoramic bird's-eye view from the perspective of the current vehicle to obtain the image to be processed.

[0073] For example, four fisheye cameras can be installed on the current vehicle, respectively located at the front, rear, left, and right sides of the vehicle. For example, they are respectively installed at the center between the front cover opening and the front license plate of the current vehicle, the center of the upper border of the rear license plate, and below the two rearview mirrors. The shooting ranges of the four fisheye cameras can cover the area around the vehicle body. Specifically, by obtaining a plurality of fisheye images with different angles collected by each fisheye camera, performing inverse perspective transformation processing on each fisheye image to generate a corresponding initial bird's-eye view, and then stitching the initial bird's-eye views to generate the panoramic bird's-eye view from the perspective of the current vehicle.

[0074] Generally, because there will be a perspective effect in the pictures taken by fisheye cameras. For example, as shown in Fig. 9(a), it is a schematic diagram of the real situation of obstacles around the current vehicle. As shown in Fig. 9(b), it is a schematic diagram of the distortion situation of the surrounding obstacles. It can be seen from Fig. 9(a) and Fig. 9(b) that the front vehicle, the right vehicle, and the square column are all deformed. Therefore, when converting a fisheye image into a panoramic bird's-eye view, inverse perspective transformation (Inverse Perspective Mapping, IPM) needs to be performed on the fisheye image.

[0075] In an embodiment of the present application, when stitching fisheye images taken by fisheye cameras at 4 installation positions into a panoramic bird's-eye view, inverse perspective transformation (Inverse Perspective Mapping, IPM) processing can be performed on each fisheye image first to facilitate restoring the image to its real state and obtaining an initial bird's-eye view.

[0076] In addition, since the factory configurations and characteristics of each fisheye camera are different, the methods for stitching the initial bird's-eye view are also different. For example, as shown in FIGS. 10(a) and 10(b), they are schematic diagrams of the panoramic bird's-eye view formed after stitching the initial bird's-eye view. In FIG. 10(a), the fisheye image captured by the rear fisheye camera is maximally retained, and in FIG. 10(b), the fisheye images captured by the left and right fisheye cameras are maximally retained.

[0077] In the embodiment of the present application, by performing inverse perspective mapping (IPM) processing on the fisheye image, the fisheye image is converted from the original fisheye view to a more intuitive bird's-eye view. The bird's-eye view provides a complete view of the surrounding environment of the current vehicle, which enables the vehicle to more accurately judge the space and obstacles around it, thereby improving the accuracy of subsequent analysis work.

[0078] In an embodiment of the present application, the panoramic bird's-eye view can be input into a preset segmentation model for segmentation to obtain a passable area (FS) and a first obstacle within a first preset range.

[0079] Among them, the preset segmentation model can include segmentation models in traditional technologies such as semantic segmentation models. There is no limitation on the segmentation model here, as long as it can segment the panoramic bird's-eye view into a passable area (FS) and a first obstacle. While segmenting the panoramic bird's-eye view, all pixel points in the panoramic bird's-eye view are labeled. For example, each pixel is labeled as a passable area (FS) or a first obstacle.

[0080] In an example, the passable area (FS) and the first obstacle within the first preset range may refer to dividing the panoramic bird's-eye view into a pre-processing area, a post-processing area, a left processing area, and a right processing area, and determining the first obstacles in the left processing area and the right processing area. For example, according to the division method shown in FIGS. 10(a) or 10(b), the panoramic bird's-eye view can be divided into processing areas according to the shooting field of view of the fisheye camera (excluding the overlapping part). For another example, according to the convenience of the developer for processing, the pre, post, left, and right processing areas can all be divided into rectangles, and the rectangles of the pre, post, left, and right processing areas can form the panoramic bird's-eye view.

[0081] In an actual application scenario, since the current vehicle itself is in the passable area (FS), and due to the driver's perspective, the obstacles in the pre-processing area can be observed. Therefore, the computing power allocated to the pre-processing area and the post-processing area can be reduced, and the computing power allocated to the left processing area and the right processing area can be increased. This method of area-based processing facilitates the allocation of computing power according to the actual situation to save computing power.

[0082] In another example, the passable area (FS) and the first obstacle within the first preset range refer to the passable area (FS) and the obstacle area within a preset range (e.g., a relatively short distance) from the current vehicle. For example, it is the area within a radius of 4 m centered on the current vehicle. Since the current vehicle will not travel to the distant passable area (FS) during parking, there is no need to detect the distant first obstacle, and thus no need to perform segmentation processing on the distant passable area (FS) and the first obstacle. Only the nearby passable area (FS) and the first obstacle are processed to save computing power.

[0083] In step S102, second obstacles of a preset type are screened out from the first obstacles.

[0084] Among them, the first obstacles are all the obstacles segmented from the image to be processed. For example, they can include any obstacles such as pedestrians, vehicles, lane lines, ground locks, houses, building square columns, cone barrels, and walls. Due to the perspective effect in the fish-eye image, before splicing the fish-eye image into a panoramic bird's-eye view, it is necessary to perform inverse perspective mapping (IPM) processing on the fish-eye image. However, the inverse perspective mapping (IPM) processing can eliminate the perspective effect of ground elements, but cannot eliminate the distortion of obstacles such as square columns, pedestrians, vehicles, ground locks, cone barrels, and walls. That is to say, there is obstacle distortion for the obstacles. Therefore, in the embodiment of the present application, second obstacles with obstacle distortion need to be screened out from the first obstacles for distortion removal processing.

[0085] In the present application, the second obstacles of the preset type can include square columns, cone barrels, pedestrians, ground locks, vehicles, and walls, etc. Since the types of the first obstacles in the output result of the segmentation model are all marked, the second obstacles of the preset type can be directly filtered out from the first obstacles. Since the distortion of these types of obstacles is relatively obvious, by using the obstacle detection method of the embodiment of the present application to perform distortion removal processing on certain types of second obstacles, a better distortion removal effect can be obtained, making the recognized obstacles more accurate.

[0086] In step S103, the outer contour of the second obstacle is obtained, and the outer contour of the second obstacle is cut into multiple edges.

[0087] Among them, the outer contour of the second obstacle includes multiple edges.

[0088] Among them, the outer contour of the second obstacle is the boundary point of the obstacle area. The outer contour information of the second obstacle may include the position information or coordinate information of each point constituting the outer contour, etc.

[0089] Further, as Figure 3 shown, the step of dividing the outer contour of the second obstacle into multiple edges in step S103 includes step S301 and step S302:

[0090] In step S301, calculate the inflection points of the outer contour of the second obstacle;

[0091] In step S302, determine the multiple edges of the outer contour according to adjacent inflection points.

[0092] When calculating the inflection points of the outer contour of the second obstacle, all boundary points on the outer contour of the obstacle area can be traversed. For example, the inflection points can be found by calculating the slope of the outer contour. After traversing all the boundary points of the second obstacle twice from the start point to the end point and then from the end point to the start point, all the inflection points of the second obstacle can be determined. Here, traversing all the boundary points of the second obstacle twice from the start point to the end point and then from the end point to the start point means that all the boundary points of the second obstacle are traversed from the start point to the end point and then from the end point to the start point to ensure that all the boundary points of the second obstacle are detected. In the embodiment of the present application, an algorithm with one computing power can traverse all the boundary points on the second obstacle from start to end, and mark the boundary points that meet the conditions as inflection points. Since it is not necessary to re-determine the obstacle and its boundary points during the second traversal, computing power is saved.

[0093] The connection line between two adjacent inflection points forms an edge. In this way, the edges of the outer contour of the second obstacle that can be seen from the perspective of the current vehicle can be determined by determining the inflection points.

[0094] In step S104, determine the edges among the multiple edges of the outer contour that are distorted.

[0095] In an embodiment of the present application, as Figure 4 shown, the step of determining the fisheye-distorted edges among the multiple edges of the outer contour in step S104 includes step S401, step S402, and step S403:

[0096] In step S401, determine the fisheye camera corresponding to the second obstacle.

[0097] In step S402, determine whether there is a first edge among the multiple edges that intersects the vehicle body edge where the fisheye camera is located, and the distance between the intersection point and the corresponding fisheye camera is within a second preset range.

[0098] In step S403, when there is such an intersection point and the distance between the intersection point and the corresponding fisheye camera is within the second preset range, determine the first edge as the distorted edge.

[0099] The shooting field of view of a camera refers to the maximum range that the camera can observe, usually expressed in terms of angle. This range determines the breadth of the scenery that the camera can capture. In camera technology, the field of view is an important parameter that affects the monitoring ability and application scope of the camera. For example, some cameras have a small field of view and can only observe a limited area, while some cameras have a very large field of view and can capture a wider scene.

[0100] For a fish-eye camera, its shooting field of view can usually reach 180 degrees, which enables it to capture a very wide scene in a single photo. Therefore, in the embodiments of this application, a fish-eye camera is used to collect the surrounding environment of the current vehicle, in order to capture more information in the surrounding environment of the vehicle.

[0101] As Figure 8 shown, it is a schematic diagram of the positional relationship between the current vehicle and the second obstacle. Continuing with the above example, fish-eye cameras are respectively installed on the vehicle body edges AB, BC, CD, and DA of the current vehicle. The square pillar is within the shooting field of view of the fish-eye camera on the BC edge (that is, the camera corresponding to the second obstacle is the camera on the BC edge). The intersection point of one side of the square pillar and the vehicle body edge BC (that is, the vehicle body edge where the camera corresponding to the second obstacle is located) is E, and the distance between the intersection point E and the fish-eye camera on the BC edge is within a second preset range (for example, the distance between E and the fish-eye camera on the BC edge is less than 2 meters). Therefore, the above-mentioned one side of the square pillar is the distorted side. The intersection point of the other side of the square pillar and the vehicle body edge BC is F. However, since the distance between F and the fish-eye camera on the BC edge is greater than the preset distance range (for example, the distance between F and the fish-eye camera on the BC edge is greater than 2 meters), it does not meet the condition for the side to be judged as distorted, and the other side of the square pillar is not the distorted side.

[0102] In one embodiment of this application, as Figure 5 shown, in step S401 of determining the shooting field of view of the fish-eye camera corresponding to the second obstacle, it includes step S501:

[0103] In step S501, when the second obstacle is within the overlapping field of view of two adjacent fish-eye cameras and the ranges occupied by the second obstacle in each shooting field are different, the shooting field with a larger occupied range is determined as the shooting field of the fish-eye camera corresponding to the second obstacle.

[0104] Continuing to combine with Figure 8, since the shooting angle of a fisheye camera can reach 180 degrees and can capture a very wide scene, there must be an overlap in the shooting fields of two adjacent fisheye cameras. If the fisheye images captured by each fisheye camera are processed, it may lead to repeated processing of obstacles, thus wasting computing power. To avoid repeated processing of the obstacle area, it is necessary to determine the attribution of the obstacle area falling into the overlapping area.

[0105] For example, when both cameras can capture the same obstacle ( Figure 8 only the case where the obstacle can be captured by only one camera is shown), in the two-dimensional plane coordinate system, the areas of the obstacle falling into the captured images of two adjacent fisheye cameras can be calculated respectively. For the convenience of description, two adjacent fisheye cameras can be called the first fisheye camera and the second fisheye camera. If the area of the obstacle in the image captured by the first fisheye camera is greater than the area in the image captured by the second fisheye camera, it means that the fisheye camera corresponding to the obstacle is the first fisheye camera. On the contrary, the fisheye camera corresponding to the obstacle is the second fisheye camera; if the area of the obstacle in the image captured by the first fisheye camera is equal to the area of the obstacle in the image captured by the second fisheye camera, one of the fisheye cameras can be designated as the corresponding fisheye camera.

[0106] In an embodiment of the present application, determining the distorted sides among the multiple sides of the outer contour includes:

[0107] Determining the distorted sides of the outer contour of the second obstacle from the perspective of the current vehicle.

[0108] Continuing with the above example, based on the positions of the four fisheye cameras installed on the current vehicle, the fisheye camera can only capture the environmental images within the field of view, and the sides of the second obstacle facing away from the vehicle cannot be captured by the fisheye camera. For example, when the vehicle is passing one of the corners of a square pillar, only the two sides of the square pillar that form the above corner can be captured, while the other two sides facing away from the vehicle cannot be captured. Therefore, it is only necessary to determine the distorted sides among the sides captured from the perspective of the current vehicle. The embodiment of the present application only determines the distorted sides and performs distortion removal processing from the perspective of the current vehicle, and does not process the sides not in the perspective of the current vehicle and the sides that are not distorted, thus saving computing power.

[0109] In step S105, label the distorted sides as the passable area (FS) to re-determine the second obstacle.

[0110] Since the distorted edge in the second obstacle does not belong to the second obstacle, the distorted edge should be labeled as a passable area (FS) to re-determine the second obstacle. In one embodiment, labeling the distorted edge as a passable area (FS) means labeling each point on the distorted edge as a passable area (FS).

[0111] Specifically, the distortion score can be calculated based on the distance between the points on the distorted edge (i.e., the edge that has undergone distortion) and the corresponding vehicle body edge, the length of the distorted edge, and a preset distortion coefficient. Here, the preset distortion coefficient can be set by the user according to the need for detection accuracy. A set of points to be de-distorted can be constructed. When traversing the distorted edge, calculate the distortion score corresponding to the points on the distorted edge. When the distortion score exceeds the preset distortion score threshold, add the point to the set of points to be de-distorted. When all the points on the distorted edge have been calculated, then uniformly label the points in the set of points to be de-distorted as passable areas or as distorted points.

[0112] As Figure 6 shown, the step of labeling the distorted edge as a passable area (FS) in step S105 to re-determine the second obstacle includes step S601, step S602, step S603, and step S604:

[0113] In step S601, calculate the first distance between any point on the distorted edge and the corresponding vehicle body edge;

[0114] In step S602, calculate the distortion score of the any point according to the first distance, the length of the distorted edge, and the preset distortion coefficient:

[0115] In step S603, when the distortion score of the any point exceeds the preset distortion score threshold, add the any point to the set of points to be de-distorted until all the points on the distorted edge have been calculated;

[0116] In step S604, uniformly label all the points in the set of points to be de-distorted as passable areas.

[0117] In the embodiment of the present application, the purpose of calculating the distortion score is to avoid incorrect de-distortion processing and unnecessary distortion processing, which can also save some computing power.

[0118] Among them, when determining the corresponding vehicle body edge, the center of the obstacle (or, regarding the obstacle as a point) can be used to calculate the distances between the four vehicle body edges and the obstacle respectively, and the vehicle body edge with the closest distance is selected as the corresponding vehicle body edge.

[0119] Specifically, the distortion coefficients may include a first distortion coefficient, a second distortion coefficient, a third distortion coefficient, a fourth distortion coefficient, and a fifth distortion coefficient.

[0120] In one embodiment, the distortion score may be calculated in the following manner:

[0121] Multiply the first distortion coefficient by the first distance to obtain a first product, subtract the first product from the second distortion coefficient to obtain a first result, and then multiply the third distortion coefficient by the first result to obtain a first distortion factor; then multiply the fourth distortion coefficient, the fifth distortion coefficient, and the length of the distortion edge where the point to be labeled is located to obtain a second distortion factor; then add the first distortion factor and the second distortion factor to obtain the distortion score. For example, the first distortion coefficient is 0.02, the second distortion coefficient is 8.96, the third distortion coefficient is 0.7, the fourth distortion coefficient is 0.02, and the fifth distortion coefficient is 0.3.

[0122] Among them, the formula for calculating the distortion score is as follows:

[0123] ……(1)

[0124] Among them, represents the distortion score of the current point to be labeled (i.e., any point on the distorted edge); represents the distance from the current point to be labeled to the corresponding vehicle body edge (i.e., the first distance); represents the length of the edge of the obstacle where the current point to be labeled is located.

[0125] After traversing all points on the distorted edge, points that meet the conditions can be determined, and a set of points to be undistorted can be obtained. By modifying the annotation information (label) of all points in the set of points to be undistorted, it can be updated to a passable area (Road) or an undistorted point type (Deformation).

[0126] In the embodiment of the present application, in the distortion score formula, the reason for taking the distorted edge where the point to be labeled (any point on the distorted edge) is located as one of the calculation factors is that when the length of the distorted edge is very small, the impact is not significant. Therefore, in the case where the distorted edge is very short (for example, less than 1 meter), it can be ignored, or rather, in the case where the distorted edge is very short, points on the distorted edge do not need to be undistorted.

[0127] In the application scenario of parking, in the spatial design of an underground parking lot, variously shaped columns are used to support the building structure and provide necessary spatial separation, among which the most common are square columns. These square columns are usually located on both sides or around the parking spaces to support the weight of the above-ground building and maintain the load in case of an emergency. Therefore, when parking, drivers may often encounter square columns as one of the structural elements of the parking lot. To more accurately identify the position of the square column, the internal parameters and distortion parameters of the fish-eye camera can be adjusted according to the distance between the square column and the vehicle body, so as to facilitate the user's parking. As Figure 7 shown, after screening out the second obstacles of a preset type from the first obstacles, the method may include the following steps S701, step S702, step S703, step S704, and step S705:

[0128] In step S701, detect whether the second obstacle is a square column;

[0129] In step S702, when the second obstacle is a square column, determine the vehicle body edge closest to the second obstacle and the distance;

[0130] In step S703, judge the size relationship between the distance and a preset distance threshold range;

[0131] In step S704, if the distance is less than the minimum value of the preset distance threshold range, adjust the internal parameters and distortion parameters of the fish-eye camera set at the vehicle body edge closest to the second obstacle to increase the shooting field of view of the fish-eye camera set at the vehicle body edge closest to the second obstacle;

[0132] In step S705, if the distance is greater than the maximum value of the preset distance threshold range, adjust the internal parameters and distortion parameters of the fish-eye camera set at the vehicle body edge closest to the second obstacle to reduce the shooting field of view of the fish-eye camera set at the vehicle body edge closest to the second obstacle.

[0133] As described above, since when the panoramic bird's-eye view is input into the preset segmentation model for segmentation, the first obstacles have been output, that is, the types of the first obstacles have been labeled. Since the second obstacles are screened out from the first obstacles, that is to say, the types of the second obstacles have also been labeled, and the obstacles labeled as "square column" can be selected from these types.

[0134] Among them, the internal parameters of the camera of the fisheye camera may include: focal lengths fu and fv, and optical centers cu and cv, etc. The external parameters of the fisheye camera may include: pitch angle Pitch, yaw angle Yaw, and the height h of the center of the fisheye camera from the ground, etc. The distortion parameters of the fisheye camera (i.e., inverse perspective transformation parameters) may include: the size of the inverse perspective image, the area of the inverse perspective transformation, and the distance from the front projection of the camera, etc.

[0135] Continue to combine Figure 8 , the current vehicle has four edges AB, BC, CD, and DA. As shown in the figure, BC is an edge facing the second obstacle square column and is also the vehicle body edge with the smallest distance from the square column. Draw a perpendicular line segment from the center point of the square column (or regard the square column as a point) to the BC side, and the length of this perpendicular line segment is the distance from the square column to the BC side. If this distance is less than the minimum value of the preset distance threshold range, then enlarge the field of view of the fisheye camera on this vehicle body edge (i.e., the camera on the BC side), and the field of view of the adjacent fisheye camera can be adjusted accordingly; if the nearest distance is greater than the maximum value of the preset distance threshold range, then reduce the field of view of the fisheye camera on this vehicle body edge, and the field of view of the adjacent fisheye camera can be adjusted accordingly. For example, the preset distance threshold range is 3 to 4 meters. When the distance from the square column to the BC side is less than 3 meters, enlarge the shooting field of view of the fisheye camera set on the vehicle body edge BC; when the distance from the square column to the BC side is greater than 4 meters, reduce the shooting field of view of the fisheye camera set on the vehicle body edge BC.

[0136] The embodiment of the present application proposes an obstacle detection method for obstacle undistortion based on a panoramic segmentation map. After extracting the traversable area (FS) and the first obstacle, then undistort the selected second obstacle, omitting the step of calculating the distortion of other obstacles except the second obstacle, saving computing power.

[0137] Moreover, the technical solution of the embodiment of the present application does not rely on the results of other deep learning models and does not require additional introduction of other 3D models or Occupancy Network (OCC) for processing.

[0138] Moreover, in the technical solution of the embodiment of the present application, the re-determined traversable area (FS) can be used for grid map fusion of the subsequent fusion module, can also be used for fusion calculation based only on the inverse perspective transformation (IPM) traversable area (FS), and can also be used for occupancy detection of the parking space.

[0139] In addition, in the embodiment of the present application, the re-determined second obstacle is beneficial to subsequent fusion calculation. For example, updating the distorted edges of the second obstacle into points of the undistorted category can perform more accurate information calculation for these undistorted category points in the fusion calculation or in the occupancy calculation.

[0140] As shown Figure 11 in the figure, it is a schematic diagram of an obstacle detection method 1100 according to another embodiment of the present application. As can be seen Figure 11 from it, four fisheye cameras are respectively arranged in front of, behind, to the left and to the right of the current vehicle to collect fisheye images within the corresponding shooting ranges. The obstacle detection method 1100 of the embodiment of the present application includes the following steps S1101, step S1102, step S1103, step S1104, step S1105 and step S1106:

[0141] In step S1101, collect fisheye images within the corresponding shooting ranges; perform inverse perspective transformation processing on the fisheye images, where the fisheye images are the images directly captured by the fisheye cameras, including front fisheye images, rear fisheye images, left fisheye images and right fisheye images.

[0142] In step S1102, splice the four fisheye images after inverse perspective transformation to obtain a panoramic bird's-eye view.

[0143] In step S1103, use a segmentation model to segment the panoramic bird's-eye view to obtain a passable area (FS) and a second obstacle. Here, the second obstacle can be an obstacle of a preset type. For example, the second obstacle can at least include obstacles such as square columns, cone barrels, pedestrians, ground locks, vehicles and walls.

[0144] In step S1104, extract the boundary points of the passable area (FS).

[0145] Among them, the outer contour of the second obstacle includes multiple edges, and among them, the multiple edges include distorted edges and non-distorted edges.

[0146] In step S1105, perform distortion removal processing on the distorted edges.

[0147] In step S1106, convert the boundary points of the passable area (FS) from the segmentation map coordinate system to the vehicle coordinate system (Vehicle Coordinate System, VCS).

[0148] As shown Figure 12 in the figure, it is a schematic diagram of the panoramic bird's-eye view formed after re-determining the passable area (FS) and the second obstacle. Figure 12 The green dotted line segments in the upper left dotted box and the lower right dotted box on the left in the figure show the processing results of the distorted edges. As can be seen from the figure, the drivable area is more in line with the actual situation. Compared with the drivable area directly segmented by the segmentation model, after using the method of the present application, the range of the drivable area is expanded, and the result shows that the processing is successful.

[0149] The obstacle detection method according to the embodiment of the present application divides the image to be processed into a passable area and a first obstacle within a first preset range, screens out a second obstacle of a preset type from the first obstacle, then processes the outer contour of the second obstacle to determine the distorted edges, and labels the distorted edges as passable areas to re-determine the second obstacle. The embodiment of the present application only performs distortion removal processing on the distorted edges of the obstacles of the preset type, and does not need to process the edges that are not distorted, thus saving computing power.

[0150] The following will be combined with Figure 13 to describe the obstacle detection system of the present application, where Figure 13 FIG. shows a schematic block diagram of an obstacle detection system 1300 according to an embodiment of the present application.

[0151] Among them, the obstacle detection system 1300 includes:

[0152] A segmentation module 1301 for dividing the image to be processed into a passable area and a first obstacle;

[0153] A screening module 1302 for screening out a second obstacle of a preset type from the first obstacle;

[0154] An acquisition module 1303 for acquiring the outer contour of the second obstacle and dividing the outer contour of the second obstacle into multiple edges;

[0155] A determination module 1304 for determining the distorted edges among the multiple edges of the outer contour;

[0156] A labeling module 1305 for labeling the distorted edges as passable areas to re-determine the second obstacle.

[0157] The following will be combined with Figure 14 to describe the electronic device of the present application, where Figure 14 FIG. shows a schematic block diagram of an electronic device according to an embodiment of the present application.

[0158] As Figure 14 shown, the electronic device 1400 includes: one or more memories 1401 and one or more processors 1402. A computer program is stored on the memory 1401 and run by the processor 1402. When the computer program is run by the processor 1402, the processor 1402 executes the obstacle detection method described above.

[0159] The electronic device 1400 may be part or all of a computer device that can implement the obstacle detection method through software, hardware, or a combination of software and hardware.

[0160] As shown Figure 14 in FIG. 1400, the electronic device 1400 includes one or more memories 1401, one or more processors 1402, a display (not shown), a communication interface, etc., and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). It should be noted that Figure 14 the components and structures of the electronic device 1400 shown in FIG. 1400 are exemplary rather than restrictive. According to needs, the electronic device 1400 may also have other components and structures.

[0161] The memory 1401 is used to store various data and executable program instructions generated during the operation of relevant programs, such as storing various application programs or algorithms for implementing various specific functions. It may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0162] The processor 1402 may be a central processing unit (CPU), an image processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may be other components in the electronic device 1400 to perform desired functions.

[0163] In one example, the electronic device 1400 further includes an output device that can output various information (such as images or sounds) to the outside (such as users), and may include one or more of a display device, a speaker, etc.

[0164] The communication interface may be an interface of any currently known communication protocol, such as a wired interface or a wireless interface. Among them, the communication interface may include one or more serial ports, USB interfaces, Ethernet ports, WiFi, wired networks, DVI interfaces, device integrated interconnection modules, or other suitable various ports, interfaces, or connections.

[0165] In addition, according to the embodiments of the present application, a storage medium is further provided. Program instructions are stored on the storage medium, and when the program instructions are run by a computer or a processor, they are used to execute the corresponding steps of the obstacle detection method according to the embodiments of the present application. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media.

[0166] Since the obstacle detection system, electronic device, and storage medium according to the embodiments of the present application can implement the foregoing obstacle detection method, they have the same advantages as the foregoing obstacle detection method.

[0167] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present application. Those of ordinary skill in the art can make various changes and modifications therein without departing from the scope and spirit of the present application. All such changes and modifications are intended to be included within the scope of the present application as claimed in the appended claims.

[0168] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0169] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0170] In the specification provided here, a large number of specific details are set forth. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and technologies have not been shown in detail so as not to obscure the understanding of this specification.

[0171] Similarly, it should be understood that, in order to streamline this application and assist in understanding one or more of the various inventive aspects, in the description of the exemplary embodiments of this application, the various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the methods of this application should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected by the corresponding claims, the inventive point lies in that the corresponding technical problems can be solved by features less than all the features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into the detailed description, where each claim itself serves as a separate embodiment of this application.

[0172] Those skilled in the art will appreciate that, except where features are mutually exclusive, any combination can be used to combine all the features disclosed in this specification (including the accompanying claims, abstract, and drawings), as well as all the processes or units of any method or apparatus so disclosed. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) can be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0173] In addition, those skilled in the art will be able to understand that, although some of the embodiments described herein include certain features included in other embodiments rather than other features, the combination of features of different embodiments means that it is within the scope of this application and forms different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0174] The various component embodiments of this application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some of the modules according to the embodiments of this application. This application can also be implemented as a device program (such as a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0175] It should be noted that the above embodiments are illustrative of the present application rather than restrictive thereof, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim listing several devices, several of these devices can be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names.

[0176] As described above, the specific embodiments of the present application are merely illustrative or illustrative of the specific embodiments, and the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should be covered within the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An obstacle detection method, characterized in that, The method includes: Segmenting the image to be processed into a passable area and a first obstacle; Selecting a second obstacle of a preset type from the first obstacles; Obtaining the outer contour of the second obstacle and dividing the outer contour of the second obstacle into multiple edges; Determining the distorted edges among the multiple edges of the outer contour; Labeling the distorted edges as passable areas to re-determine the second obstacle; Among them, determining the distorted edges among the multiple edges of the outer contour includes: Determining the camera corresponding to the second obstacle; Judging whether there is a first edge among the multiple edges of the second obstacle that intersects the edge of the vehicle body where the camera is located and the distance between the intersection point and the corresponding camera is within a second preset range; In the case where there is such an intersection point and the distance between the intersection point and the corresponding camera is within the second preset range, determining the first edge as the distorted edge.

2. The method according to claim 1, wherein Before segmenting the image to be processed, the method further includes: Obtaining fisheye images of the surrounding environment of the current vehicle collected by at least two fisheye cameras and stitching the fisheye images into a panoramic bird's-eye view image to obtain the image to be processed.

3. The method according to claim 1, wherein Dividing the outer contour of the second obstacle into multiple edges includes: Calculating the inflection points of the outer contour of the second obstacle; Determining the multiple edges of the outer contour according to adjacent inflection points.

4. The method according to claim 1, wherein Determining the shooting field of view of the fisheye camera corresponding to the second obstacle includes: In the case where the second obstacle is within the overlapping field of view range of two adjacent fisheye cameras and the occupied ranges of the second obstacle in each shooting field of view are different, determining the shooting field of view of the fisheye camera corresponding to the second obstacle as the shooting field of view with a larger occupied range.

5. The method according to claim 2, wherein Segmenting the image to be processed into a passable area and a first obstacle includes: Dividing the panoramic bird's-eye view image into a pre-processing area, a post-processing area, a left processing area, and a right processing area; Segmenting the passable area and the first obstacle in the left processing area and the right processing area.

6. The method according to claim 5, characterized in that, The current vehicle has four vehicle body edges; after selecting a second obstacle of a preset type from the first obstacles, the method further includes: In the case where the second obstacle is a square column, determining the vehicle body edge with the smallest distance from the second obstacle and the distance; Judging the size relationship between the distance and a preset distance threshold range; If the distance is less than the minimum value of the preset distance threshold range, adjusting the internal parameters and distortion parameters of the fisheye camera set on the vehicle body edge with the smallest distance from the second obstacle to increase the shooting field of view of the fisheye camera set on the vehicle body edge with the smallest distance from the second obstacle; If the distance is greater than the maximum value of the preset distance threshold range, adjusting the internal parameters and distortion parameters of the fisheye camera set on the vehicle body edge with the smallest distance from the second obstacle to reduce the shooting field of view of the fisheye camera set on the vehicle body edge with the smallest distance from the second obstacle.

7. The method according to claim 1, wherein Labeling the distorted edges as passable areas to re-determine the second obstacle includes: Calculate the first distance between any point on the distorted edge and the corresponding vehicle body edge; Calculate the distortion score of the any point according to the first distance and a preset distortion coefficient; When the distortion score of the any point exceeds a preset distortion score threshold, add the any point to the set of points to be processed for distortion removal until all points on the distorted edge are calculated; Uniformly label all points in the set of points to be processed for distortion removal as passable areas.

8. An obstacle detection system, characterized in that, The system includes: A segmentation module, configured to segment a to-be-processed image into a passable area and a first obstacle; A screening module, configured to screen out second obstacles of a preset type from the first obstacles; An obtaining module, configured to obtain the outer contour of the second obstacle and divide the outer contour of the second obstacle into multiple edges; A determining module, configured to determine the distorted edges among the multiple edges of the outer contour; A labeling module, configured to label the distorted edges as passable areas to re-determine the second obstacle; The determining module is specifically configured to determine the camera corresponding to the second obstacle; determine whether there is a first edge among the multiple edges of the second obstacle that intersects with the vehicle body edge where the camera is located and the distance between the intersection point and the corresponding camera is within a second preset range; in the case where there is the intersection point and the distance between the intersection point and the corresponding camera is within the second preset range, determine the first edge as the distorted edge.

9. A storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is run by a processor, the processor is caused to execute the obstacle detection method according to any one of claims 1 to 7.

Images