A surround obstacle detection system, method and storage medium

Abstract

Embodiments of the present application provide a detection system, method and storage medium for detecting around obstacles, a target detection module of the detection system is configured to obtain obstacle target frames and orientation angle prediction information according to images of surroundings of each vehicle; a ground segmentation module is configured to perform ground segmentation on the images of surroundings of each vehicle to obtain a ground segmentation point set; a frame point fusion module is configured to fuse the obstacle target frames and the ground segmentation point set to obtain a target segmentation point set corresponding to each obstacle target; a target segmentation point straight line fitting module is configured to perform straight line fitting on the target segmentation point set of each obstacle target to obtain surrounding line information; a pose analysis module is configured to obtain obstacle pose information in a vehicle rear axle coordinate system through coordinate transformation; and a multi-camera recognition fusion module is configured to fuse obstacle pose information corresponding to vehicle cameras in each direction of the vehicle in a consensus area in the vehicle rear axle coordinate system to obtain around position information of the obstacle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal processing, in particular to a surround obstacle detection system and method and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, automatic driving technology is becoming mature. Among them, automatic parking is an indispensable part of the automatic driving process. In order to ensure the safety of the vehicle during automatic parking, it is necessary to detect the obstacles in front of or around the vehicle body to avoid collision between the vehicle and the obstacles. In the prior art, there are some that use surround images of parking to detect obstacles, some that use laser radar and image fusion method for detection, and some that use neural network and traditional algorithm to extract image target features, and then use multi-feature fusion to detect obstacles.

[0003] However, simply using surround images of parking to detect obstacles for parking control lacks spatial position information of obstacles, and cannot distinguish and identify obstacles of different distances and sizes. The method of using laser radar and image fusion greatly increases the cost of object detection, which is not conducive to large-scale commercial application. Moreover, the laser radar is prone to blind area, the installation position is limited, the occupied space is larger than the camera, and the 3D point cloud image needs to map the labeled 3D bounding box to 2D image to label the position of the vehicle. This process increases the data labeling cost but has low labeling efficiency. In addition, the neural network or traditional algorithm is used to extract image target features, and then multi-feature fusion is used to realize three-dimensional target detection, but the actual effect is not good, and there is no real world coordinate corresponding to the camera parameter, and the error is large. SUMMARY

[0004] The present application provides a surround obstacle detection system and method and a storage medium to overcome at least one technical problem in the prior art.

[0005] In a first aspect, according to the embodiments of the present application, a surround obstacle detection system is provided, comprising:

[0006] a target detection module configured to perform target position regression and orientation angle prediction on a plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle body to obtain obstacle target frames and orientation angle prediction information, wherein the plurality of vehicle surrounding environment images collected by the vehicle cameras in each direction of the vehicle body cover the four directions of the vehicle;

[0007] a ground segmentation module configured to input each of the vehicle surrounding environment images into a semantic segmentation network to segment the ground in the vehicle surrounding environment images to obtain a ground segmentation point set;

[0008] a frame point fusion module configured to fuse the obstacle target frame and the ground segmentation point set to obtain a target segmentation point set corresponding to each obstacle target;

[0009] a target segmentation point straight line fitting module configured to perform straight line fitting on the target segmentation point set of each obstacle target to obtain surround line information of each obstacle target;

[0010] a pose analysis module configured to obtain obstacle pose information in a vehicle rear axle coordinate system according to the surround line information and the orientation angle prediction information of each obstacle target through coordinate transformation;

[0011] a multi-camera recognition fusion module configured to fuse obstacle pose information corresponding to vehicle cameras in each direction of the vehicle in the vehicle rear axle coordinate system to obtain surround position information of the obstacle.

[0012] Optionally, the target detection module is configured to perform target position regression and orientation angle prediction on a plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle, and specifically configured to,

[0013] input the plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle into a trained deep learning model, respectively, and use the trained deep learning model to perform target detection and orientation angle prediction on the obstacle to obtain obstacle information; the obstacle information includes the obstacle target frame and the orientation angle prediction information.

[0014] Further optionally, the trained deep learning model is formed through the following training steps:

[0015] constructing a deep learning model;

[0016] obtaining training images; the training images are historical vehicle surrounding environment images collected by the vehicle cameras;

[0017] performing Bounding Box labeling on historical obstacle targets in the historical vehicle surrounding environment images, and labeling orientation angle information of each historical obstacle target relative to the vehicle to obtain training data, and training the constructed deep learning model to obtain the trained deep learning model.

[0018] Optionally, the ground segmentation module is configured to input each vehicle surrounding environment image into a semantic segmentation network to segment the ground in the vehicle surrounding environment image, and specifically configured to,

[0019] obtain the vehicle surrounding environment image;

[0020] segmenting a ground in the vehicle surrounding image by using a semantic segmentation network to obtain a ground segmentation point set, wherein the semantic segmentation network is a neural network model trained based on historical vehicle surrounding images containing semantic segmentation labels, and the semantic segmentation labels include ground segmentation point labels.

[0021] Optionally, the bounding box fusion module fuses the obstacle target bounding box and the ground segmentation point set, and the fusion is specifically used for

[0022] According to the obstacle target bounding box, a target fusion range is determined.

[0023] The ground segmentation point set is extracted, and the ground segmentation point set in the target fusion range is selected to obtain a target segmentation point set corresponding to the obstacle target.

[0024] Optionally, the surrounding line information includes a surrounding line and a plurality of target line segments, and the target segmentation point straight line fitting module performs straight line fitting on the target segmentation point set of each obstacle target, and the fitting is specifically used for

[0025] The target segmentation point set of each obstacle target is obtained, and the target segmentation point set includes target segmentation points.

[0026] A random sample consensus (RANSAC) algorithm is used to perform straight line fitting on each target segmentation point set to obtain a surrounding line, and the surrounding line and the target segmentation points are used to obtain a plurality of target line segments.

[0027] Further optionally, the obstacle pose information includes obstacle position information and obstacle orientation angle information, and the pose analysis module obtains the obstacle pose information in the vehicle rear axle coordinate system according to the surrounding line information and the orientation angle prediction information of each obstacle target through coordinate transformation, and the analysis is specifically used for

[0028] The orientation angle prediction information of the obstacle target is obtained, and the target segmentation point set and the plurality of target line segments of the obstacle target are obtained; the orientation angle prediction information includes a predicted orientation angle of the obstacle target.

[0029] According to the plurality of target line segments and the target segmentation point set, the center point coordinates of each target line segment are obtained, and the normal vectors of each target line segment are obtained by using the center point coordinates of each target line segment.

[0030] Matching the predicted orientation angle of the obstacle target with the normal vector of each target line segment to obtain the included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment, and selecting the target line segment corresponding to the minimum included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment as the obstacle judgment standard;

[0031] Based on the camera parameters of the vehicle camera, performing coordinate transformation on the target segmentation point corresponding to the obstacle judgment standard, and determining the obstacle position information of the obstacle target in the vehicle rear axle coordinate system according to the pre-stored vehicle prior information;

[0032] Taking the normal vector of the obstacle judgment standard as the obstacle orientation angle information.

[0033] Optionally, the multi-camera recognition fusion module performs consensus region fusion on the obstacle pose information corresponding to the vehicle camera in each direction of the vehicle body in the vehicle rear axle coordinate system, specifically,

[0034] Obtaining the obstacle pose information corresponding to the vehicle camera in each direction of the vehicle body; the obstacle pose information includes obstacle position information and obstacle orientation angle information; the obstacle position information includes an obstacle position box;

[0035] Separating all the obstacle position information into four quadrants;

[0036] In each quadrant, it is judged whether the Euclidean distance between two adjacent obstacle position boxes is less than a threshold value;

[0037] If the Euclidean distance between two adjacent obstacle position boxes is less than a threshold value, coordinate averaging is performed on the two obstacle position boxes for consensus region fusion;

[0038] Taking the obstacle position information after consensus region fusion as the surround view position information of the obstacle.

[0039] In a second aspect, according to an embodiment of the present specification, a surround view obstacle detection method is provided, comprising:

[0040] Performing target position regression and orientation angle prediction on a plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle body to obtain an obstacle target box and orientation angle prediction information; wherein the plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle body cover the four directions of the vehicle;

[0041] Inputting each vehicle surrounding environment image into a semantic segmentation network to segment the ground in the vehicle surrounding environment image to obtain a ground segmentation point set;

[0042] fuse the obstacle target frame with the ground segmentation point set to obtain a target segmentation point set corresponding to each obstacle target;

[0043] perform straight line fitting on the target segmentation point set of each obstacle target to obtain surrounding line information of each obstacle target;

[0044] obtain obstacle pose information in a vehicle rear axle coordinate system according to the surrounding line information and the heading angle prediction information of each obstacle target through coordinate transformation;

[0045] fuse the obstacle pose information corresponding to the vehicle camera in each direction of the vehicle body in the vehicle rear axle coordinate system to obtain surround view position information of the obstacle.

[0046] In a third aspect, an embodiment of the present specification provides a storage medium storing a computer program, which, when executed, performs the surround view obstacle detection method of the second aspect.

[0047] The beneficial effects of the embodiments of the present specification are as follows:

[0048] The vehicle itself is installed with calibrated vehicle cameras to collect the vehicle surrounding environment images. The collected vehicle surrounding environment images without distortion are input into a target detection module for target position regression and vehicle heading angle prediction. Meanwhile, the vehicle surrounding environment images are input into a ground segmentation module to obtain a ground segmentation point set. Then, a frame point fusion module is used to fuse the ground segmentation point set and the obstacle target frame output by the target detection module to obtain a segmentation point set of each obstacle target in the nearby area as a target segmentation point set. Then, each target segmentation point set obtained is input into a target segmentation point straight line fitting module for straight line fitting to obtain surrounding line information of each obstacle target. A pose analysis module uses the surrounding line information and the heading angle prediction information of the obstacle target to obtain the obstacle pose in the vehicle rear axle coordinate system through coordinate transformation. Then, a multi-camera recognition fusion module is used to fuse the recognition data under multiple cameras to obtain the surround view position information of the vehicle surrounding obstacle.

[0049] The detection system estimates the pose information of the obstacle using 2D images without using 3D annotation data, positions the target object through the fusion of target detection and ground segmentation, realizes the association of detection and segmentation through a frame point fusion algorithm, fuses the obstacle pose information in each direction of the vehicle, and obtains the surround view position information of the vehicle surrounding obstacle. The data annotation cost is low, the annotation efficiency is high, and since the obstacle can be directly detected in the original image, the system can be used even if the image is distorted, the implementation effect is better, and the error is small. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present specification, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0051] Figure 1 The structural block diagram of the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure:

[0052] Figure 2 The schematic diagram of Bounding Box labeling and vehicle orientation labeling in the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure:

[0053] Figure 3 The image schematic diagram of ground segmentation in the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure:

[0054] Figure 4 The effect diagram after ground segmentation in the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure:

[0055] Figure 5 The obstacle target frame schematic diagram of the vehicle surrounding environment image in the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure:

[0056] Figure 6 The frame point fusion schematic diagram of the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure:

[0057] Figure 7 The line segment fitting schematic diagram of the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure, wherein (a) is a ground area diagram of the obstacle target, (b) is a ground segmentation point set diagram of the obstacle target, and (c) is a target segmentation point set diagram of the obstacle target.

[0058] Figure 8 The pose analysis schematic diagram of the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure, wherein (a) is a target segmentation point set diagram of the obstacle target, (b) is a target line segment diagram, and (c) is an obstacle judgment standard diagram.

[0059] Figure 9 The consensus area fusion schematic diagram of the surround obstacle detection system provided by the embodiments of the present specification is shown in the following figure, wherein (a) is a position relationship diagram of the ego vehicle and the obstacle in the vehicle rear axle coordinate system, (b) is a quadrant separation diagram, and (c) is a position relationship diagram of the obstacle after four-direction fusion.

[0060] The reference signs are explained as follows: 1 is a target detection module, 2 is a ground segmentation module, 3 is a frame point fusion module, 4 is a target segmentation point straight line fitting module, 5 is a pose analysis module, 6 is a multi-camera recognition fusion module, and 7 is a vehicle camera. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present specification will be described clearly and completely in the embodiments of the present specification in combination with the drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0062] It should be noted that the terms "include" and "have" and any variations thereof in the embodiments of the present specification and the drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to the process, method, product or device.

[0063] The embodiments of the present specification disclose a surround obstacle detection system applied to a vehicle, which uses images collected by a vehicle camera installed on the vehicle to detect obstacles in an automatic parking process. The following will be described in detail.

[0064] Figure 1 A surround obstacle detection system according to an embodiment of the present specification is shown. As shown in FIG. 1, the surround obstacle detection system includes a target detection module 1, a ground segmentation module 2, a frame point fusion module 3, a target segmentation point straight line fitting module 4, a pose analysis module 5, a multi-camera recognition fusion module 6, and a vehicle camera 7. Figure 1As shown, the detection system mainly consists of six interrelated modules, target detection module 1, ground segmentation module 2, frame point fusion module 3, target segmentation point straight line fitting module 4, pose analysis module 5 and multi-camera recognition fusion module 6. Through target detection module 1, the vehicle camera 7 collects multiple vehicle surrounding environment images in each direction of the vehicle body for obstacle object detection. At the same time, the ground segmentation module 2 finely segments the ground in each vehicle surrounding environment image. Then, the post-processed segmentation points and the coordinates of the obstacle targets are input into the frame point fusion module 3 to obtain the target segmentation point set corresponding to each obstacle target. The target segmentation point straight line fitting module 4 performs straight line fitting on the target segmentation point set of each obstacle target. The pose analysis module 5 uses the fitted surrounding line and the predicted vehicle orientation angle to obtain the pose of the obstacle target in the vehicle rear axle coordinate system through coordinate transformation. Finally, the multi-camera recognition fusion module 6 is used for multi-camera surround view fusion to obtain the surround view position information of the vehicle surrounding obstacles. According to the surround view position information, the vehicle surrounding obstacles are accurately judged to prevent collision between the vehicle and the obstacles, and further improve the safety of the parking function system. In the embodiment of the present specification, the vehicle camera 7 refers to a calibrated fisheye camera, also known as a calibrated camera.

[0065] Specifically, the target detection module 1 is connected to the vehicle camera 7 and is used for target position regression and orientation angle prediction on the multiple vehicle surrounding environment images collected by the vehicle camera 7 in each direction of the vehicle body to obtain obstacle target frames and orientation angle prediction information. It should be noted that the multiple vehicle surrounding environment images collected by the vehicle camera 7 in each direction of the vehicle body cover the four directions of the vehicle, that is, the surround view images around the vehicle can be collected by the vehicle camera 7 in each direction of the vehicle body. In addition, the vehicle surrounding environment images collected by the vehicle camera 7 in each direction of the vehicle body are all real-time collected. The target detection module 1 performs target position regression and orientation angle prediction on the vehicle surrounding environment images collected by each vehicle camera 7 to obtain the obstacle target frame and the orientation angle prediction information corresponding to the vehicle surrounding environment images collected by each vehicle camera 7.

[0066] In a specific embodiment, the target detection module 1 obtains multiple vehicle surrounding environment images through the vehicle cameras 7 in each direction of the vehicle body to obtain a surround view of the vehicle, and then inputs the multiple vehicle surrounding environment images collected by the vehicle cameras 7 in each direction of the vehicle body into the trained deep learning model to perform target detection and orientation angle prediction on the obstacles by using the trained deep learning model to obtain obstacle information. The obtained obstacle information includes obstacle target boxes and orientation angle prediction information, and the vehicle cameras 7 refer to front cameras, rear cameras, left cameras or right cameras arranged around the vehicle body, and the vehicle cameras 7 in each direction of the vehicle body perform multi-directional image collection on the obstacles around the vehicle.

[0067] Further, the trained deep learning model is formed through the following training steps: first, a deep learning model is constructed; historical vehicle surrounding environment images collected by the vehicle cameras 7 are obtained as training images; then, historical obstacle targets in the historical vehicle surrounding environment images are labeled by Bounding Box, and each historical obstacle target is labeled with orientation angle information relative to the vehicle to obtain training data, and the constructed deep learning model is trained to obtain the trained deep learning model. It should be noted and understood that the historical vehicle surrounding environment images refer to images in a previous time period before the vehicle surrounding environment images input into the deep learning model after the training is completed, which can be any vehicle surrounding environment images in the previous time period, and only represents the chronological order.

[0068] The ground segmentation module 2 is connected to the vehicle cameras 7, and is used to input each vehicle surrounding environment image into a semantic segmentation network to segment the ground in the vehicle surrounding environment image to obtain a ground segmentation point set.

[0069] The semantic segmentation network is a neural network model trained based on historical vehicle surrounding environment images containing semantic segmentation labels, and the semantic segmentation labels include ground segmentation point labels.

[0070] The semantic segmentation network can be a structure in which a multi-task detection head is attached to a backbone network, and the backbone network can be a ResNet18 network, and the detection head is used to detect ground segmentation points. The historical vehicle surrounding environment images containing semantic segmentation labels are input into an initial semantic segmentation network for model training, and when the detection head converges, the final required semantic segmentation network is obtained.

[0071] In a specific embodiment, the ground segmentation module 2 obtains multiple vehicle surrounding environment images through each vehicle camera 7, and then segments the ground in the vehicle surrounding environment image by using the trained semantic segmentation network to obtain a ground segmentation point set to distinguish the ground and the obstacle target.

[0072] The frame-point fusion module 3 is connected with the ground segmentation module 2, and is configured to fuse the obstacle target frame and the ground segmentation point set to obtain a target segmentation point set corresponding to each obstacle target. It should be noted and understood that the frame-point fusion module 3 is configured to fuse the obstacle target frame and the ground segmentation point set corresponding to the same vehicle camera 7.

[0073] In a specific embodiment, the frame-point fusion module 3 determines a target fusion range according to the obstacle target frame, and then extracts the ground segmentation point set sent by the ground segmentation module 2, selects the ground segmentation point set in the target fusion range, and thus obtains the target segmentation point set corresponding to the obstacle target. The target fusion range can be a circle with the center point of the obstacle target frame as the center, and the diameter is set according to the length and width of the obstacle target frame.

[0074] The target segmentation point straight line fitting module 4 is connected with the frame-point fusion module 3, and is configured to perform straight line fitting on the target segmentation point set of each obstacle target to obtain the surround line information of each obstacle target. The surround line information includes a surround line and a plurality of target line segments.

[0075] In a specific embodiment, the target segmentation point straight line fitting module 4 obtains the target segmentation point set of each obstacle target obtained by the frame-point fusion module 3, and the target segmentation point set includes the target segmentation point of the obstacle target. Then, the random sample consensus (RANSAC) algorithm is used to perform straight line fitting on each target segmentation point set to obtain the surround line, and the surround line and the target segmentation point are used to obtain the plurality of target line segments.

[0076] The pose analysis module 5 is connected with the target segmentation point straight line fitting module 4, and is configured to obtain the obstacle pose information in the vehicle rear axle coordinate system according to the surround line information and the heading angle prediction information of each obstacle target through coordinate transformation. The obstacle pose information includes obstacle position information and obstacle heading angle information.

[0077] In a specific embodiment, the pose analysis module 5 obtains the orientation angle prediction information of the obstacle target obtained by the target detection module 1, the orientation angle prediction information including the predicted orientation angle of the obstacle target, and obtains the target segmentation point set and the plurality of target line segments of the obstacle target. The pose analysis module 5 obtains the center point coordinates of each target line segment according to the plurality of target line segments and the target segmentation point set, and obtains the normal vector of each target line segment using the center point coordinates of each target line segment. Then, the predicted orientation angle of the obstacle target is matched with the normal vector of each target line segment to obtain the included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment, and the target line segment corresponding to the smallest included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment is selected as the obstacle judgment standard. The pose analysis module 5 performs coordinate transformation on the target segmentation point corresponding to the obstacle judgment standard based on the camera parameters of the vehicle camera 7, and determines the obstacle position information of the obstacle target in the vehicle rear axle coordinate system according to the pre-stored vehicle prior information, and takes the normal vector of the obstacle judgment standard as the obstacle orientation angle information, thereby obtaining the obstacle pose information in the vehicle rear axle coordinate system.

[0078] The multi-camera recognition fusion module 6 is connected to the pose analysis module 5, and is used to fuse the obstacle pose information corresponding to the vehicle camera 7 in each direction of the vehicle body in the consensus area in the vehicle rear axle coordinate system, so as to obtain the surround view position information of the obstacle, and thus obtain the vehicle surround view space information.

[0079] In a specific embodiment, the multi-camera recognition fusion module 6 obtains the obstacle pose information corresponding to the vehicle camera 7 in each direction of the vehicle body obtained by the pose analysis module 5, wherein the obstacle pose information includes obstacle position information and obstacle orientation angle information, and the obstacle position information includes an obstacle position box. Then, the multi-camera recognition fusion module 6 separates all the obstacle position information into four quadrants, and sets the obstacle position information of the vehicle camera 7 in each direction of the vehicle body in the four quadrants. In each quadrant, it is judged whether the Euclidean distance between two adjacent obstacle position boxes is less than a threshold value. If the Euclidean distance between two adjacent obstacle position boxes is less than a threshold value, the two obstacle position boxes are averaged in coordinates for consensus area fusion. If the Euclidean distance between two adjacent obstacle position boxes is not less than a threshold value, it means that the two obstacle position boxes are two different obstacle targets, so consensus area fusion is not needed. The obstacle position information after consensus area fusion is taken as the surround view position information of the obstacle.

[0080] The above describes the various modules of the surround view obstacle detection system provided by the embodiment, their connection relationship, and the working process of the system. The following will be described in combination with Figure 1 Figure 9 ​Taking the obstacle target including the obstacle car and the like as an example, the working principle of the detection system for looking around the obstacle is described in detail.

[0081] In the embodiment of the present specification, the vehicle to which the detection system is applied is provided with a vehicle camera 7 in each of the front, rear, left and right directions of the vehicle body, i.e., a front camera, a rear camera, a left camera and a right camera. The vehicle surrounding environment images of the vehicle are collected by the vehicle camera 7 in each direction of the vehicle body, so as to obtain the surrounding view image of the vehicle. The front camera, the rear camera, the left camera and the right camera are all calibrated fisheye cameras. The target detection module 1 performs Bounding Box labeling on the historical obstacle targets such as the obstacle car in the historical vehicle surrounding environment images, and adds the orientation angle information of each historical obstacle target relative to the vehicle, as shown in Figure 2 After the collected image data is labeled, the labeled image data is trained by using a convolutional neural network (CNN) having a feature extraction capability, so as to obtain a trained deep learning model having a generalization capability. The trained deep learning model is used for target detection and orientation angle prediction of the obstacle target, and outputs the obstacle target frame and the orientation angle prediction information, i.e., the obstacle information of the obstacle target output by the trained deep learning model, for example, the obstacle information of the obstacle car is [x, y, w, h, to, do, …, dlo, dro], wherein (x, y) represents the center point coordinates of the obstacle car in the vehicle surrounding environment image, w and h represent the width and height of the obstacle target frame respectively, and to, do, …, dlo, dro represent the orientation angle prediction information, which represents the orientation angle mark of the obstacle car relative to the ego vehicle in the image coordinate system.

[0082] The target detection module 1 can detect the specific position of the obstacle in the 2D image, and give the pixel coordinates of the obstacle target in the image. In order to further obtain the specific position of the obstacle target, as shown in Figure 3 and Figure 4 The ground segmentation module 2 segments the ground in each vehicle surrounding environment image by using a semantic segmentation network, so as to distinguish the ground and the obstacle car. The ground segmentation point set [x1, y1, x2, y2, x3, y3, …] is obtained by the ground segmentation module 2, i.e., the set of edge point pixel coordinates between the ground and the obstacle, and (x1, y1), (x2, y2), (x3, y3) and the like are the pixel coordinates of the edge points.

[0083] The pixel coordinates of each obstacle target box are obtained by the target detection module 1, and the ground segmentation point set is output after the ground segmentation by the ground segmentation module 2. Then, the outputs of the target detection module 1 and the ground segmentation module 2 are fused by the box point fusion module 3, the obstacle target box and the ground segmentation point set corresponding to the same vehicle camera 7 are fused, and the target position description of the obstacle is obtained. The box point fusion module 3 obtains the obstacle information of the obstacle target, extracts the center point coordinates of each obstacle target box and the width and height of the obstacle target box, as shown in Figure 5 and Figure 6 Taking the obstacle target box of the obstacle car as an example, the box point fusion module 3 sets the circumscribed circle of the obstacle target box with the center point of the obstacle target box as the center, as the target fusion range of the obstacle car, and selects the ground segmentation point set in the circumscribed circle of the obstacle target box, thereby fusing the obstacle target box and the ground segmentation point, completing the box point fusion, and outputting the target segmentation point set corresponding to each obstacle target, for example, the target segmentation point set of the obstacle car is [xd1, yd1, xd2, yd2, …], (xd1, yd1), (xd2, yd2) and the like are target segmentation point coordinates.

[0084] The target segmentation point set of each obstacle target can be obtained by the box point fusion module 3, and the random sample consensus (RANSAC) algorithm is used to fit a straight line to the target segmentation point set of each obstacle target. The random sample consensus (RANSAC) algorithm can estimate the parameters of a mathematical model from a set of observation data containing “outliers” by an iterative method. The target segmentation point straight line fitting module 4 uses the algorithm to fit a straight line to the target segmentation point set of each obstacle target, as shown in Figure 7 The target line segment is obtained by using the target segmentation point, so that the line segment equation of each obstacle target and the pixel coordinates of the target segmentation point on the target line segment are output by the target segmentation point straight line fitting module 4.

[0085] As shown in Figure 8As shown, the pose analysis module 5 carries out the calculation of the line segment center point coordinates of the pixel coordinates of the target line segment and the target segmentation point set obtained by the target segmentation point straight line fitting module 4, calculates the line segment center point of each obstacle target, and further obtains the normal vector of the target line segment. The predicted orientation angle of the obstacle target obtained by the target detection module 1 is compared and matched with the normal vector of the target line segment, the included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment is obtained, and the target line segment corresponding to the smallest included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment is selected as the calculation standard, i.e. the obstacle judgment standard. The target segmentation points of the selected target line segment are converted into the vehicle rear axle coordinate system based on the camera parameters of the vehicle camera 7 and the calibrated camera (i.e. the vehicle camera 7), i.e. the line segment center point and the target line segment of the target line segment are converted to the vehicle rear axle coordinate system, and the position of the obstacle in the vehicle rear axle coordinate system can be determined according to the pre-stored obstacle prior information of the pose analysis module 5, and the normal vector of the target line segment is taken as the orientation angle of the obstacle vehicle, so that the spatial position information of the obstacle vehicle is obtained.

[0086] The pose analysis module 5 can obtain the position coordinates and orientation angle information of the obstacle vehicle in the vehicle rear axle coordinate system, for example, the obstacle vehicle coordinates are [(x1, y1, z1), (x2, y2, z2), (x3, y3, z3), (x4, y4, z4), (x5, y5, z5), (x6, y6, z6), (x7, y7, z7), (x8, y8, z8)], wherein the coordinates are values in the vehicle rear axle coordinate system. Since the multiple vehicle surrounding environment images collected by the vehicle camera 7 in each direction of the vehicle body are surround view images of the vehicle surrounding environment, there are consensus regions between the left and front, front and right, right and rear, and right and left of the vehicle camera 7, that is, the obstacle vehicle may be photographed by the front camera and the left camera at the same time. In order to output accurate information and reduce redundant information, it is necessary to fuse the detection target information of the front, rear, left and right of the vehicle in the vehicle rear axle coordinate system. As shown in the figure, Figure 9 As shown, the multi-camera recognition fusion module 6 uses the obstacle pose information obtained from the pose analysis module 5 to separate the four quadrants, and calculates the Euclidean distance d between the adjacent two frames in each quadrant. If d is less than the set threshold value k, it means that the two frames are the same obstacle target object, and then the coordinate average of the two frames is calculated; if d is not less than the set threshold value k, it means that the two frames are two obstacle target objects, and then no consensus area fusion is needed. After passing through the multi-camera recognition fusion module 6, the surround view position information of the obstacle target in the vehicle rear axle coordinate system after fusion in four directions can be obtained. The threshold value k can be an empirical value or a calibrated value.

[0087] In summary, the present specification discloses a surround obstacle detection system, which uses a calibrated vehicle camera installed on the vehicle to collect vehicle surrounding environment images, inputs the collected vehicle surrounding environment images without distortion into a target detection module to regress target position and predict vehicle orientation angle, simultaneously inputs the vehicle surrounding environment images into a ground segmentation module to obtain a ground segmentation point set, and then uses a box point fusion module to fuse the ground segmentation point set and the obstacle target box output by the target detection module to obtain a segmentation point set of each obstacle target in the surrounding area as a target segmentation point set. Then, each target segmentation point set obtained is input into a target segmentation point straight line fitting module for straight line fitting to obtain the surround line information of each obstacle target. A pose analysis module uses the surround line information and the orientation angle prediction information of the obstacle target to obtain the obstacle pose in the vehicle rear axle coordinate system through coordinate transformation. A multi-camera recognition fusion module fuses the recognition data under multiple cameras to obtain the surround position information of the obstacles around the vehicle.

[0088] The detection system estimates the pose information of the obstacles using 2D images without using 3D annotation data, positions the target object through the fusion of target detection and ground segmentation, realizes the association of detection and segmentation through a box point fusion algorithm, fuses the pose information of the obstacles in each direction of the vehicle, obtains the surround position information of the obstacles around the vehicle, has low data annotation cost and high annotation efficiency, and can directly detect the obstacles in the original image, so that the effect is better and the error is small even if the image is distorted.

[0089] Corresponding to the above-mentioned surround obstacle detection system embodiment, the present embodiment also provides a surround obstacle detection method, which comprises the following steps:

[0090] The target position regression and orientation angle prediction are performed on the multiple vehicle surrounding environment images collected by the vehicle camera 7 in each direction of the vehicle body to obtain the obstacle target box and the orientation angle prediction information. The multiple vehicle surrounding environment images collected by the vehicle camera in each direction of the vehicle body cover the four directions of the vehicle.

[0091] The ground segmentation point set is obtained by inputting each vehicle surrounding environment image into a semantic segmentation network to segment the ground in the vehicle surrounding environment image.

[0092] The result fusion is performed on the obstacle target box and the ground segmentation point set to obtain the target segmentation point set corresponding to each obstacle target.

[0093] The straight line fitting is performed on the target segmentation point set of each obstacle target to obtain the surround line information of each obstacle target.

[0094] According to the surrounding line information and the heading angle prediction information of each obstacle target, obstacle pose information in a vehicle rear axle coordinate system is obtained through coordinate transformation.

[0095] The obstacle pose information corresponding to the vehicle camera 7 in each direction of the vehicle body is fused in a consensus area in the vehicle rear axle coordinate system to obtain surround view position information of the obstacle.

[0096] It should be noted that the surround view obstacle detection method provided by the embodiment of the present application has the same technical effects as the surround view obstacle detection system of the present application based on the same concept, and the description not mentioned in the present embodiment can be referred to the description in the surround view obstacle detection system embodiment of the present application, which will not be repeated here.

[0097] Correspondingly, the present embodiment also provides a storage medium, which stores a computer program, and the computer program is executed to perform the surround view obstacle detection method described in the above embodiment.

[0098] Those skilled in the art can understand that the drawings are only schematic diagrams of an embodiment, and the modules or flows in the drawings are not necessarily necessary for implementing the present application.

[0099] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed and located in one or more devices different from the present embodiment. The modules in the above embodiment can be combined as one module, or can be further split into multiple sub-modules.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the present embodiment.

Claims

1. A system for detecting around an obstacle, characterized by, The method comprises the following steps: a target detection module is configured to perform target position regression and orientation angle prediction on a plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle body to obtain obstacle target boxes and orientation angle prediction information, wherein the plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle body cover the four directions of the vehicle; a ground segmentation module is configured to input each of the vehicle surrounding environment images into a semantic segmentation network to segment the ground in the vehicle surrounding environment images to obtain a ground segmentation point set; a box point fusion module is configured to fuse the obstacle target boxes and the ground segmentation point set to obtain a target segmentation point set corresponding to each obstacle target; a target segmentation point straight line fitting module is configured to perform straight line fitting on the target segmentation point set of each obstacle target to obtain surrounding line information of each obstacle target; a pose analysis module is configured to obtain obstacle pose information in a vehicle rear axle coordinate system according to the surrounding line information and the orientation angle prediction information of each obstacle target through coordinate transformation, wherein the obstacle pose information comprises obstacle position information and obstacle orientation angle information, and the pose analysis module obtains the obstacle pose information in the vehicle rear axle coordinate system according to the surrounding line information and the orientation angle prediction information of each obstacle target through coordinate transformation, specifically as follows: obtaining the orientation angle prediction information of the obstacle target, and obtaining the target segmentation point set and a plurality of target line segments of the obstacle target; the orientation angle prediction information comprises a predicted orientation angle of the obstacle target; obtaining the center point coordinates of each target line segment according to the target line segments and the target segmentation point set, and obtaining the normal vector of each target line segment by using the center point coordinates of each target line segment; matching the predicted orientation angle of the obstacle target with the normal vector of each target line segment to obtain the included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment, and selecting the target line segment corresponding to the smallest included angle between the predicted orientation angle of the obstacle target and the normal vector of each target line segment as an obstacle judgment standard; performing coordinate transformation on the target segmentation point corresponding to the obstacle judgment standard based on the camera parameters of the vehicle camera, and determining the obstacle position information of the obstacle target in the vehicle rear axle coordinate system according to pre-stored vehicle prior information; taking the normal vector of the obstacle judgment standard as the obstacle orientation angle information; a multi-camera recognition fusion module is configured to fuse the obstacle pose information corresponding to the vehicle cameras in each direction of the vehicle body in the consensus area in the vehicle rear axle coordinate system to obtain surround position information of the obstacle.

2. The around-view obstacle detection system of claim 1, wherein, The target detection module performs target position regression and orientation angle prediction on a plurality of vehicle surrounding environment images collected by vehicle cameras in each direction of the vehicle body, specifically as follows: The vehicle camera captures a plurality of vehicle surrounding environment images in each direction of the vehicle body, and inputs the vehicle surrounding environment images into a trained deep learning model, and uses the trained deep learning model to detect the obstacles and predict the orientation angles to obtain the obstacle information. The obstacle information includes the obstacle target frame and the orientation angle prediction information.

3. The surround view obstacle detection system of claim 2, wherein, The trained deep learning model is formed through the following training steps: constructing a deep learning model; obtaining training images; the training images are historical vehicle surrounding environment images captured by the vehicle camera; performing Bounding Box labeling on historical obstacle targets in the historical vehicle surrounding environment images, and labeling the orientation angle information of each historical obstacle target relative to the vehicle to obtain training data, and training the constructed deep learning model to obtain the trained deep learning model.

4. The around-view detection system of claim 1, wherein, The ground segmentation module inputs each vehicle surrounding environment image into a semantic segmentation network to segment the ground in the vehicle surrounding environment image, specifically for, obtaining the vehicle surrounding environment image; segmenting the ground in the vehicle surrounding environment image using a semantic segmentation network to obtain a ground segmentation point set, wherein the semantic segmentation network is a neural network model trained based on historical vehicle surrounding environment images containing semantic segmentation labels, and the semantic segmentation labels include ground segmentation point labels.

5. The around-view detection system of claim 1, wherein, The frame point fusion module fuses the obstacle target frame and the ground segmentation point set, specifically for, determining a target fusion range according to the obstacle target frame; extracting the ground segmentation point set and selecting the ground segmentation point set within the target fusion range to obtain a target segmentation point set corresponding to the obstacle target.

6. The around-view detection system of claim 1, wherein, The surrounding line information includes a surrounding line and a plurality of target line segments, and the target segmentation point straight line fitting module performs straight line fitting on the target segmentation point set of each obstacle target, specifically for, obtaining a target segmentation point set of each obstacle target; the target segmentation point set includes target segmentation points; performing straight line fitting on each target segmentation point set using a random sample consensus (RANSAC) algorithm to obtain a surrounding line, and using the surrounding line and the target segmentation points to obtain a plurality of target line segments.

7. The around-view obstacle detection system of claim 1, wherein, The multi-camera recognition fusion module fuses the obstacle pose information corresponding to the vehicle camera in each direction of the vehicle body in the vehicle rear axle coordinate system, specifically for, obtaining obstacle pose information corresponding to the vehicle camera in each direction of the vehicle body; the obstacle pose information includes obstacle position information and obstacle orientation angle information; the obstacle position information includes an obstacle position frame; separating all obstacle position information into four quadrants; determining whether the Euclidean distance between two adjacent obstacle position frames is less than a threshold value in each quadrant; if the Euclidean distance between two adjacent obstacle position frames is less than a threshold value, performing coordinate averaging on the two obstacle position frames to perform consensus area fusion; the consensus area fused obstacle position information is used as the surround view position information of the obstacle.

8. A method of detecting around an obstacle, characterized by, comprises: The vehicle camera captures multiple vehicle surrounding environment images in each direction of the vehicle body, target position regression and orientation angle prediction are performed on the multiple vehicle surrounding environment images to obtain obstacle target frame and orientation angle prediction information; wherein the multiple vehicle surrounding environment images captured by the vehicle camera in each direction of the vehicle body cover the four directions of the vehicle; Each of the vehicle surrounding environment images is input into a semantic segmentation network to segment the ground in the vehicle surrounding environment image, and a ground segmentation point set is obtained; The obstacle target frame and the ground segmentation point set are fused to obtain a target segmentation point set corresponding to each obstacle target; Linear fitting is performed on the target segmentation point set of each obstacle target to obtain the surrounding line information of each obstacle target; According to the surrounding line information and the orientation angle prediction information of each obstacle target, the obstacle pose information in the vehicle rear axle coordinate system is obtained through coordinate transformation; wherein the obstacle pose information includes obstacle position information and obstacle orientation angle information, and the obstacle pose information in the vehicle rear axle coordinate system is obtained through coordinate transformation according to the surrounding line information and the orientation angle prediction information of each obstacle target. The specific steps include: obtaining the orientation angle prediction information of the obstacle target, and obtaining the target segmentation point set and multiple target line segments of the obstacle target; the orientation angle prediction information includes the predicted orientation angle of the obstacle target; according to the target line segments and the target segmentation point set, the center point coordinates of each target line segment are calculated, and the normal vectors of each target line segment are calculated respectively; the predicted orientation angle of the obstacle target is matched with the normal vectors of each target line segment to obtain the included angle between the predicted orientation angle of the obstacle target and the normal vectors of each target line segment, and the target line segment corresponding to the smallest included angle between the predicted orientation angle of the obstacle target and the normal vectors of each target line segment is selected as the obstacle judgment standard; based on the camera parameters of the vehicle camera, the target segmentation point corresponding to the obstacle judgment standard is subjected to coordinate transformation, and the obstacle position information of the obstacle target in the vehicle rear axle coordinate system is determined according to the pre-stored vehicle prior information; the normal vector of the obstacle judgment standard is used as the obstacle orientation angle information; The obstacle pose information corresponding to the vehicle camera in each direction of the vehicle body is fused in the consensus area in the vehicle rear axle coordinate system to obtain the surround position information of the obstacle.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program is executed to perform the surround obstacle detection method of claim 8.

Images