Obstacle sensing method, system and equipment based on stereoscopic vision

Abstract

The invention relates to an obstacle sensing method, system and device based on stereoscopic vision, and belongs to the field of intelligent driving. The method is specifically applied to accurate sensing of vehicle AEBS obstacles in a traffic scene, a to-be-processed image of a target object is obtained through a binocular camera, and a depth image is generated after the to-be-processed image is calculated and processed; and judging whether the target object is a virtual obstacle located on a plane based on the depth image, if the target object is the virtual obstacle, giving up identification and tracking of the target object, and otherwise, estimating the 3D pose and the motion state of the target object based on a feature extraction mode of semantic information in an improved PointNet + + model to realize perception of the obstacle in the traffic scene. According to the invention, through image and depth data fusion, false identification of AEBS on interference factors such as a virtual image is effectively prevented; the 3D pose and motion information of the obstacle target can be calculated more accurately, and data support is provided for an AEBS to make an accurate decision.

Description

Technical Field

[0001] The present invention relates to the field of vehicle intelligent driving, and in particular to an obstacle perception method, system and device based on stereoscopic vision. Background Art

[0002] With the rapid development of smart transportation technology, ensuring the safety and autonomy of the driving process has become the main driving force for the advancement of the vehicle industry. The Autonomous Emergency Braking System (AEBS) is one of the key technologies to ensure vehicle driving safety, which greatly protects the safety of life and property inside and outside the vehicle. However, the stable operation of AEBS depends on an accurate and robust environmental perception system.

[0003] LiDAR can accurately perceive 3D information in a scene and is used in a variety of vehicle perception tasks, but its high cost and significant computing power requirements still limit its application. Monocular cameras are relatively cheap, but they obtain two-dimensional image information through perspective projection, resulting in the loss of depth information in the scene and difficulty in accurately perceiving the distance field. Using only monocular images to identify targets such as vehicles or pedestrians can easily misjudge virtual elements on billboards or display screens in traffic scenes as actual obstacles, causing frequent false triggering of AEBS, greatly reducing the vehicle driving experience and increasing the risk of rear-end collisions.

[0004] Based on the above background, the binocular camera simulates the human eye binocular system and uses the parallax principle of two lenses to calculate and reconstruct the 3D structure of the surrounding environment in real time. It can effectively capture the rich color texture information and depth information in the scene while taking into account the low-cost advantage. This makes the perception information input by AEBS faster, richer and more accurate, which can effectively reduce the probability of AEBS false triggering and increase the safety of the assisted driving system.

[0005] According to the existing relevant literature, CN118343160A discloses an unmanned driving perception system, which uses a camera and a laser radar as AEBS perception sensors. CN117622075A discloses a vehicle braking control system based on a monocular camera and an ultrasonic radar. They both directly perform a simple weighted fusion of the results obtained from the two data forms at the output end, and still have problems such as high cost and computing power consumption, and do not consider how to prevent misidentification of interference factors such as virtual images on billboards.

[0006] In general, existing AEBS perception methods mainly focus on detecting 2D information of obstacles. In solving AEBS triggering problems, they still have insufficient ability to evaluate interference factors and lack research on obtaining more comprehensive 3D information. Summary of the invention

[0007] In order to solve the above technical problems, the existing AEBS perception module may cause misidentification of planar targets in billboards or display screens, and single-modal sensors may provide insufficient information. The present invention provides an obstacle perception method, system and device based on stereo vision, which can provide multi-dimensional information such as target category, 3D posture and whether it is a virtual planar target in the billboard, providing robust data support for AEBS to make accurate decision judgments.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] In a first aspect, the present invention proposes a method for obstacle perception based on stereoscopic vision, which is applied to the accurate perception of vehicle AEBS obstacles in traffic scenes, and specifically comprises the following steps:

[0010] Acquire a to-be-processed image of the target object through a binocular camera, and generate a depth image after performing computational processing on the to-be-processed image;

[0011] Based on the depth image, it is determined whether the target object is a virtual obstacle located on the plane. If it is a virtual obstacle, the recognition and tracking of the target object is abandoned. Otherwise, the 3D pose and motion state of the target object are estimated based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene.

[0012] According to the above technical means, the camera as a visual sensor is mainly used to provide visual image information in the scene, including the single visual shape and surface texture characteristics of the object. Although the monocular camera can provide high-resolution image and video information, it can only provide instantaneous visual information, making it difficult to directly perform depth calculation and estimate the distance and scale of the target in the scene. The binocular camera is composed of two monocular cameras. By performing stereo matching on the left and right cameras to obtain disparity information, the depth information of the scene can be directly calculated.

[0013] Furthermore, the step of obtaining an image to be processed of the target object by a binocular camera and generating a depth image after performing computational processing on the image to be processed specifically includes:

[0014] The binocular camera is used to obtain an image to be processed of the target object, and the image to be processed is jointly calibrated and stereo matching is calculated to obtain a disparity value of the overlapping part of the binocular image. The disparity value is converted into a depth value to obtain a depth image, wherein the pixel value corresponding to each pixel point in the depth image represents the actual distance value of the pixel point in the binocular camera coordinate system.

[0015] According to the above technical means, during the target object recognition process, each target object to be recognized is labeled with a category label and a 2D rectangular bounding box. The 2D rectangular bounding box includes the coordinate position of the upper corner of the rectangular box and the length and width of the rectangular box. In fact, the 2D bounding box includes 4-dimensional data. The 2D rectangular bounding box is used to determine whether the target object is a billboard or a virtual image target on other planes. The specific judgment condition is whether the depth value information in the target area is located in a fittable plane (traffic participants such as pedestrians or vehicles in actual application scenarios are 3D targets, and their corresponding overall depth value information cannot be located on a plane).

[0016] Further, the step of determining whether the target object is a virtual obstacle located on a plane based on the depth image specifically includes:

[0017] Acquire a target 2D rectangular bounding box area corresponding to the depth image, and preprocess the depth information of the depth image within the bounding box area to remove outlier data;

[0018] By iterating the initial plane parameters multiple times, an optimal plane is generated based on the minimum sum of the squares of the distances from all pixels in the bounding box area to the pre-built initial plane;

[0019] Calculate the root mean square error (RMSE) based on the distances from all pixels in the bounding box area to the optimal plane;

[0020] If the root mean square error RMSE is less than a set threshold, the target object is a virtual obstacle located on the plane; if the root mean square error RMSE is greater than or equal to the set threshold, the target object is not a virtual obstacle located on the plane.

[0021] According to the above technical means, the optimal plane is determined by calculating the distance from all pixels in the 2D rectangular bounding box area to the initial plane. The initial plane needs to be determined in this process. In practice, three points can determine the plane. Therefore, after dividing the area into three sub-areas, a feature point is selected in each sub-area, so that the initial plane is determined based on the three feature points.

[0022] Furthermore, the construction process of the initial plane includes the following steps:

[0023] The bounding box area is divided into three sub-areas, and the distance between each pixel and other pixels in each sub-area is calculated. The pixel with the smallest distance is used as the feature point of the sub-area, and the three feature points of the three sub-areas constitute an initial plane.

[0024] According to the above technical means, in the process of extracting 3D feature information of the target object, it is necessary to regress the 3D parameters of the depth points in the 2D bounding box area. The parameters include the 3D coordinates of the geometric center point, the length, width, height information of the bounding box, and the orientation information. Specifically, the PointNet++ model is adopted. From the perspective of improving the accuracy of 3D parameter regression, a feature extraction method with semantic information is proposed to improve the original PointNet++ model.

[0025] Furthermore, the steps of estimating the 3D position and motion state of the target object based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene specifically include:

[0026] Assigning semantic weight information of pedestrians or vehicles to the target object, fusing the semantic weight information with the distance between the pixels of the depth image for sampling, and generating a set of sampling points;

[0027] The sampling points in the sampling point set are grouped and input into the multi-level feature fusion module of the improved PointNet++ model for approximate normalization processing. The multi-level feature fusion module performs a maximum pooling operation on the features of each layer, splices the results after the pooling operation and outputs the feature information of the target object; the feature information is output through a fully connected neural network to output the 3D pose and motion state estimation information of the target object to realize the perception of obstacles in traffic scenes.

[0028] According to the above technical means, by approximately normalizing the input of the multi-level feature fusion module, the numerical size of the network parameters can be reduced and the oscillation during network training can be alleviated.

[0029] Furthermore, the relative coordinates of the sampling point in the sampling point set and other points in the neighborhood corresponding to the sampling point are divided by the neighborhood radius for approximate normalization, which is expressed by the following formula:

[0030] (P j -P i ) / r,i∈O,j∈M

[0031] Among them, P j -P i It represents the relative coordinates of the i-th sampling point and the point j with a neighborhood radius of r, r represents the sampling radius, O represents the total number of sampling points, and M represents the set of samples sorted by distance within different sampling neighborhood radii.

[0032] According to the above technical means, by performing the maximum pooling operation on the features of each layer of the multi-level feature fusion module, the pooling results are spliced ​​to obtain the final feature output. This will effectively extract feature information of different levels and dimensions and improve the accuracy of the final 3D estimation.

[0033] Furthermore, the structure of the multi-level feature fusion module is expressed by the following formula:

[0034]

[0035] in, represents the maximum pooling, δ(·) represents the feature fusion module, and f i,j Represents the features of the jth point in the neighborhood radius r of the i-th sampling point.

[0036] In a second aspect, the present invention proposes an obstacle perception system based on stereoscopic vision, wherein the system adopts the obstacle perception method based on stereoscopic vision as described in the first aspect to achieve accurate perception of vehicle AEBS obstacles in traffic scenes, and the system comprises:

[0037] A binocular camera is used to obtain the image to be processed of the target object;

[0038] A calculation module, used for performing calculation processing on the image to be processed to generate a depth image;

[0039] A processing module is used to determine whether the target object is a virtual obstacle located on a plane based on the depth image. If it is a virtual obstacle, the recognition and tracking of the target object is abandoned. Otherwise, the 3D pose and motion state of the target object are estimated based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene.

[0040] Furthermore, the improved PointNet++ model includes:

[0041] A fusion sampling module, used to assign semantic weight information of pedestrians or vehicles to the target object, fuse the semantic weight information with the distance between the pixels of the depth image to perform sampling, and generate a set of sampling points;

[0042] The multi-level feature fusion module is used to perform approximate normalization processing on the sampling points in the input sampling point set, and perform maximum pooling operation on the features of each layer, and output the feature information of the target object after splicing the results of the pooling operation; the feature information is used to output the 3D posture and motion state estimation information of the target object through a fully connected neural network to realize the perception of obstacles in traffic scenes.

[0043] In a third aspect, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0044] Beneficial effects of the present invention:

[0045] Compared with the general method system, the method proposed in the present invention is more reliable and comprehensive, which is specifically reflected in:

[0046] (1) The method proposed in the present invention achieves a more accurate understanding of obstacles in the AEBS scene by determining interference factors such as virtual images on billboards and 3D perception of targets;

[0047] (2) Through the improved feature extraction module, the estimation accuracy of the 3D pose information of the obstacle target is further improved, providing good data support for the AEBS system to make correct decisions; BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Flowchart of the obstacle perception method based on stereoscopic vision provided by the present invention

[0049] Figure 2 A technical roadmap for implementing vehicle AEBS obstacle perception methods using stereo vision algorithms;

[0050] Figure 3 Schematic diagram of the improved PointNet++ model structure

[0051] Figure 4 A schematic diagram of a left camera image collected by a binocular camera;

[0052] Figure 5 A schematic diagram of a right camera image collected by a binocular camera;

[0053] Figure 6 It is a schematic diagram of the depth image obtained by stereo matching of binocular images (left view + right view);

[0054] Figure 7 Schematic diagram of pseudo point cloud data obtained by depth image transformation;

[0055] Figure 8 A visualization diagram of the target 3D bounding box;

[0056] Fig. 9 This is a framework diagram of the obstacle perception system based on stereoscopic vision provided by the present invention. DETAILED DESCRIPTION

[0057] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0058] This embodiment proposes a method for obstacle perception based on stereoscopic vision, which is applied to the accurate perception of vehicle AEBS obstacles in traffic scenes, such as Figure 1-2 As shown, the specific steps include:

[0059] Step S101: obtaining an image to be processed of a target object through a binocular camera, and generating a depth image after performing computational processing on the image to be processed.

[0060] In this embodiment, the camera is mainly used to provide visual image information of the single visual shape and surface texture features of the target object in the scene. The binocular camera is composed of two monocular cameras. By performing stereo matching on the left and right cameras to obtain disparity information, the depth information of the scene can be directly calculated. The depth image is obtained by jointly calibrating the binocular cameras and performing stereo matching calculation of the binocular images based on the calibration results. The pixel value corresponding to each pixel in the depth image represents the actual distance value of the pixel in the binocular camera coordinate system.

[0061] Step S102: Determine whether the target object is a virtual obstacle located on a plane based on the depth image; if yes, proceed to step S103; otherwise, proceed to step S104.

[0062] In this embodiment, after obtaining the image to be processed of the target object through the binocular camera and obtaining the depth image through matching calculation processing, it is necessary to determine whether the depth image is a plane image such as the image content on the billboard. If it is determined to be a virtual obstacle image, there is no need to track the image, otherwise it is necessary to perform image feature recognition and tracking on the target object.

[0063] In this embodiment, the process of determining whether the image to be processed is a virtual obstacle on a plane specifically includes: using computer vision technologies such as convolutional neural networks or Transformer models based on category labels and 2D rectangular bounding boxes to mark each target object to be identified, so as to realize the recognition and detection of the target in the image. The 2D rectangular bounding box includes 4-dimensional data such as the coordinate position of the upper corner of the rectangular box and the length and width of the rectangular box. The 2D rectangular bounding box is used to determine whether the target object is a virtual image target on a billboard or other plane. The specific judgment condition is whether the depth value information in the target area is located in a fittable plane (traffic participants such as pedestrians or vehicles in actual application scenarios are 3D targets, and their corresponding overall depth value information cannot be located on a plane). By adopting algorithm models such as Kalman filtering or deep twin neural networks, real-time tracking of the target 2D rectangular bounding box is realized. Compared with frame-by-frame detection of the image, the tracking method can effectively reduce computing power consumption and improve real-time performance.

[0064] In this embodiment, the specific process of determining whether the target object is a virtual obstacle located on a plane based on the depth image includes:

[0065] A target 2D rectangular bounding box area corresponding to the depth image is obtained, and depth information of the depth image is preprocessed within the bounding box area to remove outlier data.

[0066] In this embodiment, the step of preprocessing the depth information includes median filtering, mean filtering, data smoothing and other processing, so as to remove outlier data in the scene depth data.

[0067] By iterating the initial plane parameters multiple times, the optimal plane is generated based on the minimum sum of the square distances of all pixels in the bounding box area to the pre-constructed initial plane.

[0068] In this embodiment, the initial plane construction process includes the following steps: dividing the bounding box area into three sub-areas, calculating the distance between each pixel point and other pixel points in each sub-area, taking the pixel point with the smallest distance as the feature point of the sub-area, and the initial plane is formed by the three feature points of the three sub-areas.

[0069] Specifically, the bounding box area is divided into three sub-areas. In each sub-area, the distance between each point and other remaining points is cumulatively calculated. The point with the smallest cumulative distance is selected as the feature point of the sub-area. A total of three feature points are selected, and the initial plane is constructed using the selected three feature points. The specific calculation formula is as follows:

[0070]

[0071] Among them, the depth point set P in the sub-area, p irepresents the i-th calculation point in the set P, p i ∈P,P'∈P-{p i}, Sd i It represents the sum of the distances from the i-th point to the remaining points in P. Compared with randomly selecting feature points, the above method can better consider the density relationship between depth points and accelerate the convergence speed of plane estimation.

[0072] After the initial plane is determined, the plane parameters are continuously iterated to obtain the optimal plane parameters in which the sum of the squares of the distances from all pixels in the 2D rectangular bounding box area to the initial plane is minimized, and the optimal plane is determined based on the optimal plane parameters.

[0073] In this embodiment, the root mean square error RMSE is calculated based on the distances from all pixels in the bounding box area to the optimal plane. The calculation formula is as follows:

[0074]

[0075] Where N represents the number of pixels and di represents the distance from the i-th point to the optimal plane.

[0076] If the root mean square error RMSE is less than a set threshold, the target object is a virtual obstacle located on the plane.

[0077] If the root mean square error RMSE is greater than or equal to a set threshold, the target object is not a virtual obstacle located on the plane.

[0078] Step S103: If it is a virtual obstacle, give up identifying and tracking the target object.

[0079] In this embodiment, if the root mean square error RMSE is less than the set threshold, it indicates that the target object is a virtual target located on a plane (such as a billboard), that is, the target object is a virtual obstacle located on a plane. Because this result does not match the actual application scenario (traffic participants such as pedestrians or vehicles in the actual application scenario are 3D targets), further tracking of the target is abandoned.

[0080] Step S104: Estimate the 3D position and motion state of the target object based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene.

[0081] In this embodiment, if the root mean square error RMSE is greater than or equal to the set threshold, it indicates that the target is not located on the plane, that is, the target object is not a virtual obstacle located on the plane. It is necessary to extract the 3D feature information of the depth point in the 2D rectangular bounding box area. After feature extraction, the perception of obstacles in the traffic scene is realized.

[0082] In this embodiment, during the process of extracting 3D feature information of the target object, it is necessary to regress the 3D parameters of the depth points in the 2D bounding box area. The parameters include the 3D coordinates of the geometric center point, the length, width, height information of the bounding box, and the orientation information. Specifically, the PointNet++ model is adopted. From the perspective of improving the accuracy of 3D parameter regression, a feature extraction method with semantic information is proposed to improve the original PointNet++ model.

[0083] The steps of estimating the 3D position and motion state of the target object based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene specifically include:

[0084] Semantic weight information of pedestrians or vehicles is assigned to the target object, and the semantic weight information is fused with the distance between the pixels of the depth image to perform sampling to generate a sampling point set.

[0085] In this embodiment, if Figure 3 The figure shows a schematic diagram of the improved PointNet++ model, including a semantic feature module, which gives additional weights to the image of the target object (the target is a depth point such as a pedestrian or vehicle that needs to realize 3D estimation). Therefore, the weight contains the semantic information of the target object, making the network pay more attention to the characteristic information such as the shape and posture of the target object. In this process, it is necessary to fuse the semantic weight information and the distance information between the pixels of the depth image for sampling, so as to obtain a total number of sampling points O. With each sampling point as the center, sampling is performed in different sampling neighborhood radii according to the distance sorting to obtain a set M.

[0086] The sampling points in the sampling point set are grouped and input into the multi-level feature fusion module of the improved PointNet++ model for approximate normalization processing. The multi-level feature fusion module performs a maximum pooling operation on the features of each layer, splices the results after the pooling operation and outputs the feature information of the target object; the feature information is output through a fully connected neural network to output the 3D pose and motion state estimation information of the target object to realize the perception of obstacles in traffic scenes.

[0087] According to the above technical means, by approximately normalizing the input of the multi-level feature fusion module, the numerical size of the network parameters can be reduced and the oscillation during network training can be alleviated.

[0088] Furthermore, the relative coordinates of the sampling point in the sampling point set and other points in the neighborhood corresponding to the sampling point are divided by the neighborhood radius for approximate normalization, which is expressed by the following formula:

[0089] (P j -P i) / r,i∈O,j∈M

[0090] Among them, P j -P i It represents the relative coordinates of the i-th sampling point and the point j with a neighborhood radius of r, r represents the sampling radius, O represents the total number of sampling points, and M represents the set of samples sorted by distance within different sampling neighborhood radii.

[0091] By performing the maximum pooling operation on the features of each layer of the multi-level feature fusion module and concatenating the pooling results, the final feature output is obtained. This will effectively extract feature information of different levels and dimensions and improve the accuracy of the final 3D estimation.

[0092] The structure of the multi-level feature fusion module is expressed as follows:

[0093]

[0094] in, represents the maximum pooling, δ(·) represents the feature fusion module, and f i,j Represents the characteristics of the jth point in the neighborhood radius r of the i-th sampling point.

[0095] In this embodiment, 3D feature information is obtained through the above steps, and then after continuous derivation operations are performed based on the position information of the depth image of the target object, motion information such as the speed and acceleration of the target object is obtained, thereby realizing the perception of obstacles. Modal fusion is achieved by combining image data and depth data. First, the 2D position recognition of the target and the determination of whether the virtual plane target is located on an object such as a billboard are completed, which effectively alleviates the misrecognition phenomenon of the plane scene of the AEBS. Then, by estimating the 3D pose information and motion information of the target, a more comprehensive scene understanding is achieved, which helps the AEBS make more accurate judgments and decisions.

[0096] Based on the above content, this embodiment provides an implementation method for using a binocular camera to test a certain car scene. The collected left and right views are as follows: Figure 4-5 As shown, the vehicle is a 3D target object, and step S101 is used to Figure 4-5 Perform stereo matching and further transform to obtain Figure 6 The depth image shown in the figure shows that the different pixel values ​​of each point in the depth image data represent the distance of the point in the actual scene. For ease of understanding, the depth data is further converted into pseudo point cloud data (see Figure 7 ). The target is estimated by plane and the 3D pose information is regressed and estimated at the same time. The result visualization is shown in Figure 8 ,It can be seen that the pose information estimation accuracy is high, and it can provide more comprehensive data support for AEBS decision-making.

[0097] This embodiment also proposes an obstacle perception system based on stereoscopic vision. The system uses an obstacle perception method based on stereoscopic vision to achieve accurate perception of vehicle AEBS obstacles in traffic scenes. Fig. 9 As shown, the system 900 includes:

[0098] Binocular camera 901, used to obtain the image to be processed of the target object;

[0099] A calculation module 902 is used to generate a depth image after performing calculation processing on the image to be processed;

[0100] The processing module 903 is used to determine whether the target object is a virtual obstacle located on the plane based on the depth image. If it is a virtual obstacle, the recognition and tracking of the target object are abandoned. Otherwise, the 3D pose and motion state of the target object are estimated based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene.

[0101] In this embodiment, the improved PointNet++ model includes:

[0102] A fusion sampling module, used to assign semantic weight information of pedestrians or vehicles to the target object, fuse the semantic weight information with the distance between the pixels of the depth image to perform sampling, and generate a set of sampling points;

[0103] The multi-level feature fusion module is used to perform approximate normalization processing on the sampling points in the input sampling point set, and perform maximum pooling operation on the features of each layer, and output the feature information of the target object after splicing the results of the pooling operation; the feature information is used to output the 3D posture and motion state estimation information of the target object through a fully connected neural network to realize the perception of obstacles in traffic scenes.

[0104] This embodiment also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for obstacle perception based on stereoscopic vision when executing the computer program.

[0105] The above embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present invention is within the protection scope of the present invention.

Claims

1. Obstacle perception method based on stereoscopic vision, characterized by: The method is applied to the accurate perception of vehicle AEBS obstacles in traffic scenes, and specifically includes the following steps: The image to be processed of the target object is obtained through a binocular camera, and a depth image is generated after computational processing of the image to be processed; based on the depth image, it is determined whether the target object is a virtual obstacle located on a plane. If it is a virtual obstacle, the recognition and tracking of the target object is abandoned; otherwise, the 3D posture and motion state of the target object are estimated based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in traffic scenes.

2. The obstacle perception method based on stereoscopic vision according to claim 1, characterized in that: The steps of obtaining a to-be-processed image of a target object by a binocular camera and generating a depth image after performing computational processing on the to-be-processed image specifically include: The binocular camera is used to obtain an image to be processed of the target object, and the image to be processed is jointly calibrated and stereo matching is calculated to obtain a disparity value of the overlapping part of the binocular image. The disparity value is converted into a depth value to obtain a depth image, wherein the pixel value corresponding to each pixel point in the depth image represents the actual distance value of the pixel point in the binocular camera coordinate system.

3. The obstacle perception method based on stereoscopic vision according to claim 1, characterized in that: The step of determining whether the target object is a virtual obstacle located on a plane based on the depth image specifically includes: Acquire a target 2D rectangular bounding box area corresponding to the depth image, and preprocess the depth information of the depth image within the bounding box area to remove outlier data; By iterating the initial plane parameters multiple times, an optimal plane is generated based on the minimum sum of the squares of the distances from all pixels in the bounding box area to the pre-built initial plane; Calculate the root mean square error (RMSE) based on the distances from all pixels in the bounding box area to the optimal plane; If the root mean square error RMSE is less than a set threshold, the target object is a virtual obstacle located on the plane; if the root mean square error RMSE is greater than or equal to the set threshold, the target object is not a virtual obstacle located on the plane.

4. The obstacle perception method based on stereoscopic vision according to claim 3, characterized in that: The construction process of the initial plane includes the following steps: The bounding box area is divided into three sub-areas, and the distance between each pixel and other pixels in each sub-area is calculated. The pixel with the smallest distance is used as the feature point of the sub-area, and the three feature points of the three sub-areas constitute an initial plane.

5. The obstacle perception method based on stereoscopic vision according to claim 4 or 3, characterized in that: The steps of estimating the 3D position and motion state of the target object based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene specifically include: Assigning semantic weight information of pedestrians or vehicles to the target object, fusing the semantic weight information with the distance between the pixels of the depth image for sampling, and generating a set of sampling points; The sampling points in the sampling point set are grouped and input into the multi-level feature fusion module of the improved PointNet++ model for approximate normalization processing. The multi-level feature fusion module performs a maximum pooling operation on the features of each layer, splices the results after the pooling operation and outputs the feature information of the target object; the feature information is output through a fully connected neural network to output the 3D pose and motion state estimation information of the target object to realize the perception of obstacles in traffic scenes.

6. The obstacle perception method based on stereoscopic vision according to claim 5, characterized in that: The step of the approximate normalization processing is specifically: performing an approximate normalization processing based on the relative coordinates of the sampling point in the sampling point set and other points in the neighborhood corresponding to the sampling point divided by the neighborhood radius, which is expressed by the following formula: (P j -P i ) / r,i∈O,j∈M Among them, P j -P i It represents the relative coordinates of the i-th sampling point and the point j with a neighborhood radius of r, r represents the sampling radius, O represents the total number of sampling points, and M represents the set of samples sorted by distance within different sampling neighborhood radii.

7. The obstacle perception method based on stereoscopic vision according to claim 5, characterized in that: The structure of the multi-level feature fusion module is expressed as follows: in, represents the maximum pooling, δ(·) represents the feature fusion module, and f i,j Represents the characteristics of the jth point in the neighborhood radius r of the i-th sampling point.

8. Obstacle perception system based on stereoscopic vision, characterized by: The system adopts the obstacle perception method based on stereoscopic vision as described in any one of claims 1 to 7 to realize accurate perception of vehicle AEBS obstacles in traffic scenes, and the system includes: A binocular camera is used to obtain the image to be processed of the target object; A calculation module, used for performing calculation processing on the image to be processed to generate a depth image; A processing module is used to determine whether the target object is a virtual obstacle located on a plane based on the depth image. If it is a virtual obstacle, the recognition and tracking of the target object is abandoned. Otherwise, the 3D pose and motion state of the target object are estimated based on the feature extraction method of semantic information in the improved PointNet++ model to realize the perception of obstacles in the traffic scene.

9. The obstacle perception system based on stereoscopic vision according to claim 8, characterized in that: The improved PointNet++ model includes: A fusion sampling module, used to assign semantic weight information of pedestrians or vehicles to the target object, fuse the semantic weight information with the distance between the pixels of the depth image to perform sampling, and generate a set of sampling points; The multi-level feature fusion module is used to perform approximate normalization processing on the sampling points in the input sampling point set, and perform maximum pooling operation on the features of each layer, and output the feature information of the target object after splicing the results of the pooling operation; the feature information is used to output the 3D posture and motion state estimation information of the target object through a fully connected neural network to realize the perception of obstacles in traffic scenes.

10. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

Images