A vehicle-mounted obstacle precise sensing method, system and storage medium

Abstract

The present invention discloses a method, system and storage medium for precise on-vehicle obstacle perception. The method includes acquiring images through a binocular camera, obtaining camera perception results through a target detection model, and obtaining a disparity map through a deep stereo matching model, thereby obtaining a final camera perception result; acquiring and processing raw point cloud data to obtain radar perception results; fusing the two perception results and obtaining an obstacle perception result through a perception strategy; obtaining the instantaneous speed of the obstacle based on the obstacle perception result and optimizing it to obtain the obstacle state of the obstacle; and controlling the driving of the vehicle based on the obstacle state. This application improves the redundancy of obstacle perception results, solves the problem of blind spots in laser radar detection at close range, and can perform state perception and trajectory tracking of obstacles in motion, so as to provide data support for subsequent vehicle decision-making and path planning, meeting the safety and stability requirements of autonomous driving.

Description

Technical Field

[0001] The present invention relates to the field of automobile safe driving technology, and in particular to a method, system and storage medium for accurately sensing vehicle-mounted obstacles. Background Art

[0002] The four core technologies of autonomous driving are environmental perception, precise positioning, path planning, and drive-by-wire execution. Environmental perception primarily encompasses three aspects: target detection, object state estimation, and object tracking. Environmental perception technology primarily uses information about the external environment and the current vehicle state acquired by onboard sensors to calculate environmental perception results. Commonly used sensors include cameras, lidar, millimeter-wave radar, ultrasonic radar, inertial navigation systems, and GPS / RTK positioning systems. Each sensor has its own advantages and disadvantages, and the coordinated operation of multiple sensors is generally required to achieve the environmental perception requirements of autonomous driving. Therefore, sensor data fusion has become a key perception technology.

[0003] However, current environmental perception technologies are mostly implemented through a single sensor or a fusion of multiple sensors. The processing of sensor data is relatively simple, and some sensors have blind spots, resulting in low accuracy and inability to achieve precise environmental perception. Furthermore, when detecting obstacles in the vehicle's surroundings, obtaining information about moving obstacles is difficult. This makes it difficult to perceive and collect this information, failing to meet the requirements for state perception and trajectory tracking of moving obstacles, hindering subsequent vehicle decision-making and path planning. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, system and storage medium for accurately sensing vehicle-mounted obstacles to solve one or more technical problems existing in the prior art and at least provide a beneficial option or create conditions.

[0005] The present invention solves the technical problem by providing a method for accurately sensing vehicle-mounted obstacles, comprising the following steps:

[0006] Obtaining a long-range image and a medium-range image through a binocular camera, inputting the long-range image into a target detection model to obtain a camera perception result, wherein the camera perception result includes an image detection frame of the target obstacle and category information of the target obstacle;

[0007] Inputting the long-range image and the medium-range image into a depth stereo matching model, outputting a disparity map, and projecting the image detection frame into the disparity map to obtain distance information between the target obstacle and the vehicle and first volume information of the target obstacle, and adding the distance information and the first volume information to the camera perception result;

[0008] Obtaining raw point cloud data from a laser radar, processing the raw point cloud data using a point cloud ground-non-ground segmentation method and a point cloud clustering algorithm, and outputting a radar perception result, the radar perception result including location information of the target obstacle, second volume information, and a three-dimensional detection frame;

[0009] fusing the camera perception result and the radar perception result to obtain a fused radar perception result; when the distance information is less than a preset distance threshold, using the camera perception result as the final obstacle perception result; and when the distance information is greater than or equal to the distance threshold, using the fused radar perception result as the final obstacle perception result;

[0010] The fused radar perception result includes the radar perception result and category information of the target obstacle;

[0011] Collecting the UTM coordinate information of the vehicle, combining the UTM coordinate information of the vehicle and the obstacle perception result to calculate the UTM coordinate information of the target obstacle, and calculating the instantaneous speed and movement direction of the target obstacle at the current moment based on the UTM coordinate information of the target obstacle;

[0012] The instantaneous speed of the target obstacle is optimized by Kalman filtering to obtain stable speed information of the target obstacle, and the obstacle state of the target obstacle is updated using the Hungarian algorithm according to the stable speed information, and the vehicle is controlled according to the obstacle state.

[0013] On the other hand, the present application provides a vehicle-mounted obstacle precision perception system, comprising:

[0014] Binocular cameras, including a long-range camera and a medium-range camera, are used to detect the environment in front of the vehicle and obtain long-range images and medium-range images;

[0015] Wherein, the long-range camera and the medium-range camera are respectively installed at both ends of the front windshield of the vehicle;

[0016] LiDAR, located on top of the vehicle, is used to detect target obstacles at medium and long distances and generate raw point cloud data;

[0017] GPS positioning system, used to obtain the vehicle's UTM coordinate information;

[0018] On-board domain controller, including:

[0019] The target detection unit is equipped with a target detection model, which is used to detect targets in long-range images through the target detection model to obtain camera perception results;

[0020] The camera perception result includes an image detection frame of the target obstacle and category information of the target obstacle;

[0021] A stereo matching unit equipped with a deep stereo matching model is used to perform stereo matching on long-range images and medium-range images to obtain a disparity map.

[0022] an image processing unit, configured to project the image detection frame into the disparity map, obtain distance information between the target obstacle and the vehicle, orientation information of the target obstacle, and first volume information, and add the distance information between the target obstacle and the vehicle, the orientation information of the target obstacle, and the first volume information to a camera perception result;

[0023] a radar data processing unit, configured to process the raw point cloud data using a point cloud ground-non-ground segmentation method and a point cloud clustering algorithm to obtain a radar perception result;

[0024] The radar sensing result includes the location information of the target obstacle, the second volume information and the three-dimensional detection frame;

[0025] a first data processing unit, configured to fuse a camera perception result and a radar perception result to obtain a fused radar perception result; and when the distance information is less than a preset distance threshold, using the camera perception result as the final obstacle perception result; and when the distance information is greater than or equal to the distance threshold, using the fused radar perception result as the final obstacle perception result;

[0026] a second data processing unit, configured to calculate the UTM coordinate information of the target obstacle by combining the UTM coordinate information of the vehicle and the obstacle sensing result, and to infer the instantaneous speed and movement direction of the target obstacle at a current moment based on the UTM coordinate information of the target obstacle;

[0027] a third data processing unit, configured to optimize the instantaneous speed of the target obstacle through a Kalman filter to obtain stable speed information of the target obstacle, and update the state of the obstacle using a Hungarian algorithm according to the stable speed information;

[0028] The control unit is used to control the vehicle driving according to the updated state of the obstacle.

[0029] On the other hand, the present application also provides a storage medium storing processor-executable instructions, which are used to execute the method for accurately sensing vehicle-mounted obstacles when executed by the processor.

[0030] The beneficial effects of the present invention are: providing a method, system and storage medium for accurate on-vehicle obstacle perception, using a lidar and a binocular camera as target perception sensors to perceive and detect obstacles during vehicle travel, fusing the perception results of the binocular camera with the perception results of the lidar to obtain a final obstacle perception result, so that obstacle perception has a certain degree of redundancy, solving the problem that the lidar has a detection blind spot at close range and cannot effectively perceive obstacles, and can perceive the state and track the trajectory of obstacles in motion, so as to provide data support for subsequent vehicle decision-making and path planning, and meet the safety and stability requirements of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a vehicle-mounted obstacle precision sensing system provided in an embodiment of the present application;

[0032] Figure 2 A flowchart of a method for accurately sensing vehicle-mounted obstacles provided in an embodiment of the present application;

[0033] Figure 3A A schematic diagram of a medium-range image provided in an embodiment of the present application;

[0034] Figure 3B A schematic diagram of a long-range image provided in an embodiment of the present application;

[0035] Figure 3C A schematic diagram of a disparity map obtained through a medium-range image and a long-range image provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram illustrating the principle of three-dimensional mapping of a disparity map and an image detection frame provided in an embodiment of the present application;

[0037] Figure 5 A flowchart of the fusion radar perception results provided in an embodiment of the present application;

[0038] Figure 6 Schematic diagram of the method for accurately sensing vehicle-mounted obstacles provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0040] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be considered as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0041] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0043] The four core technologies of autonomous driving are environmental perception, precise positioning, path planning, and drive-by-wire control. Environmental perception primarily encompasses target detection, object state estimation, and object trajectory tracking. Environmental perception is a core technology in the fields of autonomous driving and robotics. It involves utilizing on-board sensors, networked communications, and other technologies to acquire environmental information, transmitting this information—including road, vehicle, and obstacle information—to a decision-making center, providing a basis for intelligent decision-making. Commonly used on-board sensors in this field include cameras, lidar, millimeter-wave radar, ultrasonic radar, inertial navigation systems, and GPS / RTK positioning systems. Each sensor has its own advantages and disadvantages, and a single sensor alone cannot accurately detect environmental information during environmental perception. The collaborative work of multiple sensors is generally required to achieve the environmental perception requirements of autonomous driving. Therefore, sensor data fusion has become a key technology in environmental perception.

[0044] Currently, there has been some research in this field on environmental perception for intelligent vehicles. For example, multiple cameras and millimeter-wave radars are used to perceive the surrounding environment of the vehicle. Another example is the use of multiple sensors to build a multi-sensor fusion environmental perception platform to obtain integrated environmental perception information. However, current vehicle environmental perception technology still has the following problems and defects:

[0045] First, due to the high precision required for vehicle environmental perception, radar is often used as one of the onboard sensors used for environmental perception. However, when LiDAR is close to a target, it cannot detect the target, reducing the accuracy of environmental perception and failing to meet the vehicle's environmental perception requirements.

[0046] Second, after obtaining environmental information, the existing technology's processing methods for environmental information are relatively simple, resulting in low accuracy of the environmental information ultimately used for vehicle decision-making and planning, affecting the decision-making and planning effects of intelligent vehicles.

[0047] Third, when detecting targets, it is difficult to obtain the status information of the target in motion, and it is difficult to collect the actual status information of the target. This cannot meet the needs of state perception and trajectory tracking of the target in motion, and thus cannot achieve subsequent vehicle decision-making and path planning.

[0048] In response to the problems existing in the above-mentioned existing technologies, this application proposes a method, system and storage medium for accurate on-vehicle obstacle perception, which obtains the position and speed information of target obstacles in motion or stationary state through binocular camera and lidar detection, thereby providing accurate data support for subsequent intelligent vehicle decision-making and planning.

[0049] Reference Figure 1 As shown, Figure 1 The following is a schematic diagram of a vehicle-mounted precise obstacle perception system provided by an embodiment of the present application. This is an embodiment of the present application. The following describes and elaborates on the structure of the perception system. The perception system includes, but is not limited to, a binocular camera, a lidar 101, a GPS positioning system, and a vehicle-mounted domain controller.

[0050] The binocular camera system consists of a long-range camera 103 and a medium-range camera 102. The medium-range camera 102 has a wider field of view, capable of observing the environment at medium and close distances in front of the vehicle. Compared to the medium-range camera 102, the long-range camera 103 can obtain environmental conditions at a long distance in front of the vehicle, thereby obtaining more information about the field of view in front of the vehicle. Optionally, the medium-range camera 102 is mounted on one end of the vehicle's front windshield, and the long-range camera 103 is mounted on the other end of the vehicle's front windshield. The medium-range camera 102 and the long-range camera 103 are mounted symmetrically to ensure that the entire field of view in front of the vehicle is captured, avoiding blind spots.

[0051] In this embodiment, the binocular camera is used to detect the environment in front of the vehicle, such as whether there are pedestrians or obstacles in front of the vehicle, and then obtain long-range images and medium-range images. Specifically, the long-range image is obtained by the long-range camera 103, and the medium-range image is obtained by the medium-range camera 102.

[0052] The function of the laser radar 101 is to detect target obstacles at medium and long distances and obtain original point cloud data.

[0053] Optionally, the laser radar 101 is mounted on the roof of the vehicle. This installation method ensures the ability to perceive targets at medium and long distances. In this specific embodiment, the laser radar 101 is a 32-line laser radar. In other embodiments of the present application, the type of laser radar 101 can also be other types, and this application does not specifically limit this.

[0054] The function of the GPS positioning system is to obtain the current position information of the vehicle. In this embodiment, the current position information is represented by UTM coordinate information.

[0055] The vehicle domain controller is equipped with:

[0056] The target detection unit, which is equipped with a target detection model, inputs long-range images into the target detection model and performs target detection on the long-range images to obtain camera perception results.

[0057] The stereo matching unit is equipped with a deep stereo matching model. The function of the stereo matching unit is to input the medium-range image and the long-range image into the deep stereo matching model, perform stereo matching on the medium-range image and the long-range image, and output a disparity map.

[0058] The image processing unit projects the image detection frame into the disparity map to obtain the distance information between the target obstacle and the vehicle, the orientation information of the target obstacle, and the first volume information, and adds these three types of information to the camera perception result.

[0059] The radar data processing unit processes the original point cloud data through the point cloud ground-non-ground segmentation method and point cloud clustering algorithm, and outputs the radar perception results.

[0060] The first data processing unit is used to fuse the camera perception results and the radar perception results to obtain the fused radar perception results, and determine the final obstacle perception results from the fused radar perception results and the camera perception results through a set distance threshold.

[0061] Specifically, when the distance information is less than a preset distance threshold, the camera perception result is used as the final obstacle perception result; when the distance information is greater than or equal to the distance threshold, the fused radar perception result is used as the final obstacle perception result.

[0062] The second data processing unit is used to combine the vehicle's UTM coordinate information and the obstacle perception results to calculate the UTM coordinate information of the target obstacle, and to infer the instantaneous speed and movement direction of the target obstacle at the current moment based on the UTM coordinate information of the target obstacle.

[0063] The third data processing unit optimizes the instantaneous speed of the target obstacle through Kalman filtering to obtain the stable speed information of the target obstacle, and updates the state of the obstacle using the Hungarian algorithm according to the stable speed information.

[0064] The control unit is used to control the vehicle's movement according to the updated obstacle status.

[0065] Reference Figure 2 As shown, Figure 2 The flowchart of the vehicle-mounted obstacle accurate perception method provided by the embodiment of the present application is shown below. The implementation process of the perception method is described and explained below. The vehicle-mounted obstacle accurate perception method may include but is not limited to the following steps.

[0066] S100, obtain long-range images and medium-range images through a binocular camera, input the long-range image into the target detection model, and obtain the camera perception result.

[0067] It should be noted that the camera perception results include the image detection frame of the target obstacle and the category information of the target obstacle. The target detection model is a pre-trained neural network model based on YOLOv5.

[0068] This step involves visual obstacle detection. Both the long-range and medium-range images are 2D images. The images captured by the medium-range camera are used as input for the object detection model. The model then infers the target obstacle and obtains the detection bounding box and category label for the obstacle in front of the vehicle in the 2D image.

[0069] S200: Input the long-range image and the medium-range image into the deep stereo matching model, output the disparity map, and project the image detection frame into the disparity map to obtain the distance information between the target obstacle and the vehicle, as well as the orientation information and first volume information of the target obstacle. The distance information, orientation information, and first volume information are added to the camera perception result.

[0070] It should be noted that the deep stereo matching model is a pre-trained neural network model based on AANET.

[0071] In this step, the information captured by the medium-range and long-range cameras is used as input to the binocular stereo matching model, also known as the deep stereo matching model. The target obstacle is inferred and a disparity map corresponding to the area in front of the vehicle is obtained. The detection box output by the target detection model is then directly used as the reference point for the obstacle. This detection box, combined with the disparity map, is processed to determine the distance between the vehicle and the target obstacle and the obstacle's orientation. Finally, based on the 2D images (i.e., the long-range and medium-range images), the actual size of the target obstacle, known as the first volumetric information, is inferred. These three pieces of information are then incorporated into the camera perception results.

[0072] S300 obtains the original point cloud data of the lidar, processes the original point cloud data through the point cloud ground-non-ground segmentation method and point cloud clustering algorithm, and outputs the radar perception result.

[0073] It should be noted that the radar perception results include the position information of the target obstacle relative to the vehicle within the medium range, the target obstacle's second volume information, and the 3D detection frame. The second volume information reflects the actual size of the obstacle.

[0074] This step complements visual obstacle detection, aiming to ensure accuracy and reduce errors. In this step, a single 32-line LiDAR is used for obstacle detection, generating raw point cloud data. First, a point cloud ground-non-ground segmentation method is used to divide the point cloud into ground and non-ground categories. A point cloud clustering algorithm is then used to process the non-ground point cloud, ultimately yielding radar perception results.

[0075] S400, fusing the camera perception result and the radar perception result to obtain a fused radar perception result. When the distance information is less than a preset distance threshold, the camera perception result is used as the final obstacle perception result; when the distance information is greater than or equal to the distance threshold, the fused radar perception result is used as the final obstacle perception result.

[0076] In this specific embodiment, the distance threshold is 15 meters.

[0077] In this step, since the radar perception results do not contain a category label corresponding to the target obstacle, the camera perception results must be fused with the radar perception results. This fusion results in a fused radar perception result. Next, a perception strategy is developed: using a 15-meter distance threshold, the relationship between the distance between the vehicle and the target obstacle and this distance threshold is determined. If the distance is less than 15 meters, the final obstacle perception result is the camera perception result. If the distance is greater than 15 meters, the final obstacle perception result is the fused radar perception result.

[0078] S500: Collect the UTM coordinate information of the vehicle, combine the UTM coordinate information of the vehicle and the obstacle perception result, calculate the UTM coordinate information of the target obstacle, and calculate the instantaneous speed and movement direction of the target obstacle at the current moment based on the UTM coordinate information of the target obstacle.

[0079] In this step, the vehicle's coordinates are obtained through the GPS positioning system. If the obstacle perception result is a camera perception result, the UTM coordinates of the target obstacle are calculated using the distance information and the vehicle's UTM coordinates. If the obstacle perception result is a fused radar perception result, the UTM coordinates of the target obstacle are calculated using the target obstacle's location information and the vehicle's UTM coordinates.

[0080] S600: Optimizing the instantaneous speed of the target obstacle through Kalman filtering to obtain stable speed information of the target obstacle, updating the obstacle state of the target obstacle using a Hungarian algorithm based on the stable speed information, and controlling vehicle travel based on the obstacle state.

[0081] In this step, because the instantaneous speed information obtained from the perception results and the obstacle's UTM coordinates may fluctuate, a Kalman filter is used to filter out these fluctuations to obtain stable speed information, facilitating more precise control decisions and planning. After obtaining stable speed information, the Hungarian algorithm updates the current state of the obstacle, allowing the vehicle's control unit to make decisions and plans for autonomous driving based on the obstacle's current state.

[0082] Before obtaining the medium-range image and the long-range image in S100, a calibration step for the medium-range camera and the long-range camera is also included. Camera calibration refers to establishing the relationship between the pixel position of the camera image and the position of the scene point. According to the camera imaging model, the intrinsic parameter matrix, the extrinsic parameter matrix and the distortion coefficient of the camera model are solved by the correspondence between the coordinates of the feature points in the image and the world coordinates. Therefore, in this application, the Zhang Zhengyou calibration method is used to implement camera calibration. The Zhang Zhengyou calibration method uses the pixel coordinates of each corner point and its physical coordinates in the world coordinate system to calibrate the camera, and obtain the intrinsic parameters, extrinsic parameters and distortion coefficients of the medium-range camera and the long-range camera, so as to facilitate the elimination of the distortion of the medium-range image and the long-range image in the subsequent image processing process.

[0083] In one embodiment of the present application, the calibration process of the binocular camera will be further described and elaborated below. In this application, a calibration plate composed of several two-dimensional grids is used to calibrate the medium-range camera and the long-range camera. The calibration process is as follows:

[0084] The first step is to obtain different pose images of the calibration plate and extract the pixel coordinates of the corner points in the image.

[0085] In this step, the corners of the pose image are detected and extracted using a corner detection algorithm. The initial values ​​of the intrinsic and extrinsic parameters of the long-range and medium-range cameras are then obtained by calculating the corners of the pose image. A corner is usually defined as the intersection of two edges, and the local neighborhood of a corner should have boundaries in two different regions in different directions. Corner detection is a method used in computer vision systems to obtain image features and is widely used in motion detection, image matching, video tracking, 3D modeling, and target recognition. Corner detection is also called feature point detection.

[0086] Optionally, the corner detection algorithm is any one of the Moravec corner detection algorithm, the FAST corner detection algorithm, or the Harris corner detection algorithm. However, in other embodiments of the present application, other corner detection algorithms may be selected according to actual circumstances, and the present application does not specifically limit this.

[0087] In the second step, the pixel coordinates of the corner points are calculated through the homography matrix to obtain the initial values ​​of the intrinsic parameters and the initial values ​​of the extrinsic parameters of the binocular camera.

[0088] It should be noted that each pose image has a corresponding homography matrix. The homography matrix can be understood as the position mapping relationship between the world coordinate system and the pixel coordinate system. The homography matrix H satisfies:

[0089] H=[h1, h2, h3]=γA[r1 r2 t];

[0090] Where γ is an arbitrary scaling factor. r1 and r2 are column vectors that are orthogonal and have a modulus of 1. The homography matrix is ​​a 3x3 matrix with one element as a homogeneous coordinate. Therefore, H has eight degrees of freedom.

[0091] In this step, the Zhang Zhengyou calibration method constructs a geometric model of the camera imaging. Combined with this geometric model, the homography matrix from the pixel coordinate system to the world coordinate system is calculated. Since the homography matrix is ​​a combination of extrinsic and intrinsic parameters, the original intrinsic and extrinsic parameters of the camera can be obtained by simply solving the intrinsic and extrinsic parameters in the homography matrix separately.

[0092] The third step is to estimate the initial value of the distortion coefficient of the binocular camera by nonlinear least squares method.

[0093] Due to the presence of the lens in the camera lens, the propagation of light during the imaging process will have two effects. One is the influence of the lens shape on the propagation of light, which is called radial distortion; the other is the influence of the lens position on the propagation of light, which is called tangential distortion. The calibration method used in this application is the Zhang Zhengyou calibration method, which only focuses on radial distortion. In actual situations, the radial distortion is small, so it can be described by the first few terms of the Taylor series expansion around the principal point. The first two terms are used in this application to determine the distortion coefficient of radial distortion. The mathematical expression is:

[0094]

[0095]

[0096] Among them, (u, v) is the ideal undistorted pixel coordinate, are the pixel coordinates under actual radial distortion, (u0, v0) is the principal point, (x, y) is the continuous image coordinate under ideal distortion-free conditions, and k1 and k2 represent the distortion coefficients of the first two orders.

[0097] Then, for any point on the pose image, there exists the following matrix equation:

[0098]

[0099] In the above matrix equation, the values ​​of (x, y) and (u, v) can be obtained through the camera imaging geometry model and the initial values ​​of the internal and external parameters. The principal point can be obtained through the initial values ​​of the internal parameters, and the distorted pixel coordinates can be read through the pose image.

[0100] Then, assuming there are n images and each image has m corner points, we can get 2mn matrix equations. Let D be the coefficient matrix on the left side of the matrix equation, D is a 2mn×2 matrix; k = [k1, k2] T , d is the matrix consisting of the difference between the pixel coordinates with distortion and the pixel coordinates without distortion on the right side of the matrix equation. The initial value of the distortion coefficient is estimated by nonlinear least squares method: k = (D T D) -1 D T d.

[0101] In the fourth step, the initial values ​​of the intrinsic parameters, the initial values ​​of the extrinsic parameters, and the initial values ​​of the distortion coefficients are optimized by the maximum likelihood estimation method to obtain the final intrinsic parameters, extrinsic parameters, and distortion coefficients of the binocular camera.

[0102] In this step, after obtaining the initial values ​​of the internal and external parameters and the distortion coefficient, it is necessary to obtain the true internal and external parameters and the distortion coefficient through the maximum likelihood estimation strategy. The maximum likelihood estimation method satisfies:

[0103]

[0104] In this specific embodiment, the Zhang Zhengyou calibration method is used to calibrate the intrinsic parameters, extrinsic parameters, and distortion coefficients of the medium-range camera and the long-range camera, so as to facilitate the subsequent elimination of distortion of the medium-range image and the long-range image in S200.

[0105] In one embodiment of the present application, the target detection model in S100 is a pre-trained model, and this target detection model is a target detection model based on YOLOv5. The training process of the target detection model will be described and explained below. The training process of the target detection model includes:

[0106] First, obtain the COCO dataset and select categories with high interest in autonomous driving scenarios from the COCO dataset to form an object detection dataset. Optionally, the categories include pedestrians, cars, bicycles, etc.

[0107] It should be noted that the COCO dataset is a large, rich dataset for object detection, segmentation, and captioning. This dataset aims to understand scenes and is primarily captured from complex everyday scenes. Objects in the images are accurately segmented and their positions are calibrated.

[0108] Next, we build a YOLOv5 model and divide the object detection dataset into a training set and a test set. The training set is used to train the YOLOv5 model, and the test set is used to test the performance of the trained YOLOv5 model. The training set is input into the YOLOv5 model for training, and the trained YOLOv5 model is output.

[0109] It's important to note that YOLOv5 is a widely used object detection model in the machine vision industry. The YOLOv5 network consists of four modules: the input, backbone, neck, and prediction modules. It uses non-maximum suppression (NMS) as a loss function to filter object boxes.

[0110] Among them, GIOU meets:

[0111]

[0112]

[0113] Where M represents the intersection between the target real box and the predicted box, N represents the union between the target real box and the predicted box, and IOU represents the ratio of the intersection to the union. The minimum bounding rectangle of the target real box and the predicted box is represented by C, and D is the difference between C and the union N. Therefore, GIOU_Loss can be expressed by the following formula:

[0114]

[0115] Afterwards, the performance of the trained YOLOv5 model is tested on the test set, where the confidence level is used to characterize the performance of the YOLOv5 model.

[0116] Specifically, during the performance test, if the confidence level of the results output by the trained YOLOv5 model is lower than the confidence threshold, the YOLOv5 model is considered to have failed to achieve the expected performance, the network hyperparameters are optimized, and the YOLOv5 model is retrained. If the confidence level of the results output by the trained YOLOv5 model is greater than or equal to the confidence threshold, the YOLOv5 model is considered to have achieved the expected performance and is output as the object detection model. Optionally, the confidence threshold is set to 0.5.

[0117] In one embodiment of the present application, the process of obtaining the disparity map in S200 will be further described and elaborated below. The process of obtaining the disparity map may include but is not limited to the following steps.

[0118] S210 , performing stereo correction on the long-range image and the medium-range image.

[0119] If you want to estimate the depth information of an object point using two images, you must accurately match the same object point in both images. This way, you can calculate the depth of the object based on the positional relationship between the object point in the two images. Therefore, before inferring the disparity map, this application needs to perform stereo correction on the medium-range image and the long-range image. The specific steps of stereo correction are:

[0120] S211, obtaining original pixel points of the long-range image and the medium-range image, and converting the pixel coordinate systems of the long-range image and the medium-range image into the camera coordinate system using the intrinsic parameters of the binocular camera;

[0121] S212 , calculating the distortion amounts of the long-range image and the medium-range image, obtaining the distorted positions of the long-range image and the medium-range image in the camera coordinate system, and performing dedistortion processing on the long-range image and the medium-range image according to the distorted positions.

[0122] In the above steps, distortion coefficients, intrinsic parameters, and extrinsic parameters obtained using the Zhang Zhengyou calibration method are used to correct the distortion of the medium-range and long-range images to eliminate distortion. After distortion elimination, the medium-range and long-range images undergo stereo correction. Stereo correction is defined as correcting two images that are not actually coplanarly aligned to coplanar alignment, thereby improving matching search efficiency.

[0123] S213, converting the camera coordinate system into a pixel coordinate system, and re-interpolating the pixel points of the long-range image and the medium-range image in the pixel coordinate system using the original pixel points;

[0124] S214 , adjusting the sizes of the medium-distance image and the long-distance image so that the sizes of the medium-distance image and the long-distance image are consistent.

[0125] S220: Input the long-range image and the medium-range image into a depth stereo matching model to obtain a disparity map.

[0126] It should be noted that the deep stereo matching model is a neural network model based on AANET (Adaptive Aggregation Network). AANET is commonly used in binocular stereo matching and can be used to replace 3D convolution based on matching cost volumes, speeding up inference while maintaining high accuracy. AANET can be divided into three parts. The first part is feature extraction, which uses convolutional neural networks to perform convolution to obtain features of different sizes; the second part is cost aggregation, which includes matching cost calculation and multi-scale cost fusion; and the third part is depth map optimization, which is the optimization of the disparity map. By using the stereo-rectified medium-range image and long-range image as input to AANET, the corresponding disparity map directly in front of the vehicle is inferred.

[0127] In this specific embodiment, refer to Figures 3A to 3C As shown, Figure 3A This is a schematic diagram of a medium-range image provided in an embodiment of the present application. Figure 3B This is a schematic diagram of a long-range image provided by an embodiment of the present application. Figure 3A The mid-range image shown and the Figure 3B The long-range image shown in the figure is stereo rectified and input into AANET to obtain the following Figure 3C The disparity map shown.

[0128] Reference Figure 4 As shown, Figure 4 The following is a schematic diagram of the three-dimensional mapping principle of the disparity map and the image detection frame provided by the embodiment of the present application. Figure 4 The process of obtaining the distance information, the first volume information, and the orientation information in S200 is further described and elaborated. The process of obtaining the distance information, the first volume information, and the orientation information may include but is not limited to the following steps.

[0129] S230: Use the center point of the image detection frame as the reference point of the target obstacle, calculate the coordinates of the reference point of the target obstacle in the pixel coordinate system, and project the image detection frame into the disparity map to obtain the projection point coordinates of the reference point in the disparity map. The distance between the target obstacle and the vehicle is obtained through the projection point coordinates.

[0130] In this step, considering the small deviation between the disparity maps output by the long-range camera and the depth stereo matching model, the center point of the detection frame output by the object detection model is directly used as the obstacle reference point to calculate its coordinates in the pixel coordinate system. After calculating the coordinates of the reference point in the coordinate system, its coordinate value in the depth map is read to obtain the distance between the obstacle and the vehicle.

[0131] S240 , based on the intrinsic parameters of the binocular camera, convert the projection point coordinates from the pixel coordinate system to the world coordinate system to obtain the orientation information of the target obstacle relative to the vehicle.

[0132] In this embodiment, Figure 4 As shown, Figure 4 The coordinate system XYZ is the local three-dimensional coordinate system of the camera, that is, the world coordinate system; the coordinate system x′y′ is the pixel coordinate system, point P is the center point of the long-range image corresponding to the image detection frame; point C is the position of the binocular camera; f is the focal length of the camera; point X M is the coordinate of the projection point of the reference point in the disparity map. In this embodiment, the image corresponding to the target detection frame is a long-range image, and the point X in the long-range image M is the coordinate in the pixel coordinate system, and the point M in the world coordinate system can be obtained by conversion, that is, the orientation information of the obstacle relative to the vehicle.

[0133] S250 , inferring first volume information of the target obstacle based on the long-range image and the medium-range image in combination with the distance information and the orientation information.

[0134] In this specific embodiment, after performing target detection and stereo matching inference on pre-calibrated camera images, the target obstacle's category information, image detection frame, distance from the vehicle, and first volume information together constitute the camera perception result. This application uses the camera perception result to accurately perceive the distance and size of obstacles that are within the lidar blind spot, close to the vehicle, and low in height. This solves the problem of lidar detection blind spots at close ranges and further expands the range of obstacle information perception.

[0135] Based on the above embodiment, the perception of the laser radar supplements the detection of visual obstacles of the camera. In one embodiment of the present application, the processing process of the original point cloud data in S300 will be further described and elaborated below. The point cloud data in S300 is a data set of spatial points obtained by scanning a three-dimensional laser radar device. Each point contains three-dimensional coordinate information and may also contain color information, reflection intensity information, echo number information, etc. Point cloud data has the characteristics of sparsity, disorder, and uneven spatial distribution. Therefore, after obtaining the original point cloud data in S300, the following processing is performed in succession:

[0136] The first processing step is ground-non-ground segmentation of the point cloud. When using LiDAR for obstacle detection, traditional point cloud processing algorithms are typically used for obstacle segmentation, fitting, and tracking. Clustering algorithms are used for obstacle segmentation during this process, and these algorithms measure attributes based on Euclidean distance between points. Consequently, ground point cloud data can interfere with the ground object point set, thus failing to achieve the desired obstacle detection results.

[0137] To address this issue, this application uses a segmentation method to separate point cloud data into ground point cloud data and non-ground point cloud data before performing point cloud clustering. This method eliminates the interference caused by the ground point cloud data on the point set of objects located on the ground. The specific steps of point cloud segmentation are to divide the space in front of the vehicle into several parts based on the distance from the front of the vehicle, and then set a threshold for the point cloud in different spaces. If the point cloud height is lower than the threshold, it is judged as a ground point; otherwise, it is a non-ground point.

[0138] In this application, the process of point cloud ground-non-ground segmentation includes:

[0139] First, the space in front of the vehicle is divided into several subspaces according to the distance between the lidar and the head of the vehicle.

[0140] It should be noted that each subspace contains a number of original point cloud data located in this subspace.

[0141] Furthermore, a height threshold corresponding to each subspace is set, and the distance between the original point cloud data of each subspace and the ground is calculated, and the distance is recorded as the point cloud height of the original point cloud data.

[0142] It should be noted that the height threshold can be set according to actual conditions, and this application does not make any specific limitations on this.

[0143] In this step, a height threshold corresponding to the subspace is set according to the subspace, and there is a corresponding relationship between the original point cloud data in the subspace and the height threshold of the subspace.

[0144] Then, the point cloud height of each raw point cloud data is determined to be greater than its corresponding height threshold. Based on the judgment results, each raw point cloud data is divided into ground point data and non-ground point data. Specifically, when the point cloud height is less than the height threshold, the raw point cloud data is ground point data; when the point cloud height is greater than the height threshold, the raw point cloud data is non-ground point data, thus completing the segmentation of ground points and non-ground points.

[0145] The second step of processing is point cloud clustering. The purpose of clustering is to classify non-ground point data, and then realize the classification of different target obstacles. This application uses the Euclidean point cloud clustering algorithm to realize point cloud clustering, and then obtain radar perception results. The Euclidean clustering algorithm is based on the Euclidean distance clustering method. This algorithm needs to design a distance threshold parameter, and the distance sensitivity is better than the traditional clustering algorithm, and it has the ability to automatically filter noise. During the search process of the algorithm, the Kd-tree is used for acceleration to improve the efficiency of data search. This application obtains the position and size information of different obstacles in this way, and obtains the position and volume information of obstacles within the medium distance range and the three-dimensional detection frame. This information constitutes the radar perception result. It should be noted that the volume information maps the size information of the target obstacle.

[0146] In this application, the process of point cloud clustering using the Euclidean clustering algorithm is as follows:

[0147] In each subspace, any non-ground point data is randomly selected as the center of the sphere. The radius of the sphere is set according to the distance between the subspace and the head of the vehicle to form a sphere that maps the obstacle set corresponding to the non-ground point data. The original point cloud data within the sphere under this radius are all classified into the corresponding obstacle set; each original point cloud data within the sphere is expanded outward with the same set radius, and other original point cloud data are included in the obstacle set; the above steps are repeated until all original point cloud data are included in the corresponding obstacle set, and clustering is terminated.

[0148] In this specific embodiment, a set A containing all non-ground point data is initialized, and objects in A are marked as "unvisited". Then, any non-ground point data p is selected from A. i , mark "visited", and create a set C i , set C i The position and size of the obstacles reflected by the data in are the same or similar. i Add to this collection C i In. Use Kd-tree structure to quickly find the non-ground point data p obtained by distance selection i The set B of the nearest n points i . Traverse set B i If the point is far from the selected non-ground point data p i If the distance is less than the set threshold, it is placed in set C i If it is greater than the threshold, the point is determined to be noise. According to the formula "A=AC i "Update set A and repeat the above steps until set A is empty, and return the clustering results.

[0149] Reference Figure 5 As shown, Figure 5 The flowchart shown is for fusing radar perception results provided by an embodiment of the present application. In one embodiment of the present application, S400 is the step of fusing radar perception results with camera perception results to obtain obstacle perception results. S400 is broadly divided into two steps: one for obtaining the fused radar perception results and the other for obtaining the final obstacle perception results. The process of obtaining the fused radar perception results may include, but is not limited to, the following steps.

[0150] S410: Jointly calibrate the binocular camera and the lidar to obtain an extrinsic calibration matrix.

[0151] In this step, before fusion, you need to use the AUTOWARE tool to jointly calibrate the binocular camera and lidar to obtain the extrinsic calibration matrix.

[0152] It should be noted that AUTOWARE is a mainstream open source project framework for autonomous driving. AUTOWARE primarily consists of the following modules: Positioning module: This module achieves positioning by combining 3D maps and SLAM algorithms with GNSS and IMU sensors. Detection module: This module uses cameras and lidar for detection through sensor fusion algorithms and deep neural networks. Prediction and planning module: This module is based on probabilistic robotics models and rule-based systems, and some also use deep neural networks. Control module: Autoware outputs the vehicle's distortion of speed and angular velocity. Although the bulk of the control volume typically resides in the vehicle's drive-by-wire controller, these are part of the control module.

[0153] S420: Project the three-dimensional detection frame onto the long-range image to obtain a two-dimensional detection frame.

[0154] It should be noted that the two-dimensional detection frame refers to the frame formed by projecting the 3D detection frame obtained by lidar detection onto the 2D image.

[0155] S430: Traverse all two-dimensional detection frames to find the target obstacle with the highest overlap with the two-dimensional detection frame in the long-range image, and add the category information corresponding to the target obstacle with the highest overlap to the radar perception result to obtain a fused radar perception result.

[0156] In the above steps, since the original radar perception results do not include the category of the target obstacle, it is necessary to fuse the radar perception results with the camera perception results, so that the radar perception results can more comprehensively reflect the actual situation of the obstacle. To this end, this application traverses all 3D detection frame projections for each image detection frame output by the target detection model, and searches for the object with the highest degree of overlap in the long-range image plane. At the same time, the category information of the object with the highest degree of overlap is obtained, and this category information is added to the radar perception result, thereby forming a fused radar perception result.

[0157] After obtaining the fused radar perception results, the obstacle perception results are inferred through the perception strategy. In this application, when the distance between the obstacle and the head of the vehicle is less than 15 meters, the laser radar is installed on the top of the vehicle and is relatively high, so the laser radar has poor perception of short and small obstacles. In addition, the laser radar perception results obtained by clustering do not have object category information, which makes it inconvenient to obtain specific information and data of the target obstacle. The camera perception results output by the target detection module contain object category information. Therefore, the perception strategy is used to fuse the camera perception results and the radar perception results to improve the positioning accuracy of the target obstacle. This application adopts a perception strategy that uses the distance between the obstacle and the head of the vehicle as 15 meters as the distinction. For obstacles within 15 meters, the camera perception result is used as the final result. For obstacles beyond 15 meters, the radar perception result fused with the obstacle category label is used as the final result.

[0158] In one embodiment of the present application, S500 will be further described and elaborated below. S500 may include but is not limited to the following steps.

[0159] S510 , collecting the UTM coordinate information of the vehicle, and combining it with the distance information between the target obstacle and the vehicle to obtain the UTM coordinate information of the target obstacle.

[0160] It should be noted that the vehicle's real-time UTM (Universal Transverse Mercator Grid System) coordinate information is obtained through the GPS positioning system installed at the center of the vehicle's rear axle.

[0161] S520 , using the Hungarian algorithm to match and track the target obstacle in each time frame at a preset matching frequency, obtain the change in the UTM coordinate information of the target obstacle, and then obtain the instantaneous speed and movement direction of the target obstacle in the current time frame.

[0162] It should be noted that the Hungarian algorithm is the most common algorithm for partial graph matching. It uses augmenting paths to find the maximum matching of a bipartite graph. A bipartite graph is a special model in graph theory. If the vertex set V of an undirected graph G = (V, E) can be partitioned into two mutually disjoint subsets V1 and V2, and the endpoints of any edge belong to both sets, then the graph G is called a bipartite graph. The maximum matching problem is to select the subset with the largest number of edges among these subsets, and the maximum matching number is called a maximum matching.

[0163] In this step, the matching frequency is set to one second. The UTM coordinate changes of obstacles are used to roughly estimate and calculate the speed and direction of the obstacles. At this point, the vehicle's perception system, or domain controller, has obtained the position, volume, type, and speed of all obstacles in front of the vehicle in the current time frame.

[0164] In one embodiment of the present application, the optimization process of the instantaneous speed of a target obstacle is described and elaborated below. After obtaining the position, volume, category, and speed information of all obstacles in front of the vehicle, the present application uses a Kalman filter method to optimize the instantaneous speed of the target obstacle. Specifically, in S600, optimizing the instantaneous speed of the target obstacle includes the following steps:

[0165] First, based on the obstacle perception results, a single obstacle state x is constructed k =(p x , p y , v x , v y ) T ;

[0166] Then, construct the constant velocity model:

[0167]

[0168] in, is the state matrix, is the noise matrix.

[0169] Among them: The coordinate information of the target obstacle in the x direction and y direction is respectively obtained through p x and p y To express, the speed information of the target obstacle in the x direction and y direction is respectively expressed by v x 、p y The speed of the target obstacle at the current moment is expressed by x k To express, the speed of the target obstacle at the next moment is expressed by x k+1 The time difference between the current time k and the next time k+1 is represented by Δt.

[0170] Then we get P k =AP k-1 A T .

[0171] Afterwards, according to the Kalman filter update equation G k =P k C T (CP k C T +R) -1 , and the solution is as follows:

[0172] P k ←(1-G k C)P k .

[0173] Where C is the measurement matrix, which is expressed by the following formula: The measurement noise covariance matrix is ​​represented by R, which satisfies the following formula: in, and All of these can be obtained through actual measurement.

[0174] By formula P k ←(1-G k C)P k Reduce the fluctuation of the instantaneous speed of the target obstacle and obtain the stable speed information of the target obstacle.

[0175] Based on the above embodiment, in S600, after obtaining stable target obstacle speed information, the target obstacle speed information of each time frame is matched and updated using the Hungarian algorithm, thereby automatically controlling the vehicle's travel. Specifically, the process of automatically controlling the vehicle's travel includes the following steps:

[0176] First, the detection range is divided. In this specific embodiment, the detection range is within five meters in the left and right directions and within forty meters in the front, centered on the vehicle.

[0177] For obstacles outside the detection range, considering that the vehicle is traveling at a low speed, it is assumed that obstacles outside the monitoring range will not affect the subsequent decision-making and planning of autonomous driving. Therefore, the perception information of such obstacles is not output. In addition, it will not affect the current driving state of the vehicle.

[0178] Obstacles within the detection range are considered target obstacles. Each time a target obstacle's obstacle status is updated, the state matrix and noise matrix of its velocity model are recorded. This allows the Kalman filter to continuously update the velocity information and status of each individual target obstacle as the UTM coordinates of the target obstacle are received in each time frame. Simultaneously, based on the updated velocity model and obstacle status of the target obstacle, the vehicle's domain controller plans, makes decisions, and controls the vehicle's current driving state.

[0179] Reference Figure 6 As shown, Figure 6 The following is a schematic diagram of the method for accurately sensing obstacles in a vehicle provided by an embodiment of the present application. The following example illustrates the technical solution of the present application. During the process of automatic driving or driving of the vehicle, the following steps are performed:

[0180] Image acquisition: The Zhang Zhengyou calibration method is used to establish the camera imaging geometry model, obtain the intrinsic parameter matrix, extrinsic parameter matrix, and distortion coefficient of the binocular camera. At the same time, the binocular camera is used to obtain medium-range and long-range images.

[0181] 2D Image Object Detection: Long-range images are fed into a pre-trained YOLOv5-based object detection model to obtain preliminary 2D obstacle locations and categories, representing the camera's perception results. The 2D locations of obstacles are represented by detection boxes in the image output by the model.

[0182] 2D disparity estimation: Input the medium-range image and the long-range image into the trained deep stereo matching model and output a 2D disparity map.

[0183] 2D Image Depth Matching: Adjust parameters to detect the depth of obstacles using a binocular camera as input by finding the depth information of the projection point of the center pixel of the image detection frame on the 2D disparity map. The distance information and the first volume information obtained from depth matching are added to the camera perception results.

[0184] LiDAR perception: Processes the raw point cloud data detected by the LiDAR and outputs the radar perception results.

[0185] Sensor data fusion: The 3D detection frame perceived by the LiDAR is projected onto the plane of the 2D image. By finding the object with the highest overlap, the radar perception results are fused and modified to obtain a fused radar perception result. Furthermore, the final obstacle perception result is filtered based on the perception strategy.

[0186] Obstacle trajectory prediction: The vehicle's UTM coordinates are collected in real time through the GPS positioning system. Combined with obstacle perception results, the instantaneous speed and direction of the obstacle are calculated. A constant velocity model is then established, incorporating the noise matrix and measurement matrix, and the Kalman gain is calculated to update the obstacle status in real time.

[0187] Vehicle Control: Obtains the real-time obstacle status of obstacles within the detection range based on the detection range. The vehicle's domain controller controls the vehicle's driving state in real time based on this real-time obstacle status.

[0188] This application uses lidar and binocular cameras as target perception sensors to sense and detect obstacles while the vehicle is in motion. The binocular camera's perception results are integrated with the lidar's to obtain the final obstacle perception result, making obstacle perception redundant and meeting the safety and stability requirements of autonomous driving. This application solves the problem of lidar's blind spots at close range and its inability to effectively perceive obstacles. It can sense and track the trajectory of moving obstacles, providing data support for subsequent vehicle decision-making and path planning.

[0189] First, by performing target detection and stereo matching inference on pre-calibrated camera images, it is possible to accurately perceive the distance and size of obstacles that are in the lidar blind spot, close to the vehicle, and at a low height. This solves the problem of lidar detection blind spots at close ranges and expands the perception range of obstacle information.

[0190] LiDAR is then selected as a complementary sensor to camera perception. By integrating radar perception results, the accuracy of LiDAR perception is improved. Obstacle perception results are obtained based on the perception strategy, improving the accuracy of environmental information used for vehicle decision-making and planning. This increases the precision of environmental perception and thus meets the vehicle's environmental perception requirements.

[0191] Finally, based on the obstacle perception results, the instantaneous speed of the obstacle is obtained and Kalman filtering is performed to collect the actual state information of the obstacle, which can meet the needs of state perception and trajectory tracking of the target object in motion, and then realize subsequent vehicle decision-making and path planning.

[0192] In addition, an embodiment of the present application further provides a storage medium storing processor-executable instructions, which, when executed by the processor, are used to execute a method for accurately sensing vehicle obstacles.

[0193] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0194] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk.

[0199] The step numbers in the above method embodiment are only provided for the convenience of explanation and do not limit the order of the steps. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

Claims

1. A method for accurately sensing vehicle-mounted obstacles, characterized in that: The steps include: Obtaining a long-range image and a medium-range image through a binocular camera, inputting the long-range image into a target detection model to obtain a camera perception result, wherein the camera perception result includes an image detection frame of the target obstacle and category information of the target obstacle; Inputting the long-range image and the medium-range image into a depth stereo matching model, outputting a disparity map, and projecting the image detection frame into the disparity map to obtain distance information between the target obstacle and the vehicle, as well as orientation information and first volume information of the target obstacle, and adding the distance information, orientation information, and first volume information to the camera perception result; Obtaining raw point cloud data from a laser radar, processing the raw point cloud data using a point cloud ground-non-ground segmentation method and a point cloud clustering algorithm, and outputting a radar perception result, the radar perception result including location information of the target obstacle, second volume information, and a three-dimensional detection frame; fusing the camera perception result and the radar perception result to obtain a fused radar perception result; when the distance information is less than a preset distance threshold, using the camera perception result as the final obstacle perception result; and when the distance information is greater than or equal to the distance threshold, using the fused radar perception result as the final obstacle perception result; The fused radar perception result includes the radar perception result and category information of the target obstacle; Collecting the UTM coordinate information of the vehicle, combining the UTM coordinate information of the vehicle and the obstacle perception result to calculate the UTM coordinate information of the target obstacle, and calculating the instantaneous speed and movement direction of the target obstacle at the current moment based on the UTM coordinate information of the target obstacle; The instantaneous speed of the target obstacle is optimized by Kalman filtering to obtain stable speed information of the target obstacle, and the obstacle state of the target obstacle is updated using the Hungarian algorithm according to the stable speed information, and the vehicle is controlled according to the obstacle state.

2. The method for accurately sensing vehicle-mounted obstacles according to claim 1, characterized in that: Before obtaining the long-range image and the medium-range image by the binocular camera, the method includes: Collecting a pose image of a calibration plate, the calibration plate consisting of a plurality of two-dimensional squares, and extracting pixel coordinates of corner points of the pose image; Calculating the pixel coordinates of the corner points using a homography matrix to obtain initial values ​​of the intrinsic parameters and initial values ​​of the extrinsic parameters of the binocular camera; estimating an initial value of the distortion coefficient of the binocular camera by a nonlinear least squares method; The initial values ​​of the intrinsic parameters, the initial values ​​of the extrinsic parameters, and the initial values ​​of the distortion coefficients are optimized by the maximum likelihood estimation method to obtain the final intrinsic parameters, extrinsic parameters, and distortion coefficients of the binocular camera.

3. The method for accurately sensing vehicle-mounted obstacles according to claim 1, characterized in that: The step of inputting the long-range image and the medium-range image into a depth stereo matching model to obtain a disparity map includes: performing stereo correction on the long-range image and the medium-range image; Inputting the long-range image and the medium-range image into a depth stereo matching model to obtain a disparity map; Wherein, the deep stereo matching model is a neural network model based on AANET; The performing stereo correction on the depth stereo matching model includes: Obtaining original pixel points of the long-range image and the medium-range image, and converting the pixel coordinate systems of the long-range image and the medium-range image into a camera coordinate system using the intrinsic parameters of the binocular camera; calculating distortion amounts of the long-range image and the medium-range image, obtaining distorted positions of the long-range image and the medium-range image in the camera coordinate system, and performing dedistortion processing on the long-range image and the medium-range image according to the distorted positions; Converting the camera coordinate system to the pixel coordinate system, and re-interpolating pixel points of the long-range image and the medium-range image in the pixel coordinate system using the original pixel points; The sizes of the medium-distance image and the long-distance image are adjusted so that the sizes of the medium-distance image and the long-distance image are consistent.

4. The method for accurately sensing vehicle-mounted obstacles according to claim 3, characterized in that: The projecting the image detection frame onto the disparity map to obtain distance information between the target obstacle and the vehicle, as well as orientation information and first volume information of the target obstacle, includes: Using the center point of the image detection frame as the reference point of the target obstacle, calculating the coordinates of the reference point of the target obstacle in a pixel coordinate system, and projecting the image detection frame onto the disparity map to obtain the projected point coordinates of the reference point in the disparity map, and obtaining the distance between the target obstacle and the vehicle using the projected point coordinates; Converting the projection point coordinates from the pixel coordinate system to the world coordinate system according to the intrinsic parameters of the binocular camera to obtain the orientation information of the target obstacle relative to the vehicle; First volume information of the target obstacle is inferred based on the long-range image and the medium-range image in combination with the distance information and the orientation information.

5. The method for accurately sensing vehicle-mounted obstacles according to claim 1, characterized in that: The processing of the original point cloud data by using the point cloud ground-non-ground segmentation method and the point cloud clustering algorithm to output the radar perception result includes: Dividing the space in front of the vehicle into a plurality of subspaces according to the distance between the laser radar and the head of the vehicle, each of the subspaces comprising a plurality of original point cloud data located within the subspace; Setting a height threshold corresponding to each subspace, calculating the distance between the original point cloud data of each subspace and the ground, and recording the distance as the point cloud height of the original point cloud data; sequentially determining whether the point cloud height of each piece of the original point cloud data is greater than its corresponding height threshold, and dividing each piece of the original point cloud data into ground point data and non-ground point data according to the determination result; Clustering all the original point cloud data according to the non-ground point data in combination with the Euclidean clustering algorithm to obtain a radar perception result; The step of dividing each original point cloud data into ground point data and non-ground point data according to the judgment result includes: When the point cloud height is less than the height threshold corresponding to the original point cloud data, the original point cloud data is ground point data; When the point cloud height is greater than or equal to a height threshold corresponding to the original point cloud data, the original point cloud data is non-ground point data.

6. The method for accurately sensing vehicle-mounted obstacles according to claim 1, characterized in that: The fusing the camera perception result and the radar perception result to obtain a fused radar perception result includes: Jointly calibrating the binocular camera and the laser radar to obtain an extrinsic calibration matrix; Projecting the three-dimensional detection frame onto the long-range image to convert it into a two-dimensional detection frame; Traverse all the two-dimensional detection frames to find the target obstacle in the long-range image that has the highest overlap with the two-dimensional detection frame, and add the category information corresponding to the target obstacle with the highest overlap to the radar perception result to obtain a fused radar perception result.

7. The method for accurately sensing vehicle-mounted obstacles according to claim 1, characterized in that: The calculating, based on the UTM coordinate information of the target obstacle, the instantaneous speed and movement direction of the target obstacle at the current moment includes: Using the Hungarian algorithm, the target obstacle in each time frame is matched and tracked at a preset matching frequency to obtain the change value of the UTM coordinate information of the target obstacle; Calculate the instantaneous speed and movement direction of the target obstacle in the current time frame based on the change value of the UTM coordinate information; The matching frequency is set to one second.

8. The method for accurately sensing vehicle-mounted obstacles according to claim 1, characterized in that: Optimizing the instantaneous speed of the target obstacle by using Kalman filtering to obtain stable speed information of the target obstacle includes: According to the obstacle perception results, a single obstacle state x is constructed k =(p x , p y , v x , v y ) T ; Construct a constant velocity model: in: is the state matrix; is the noise matrix; Where: p x is the coordinate information of the target obstacle in the x direction, p y is the coordinate information of the target obstacle in the y direction, v x and v y Respectively represent the speed information of the target obstacle in the x direction and y direction, x k is the speed of the target obstacle at the current moment, x k+1 is the speed of the target obstacle at the next moment, Δt is the time difference between the current moment and the next moment; According to the Kalman filter update equation G k =P k C T (CP k C T +R) -1 , and we get the following formula: P k ←(1-G k C)P k ; in: is the measurement matrix, is the measurement noise covariance matrix; By P k ←(1-G k C)P k To obtain the stable speed information of the target obstacle.

9. A vehicle-mounted obstacle precision sensing system, characterized in that: include: Binocular cameras, including a long-range camera and a medium-range camera, are used to detect the environment in front of the vehicle and obtain long-range images and medium-range images; Wherein, the long-range camera and the medium-range camera are respectively installed at both ends of the front windshield of the vehicle; LiDAR, located on top of the vehicle, is used to detect target obstacles at medium and long distances and generate raw point cloud data; GPS positioning system, used to obtain the vehicle's UTM coordinate information; On-board domain controller, including: The target detection unit is equipped with a target detection model, which is used to detect targets in long-range images through the target detection model to obtain camera perception results; The camera perception result includes an image detection frame of the target obstacle and category information of the target obstacle; A stereo matching unit equipped with a deep stereo matching model is used to perform stereo matching on long-range images and medium-range images to obtain a disparity map. an image processing unit, configured to project the image detection frame into the disparity map, obtain distance information between the target obstacle and the vehicle, orientation information of the target obstacle, and first volume information, and add the distance information between the target obstacle and the vehicle, the orientation information of the target obstacle, and the first volume information to a camera perception result; a radar data processing unit, configured to process the raw point cloud data using a point cloud ground-non-ground segmentation method and a point cloud clustering algorithm to obtain a radar perception result; The radar sensing result includes the location information of the target obstacle, the second volume information and the three-dimensional detection frame; a first data processing unit, configured to fuse a camera perception result and a radar perception result to obtain a fused radar perception result; and when the distance information is less than a preset distance threshold, using the camera perception result as the final obstacle perception result; and when the distance information is greater than or equal to the distance threshold, using the fused radar perception result as the final obstacle perception result; a second data processing unit, configured to calculate the UTM coordinate information of the target obstacle by combining the UTM coordinate information of the vehicle and the obstacle sensing result, and to infer the instantaneous speed and movement direction of the target obstacle at a current moment based on the UTM coordinate information of the target obstacle; a third data processing unit, configured to optimize the instantaneous speed of the target obstacle through a Kalman filter to obtain stable speed information of the target obstacle, and update the state of the obstacle using a Hungarian algorithm according to the stable speed information; The control unit is used to control the vehicle driving according to the updated state of the obstacle.

10. A storage medium storing instructions executable by a processor, characterized in that: The processor-executable instructions, when executed by the processor, are used to execute the method for accurately sensing vehicle-mounted obstacles as described in any one of claims 1-8.

Images