Method, system, medium and equipment for unmanned vehicle to perceive obstacle target

Abstract

The invention belongs to the technical field of vehicle automatic driving, and provides a method, a system, a medium and equipment for an unmanned vehicle to perceive an obstacle target, and the method comprises the steps: obtaining the image data of a camera, and extracting a first target set and the first feature information of a first target from the image data through a target detection algorithm; extracting first position information of the first target from the image data by using a camera projection model to obtain first comprehensive information of the first target; acquiring three-dimensional point cloud data of the laser radar, and extracting a second target set and second comprehensive information of a second target from the three-dimensional point cloud data by using an Euclidean clustering method; and determining an obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information. According to the invention, the perception capability of the vehicle system to the obstacle target is improved, the matching capability and the position estimation capability of the obstacle target are optimized, and the collision risk of the vehicle and the obstacle target is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle automatic driving, and in particular relates to a method, system, medium and equipment for an unmanned vehicle to perceive an obstacle target. Background Art

[0002] The IGV (Intelligent Guided Vehicle) operating in the railway freight yard is an autonomous transport vehicle that travels along a preset route and performs container transshipment tasks in the railway freight yard. There are staff and various movable vehicles and equipment such as container trucks, reach stackers, forklifts, and rail-mounted gantry cranes in the freight yard, which together constitute a mixed traffic flow. Since no traffic rules have been formulated in the freight yard, there is no regularity for vehicles in the freight yard. There are many driving situations that do not comply with conventional traffic rules, such as crossing lanes, driving in the opposite direction, and driving sideways. The traffic environment is complex and changeable. In order to ensure the safe and orderly operation of the IGV in the freight yard, its operating rules and lanes have been clearly defined, but other manually driven vehicles still do not follow any rules and often intrude into the IGV lanes. In addition, there are large vehicles in the freight yard that do not appear on conventional roads, such as container trucks and reach stackers. There are fewer data samples of these large vehicles in the visual recognition dataset, and the large size makes it difficult for the camera to capture enough effective information to accurately judge the type and location of the vehicle. These large vehicles bring challenges to the autonomous driving perception system. In this complex and unconventional mixed traffic environment, conventional autonomous driving vehicle perception strategies are difficult to support the safe operation of IGVs. Therefore, in daily operations, a new customized perception strategy deployment solution is needed to ensure the perception capabilities of the autonomous driving system. Summary of the invention

[0003] The present invention provides a method, system, medium and equipment for an unmanned vehicle to perceive an obstacle target.

[0004] A method for an unmanned vehicle to perceive an obstacle target of the present invention comprises: Acquire image data from a camera, and extract a first target set and first feature information of the first target from the image data using a target detection algorithm; Extracting first position information of a first target from the image data using a camera projection model, and defining the first feature information and the first position information as first comprehensive information of the first target; Acquire three-dimensional point cloud data of the laser radar, and extract a second target set and second comprehensive information of the second target from the three-dimensional point cloud data using a Euclidean clustering method; The operation information of the unmanned vehicle is obtained, and the obstacle target is determined based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information.

[0005] Further, The step of acquiring image data from a camera and extracting a first target set and first feature information of the first target from the image data using a target detection algorithm includes: Acquire image data from the camera, detect the image data using a pre-trained model in the target detection algorithm, and obtain a first original target set and first feature information of the first original target; Based on the type and confidence in the first feature information and using the confidence threshold in the target detection algorithm, the first original target set is filtered to obtain the first target set and the first feature information of the first target.

[0006] Further, The method of acquiring the three-dimensional point cloud data of the laser radar and extracting the second target set and the second comprehensive information of the second target from the three-dimensional point cloud data by using the Euclidean clustering method includes: Acquire the three-dimensional point cloud data of the laser radar, and define the three-dimensional point cloud data of the laser radar as original three-dimensional point cloud data; Preprocessing the original three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data; Based on the spatial distribution characteristics of the three-dimensional point cloud data, the pre-processed three-dimensional point cloud data is divided into a plurality of regions according to the spatial position; Clustering the preprocessed three-dimensional point cloud data in each of the regions using the Euclidean clustering method to obtain a clustering result; The clustering result is optimized to obtain the second target set and second comprehensive information of the second target.

[0007] Further, The Euclidean clustering method is used to cluster the pre-processed three-dimensional point cloud data in each area to obtain a clustering result, including: Calculating the variance of all data points in each dimension of the preprocessed three-dimensional point cloud data, and selecting the dimension corresponding to the largest variance as the segmentation dimension of the current node; Calculate the median of all the data points in the split dimension, and use the median as the split point of the current node; Constructing a hyperplane, wherein the hyperplane is perpendicular to the segmentation dimension and passes through the segmentation point; Recursively construct left and right subtrees based on the segmentation dimension, the segmentation point and the hyperplane.

[0008] Further, The step of optimizing the clustering results to obtain a second target set and second comprehensive information of the second target includes: Optimizing the clustering result to obtain an optimized clustering result; In the optimized clustering result, one cluster is one second target, and one or more second targets constitute the second target set; Determine the second position information of the second target by the three-dimensional coordinates of the center point of each cluster; Extracting bounding box coordinates from the three-dimensional point cloud data in each of the clusters, determining a bounding box through the bounding box coordinates, and thereby determining second shape information of a second object through the bounding box; Determine the orientation of the second target by using the normal vector of the three-dimensional point cloud data in each of the clusters, thereby determining second posture information of the second target; The second position information of the second object, the second shape information of the second object, and the second posture information of the second object are collectively defined as the second comprehensive information of the second object.

[0009] Further, The obtaining of the operation information of the unmanned vehicle and determining the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information include: Aligning the time frames of acquisition of the operation information of the unmanned vehicle, the first comprehensive information, and the second comprehensive information; Fusing the first target set and the second target set to obtain a first fused target set and first fused comprehensive information of the first fused targets; Using the Kalman filter method to update the first fusion comprehensive information, to obtain updated first fusion comprehensive information; Further, The step of obtaining the operation information of the unmanned vehicle and determining the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information further includes: Based on the updated first fusion comprehensive information, the first fusion target set is matched between frames by using the Hungarian algorithm to obtain a final fusion target set and final fusion information of the final fusion target, wherein the final fusion information includes final fusion position information, final fusion shape information and final fusion posture information; The obstacle target is determined through the final fusion information.

[0010] A system for an unmanned vehicle to perceive an obstacle target of the present invention is used to implement the above-mentioned method for an unmanned vehicle to perceive an obstacle target, and the system comprises: A first feature information acquisition module, used to acquire image data from a camera, and extract a first target set and first feature information of the first target from the image data using a target detection algorithm; A first comprehensive information acquisition module, configured to extract first position information of a first target from the image data by using a camera projection model, and to define the first feature information and the first position information as first comprehensive information of the first target; A second comprehensive information acquisition module is used to acquire three-dimensional point cloud data of the laser radar, and extract a second target set and second comprehensive information of the second target from the three-dimensional point cloud data using a Euclidean clustering method; The obstacle target determination module is used to obtain the operation information of the unmanned vehicle and determine the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information.

[0011] A computer-readable storage medium of the present invention stores a program or instruction. When the program or instruction is executed on a computer, the computer executes the aforementioned method for an unmanned vehicle to perceive an obstacle target.

[0012] An electronic device of the present invention includes a processor, which is coupled to a memory; the processor is used to read and execute a computer program stored in the memory to implement the above-mentioned method for unmanned vehicle to perceive obstacle targets.

[0013] Compared with the prior art, the present invention has the following beneficial effects: Aiming at the complex traffic environment of railway freight yards, especially the chaotic driving of large manned vehicles, a barrier target perception method integrating lidar, camera and vehicle operation data is designed. This method improves the vehicle system's ability to perceive obstacles and is particularly suitable for the dynamic identification and positioning of large vehicle obstacles in railway freight yards. At the same time, it reduces the frequent processing of long-distance obstacle targets and saves computing resources.

[0014] Through position projection technology, 2D pixel coordinates are accurately converted into 3D world coordinates, improving the accuracy of spatial information.

[0015] The processing method of fusion of visual and laser perception results integrates Hungarian matching and Kalman filtering algorithms to optimize the obstacle matching ability and position estimation ability.

[0016] A priority processing mechanism is designed to enable the system to give priority to obstacle targets that have the greatest impact on the safe driving of the vehicle, improve response efficiency, and reduce the risk of collision between the vehicle and obstacle targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of the bidirectional vehicle and the sensor arrangement structure of the present invention; Figure 2 A flow chart of a method for an unmanned vehicle to perceive an obstacle target according to the present invention; Figure 3 A schematic diagram of the structure of a system for an unmanned vehicle to perceive an obstacle target according to the present invention; Figure 4 It is a schematic diagram of the structure of the electronic device of the present invention.

[0019] Description of reference numerals: 201 - first feature information acquisition module, 202 - first comprehensive information acquisition module, 203 - second comprehensive information acquisition module, 204 - obstacle target determination module, 301 - processor, 302 - memory. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, the unmanned vehicle applicable to the embodiment of the present invention is an intelligent guided vehicle (i.e., "IGV"), which has the following characteristics: the front, rear, left, and right structures of the frame are completely symmetrical, and there is no difference in the front and rear driving capabilities; the front and rear wheels of the vehicle are respectively equipped with a steering motor and a drive motor, and the steering and driving of the front and rear wheels can be independently controlled; the vehicle is equipped with sensors.

[0022] The sensors include cameras and laser radars. The camera is used to obtain image data. The camera is divided into a front camera and a side camera. The front camera is arranged at the front of the vehicle at both ends, looking directly in front of the vehicle. Two front cameras are arranged at each end of the front of the vehicle, one is a normal camera for sensing obstacles in front of the vehicle, and the other is a fisheye camera for sensing obstacles near the front of the vehicle. The side cameras are arranged at the four corners of the vehicle, facing the middle of the vehicle, and are used to sense obstacles around the vehicle and on the sides. The laser radar is used to obtain three-dimensional point cloud data. The laser radar is arranged at the four corners of the vehicle to monitor the obstacles in the laser point cloud in real time, realizing all-round obstacle detection centered on the vehicle.

[0023] Figure 2 1 is a flow chart of a method for an unmanned vehicle to perceive an obstacle target provided by an embodiment of the present invention. In one embodiment, the method specifically includes the following steps: S101: Acquire image data from a camera, and extract a first target set and first feature information of the first target from the image data using a target detection algorithm.

[0024] S101 - 1: Acquire image data from a camera, use a pre-trained model in a target detection algorithm to detect the image data, and obtain a first original target set and first feature information of the first original target.

[0025] The image data is captured in real time by the camera equipped on the IGV.

[0026] The pre-trained model is a pre-trained target detection model. The training method is: use the image data containing the obstacle target (hereinafter referred to as the "target") as the training data set, and use the training data set to train the target detection model to obtain the pre-trained model. The targets include common targets on daily roads such as cars, pedestrians, and triangular cones, as well as unique targets in railway freight yards such as container trucks, forklifts, reach loaders, and gantry cranes. Training the target detection model with a training set composed of image data containing multiple targets can improve the detection accuracy of the pre-trained model.

[0027] Using the pre-trained model to detect image data, one or more first original targets are detected from the image data. The one or more first original targets constitute a first original target set. Each first original target has corresponding feature information, and the feature information includes bounding box coordinates, type, and confidence , determine the bounding box through the bounding box coordinates, determine the shape information of the first original target through the bounding box, and determine the posture information of the first original target through the type. , shape information, and posture information are uniformly defined as the first feature information.

[0028] S101 - 2: Based on the type and confidence in the first feature information, and using the confidence threshold in the target detection algorithm, the first original target set is filtered to obtain the first target set and the first feature information of the first target.

[0029] The confidence threshold in the target detection algorithm is used to filter out erroneous targets or low-quality targets, thereby retaining high-quality targets. The confidence threshold configuration table is shown in Table 1. Table 1 Confidence threshold configuration table

[0030] Compare the confidence of each first original target according to the confidence threshold configuration table With confidence threshold The size of the confidence ≤Confidence Threshold , remove the confidence The corresponding first original target is obtained by removing the first original target and retaining the first original target. The removed first original targets constitute the first filtered target set, and the retained first original targets constitute the first retained target set. The “retained first original target” is referred to as the “first target”, and the “first retained target set” is referred to as the “first target set”.

[0031] Since this step only adjusts the data range of the first original target set and does not change the data range of the first feature information, the first feature information of the first target in the first target set and the first feature information of the first original target removed in the first filtered target set are the same as the first feature information of the first original target in the first original target set.

[0032] By comparing the confidence level with the confidence threshold, small targets such as pedestrians and cones as well as large targets with extremely high confidence such as vehicles can be retained, eliminating misjudgments caused by large target size, non-standard target posture, and vehicles in the target not complying with traffic rules, thereby ensuring the accuracy of target recognition.

[0033] S102: extracting first position information of the first target from the image data using a camera projection model, and uniformly defining the first feature information and the first position information as first comprehensive information of the first target.

[0034] Obtain the intrinsic and extrinsic parameters of the camera equipped with the IGV. The intrinsic parameters include focal length and principal point coordinates, and the extrinsic parameters refer to the position and posture of the camera relative to the IGV.

[0035] Get the image data from the camera and establish a pixel coordinate system to describe the image data. The pixel coordinates in the pixel coordinate system are express.

[0036] The pixel coordinates of each first target Convert to world coordinates , the conversion formula is as follows, (1), In the formula, is the intrinsic parameter matrix of the camera; is the rotation matrix of the camera; is the camera's translation vector.

[0037] Through this step, the pixel coordinates of the first target can be converted into world coordinates, and the first position information of the first target can be obtained through the world coordinates.

[0038] The first feature information of the first target (confidence , shape information, posture information) and the first position information of the first target are collectively defined as the first comprehensive information of the first target.

[0039] S103: Acquire three-dimensional point cloud data of the laser radar, and extract a second target set and second comprehensive information of the second target from the three-dimensional point cloud data using a Euclidean clustering method.

[0040] S103-1: Obtain the three-dimensional point cloud data scanned in real time by the laser radar equipped with the IGV, and define the three-dimensional point cloud data scanned in real time by the laser radar as the original three-dimensional point cloud data. The three-dimensional point cloud data refers to the point cloud data of the three spatial dimensions of horizontal (X axis), vertical (Y axis), and height (Z axis).

[0041] S103-2: Preprocessing the original three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data, where the preprocessing method includes filtering, denoising, downsampling, etc. Preprocessing can reduce the amount of data and improve the data quality.

[0042] S103-3: Based on the spatial distribution characteristics of the three-dimensional point cloud data, the pre-processed three-dimensional point cloud data is divided into a plurality of regions according to the spatial positions.

[0043] S103-4: Cluster the pre-processed three-dimensional point cloud data in each area using the Euclidean clustering method to obtain a clustering result.

[0044] Euclidean clustering is a clustering algorithm based on Euclidean distance metric. In order to improve clustering efficiency, KD-Tree (full name "K-dimension tree") is usually constructed as a spatial index structure. KD-Tree is a balanced binary tree in which each node is a k-dimensional data point. Each node on the balanced binary tree can construct a hyperplane, which is perpendicular to the current partition dimension and divides the space into two parts based on the current partition dimension, one part in the left subtree of the current partition dimension, and the other part in the right subtree of the current partition dimension.

[0045] In the present invention, the preprocessed three-dimensional point cloud data includes not only the three spatial dimensions mentioned in S103-1, but also the reflection intensity dimension. The reflection intensity refers to the intensity of the laser pulse light reflection, which can provide information related to the material of the object.

[0046] The clustering process of the preprocessed 3D point cloud data is as follows: Step 1: Determine the segmentation dimensions and points.

[0047] Calculate the variance of all data points in each dimension of the preprocessed 3D point cloud data. The calculation formula is as follows: (2), In the formula, is the number of all data points in a single dimension; is the value of a single data point; is the average of all data points in a single dimension.

[0048] Select the dimension corresponding to the largest variance as the segmentation dimension of the current node, and record the segmentation dimension as .

[0049] Calculate the median of all data points in the split dimension, and use the median of all data points in the split dimension as the split point of the current node. The calculation formula is as follows: (3), In the formula, is the dividing point; is the set of values ​​for all data points in a single dimension.

[0050] Step 2: Construct the hyperplane.

[0051] Create a hyperplane perpendicular to the splitting dimension and passes through the split point .

[0052] Step 3: Construct left and right subtrees based on the split dimension, split point and hyperplane.

[0053] In the split dimension Middle and less than the split point The data points are placed on the left side of the hyperplane, and the left side of the hyperplane is the left subtree; In and greater than or equal to the split point The data points are placed on the right side of the hyperplane, and the right side of the hyperplane is the right subtree.

[0054] Repeat the first to third steps for the left subtree and the right subtree, that is, recursively divide all data points in the left subtree and the right subtree until the number of data points contained in each node reaches the predetermined number.

[0055] Preferably, the predetermined number is one.

[0056] S103-5: Optimize the clustering results to obtain a second target set and second comprehensive information of the second target.

[0057] In order to improve the accuracy of clustering results, the clustering results are optimized. The optimization methods are as follows: Method 1: Set a distance threshold. When the distance between the center points of two clusters is less than this distance threshold, the two clusters are considered too close and are merged into a new cluster.

[0058] Method 2: A density threshold is set. When the density of the 3D point cloud data in a cluster is lower than this density threshold, the cluster is considered to be too dispersed and the cluster is merged with the cluster closest to the distance.

[0059] Method 3: The maximum threshold and the minimum threshold are set. When the number of 3D point cloud data in a cluster is less than the minimum threshold, the cluster is considered incomplete and the cluster is merged with the cluster closest to it. When the number of 3D point cloud data in a cluster is greater than or equal to the maximum threshold, the cluster is further subdivided.

[0060] Method 4: A similarity threshold is set and the shape features of the clusters are analyzed. When the shape feature similarity of two clusters is greater than or equal to the similarity threshold, the two clusters are considered to belong to the same target and are merged into a new cluster.

[0061] Method 5: An error threshold is set to analyze whether the directions of the normal vectors of the 3D point cloud data in the two clusters are consistent. When the directions of the normal vectors of the 3D point cloud data in the two clusters are consistent or the difference between the directions is less than the error threshold, the two clusters are considered to belong to the same target and are merged into a new cluster.

[0062] The optimized clustering result is obtained through the above method.

[0063] In the optimized clustering results, one cluster is a second target, and one or more second targets constitute a second target set. The second position information of the second target is determined by the three-dimensional coordinates of the center point of each cluster; the bounding box coordinates are extracted from the three-dimensional point cloud data of each cluster, and the bounding box is determined by the bounding box coordinates, so as to determine the second shape information of the second target by the bounding box; the normal vector of the three-dimensional point cloud data in each cluster is analyzed to determine the orientation of the second target, so as to determine the second posture information of the second target.

[0064] The second position information of the second target, the second shape information of the second target, and the second posture information of the second target extracted from the three-dimensional point cloud data are collectively defined as the second comprehensive information of the second target.

[0065] S104: Acquire the operation information of the unmanned vehicle, and determine the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information.

[0066] S104-1: Acquire the operation information of the unmanned vehicle, which specifically refers to the operation information of the IGV, and includes the vehicle position information, vehicle head direction information, and vehicle speed information. Align the time frame of the acquisition of the operation information of the unmanned vehicle, the acquisition time of the first comprehensive information of the first target, and the acquisition time of the second comprehensive information of the second target to ensure that the acquisition time of the above three pieces of information is consistent.

[0067] S104-2: Fusing the first target set and the second target set to obtain a first fused target set and first fused comprehensive information of the first fused targets.

[0068] For the first target bounding box and the second target bounding box at the same time (the same time is also called the "same frame"), first obtain the intersection area of ​​the two bounding boxes and the union area of ​​the two bounding boxes, and then divide the intersection area by the union area to obtain the calculation result, which is the intersection-union ratio of the two bounding boxes, that is, the degree of overlap of the two bounding boxes. The degree of overlap of the two bounding boxes is the degree of overlap of the first target and the second target at the same time.

[0069] An intersection-and-union ratio threshold is set to determine the overlap degree of the two bounding boxes and the size of the intersection-and-union ratio threshold. When the overlap degree of the two bounding boxes is greater than or equal to the intersection-and-union ratio threshold, the first target and the second target are fused into the same target to obtain a fused target. The shape information, posture information and position information of the fused target respectively inherit the second shape information, second posture information and second position information of the second target. When the overlap degree of the two bounding boxes is less than the intersection-and-union ratio threshold, it is considered that the first target and the second target respectively corresponding to the two bounding boxes do not belong to the same target. At this time, the first target and the second target are retained at the same time to obtain a retained first target and a retained second target.

[0070] The fusion target, the retained first target and the retained second target are uniformly defined as the first fusion target, each first fusion target has corresponding first fusion position information, first fusion shape information and first fusion posture information, and the first fusion position information, first fusion shape information and first fusion posture information are uniformly defined as the first fusion comprehensive information.

[0071] All first targets in the first target set and all second targets in the second target set are fused according to the method described in step S104-2 to obtain one or more first fused targets. One or more first fused targets constitute a first fused target set, and then a first fused target set at multiple different moments (different moments are also called "different frames") within a specified time range is obtained.

[0072] This step ensures that the first target and the second target are correctly identified and associated.

[0073] S104-3: Use the Kalman filter method to update the first fusion comprehensive information to obtain updated first fusion comprehensive information.

[0074] The Kalman filter is a recursive filter used to track and predict the first fusion position information, the first fusion shape information and the first fusion attitude information respectively to obtain updated first fusion position information, updated first fusion shape information and updated first fusion attitude information.

[0075] The specific process is as follows: Step 1: Clarify the state equation and observation equation.

[0076] The state equation is, (4), In the formula, is the real value at the current moment; is the state transfer matrix; is the true value of the previous moment; is the control matrix; The control instructions for the unmanned vehicle at the current moment; is the process noise at the previous moment.

[0077] The true value includes the true first fusion position information, the true first fusion shape information and the true first fusion attitude information. These three pieces of information are all obtained through field surveys.

[0078] The control instructions of the unmanned vehicle include vehicle acceleration, vehicle steering angle and vehicle speed. When the real value at the current moment is calculated by formula (4), the vehicle acceleration, vehicle steering angle velocity and vehicle speed at the current moment are substituted into formula (4) for subsequent calculation.

[0079] The observation equation is, (5), In the formula, is the observed value at the current moment; is the observation matrix; is the real value at the current moment; is the observation noise at the current moment.

[0080] The observation values ​​include the first fusion position information, the first fusion shape information and the first fusion attitude information.

[0081] Step 2: Perform Kalman filtering recursion based on the state equation and observation equation to obtain the updated value at the current moment.

[0082] The filtering process is as follows: (6), In the formula, is the predicted value at the current moment; is the state transfer matrix; is the updated value of the previous moment; is the control matrix; It is the control instruction of the unmanned vehicle at the current moment.

[0083] The predicted values ​​include predicted first fusion position information, predicted first fusion shape information, and predicted first fusion posture information.

[0084] The updated values ​​include updated first fusion position information, updated first fusion shape information, and updated first fusion posture information.

[0085] (7), In the formula, is the error between the predicted value at the current moment and the true value at the current moment; is the state transfer matrix; is the error between the updated value at the previous moment and the true value at the previous moment; is the process noise covariance.

[0086] (8), In the formula, is the Kalman gain; is the error between the predicted value at the current moment and the true value at the current moment; is the observation matrix; is the observation noise covariance.

[0087] (9), In the formula, is the updated value at the current moment; is the predicted value at the current moment; is the Kalman gain; is the observed value at the current moment; is the observation matrix.

[0088] (10), In the formula, is the error between the updated value at the current moment and the true value at the current moment; is the identity matrix; is the Kalman gain; is the observation matrix; is the error between the predicted value at the current moment and the true value at the current moment.

[0089] Through Kalman filtering, the updated first fusion position information, the updated first fusion shape information and the updated first fusion attitude information at different times can be obtained, and the updated first fusion position information, the updated first fusion shape information and the updated first fusion attitude information are uniformly defined as the updated first fusion comprehensive information.

[0090] Through this step, more accurate information input is provided for S104-4.

[0091] S104-4: Based on the updated first fusion comprehensive information, the first fusion target set is matched between frames using the Hungarian algorithm to obtain the final fusion target set and the final fusion information of the final fusion target.

[0092] Inter-frame matching means: when performing the first match on the data at different times within the specified time range, the data of the next frame is matched with the data of the previous frame, and the previous set frame is obtained through the first match. When performing subsequent matches, the data of the next frame is matched with the data of the previous set frame. And so on, until the inter-frame matching of the data at different times within the specified time range is completed.

[0093] The specific process is as follows: Step 1: Construct multiple cost matrices.

[0094] Different difference measures are used as different cost types, and different cost matrices are constructed based on different cost types.

[0095] Cost types include size difference, area difference, shape difference, and distance difference, specifically: A. Size difference: refers to the size difference between the first fusion target of the next frame and the first fusion target of the previous frame.

[0096] B. Area difference: refers to the area difference between the first matching target in the latter frame and the first fusion target in the previous set frame.

[0097] C. Shape difference: refers to the shape difference between the first fusion target of the latter frame and the first fusion target of the previous set frame. The shape difference is calculated based on the updated first fusion shape information.

[0098] D. Distance difference: refers to the Euclidean distance between the first fusion target of the latter frame and the first fusion target of the previous set frame. The Euclidean distance is calculated based on the updated first fusion position information.

[0099] Different cost matrices constructed based on different cost types are expressed as follows: (11), In the formula, Cost matrix corresponding to different cost types; Matching costs corresponding to different cost types; is the first fusion target of the next frame; It is the first fusion target of the previous set frame.

[0100] Take the distance difference as an example to explain the meaning of different parameters: is the cost matrix corresponding to the distance difference, is the Euclidean distance between the first first fusion target of the next frame and the first first fusion target of the previous set frame, is the first fusion target of the next frame and the first fusion target of the previous frame. The Euclidean distance between the first fusion targets, The next frame The first fusion target and the previous set frame The Euclidean distance between the first fusion targets.

[0101] Step 2: Perform row subtraction and column subtraction operations on each cost matrix to obtain the cost matrix after column subtraction.

[0102] Perform a row subtraction operation on each cost matrix, that is, subtract the minimum value of each row from the cost matrix so that each row has at least one zero, and obtain the cost matrix after row subtraction. The cost matrix after row subtraction is defined as .

[0103] Perform column subtraction on the cost matrix after row subtraction, that is, subtract the minimum value of each column from the cost matrix so that each column has at least one zero, and obtain the cost matrix after column subtraction. The cost matrix after column subtraction is defined as .

[0104] Performing row subtraction and column inspection operations makes the cost matrix more sparse.

[0105] Step 3: Perform a zero-element cover operation on each column-subtracted cost matrix.

[0106] Cover each column of the subtracted cost matrix with as few lines as possible All zero elements in , including horizontal and vertical lines, if the number of lines covering all zero elements is equal to the cost matrix after column subtraction If the order is , the optimal solution is found; otherwise, the cost matrix after column reduction is further adjusted , see step 4 for details.

[0107] Step 4: Adjust the cost matrix after each column subtraction to obtain the final adjusted cost matrix.

[0108] Get the cost matrix after column subtraction All the elements not covered by the line in , find the minimum element from all the elements not covered by the line, subtract the minimum element from all the elements not covered by the line, and add the minimum element to the elements covered by two lines to get the adjusted cost matrix. Through adjustment, more positions in the cost matrix after column subtraction become zero.

[0109] The zero-element covering operation is performed again on the adjusted cost matrix until the number of lines covering all zero elements is equal to the order of the adjusted cost matrix, and the final adjusted cost matrix is ​​obtained. The final adjusted cost matrix is ​​recorded as .

[0110] Step 5: Calculate the sum of all elements in each final adjusted cost matrix to obtain the minimum sum of all elements and the cost matrix corresponding to the minimum sum of all elements.

[0111] Calculate each final adjusted cost matrix The sum of all elements in the final adjusted cost matrix The sum of all elements in is denoted by , compare and get the smallest sum of all elements, thus getting the cost matrix corresponding to the smallest sum of all elements. The smallest sum of all elements represents the lowest matching cost between targets in different sensors.

[0112] Step 6: Based on the cost type corresponding to the cost matrix corresponding to the minimum sum of all elements, perform inter-frame matching on the first fusion target sets at different times to obtain the final fusion target set.

[0113] When the cost type corresponding to the cost matrix corresponding to the minimum sum of all elements is size difference, inter-frame matching is performed on the first fusion target sets at different times based on the size difference, and the same applies to other cost types.

[0114] The inter-frame matching process is: The cost matrix is ​​constructed based on the first fusion target of the next frame and the first fusion target of the previous set frame. If the cost matrix and If they are equal, first obtain the minimum value in the cost matrix, determine the target corresponding to the minimum value as the same target, and remove the row and column where the minimum value is located; continue to obtain the minimum value in the remaining values, determine the target corresponding to the minimum value obtained in the remaining values ​​as the same target, and remove the row and column where the minimum value is located; and so on, until the inter-frame matching of all targets is completed; if the cost matrix and If they are not equal, then for the cost matrix = For the part, inter-frame matching is performed according to the above method. For the newly added row or column, the target corresponding to the row or column is directly recorded, that is, the target of the row or column does not participate in the matching and is directly used as the matched object in the next inter-frame matching. When it is determined to be the same target, the first fusion comprehensive information of the first fusion target of the next frame is inherited.

[0115] Similarly, all first fusion targets in the first fusion target set at different times within the specified time range are matched between frames to obtain the final fusion target set.

[0116] The final fusion target set includes one or more final fusion targets, each of which has corresponding final fusion position information, final fusion shape information and final fusion posture information. The final fusion position information, final fusion shape information and final fusion posture information are uniformly defined as final fusion comprehensive information.

[0117] The obstacle target is finally determined by finally fusing the information.

[0118] Through this step, it is ensured that the same target at different times is correctly identified and associated, and the obstacle target is finally determined.

[0119] It is worth noting that when there are multiple targets in the same direction and one target can completely block another target in the same direction that is farther away from the IGV, priority is given to the target that is closer to the IGV; when the position of the blocked target changes to an unblocked position or is partially blocked, attention is added to this target.

[0120] After completing the calculations of S101 to S104, the final target comprehensive information can be continuously optimized based on the real-time changing operation information of the unmanned vehicle and the real-time changing sensor data.

[0121] In another embodiment of the present invention, S103 can be executed first, that is: first obtain the three-dimensional point cloud data of the laser radar, and use the Euclidean clustering method to extract the second comprehensive information of the second target from the three-dimensional point cloud data, and then execute S101 and S102, that is: then obtain the image data of the camera, use the target detection algorithm to extract the first feature information of the first target from the image data, and then use the camera projection model to extract the first position information of the first target from the image data, and uniformly define the first feature information of the first target and the first position information of the first target as the first comprehensive information of the first target.

[0122] The embodiment of the present invention also provides a system for an unmanned vehicle to perceive an obstacle target, such as Figure 3 As shown, including: The first feature information acquisition module 201 is used to acquire image data from a camera and extract a first target set and first feature information of the first target from the image data using a target detection algorithm.

[0123] The first comprehensive information acquisition module 202 is used to extract the first position information of the first target from the image data by using the camera projection model, and to define the first feature information and the first position information as the first comprehensive information of the first target.

[0124] The second comprehensive information acquisition module 203 is used to acquire the three-dimensional point cloud data of the laser radar, and extract the second target set and the second comprehensive information of the second target from the three-dimensional point cloud data using the Euclidean clustering method.

[0125] The obstacle target determination module 204 is used to obtain the operation information of the unmanned vehicle and determine the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information.

[0126] It should be noted here that the above-mentioned first feature information acquisition module 201, first comprehensive information acquisition module 202, second comprehensive information acquisition module 203 and obstacle target determination module 204 correspond to steps S101 to S104 in the method embodiment of the unmanned vehicle perceiving obstacle targets, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiments.

[0127] An embodiment of the present invention also provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed on a computer, the computer executes the method for an unmanned vehicle to perceive an obstacle target as described in the above method embodiment.

[0128] like Figure 4 As shown, an embodiment of the present invention further provides an electronic device, including: a processor 301, the processor 301 is coupled to a memory 302, and the processor 301 is used to read and execute a computer program stored in the memory 302 to implement the method for an unmanned vehicle to perceive an obstacle target as described in the above method embodiment.

[0129] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for an unmanned vehicle to perceive an obstacle target, characterized in that: include: Acquire image data from a camera, and extract a first target set and first feature information of the first target from the image data using a target detection algorithm; Extracting first position information of a first target from the image data using a camera projection model, and defining the first feature information and the first position information as first comprehensive information of the first target; Acquire three-dimensional point cloud data of the laser radar, and extract a second target set and second comprehensive information of the second target from the three-dimensional point cloud data using a Euclidean clustering method; The operation information of the unmanned vehicle is obtained, and the obstacle target is determined based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information.

2. The method according to claim 1, characterized in that The step of acquiring image data from a camera and extracting a first target set and first feature information of the first target from the image data using a target detection algorithm includes: Acquire image data from the camera, detect the image data using a pre-trained model in the target detection algorithm, and obtain a first original target set and first feature information of the first original target; Based on the type and confidence in the first feature information and using the confidence threshold in the target detection algorithm, the first original target set is filtered to obtain the first target set and the first feature information of the first target.

3. The method according to claim 1, characterized in that The method of acquiring three-dimensional point cloud data of a laser radar and extracting a second target set and second comprehensive information of the second target from the three-dimensional point cloud data using a Euclidean clustering method includes: Acquire the three-dimensional point cloud data of the laser radar, and define the three-dimensional point cloud data of the laser radar as original three-dimensional point cloud data; Preprocessing the original three-dimensional point cloud data to obtain preprocessed three-dimensional point cloud data; Based on the spatial distribution characteristics of the three-dimensional point cloud data, the pre-processed three-dimensional point cloud data is divided into a plurality of regions according to the spatial position; Clustering the preprocessed three-dimensional point cloud data in each of the regions using the Euclidean clustering method to obtain a clustering result; The clustering result is optimized to obtain the second target set and second comprehensive information of the second target.

4. The method according to claim 3, characterized in that The pre-processed three-dimensional point cloud data in each region is clustered using the Euclidean clustering method to obtain a clustering result, including: Calculate the variance of all data points in each dimension of the preprocessed three-dimensional point cloud data, and select the dimension corresponding to the largest variance as the segmentation dimension of the current node; Calculate the median of all the data points in the split dimension, and use the median as the split point of the current node; Constructing a hyperplane, wherein the hyperplane is perpendicular to the segmentation dimension and passes through the segmentation point; Recursively construct left and right subtrees based on the segmentation dimension, the segmentation point and the hyperplane.

5. The method according to claim 3, characterized in that: Optimizing the clustering result to obtain the second target set and second comprehensive information of the second target, including: Optimizing the clustering result to obtain an optimized clustering result; In the optimized clustering result, one cluster is one second target, and one or more second targets constitute the second target set; Determine the second position information of the second target by the three-dimensional coordinates of the center point of each cluster; Extracting bounding box coordinates from the three-dimensional point cloud data in each of the clusters, determining a bounding box through the bounding box coordinates, and thereby determining second shape information of a second object through the bounding box; Determine the orientation of the second target by using the normal vector of the three-dimensional point cloud data in each of the clusters, thereby determining second posture information of the second target; The second position information of the second object, the second shape information of the second object, and the second posture information of the second object are collectively defined as the second comprehensive information of the second object.

6. The method according to claim 1, characterized in that The obtaining of the operation information of the unmanned vehicle and determining the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information includes: Aligning the time frames of acquisition of the operation information of the unmanned vehicle, the first comprehensive information, and the second comprehensive information; Fusing the first target set and the second target set to obtain a first fused target set and first fused comprehensive information of the first fused targets; The first fusion comprehensive information is updated by using the Kalman filtering method to obtain updated first fusion comprehensive information.

7. The method according to claim 6, characterized in that The obtaining of the operation information of the unmanned vehicle, and determining the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information, further includes: Based on the updated first fusion comprehensive information, the first fusion target set is matched between frames by using the Hungarian algorithm to obtain a final fusion target set and final fusion information of the final fusion target, wherein the final fusion information includes final fusion position information, final fusion shape information and final fusion posture information; The obstacle target is determined through the final fusion information.

8. A system for unmanned vehicle to perceive obstacle targets, characterized in that: include: A first feature information acquisition module, used to acquire image data from a camera, and extract a first target set and first feature information of the first target from the image data using a target detection algorithm; A first comprehensive information acquisition module, configured to extract first position information of a first target from the image data by using a camera projection model, and to define the first feature information and the first position information as first comprehensive information of the first target; A second comprehensive information acquisition module is used to acquire three-dimensional point cloud data of the laser radar, and extract a second target set and second comprehensive information of the second target from the three-dimensional point cloud data using a Euclidean clustering method; The obstacle target determination module is used to obtain the operation information of the unmanned vehicle and determine the obstacle target based on the operation information of the unmanned vehicle, the first comprehensive information and the second comprehensive information.

9. A computer-readable storage medium, characterized in that: A program or instruction is stored, and when the program or instruction is executed on a computer, the computer is caused to execute the method for an unmanned vehicle to perceive an obstacle target as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: comprising a processor coupled to a memory; The processor is used to read and execute the computer program stored in the memory to implement the method for an unmanned vehicle to perceive an obstacle target as described in any one of claims 1-7.

Images