Obstacle detection and tracking method, device, computer equipment and storage medium

Abstract

The present invention discloses an obstacle detection and tracking method, device, equipment and medium, including: calibrating camera internal parameters and external parameters between laser radar and camera, and collecting image data and point cloud data through camera and laser radar respectively; using 3D target detection algorithm to perform target detection on point cloud data, obtain 3D detection frame and target category, and remove the point cloud inside the 3D detection frame to extract the 3D detection frame of the obstacle; using 2D instance segmentation algorithm to perform instance segmentation on image data, obtain 2D detection result, 2D detection result includes 2D detection frame, segmentation polygon and target category; mapping point cloud data to image data according to external parameters, correspondingly matching 3D detection frame and 2D detection result; matching according to the detection results of the previous and next two frames, using Kalman filter to filter the comprehensive prediction value and observation value results, and determining the obstacle tracking result. The present invention improves the accuracy of obstacle detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an obstacle detection and tracking method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer vision technology, more and more intelligent robots, unmanned driving and other technologies are appearing in people's lives. Obstacle detection, as one of the important research directions in the field of computer vision, has broad application prospects in the fields of autonomous driving, intelligent transportation, industrial automation, and intelligent security. With the continuous development of artificial intelligence technology, obstacle detection systems play a vital role in realizing intelligent perception and autonomous decision-making automation systems. An important indicator for measuring the degree of intelligence of robots or drones is the ability to process unknown environmental information. In practical applications, the ability of robots or unmanned vehicles to judge obstacles is an important indicator, and semantic information and depth information are crucial and indispensable in scene obstacle analysis.

[0003] In the existing method, obstacle recognition is mainly performed through the semantic segmentation technology of deep convolutional neural network. However, the inventors found in the process of realizing the present invention that in dynamic and complex scenes, the obstacle determination process is affected by many factors such as the large number of object types, easy occlusion, large light changes, etc., and the existing method is not accurate enough for obstacle detection. Summary of the invention

[0004] Embodiments of the present invention provide an obstacle detection and tracking method, apparatus, computer equipment, and storage medium to improve the accuracy of obstacle detection.

[0005] In order to solve the above technical problems, the present application provides an obstacle detection and tracking method, including:

[0006] Calibrate the camera's intrinsic parameters and the extrinsic parameters between the LiDAR and the camera, and collect image data and point cloud data through the camera and LiDAR respectively;

[0007] Using a 3D target detection algorithm to perform target detection on the point cloud data, obtain a 3D detection frame and a target category, and remove the point cloud inside the 3D detection frame to extract a 3D detection frame of the obstacle;

[0008] Perform instance segmentation on the image data using a 2D instance segmentation algorithm to obtain a 2D detection result, wherein the 2D detection result includes a 2D detection box, a segmentation polygon, and a target category;

[0009] Mapping the point cloud data to the image data according to an external reference, and correspondingly matching the 3D detection bounding box and the 2D detection result;

[0010] The obstacle tracking result is determined by matching the detection results of the previous and next frames and filtering the comprehensive prediction value and observation value using Kalman filtering.

[0011] Optionally, the adopting a 3D target detection algorithm to perform target detection on the point cloud data to obtain a 3D detection box and a target category includes:

[0012] Acquire sample data, where the sample data is a data set collected and marked by a laser radar;

[0013] Performing cluster analysis on the borders in the sample data to obtain a priori borders;

[0014] Perform data enhancement on the prior frame, retrain using pre-trained weights, save the best weights after multiple iterations, and obtain a trained 3D object detection model;

[0015] The trained 3D target detection model is used to perform target detection on the point cloud data to obtain a 3D detection box and a target category.

[0016] Optionally, removing the point cloud inside the 3D detection frame to extract a 3D detection frame of the obstacle includes:

[0017] Performing data preprocessing on the point cloud data to obtain preprocessed data;

[0018] Performing plane fitting on the preprocessed data to obtain fitting data;

[0019] Removing the ground point cloud and the point cloud inside the 3D detection frame from the fitting data to obtain target data;

[0020] The target data is subjected to Euclidean clustering to remove discrete points and extract the 3D detection bounding box.

[0021] Optionally, performing plane fitting on the preprocessed data to obtain fitting data comprises:

[0022] Dividing the point cloud data into a plurality of sectors having regularly spaced radial and azimuthal angles according to a center point;

[0023] Each sector performs regional ground plane fitting and then merges some ground points;

[0024] Through ground likelihood estimation, it is judged whether the divided ground belongs to the actual ground, and fitting data including ground point cloud data and non-ground point cloud data are obtained.

[0025] Optionally, performing Euclidean clustering on the target data, removing discrete points, and extracting the 3D detection border includes:

[0026] Set different radius thresholds for clustering to obtain point cloud clusters of target contours, and merge different point cloud clusters;

[0027] After clustering, the outline of the obstacle is obtained, and convex polygon fitting is used to calculate the oblique rectangle based on the vertices of the convex polygon to obtain the center of the obstacle frame;

[0028] The length, width, height, center and orientation of the obstacle frame are adjusted to obtain the 3D detection frame.

[0029] Optionally, mapping the point cloud data to the image data according to the external reference, and correspondingly matching the 3D detection bounding box and the 2D detection result includes:

[0030] The point cloud data of the ground is removed and the point cloud in the opposite direction of the camera view is removed, and the points of the clipped point cloud data are projected onto the image data plane according to the external parameters between the laser radar and the camera;

[0031] intercepting point cloud data according to the segmented polygon to obtain intercepted data;

[0032] Get the 3D detection frame with the largest distribution value of the intercepted data as the target detection frame that matches successfully.

[0033] Optionally, matching the detection results of the two previous and next frames, filtering the comprehensive prediction value and the observation value result by using Kalman filtering, and determining the obstacle tracking result includes:

[0034] According to the detection results of the previous and next two frames, the initial position and speed of the obstacle are determined, and the state vector and covariance matrix are constructed;

[0035] Based on the state vector and the covariance matrix, predict a next state of the obstacle as a prediction result;

[0036] Using sensors to measure the position of obstacles as a measurement result;

[0037] Using a Kalman filter formula, the measurement result is combined with the prediction result to obtain an updated state estimate and covariance matrix;

[0038] The step of predicting the next state of the obstacle based on the state vector and the covariance matrix is ​​returned and continued to be executed as the prediction result to obtain a continuous trajectory tracking result of the obstacle.

[0039] In order to solve the above technical problems, the embodiment of the present application further provides an obstacle detection and tracking device, comprising:

[0040] The data acquisition module is used to calibrate the camera's internal parameters and the external parameters between the laser radar and the camera, and to collect image data and point cloud data through the camera and laser radar respectively;

[0041] A 3D detection module is used to perform target detection on the point cloud data using a 3D target detection algorithm to obtain a 3D detection frame and a target category, and remove the point cloud inside the 3D detection frame to extract a 3D detection frame of the obstacle;

[0042] A 2D detection module, used to perform instance segmentation on the image data using a 2D instance segmentation algorithm to obtain a 2D detection result, wherein the 2D detection result includes a 2D detection box, a segmentation polygon and a target category;

[0043] A mapping and matching module, used to map the point cloud data to the image data according to external parameters, and to correspond to the 3D detection frame and the 2D detection result;

[0044] The dynamic tracking module is used to match the detection results of the previous and next frames, and use Kalman filtering to filter the comprehensive prediction value and observation value results to determine the obstacle tracking result.

[0045] Optionally, the 3D detection module includes:

[0046] A sample collection unit, used to obtain sample data, where the sample data is a data set collected and marked by the laser radar;

[0047] A sample clustering unit, used for performing cluster analysis on the borders in the sample data to obtain a priori borders;

[0048] An iterative training unit is used to perform data enhancement on the prior frame, retrain using pre-trained weights, save the best weights after multiple iterations, and obtain a trained 3D object detection model;

[0049] The target detection unit is used to perform target detection on the point cloud data using the trained 3D target detection model to obtain a 3D detection frame and a target category.

[0050] Optionally, the 3D detection module further includes:

[0051] A preprocessing unit, used for performing data preprocessing on the point cloud data to obtain preprocessed data;

[0052] A plane fitting unit, used for performing plane fitting on the preprocessed data to obtain fitting data;

[0053] A data screening unit, used to remove the ground point cloud and the point cloud inside the 3D detection box from the fitting data to obtain target data;

[0054] The data clustering unit is used to perform Euclidean clustering on the target data, remove discrete points, and extract the 3D detection border.

[0055] Optionally, the plane fitting unit includes:

[0056] A slicing subunit, for dividing the point cloud data into a plurality of sectors with regularly spaced radial and azimuthal angles according to a center point;

[0057] A fitting subunit, used to fit the regional ground plane for each sector and then merge some ground points;

[0058] The determination subunit is used to determine whether the divided ground belongs to the actual ground through ground likelihood estimation, and obtain fitting data including ground point cloud data and non-ground point cloud data.

[0059] Optionally, the data clustering unit includes:

[0060] The clustering subunit is used to set different radius thresholds for clustering, obtain point cloud clusters of the target contour, and merge different point cloud clusters;

[0061] The center calculation subunit is used to obtain the outline of the obstacle after clustering is completed, and to calculate the oblique rectangle according to the vertices of the convex polygon by using convex polygon fitting to obtain the center of the obstacle frame;

[0062] The frame generation subunit is used to adjust the length, width, height, center and orientation of the obstacle frame to obtain the 3D detection frame.

[0063] Optionally, the mapping matching module includes:

[0064] A projection unit, used to remove the point cloud data of the ground and the point cloud in the opposite direction of the camera's viewing angle, and project the points of the clipped point cloud data onto the image data plane according to the external parameters between the laser radar and the camera;

[0065] An interception unit, used for intercepting point cloud data according to the segmentation polygon to obtain intercepted data;

[0066] The matching unit is used to obtain the 3D detection frame with the largest distribution value of the intercepted data as the target detection frame that is successfully matched.

[0067] Optionally, the dynamic tracking module includes:

[0068] A construction unit, used to determine the initial position and speed of the obstacle according to the detection results of the previous and next two frames, and to construct a state vector and a covariance matrix;

[0069] A prediction unit, configured to predict a next state of the obstacle based on the state vector and the covariance matrix as a prediction result;

[0070] a measuring unit, for measuring the position of the obstacle using a sensor as a measurement result;

[0071] An updating unit, configured to combine the measurement result with the prediction result using a Kalman filter formula to obtain an updated state estimate and a covariance matrix;

[0072] The tracking unit is used to return the next state of the obstacle predicted based on the state vector and the covariance matrix, and continue to execute the step of predicting the result to obtain a continuous trajectory tracking result of the obstacle.

[0073] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above obstacle detection and tracking method when executing the computer program.

[0074] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above obstacle detection and tracking method are implemented.

[0075] The obstacle detection and tracking method, device, computer equipment and storage medium provided by the embodiment of the present invention calibrates the camera internal parameters and the external parameters between the laser radar and the camera, and collects image data and point cloud data through the camera and the laser radar respectively; uses a 3D target detection algorithm to perform target detection on the point cloud data to obtain a 3D detection frame and target category, and removes the point cloud inside the 3D detection frame to extract the 3D detection frame of the obstacle; uses a 2D instance segmentation algorithm to perform instance segmentation on the image data to obtain a 2D detection result, and the 2D detection result includes a 2D detection frame, a segmentation polygon and a target category; maps the point cloud data to the image data according to the external parameters, and matches the 3D detection frame and the 2D detection result accordingly; matches the detection results of the previous and next two frames, and uses the Kalman filter to filter the comprehensive prediction value and the observation value results to determine the obstacle tracking result. It realizes the collection of different types of data through multiple sensors, and performs 2D and 3D fusion detection, thereby improving the accuracy of obstacle detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0077] Figure 1 is a flow chart of an embodiment of the obstacle detection and tracking method of the present application;

[0078] Figure 2 is a schematic structural diagram of an embodiment of an obstacle detection and tracking device according to the present application;

[0079] Figure 3 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0081] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] See also Figure 1 , Figure 1 An obstacle detection and tracking method provided by an embodiment of the present invention is shown. The method is applied in Figure 1 The server in is used as an example to illustrate the details as follows:

[0084] S201: Calibrate the camera's internal parameters and the external parameters between the laser radar and the camera, and collect image data and point cloud data through the camera and the laser radar respectively.

[0085] Specifically, the camera intrinsic parameter calibration method is as follows: a homemade checkerboard is used to calibrate using the calibration tool of the ROS system to obtain the camera's intrinsic parameter matrix K and distortion coefficient D; the camera and lidar extrinsic parameter calibration method is as follows: a homemade calibration plate is used as the target, the position of the calibration plate is observed by the camera and radar, and the error of the target in each coordinate system is detected to complete the estimation of the relative posture between the coordinate systems.

[0086] Among them, the ROS system (Robot Operating System) is a computer operating system architecture designed specifically for robot software development. It is an open source meta-level operating system (post-operating system) that provides services similar to operating systems, including hardware abstract description, underlying driver management, execution of shared functions, inter-program message passing, program distribution package management, etc. It also provides some tools and libraries for acquiring, building, writing and executing multi-machine fusion programs.

[0087] Point cloud data refers to a set of vectors in a three-dimensional coordinate system. Scanning data is recorded in the form of points, each of which contains three-dimensional coordinates. Some may contain color information (RGB) or reflection intensity information (Intensity). In addition to geometric positions, point cloud data also has color information. Color information is usually obtained by acquiring color images through a camera, and then the color information (RGB) of the pixels at the corresponding positions is assigned to the corresponding points in the point cloud. The acquisition of intensity information is the echo intensity collected by the laser scanner receiving device. This intensity information is related to the surface material, roughness, incident angle direction of the target, as well as the emission energy of the instrument and the laser wavelength.

[0088] S202: Use a 3D target detection algorithm to perform target detection on the point cloud data to obtain a 3D detection frame and a target category, and remove the point cloud inside the 3D detection frame to extract the 3D detection frame of the obstacle.

[0089] In a specific optional implementation, in step S202, a 3D target detection algorithm is used to perform target detection on the point cloud data, and the 3D detection box and target category obtained include:

[0090] Obtain sample data, which is a data set collected and marked by the laser radar;

[0091] Perform cluster analysis on the bounding boxes in the sample data to obtain the prior box;

[0092] Perform data enhancement on the prior box, retrain using pre-trained weights, save the best weights after multiple iterations, and obtain a trained 3D object detection model;

[0093] Use the trained 3D target detection model to perform target detection on point cloud data to obtain 3D detection boxes and target categories.

[0094] In a specific optional implementation, in step S202, removing the point cloud inside the 3D detection frame to extract the 3D detection frame of the obstacle includes:

[0095] Perform data preprocessing on the point cloud data to obtain preprocessed data;

[0096] Perform plane fitting on the preprocessed data to obtain fitting data;

[0097] Remove the ground point cloud and the point cloud inside the 3D detection box from the fitting data to obtain the target data;

[0098] Perform Euclidean clustering on the target data, remove discrete points, and extract the 3D detection bounding box.

[0099] In a specific optional implementation manner, performing plane fitting on the preprocessed data to obtain the fitting data includes:

[0100] Divide the point cloud data into multiple sectors with regularly spaced radial and azimuthal angles according to the center point;

[0101] Each sector performs regional ground plane fitting and then merges some ground points;

[0102] Through ground likelihood estimation, it is judged whether the divided ground belongs to the actual ground, and fitting data including ground point cloud data and non-ground point cloud data are obtained.

[0103] In a specific optional implementation, performing Euclidean clustering on the target data, removing discrete points, and extracting a 3D detection border includes:

[0104] Set different radius thresholds for clustering to obtain point cloud clusters of target contours, and merge different point cloud clusters;

[0105] After clustering, the outline of the obstacle is obtained, and convex polygon fitting is used to calculate the oblique rectangle based on the vertices of the convex polygon to obtain the center of the obstacle frame;

[0106] Adjust the length, width, height, center and orientation of the obstacle frame to obtain a 3D detection frame.

[0107] S203: Perform instance segmentation on the image data using a 2D instance segmentation algorithm to obtain a 2D detection result, which includes a 2D detection box, a segmentation polygon, and a target category.

[0108] Specifically, a data set is created by marking the data collected by the camera used.

[0109] Perform data enhancement, use the official pre-trained weights for retraining, save the best weights after multiple iterations, obtain the trained segmentation model, and then use the trained segmentation model to perform instance segmentation on the image data.

[0110] The segmentation polygon is a polygon composed of lines and is used to segment the image. The 2D detection box is the entire outer frame outline of the object in the 2D plane, and the target category is the category of the obstacle.

[0111] S204: Map the point cloud data to the image data according to the external reference, and correspond to the matching of the 3D detection bounding box and the 2D detection result.

[0112] In a specific optional implementation, in step S204, the point cloud data is mapped to the image data according to the external reference, and the corresponding matching of the 3D detection frame and the 2D detection result includes:

[0113] Remove the point cloud data from the ground and the point cloud in the opposite direction of the camera's view, and project the points of the clipped point cloud data onto the image data plane based on the external parameters between the lidar and the camera;

[0114] Intercepting point cloud data according to the segmentation polygon to obtain intercepted data;

[0115] Get the 3D detection frame with the largest distribution value of the intercepted data as the target detection frame that matches successfully.

[0116] S205: Match the detection results of the previous and next two frames, use Kalman filtering to filter the comprehensive prediction value and observation value results, and determine the obstacle tracking result.

[0117] In a specific optional implementation, in step S205, matching is performed based on the detection results of the previous and next two frames, and the Kalman filter is used to filter the comprehensive prediction value and the observation value result to determine the obstacle tracking result, including:

[0118] According to the detection results of the previous and next two frames, the initial position and speed of the obstacle are determined, and the state vector and covariance matrix are constructed;

[0119] Based on the state vector and the covariance matrix, the next state of the obstacle is predicted as a prediction result;

[0120] Using sensors to measure the position of obstacles as a measurement result;

[0121] Using the Kalman filter formula, the measurement results are combined with the prediction results to obtain the updated state estimate and covariance matrix;

[0122] Return the next state of the obstacle based on the state vector and covariance matrix, and continue to execute the steps as the prediction result to obtain the continuous trajectory tracking result of the obstacle.

[0123] In this embodiment, the camera internal parameters and the external parameters between the laser radar and the camera are calibrated, and the image data and point cloud data are collected by the camera and the laser radar respectively; the 3D target detection algorithm is used to perform target detection on the point cloud data to obtain the 3D detection frame and target category, and the point cloud inside the 3D detection frame is removed to extract the 3D detection frame of the obstacle; the image data is segmented by the 2D instance segmentation algorithm to obtain the 2D detection result, which includes the 2D detection frame, segmentation polygon and target category; the point cloud data is mapped to the image data according to the external parameters, and the 3D detection frame and the 2D detection result are matched accordingly; the detection results of the previous and next two frames are matched, and the Kalman filter is used to filter the comprehensive prediction value and the observation value results to determine the obstacle tracking result. It is realized that different types of data are collected through multiple sensors, and 2D and 3D fusion detection is performed, which improves the accuracy of obstacle detection.

[0124] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0125] Figure 2 FIG. 2 is a block diagram showing a principle of an obstacle detection and tracking device corresponding to the obstacle detection and tracking method in the above embodiment. Figure 2 As shown, the obstacle detection and tracking device includes a data acquisition module 31, a 3D detection module 32, a 2D detection module 33, a mapping matching module 34 and a dynamic tracking module 35. The functional modules are described in detail as follows:

[0126] The data acquisition module 31 is used to calibrate the camera internal parameters and the external parameters between the laser radar and the camera, and to collect image data and point cloud data through the camera and the laser radar respectively;

[0127] The 3D detection module 32 is used to perform target detection on the point cloud data using a 3D target detection algorithm to obtain a 3D detection frame and target category, and remove the point cloud inside the 3D detection frame to extract the 3D detection frame of the obstacle;

[0128] A 2D detection module 33 is used to perform instance segmentation on the image data using a 2D instance segmentation algorithm to obtain a 2D detection result, wherein the 2D detection result includes a 2D detection box, a segmentation polygon and a target category;

[0129] A mapping and matching module 34 is used to map the point cloud data to the image data according to the external parameters, and to correspond to the matching of the 3D detection bounding box and the 2D detection result;

[0130] The dynamic tracking module 35 is used to match the detection results of the previous and next two frames, and use Kalman filtering to filter the comprehensive prediction value and observation value results to determine the obstacle tracking result.

[0131] Optionally, the 3D detection module 32 includes:

[0132] A sample collection unit is used to obtain sample data, where the sample data is a data set collected and marked by the laser radar;

[0133] The sample clustering unit is used to perform cluster analysis on the bounding boxes in the sample data to obtain a priori boxes;

[0134] Iterative training unit, used to perform data enhancement on the prior box, retrain using pre-trained weights, save the best weights after multiple iterations, and obtain a trained 3D object detection model;

[0135] The target detection unit is used to perform target detection on point cloud data using the trained 3D target detection model to obtain the 3D detection box and target category.

[0136] Optionally, the 3D detection module 32 further includes:

[0137] A preprocessing unit, used for preprocessing the point cloud data to obtain preprocessed data;

[0138] A plane fitting unit is used to perform plane fitting on the preprocessed data to obtain fitting data;

[0139] A data screening unit is used to remove the ground point cloud and the point cloud inside the 3D detection box from the fitting data to obtain the target data;

[0140] The data clustering unit is used to perform Euclidean clustering on the target data, remove discrete points, and extract 3D detection bounding boxes.

[0141] Optionally, the plane fitting unit includes:

[0142] A slicing subunit, for dividing the point cloud data into a plurality of sectors with regularly spaced radial and azimuthal angles according to a center point;

[0143] A fitting subunit, used to fit the regional ground plane for each sector and then merge some ground points;

[0144] The determination subunit is used to determine whether the divided ground belongs to the actual ground through ground likelihood estimation, and obtain fitting data including ground point cloud data and non-ground point cloud data.

[0145] Optionally, the data clustering unit includes:

[0146] The clustering subunit is used to set different radius thresholds for clustering, obtain point cloud clusters of the target contour, and merge different point cloud clusters;

[0147] The center calculation subunit is used to obtain the outline of the obstacle after clustering is completed, and to calculate the oblique rectangle according to the vertices of the convex polygon by using convex polygon fitting to obtain the center of the obstacle frame;

[0148] The bounding box generation subunit is used to adjust the length, width, height, center and orientation of the obstacle box to obtain a 3D detection bounding box.

[0149] Optionally, the mapping matching module 34 includes:

[0150] A projection unit is used to remove the point cloud data of the ground and the point cloud in the opposite direction of the camera's view, and project the points of the clipped point cloud data onto the image data plane according to the external parameters between the laser radar and the camera;

[0151] An interception unit, used for intercepting point cloud data according to the segmentation polygon to obtain intercepted data;

[0152] The matching unit is used to obtain the 3D detection frame with the largest distribution value of the intercepted data as the target detection frame that is successfully matched.

[0153] Optionally, the dynamic tracking module 35 includes:

[0154] A construction unit, used to determine the initial position and speed of the obstacle according to the detection results of the previous and next two frames, and to construct a state vector and a covariance matrix;

[0155] A prediction unit, used for predicting the next state of the obstacle based on the state vector and the covariance matrix as a prediction result;

[0156] a measuring unit, for measuring the position of the obstacle using a sensor as a measurement result;

[0157] An updating unit, used to combine the measurement result with the prediction result using a Kalman filter formula to obtain an updated state estimate and a covariance matrix;

[0158] The tracking unit is used to return the next state of the obstacle based on the state vector and the covariance matrix, and continue to execute the step of the prediction result to obtain the continuous trajectory tracking result of the obstacle.

[0159] For the specific definition of the obstacle detection and tracking device, please refer to the definition of the obstacle detection and tracking method above, which will not be repeated here. Each module in the above obstacle detection and tracking device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0160] To solve the above technical problems, the present application also provides a computer device. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0161] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0162] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0163] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code of the obstacle detection and tracking method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0164] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code of the obstacle detection and tracking method.

[0165] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0166] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the obstacle detection and tracking method as described above.

[0167] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0168] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. An obstacle detection and tracking method, applied to a multi-sensor intelligent robot, the multi-sensor intelligent robot comprising a laser radar and a camera, characterized in that: The method comprises: Calibrate the camera's intrinsic parameters and the extrinsic parameters between the LiDAR and the camera, and collect image data and point cloud data through the camera and LiDAR respectively; Using a 3D target detection algorithm to perform target detection on the point cloud data, obtain a 3D detection frame and a target category, and remove the point cloud inside the 3D detection frame to extract a 3D detection frame of the obstacle; Perform instance segmentation on the image data using a 2D instance segmentation algorithm to obtain a 2D detection result, wherein the 2D detection result includes a 2D detection box, a segmentation polygon, and a target category; Mapping the point cloud data to the image data according to an external reference, and correspondingly matching the 3D detection bounding box and the 2D detection result; Match the detection results of the previous and next two frames, use Kalman filtering to filter the comprehensive prediction value and observation value results, and determine the obstacle tracking result; The step of mapping the point cloud data to the image data according to the external reference and correspondingly matching the 3D detection frame and the 2D detection result includes: The point cloud data of the ground is removed and the point cloud in the opposite direction of the camera view is removed, and the points of the clipped point cloud data are projected onto the image data plane according to the external parameters between the laser radar and the camera; intercepting point cloud data according to the segmented polygon to obtain intercepted data; Get the 3D detection frame with the largest distribution value of the intercepted data as the target detection frame that matches successfully.

2. The obstacle detection and tracking method according to claim 1, characterized in that: The 3D target detection algorithm is used to perform target detection on the point cloud data to obtain a 3D detection frame and target category, including: Acquire sample data, where the sample data is a data set collected and marked by a laser radar; Performing cluster analysis on the borders in the sample data to obtain a priori borders; Performing data enhancement on the prior frame, retraining using pre-trained weights, and obtaining a trained 3D object detection model after multiple iterations; The trained 3D target detection model is used to perform target detection on the point cloud data to obtain a 3D detection box and a target category.

3. The obstacle detection and tracking method according to claim 1, characterized in that: The step of removing the point cloud inside the 3D detection frame and extracting the 3D detection frame of the obstacle includes: Performing data preprocessing on the point cloud data to obtain preprocessed data; Performing plane fitting on the preprocessed data to obtain fitting data; Removing the ground point cloud and the point cloud inside the 3D detection frame from the fitting data to obtain target data; The target data is subjected to Euclidean clustering to remove discrete points and extract the 3D detection bounding box.

4. The obstacle detection and tracking method according to claim 3, characterized in that: The performing plane fitting on the preprocessed data to obtain fitting data comprises: Dividing the point cloud data into a plurality of sectors having regularly spaced radial and azimuthal angles according to a center point; Each sector performs regional ground plane fitting and then merges some ground points; Through ground likelihood estimation, it is judged whether the divided ground belongs to the actual ground, and fitting data including ground point cloud data and non-ground point cloud data are obtained.

5. The obstacle detection and tracking method according to claim 3, characterized in that: The performing Euclidean clustering on the target data, removing discrete points, and extracting the 3D detection border comprises: Set different radius thresholds for clustering to obtain point cloud clusters of target contours, and merge different point cloud clusters; After clustering, the outline of the obstacle is obtained, and convex polygon fitting is used to calculate the oblique rectangle based on the vertices of the convex polygon to obtain the center of the obstacle frame; The length, width, height, center and orientation of the obstacle frame are adjusted to obtain the 3D detection frame.

6. The obstacle detection and tracking method according to claim 1, characterized in that: The matching of the detection results of the two frames before and after, filtering the comprehensive prediction value and the observation value results using Kalman filtering, and determining the obstacle tracking result includes: According to the detection results of the previous and next two frames, the initial position and speed of the obstacle are determined, and the state vector and covariance matrix are constructed; Based on the state vector and the covariance matrix, predict a next state of the obstacle as a prediction result; Using sensors to measure the position of obstacles as a measurement result; Using a Kalman filter formula, the measurement result is combined with the prediction result to obtain an updated state estimate and covariance matrix; The step of predicting the next state of the obstacle based on the state vector and the covariance matrix is ​​returned and continued to be executed as the prediction result to obtain a continuous trajectory tracking result of the obstacle.

7. An obstacle detection and tracking device, characterized in that: include: The data acquisition module is used to calibrate the camera's internal parameters and the external parameters between the laser radar and the camera, and to collect image data and point cloud data through the camera and laser radar respectively; A 3D detection module is used to perform target detection on the point cloud data using a 3D target detection algorithm to obtain a 3D detection frame and a target category, and remove the point cloud inside the 3D detection frame to extract a 3D detection frame of the obstacle; A 2D detection module, used to perform instance segmentation on the image data using a 2D instance segmentation algorithm to obtain a 2D detection result, wherein the 2D detection result includes a 2D detection box, a segmentation polygon and a target category; A mapping and matching module, used to map the point cloud data to the image data according to external parameters, and to correspond to the 3D detection frame and the 2D detection result; The dynamic tracking module is used to match the detection results of the previous and next frames, and use Kalman filtering to filter the comprehensive prediction value and observation value results to determine the obstacle tracking result; Wherein, the mapping matching module includes: The point cloud data of the ground is removed and the point cloud in the opposite direction of the camera view is removed, and the points of the clipped point cloud data are projected onto the image data plane according to the external parameters between the laser radar and the camera; intercepting point cloud data according to the segmented polygon to obtain intercepted data; Get the 3D detection frame with the largest distribution value of the intercepted data as the target detection frame that matches successfully.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the obstacle detection and tracking method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the obstacle detection and tracking method according to any one of claims 1 to 6 is implemented.

Images