Target object identification method and device, system integrated with laser radar and medium

Abstract

The invention relates to the technical field of computer vision, and discloses a target object recognition method and device, a laser radar integrated system and a medium, which are used for remarkably reducing the calculation time of point cloud processing on the premise of ensuring the recognition precision. The target object identification method comprises the following steps: acquiring initial point cloud data and an original image; performing image recognition on the original image to obtain an image recognition result; determining a target constraint parameter based on the contour feature of each object in the image recognition result, wherein the target constraint parameter at least comprises the grid scale of the target image; distributing the initial point cloud data to each target grid of the target image according to a preset mapping matrix, an image recognition result and a target constraint parameter to obtain target point cloud data; and performing real-time target identification on the target point cloud data based on the target constraint parameter to obtain a target object identification result.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target object recognition method, device, system and medium integrated with laser radar. Background Art

[0002] Computer vision can realize automatic environmental perception to achieve tasks such as target object recognition, improve recognition accuracy, and greatly reduce the workload of personnel. While ensuring the recognition accuracy of target objects, the recognition speed is improved. Real-time target object recognition is crucial for specific application scenarios such as power safety management and control, anti-external damage monitoring, intelligent monitoring, autonomous driving, human-computer interaction, and intelligent manufacturing.

[0003] The relevant technologies for target object recognition mainly include image-based semantic recognition and point cloud-based semantic recognition. The former can be suitable for real-time recognition on low-computing power platforms, but cannot accurately represent the true three-dimensional distribution of the target. Point cloud data can provide rich environmental information. Point cloud clustering is usually used to extract point clouds related to the identified object and determine the three-dimensional spatial distribution of the target from them. However, it is time-consuming, does not meet real-time requirements, and cannot be applied to low-computing power platforms.

[0004] However, point cloud clustering usually requires traversing each point to achieve object recognition. As the amount of collected point cloud data increases dramatically, the time required for point cloud mapping extraction and clustering processing increases significantly, and the computing resources consumed are huge. Especially in multi-target object recognition scenarios, it is often difficult to achieve real-time recognition of multiple target objects on low-computing power platforms. Summary of the invention

[0005] The present application provides a target object recognition method, device, system and medium with integrated laser radar, which are used for target object recognition methods based on images, which lack three-dimensional information and lead to recognition limitations, while target object recognition methods based on point clouds provide rich three-dimensional information but are difficult to achieve real-time recognition of multiple targets on low computing power platforms due to computationally intensive processing and high resource consumption.

[0006] The first aspect of the present application provides a target object recognition method, comprising: acquiring initial point cloud data and an original image; performing image recognition on the original image to obtain an image recognition result; determining target constraint parameters based on contour features of each object in the image recognition result, the target constraint parameters at least including a grid scale of the target image; allocating the initial point cloud data to each target grid of the target image according to a preset mapping matrix, the image recognition result and the target constraint parameters to obtain target point cloud data; performing real-time target recognition on the target point cloud data based on the target constraint parameters to obtain a target object recognition result.

[0007] The second aspect of the present application provides a target object recognition device, including: an acquisition module, used to acquire initial point cloud data and an original image. A first recognition module, used to perform image recognition on the original image to obtain an image recognition result. A determination module, used to determine target constraint parameters based on the contour features of each object in the image recognition result, wherein the target constraint parameters include at least the grid scale of the target image; an allocation module, used to allocate the initial point cloud data to each target grid of the target image according to a preset mapping matrix, the image recognition result and the target constraint parameters to obtain target point cloud data; a second recognition module, used to perform real-time target recognition on the target point cloud data based on the target constraint parameters to obtain a target object recognition result.

[0008] The third aspect of the present application provides a system with an integrated laser radar, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the system with an integrated laser radar executes the above-mentioned target object recognition method.

[0009] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned target object recognition method.

[0010] In the technical solution provided by the present application, the target constraint parameters are determined by the contour features of each object in the image recognition results, and the constraint parameters are set for each object in a targeted manner, so that the subsequent point cloud data processing process is constrained by the key features of the target object; by introducing a new grid scale and other constraint parameters of the target image, the calculation time of point cloud processing can be significantly reduced while ensuring the recognition accuracy; the grid scale is dynamically updated according to the actual observation conditions of the camera, so that the processing algorithm is insensitive to changes in the order of magnitude of the point cloud, thereby effectively reducing the computational complexity; the optimized method takes into account the advantages of small computational complexity and fast speed of image data processing, as well as the advantage of rich three-dimensional spatial information of point cloud data, and can realize real-time spatial range recognition and extraction of multiple targets to meet application scenarios with high real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A schematic diagram of an embodiment of a target object recognition method in an embodiment of the present application;

[0012] Figure 2 This is another schematic diagram of a target object identification method in an embodiment of the present application;

[0013] Figure 3 This is a schematic diagram of an embodiment of a target object recognition device in an embodiment of the present application;

[0014] Figure 4 This is a schematic diagram of another embodiment of the target object recognition device in the embodiment of the present application;

[0015] Figure 5 This is a schematic diagram of an embodiment of a system integrating a laser radar in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The present application provides a target object recognition method, device, system and medium integrated with laser radar, which are used to optimize the target object extraction process, shorten the operation time while ensuring the accuracy of the extraction results.

[0017] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0018] For ease of understanding, the specific process of the embodiment of the present application is described below. Figure 1 , an embodiment of the target object recognition method in the embodiment of the present application includes:

[0019] 101. Obtain initial point cloud data and original images.

[0020] It is understandable that the executor of the present application may be a target object identification device, or a security monitoring system or a terminal such as a server, a vehicle or a robot, and the specifics are not limited here.

[0021] The above-mentioned execution entity is equipped with an image acquisition device (such as a camera, etc.) and a laser radar, or a communication connection is used to obtain the initial point cloud data and original images in the same scene to execute the target object recognition method provided by this application. This application can be applied to various target object recognition scenarios with high real-time requirements, especially real-time object recognition on low computing power platforms. The embodiment of this application is illustrated by taking the terminal as the execution entity as an example.

[0022] Specifically, for the same target scene, cameras and lidar are used to collect multi-frame data in real time to obtain initial point cloud data and original images.

[0023] It can be understood that due to the different data acquisition frequencies of the camera and the lidar, the acquisition devices are time synchronized. When the camera acquisition frequency is higher than the radar acquisition frequency, each frame of the lidar point cloud frame is matched with the camera image frame that is closest in matching time and whose time error is less than a certain preset threshold based on the time of the image frame and the radar point cloud frame.

[0024] 102. Perform image recognition on the original image to obtain an image recognition result.

[0025] This embodiment takes into account the real-time processing requirements and the computing power limitations of the computing platform. The original image is identified and segmented through image semantic recognition, and the possible positions of the target objects are preliminarily divided. The image recognition results of this embodiment include the object contours corresponding to each object.

[0026] The target scene includes at least one object to be identified, which may be a person, a car, an obstacle, etc. This embodiment does not limit the number and type of objects in the target scene, that is, the image recognition result includes the contours of each object and the labels of each object.

[0027] It is understandable that the semantic recognition method of an image can be a feature-based target object recognition method: such as SIFT, SURF, etc., a target object recognition method based on image segmentation: such as threshold segmentation, region growing, edge detection, etc., and other applicable image recognition algorithms can also be used, and this embodiment does not make specific restrictions.

[0028] 103. Determine target constraint parameters based on contour features of each object in the image recognition result, where the target constraint parameters at least include a grid scale of the target image.

[0029] In this embodiment, the contour feature is used to indicate the contour characteristics corresponding to each object extracted based on the image processing technology, which can be the contour area, main direction of the contour, contour shape and other information on the original image; the target constraint parameter is used to indicate the parameters for constraining and optimizing the target object recognition process of the point cloud data, and the target constraint parameter includes at least the grid scale of the target image. Through the contour features of each object, this embodiment can set more precise constraints in the target recognition process for each object, that is, the target constraint parameter, so that the subsequent point cloud data processing process is constrained by the key features of the target object, thereby improving the recognition speed, accuracy and robustness, and can effectively reduce unnecessary data processing.

[0030] Optionally, target constraint parameters are determined based on contour features of each object in the image recognition result, and the target constraint parameters include at least a grid scale of the target image, including: determining the grid scale corresponding to each object in the target image based on the contour features corresponding to each object in the image recognition result. In this embodiment, a new grid scale is determined by the contour features of each object in the image recognition result in the original image to generate a target image, so as to ensure the distribution uniformity and data validity of the point cloud after downsampling at the current observation angle of the camera.

[0031] In this embodiment, the grid scale of the target image refers to the grid size of the target image after rasterization processing under the camera angle. The grid scale of the target image is a two-dimensional constraint parameter, and the grid scales corresponding to the objects in the target image may be the same or different, that is, the target image of the present application may have multiple grids of the same scale, or may have multi-scale grids. The grid scale of the present application can effectively constrain the point cloud in both downsampling and point cloud clustering.

[0032] Specifically, the grid scale corresponding to each object in the target image is determined based on the contour features corresponding to each object in the image recognition result, including: determining the contour area corresponding to each object according to the contour corresponding to each object in the image recognition result; and determining the grid scale corresponding to each object in the target image by taking the ratio of the contour area corresponding to each object to the corresponding preset area threshold.

[0033] In this embodiment, the target constraint parameters may include two-dimensional constraint parameters such as grid scale, downsampling distance threshold, maximum point number threshold, and allowed point number range, that is, parameters for constraining downsampling and point cloud clustering of point cloud data from the two-dimensional image angle and camera angle, and may also include three-dimensional constraint parameters such as direction constraint parameters and angle threshold, that is, parameters for constraining point cloud clustering mainly from the three-dimensional point cloud perspective.

[0034] Optionally, after determining the grid scale corresponding to each object in the target image based on the contour features corresponding to each object in the image recognition result, the method further includes: obtaining the default value of the constraint parameter set for each object based on the label of each object in the image recognition result. This embodiment further determines other constraint parameters in combination with the object label, and further constrains the point cloud together with the grid scale, significantly reducing the processing amount of subsequent point cloud clustering and improving processing accuracy.

[0035] Among them, the default value of the constraint parameter is used to indicate the empirical value of the constraint parameter associated with the object label, that is, the parameter for constraining and optimizing each pre-labeled object in the target object recognition process of the point cloud data. For example, if the object label is a person, the default value of the downsampling distance corresponding to the pre-stored "person" label can be directly obtained, which is 0.1 meters, that is, the target constraint parameter also includes the downsampling distance threshold.

[0036] It should be further explained that, other constraint parameters except the grid scale can be fixed empirical values, that is, obtained by association through object labels, or can be further calculated and determined based on image recognition results and initial point cloud data to improve the adaptability of other constraint parameters to actual scenes. This embodiment does not impose specific restrictions.

[0037] The above-mentioned downsampling distance threshold is used to indicate the minimum distance between any two points in each grid. For example, the minimum distance between any two points in each two-dimensional grid in the target image is mapped to the target image and then two-dimensional angle downsampling is performed through the downsampling distance threshold to reduce the consumed computing resources. The downsampling distance threshold corresponding to each object may be the same or different. Downsampling through the downsampling distance threshold can maintain a certain distance between any two points in each grid, thereby reducing the number of points in each grid and significantly reducing the processing volume of subsequent clustering. In order to increase the downsampling rate, the downsampling distance threshold of the present application is mainly through

[0038] The above-mentioned direction constraint parameters are used to indicate the main orientation or posture information of the object. The direction constraint parameters corresponding to each object may be the same or different. By limiting the point cloud clustering direction through the direction constraint parameters, the search space in the clustering process can be reduced, thereby improving the clustering speed. For example, when identifying pedestrians, the main orientation of the point cloud in the standing direction of the pedestrian is roughly perpendicular to the ground. When clustering the point cloud, the direction constraint parameters are used to focus on the point cloud data in this direction, and the point clouds in other directions are constrained, which can improve the point cloud clustering speed.

[0039] The above maximum point threshold is used to indicate the maximum number of points allowed in each grid, for example, the maximum number of points allowed in each two-dimensional grid in the target image. The maximum point threshold corresponding to each object may be the same or different. By setting the maximum point threshold for downsampling, the number of points in each grid can be limited, reducing the processing volume of subsequent clustering.

[0040] The above-mentioned allowed point range is used to indicate the allowed point range in each grid. For example, the allowed point range in each two-dimensional grid in the target image may be the same or different for each object. By setting the allowed point range for downsampling, an allowed range can be provided for the points in each grid, so that the point cloud data in the grid will neither be too dense, resulting in high computational complexity, nor too sparse, resulting in loss of important information.

[0041] The above-mentioned angle threshold is used to indicate the maximum angle difference between adjacent points in each grid, for example, the angle difference between two adjacent points compared to a reference direction (such as the depth direction of the acquisition device), or the maximum angle difference formed by adjacent points on both sides of any point. By setting the angle threshold and retaining only those point pairs whose angle difference is lower than the set threshold, the number of points that need further analysis can be effectively reduced, thereby accelerating the subsequent clustering process.

[0042] It can be understood that the target constraint parameters can be constrained from the perspective of two-dimensional images or from the perspective of three-dimensional point clouds. For example, under appropriate circumstances, the above-mentioned angle threshold can also constrain the point cloud data from a two-dimensional perspective, and the maximum point count threshold and the allowed point count range can also constrain the point cloud data from a three-dimensional perspective. The present application can execute the target object recognition method of the present application through one or more of the above-mentioned target constraint parameters, which can achieve downsampling of point cloud data, reduce processing volume, and improve the speed and accuracy of point cloud clustering.

[0043] 104. Allocate the initial point cloud data to each target grid of the target image according to the preset mapping matrix, the image recognition result and the target constraint parameter to obtain the target point cloud data.

[0044] In practical applications, there may be multiple object contours in the image recognition result of the original image. The pixel values ​​outside the contours are set to 0 to generate a target image of the same size as the original image; based on the mapping matrix, the point cloud coordinates corresponding to each point in the initial point cloud data are converted to the camera coordinate system to obtain the two-dimensional pixel coordinates corresponding to each point; the initial point cloud data is traversed according to the contours corresponding to each object in the image recognition result and the two-dimensional pixel coordinates corresponding to each point, the point cloud data corresponding to each contour is determined, and the target point cloud data is obtained. In this embodiment, the point cloud data corresponding to each contour area is obtained, and irrelevant point cloud data is eliminated at the same time to reduce the amount of calculation for subsequent point cloud processing; and downsampling is performed based on the constraints of the grid scale to achieve uniform distribution of point clouds based on the camera observation angle.

[0045] The above-mentioned target image can be a labeled image, that is, multiple contours are marked with serial numbers starting from 1, and the pixel value in each contour is its corresponding serial number. That is, a labeled image usually refers to an image with specific marks or annotations added to the original image, such as watermarks, text annotations, legends, highlighted areas, etc., in order to highlight certain features or information.

[0046] In this embodiment, the mapping matrix is ​​used to indicate the conversion relationship between the coordinate systems of the camera and the lidar, that is, the coordinate conversion relationship between the camera coordinate system and the point cloud coordinate system. For the successfully matched point cloud frames and image frames, according to the pre-performed camera radar calibration, the coordinate conversion relationship between the image coordinate system and the point cloud coordinate system can be established through the corresponding parameter matrix.

[0047] Optionally, after traversing the initial point cloud data according to the contours corresponding to each object and the two-dimensional pixel coordinates corresponding to each point in the image recognition result to determine the point cloud data corresponding to each contour, it also includes: downsampling the point cloud data corresponding to each contour according to the default values ​​of each constraint parameter to obtain the target point cloud data. Based on the new grid scale, this embodiment further constrains the point cloud according to the default values ​​of the constraint parameters preset by the object label to reduce the processing volume of subsequent point cloud clustering. By jointly optimizing the recognition results of the target object of the point cloud data through the grid scale and other constraint parameters, the processing volume can be reduced while ensuring the recognition accuracy, and real-time recognition can be achieved.

[0048] Optionally, a fixed empirical value can be set as the downsampling distance threshold, and the point cloud data corresponding to each contour is downsampled according to the default values ​​of each constraint parameter to obtain the target point cloud data, including: downsampling the point cloud data in each grid according to the downsampling distance threshold to obtain the target point cloud data. After the point cloud downsampling is implemented, the time consumption of subsequent target recognition is greatly reduced. At the same time, since the above-mentioned downsampling process relies on a fixed threshold, for situations where the point cloud density is different due to factors such as distance, the point cloud density is finally equivalent after downsampling, and the time consumption is equivalent.

[0049] 105. Based on the target constraint parameters, real-time target recognition is performed on the target point cloud data to obtain the target object recognition result.

[0050] Specifically, the target point cloud data is clustered based on the grid constraint of the target image to obtain the clustering results corresponding to each object; and the target object recognition result is output according to the clustering results corresponding to each object. In this embodiment, the point cloud clustering constraint is performed through the grid constraint so that the point cloud clustering is traversed only within the range corresponding to the object, which speeds up the clustering process and avoids unnecessary traversal calculations.

[0051] In this embodiment, the target object recognition result includes at least one object, and various target object recognition algorithms can be used to perform target recognition on the clustering results. The clustering algorithm in this embodiment can be a hierarchical clustering algorithm, a Dbscan clustering algorithm, or other applicable clustering algorithms. By adjusting the appropriate clustering distance threshold, the target point cloud data can be grouped into different classes or clusters; the target recognition algorithm in this embodiment can be a target object recognition algorithm based on a deep learning algorithm, such as a three-dimensional convolutional neural network, VoteNet, or other applicable target object recognition algorithms, which are not specifically limited in this embodiment.

[0052] In an embodiment of the present application, the target constraint parameters are determined by the contour features of each object in the image recognition results, and the constraint parameters can be set for each object in a targeted manner, so that the subsequent point cloud data processing process is constrained by the key features of the target object; by introducing a new grid scale and other constraint parameters of the target image, the calculation time of point cloud processing can be significantly reduced while ensuring the recognition accuracy; the grid scale is dynamically updated according to the actual observation conditions of the camera, so that the processing algorithm is insensitive to changes in the order of magnitude of the point cloud, thereby effectively reducing the computational complexity; the optimized method takes into account the advantages of small image data processing computational complexity and high speed, as well as the advantage of rich three-dimensional spatial information of point cloud data, and can realize real-time spatial range recognition and extraction of multiple targets, meet application scenarios with high real-time requirements, and can also realize real-time multi-target recognition on low computing power platforms.

[0053] See also Figure 2 Another embodiment of the target object recognition method in the embodiment of the present application includes:

[0054] 201. Obtain initial point cloud data and original image.

[0055] 202. Perform image recognition on the original image to obtain an image recognition result.

[0056] Steps 201 and 202 may be performed with reference to steps 101 and 102 and will not be described in detail here.

[0057] 203. Determine target constraint parameters based on the image recognition result and the initial point cloud data, where the target constraint parameters at least include a grid scale and a downsampling distance threshold corresponding to each object.

[0058] Taking the target constraint parameters as the grid scale and the downsampling distance threshold as an example, specifically, the target area ratio corresponding to each object is determined based on the image recognition result, and the grid scale corresponding to each object in the target image is determined based on the target area ratio corresponding to each object; the point unit volume corresponding to each object is determined based on the image recognition result and the initial point cloud data, and the downsampling distance threshold corresponding to each object is determined according to the point unit volume.

[0059] In this embodiment, the contour feature can be represented by the ratio of the area of ​​the rectangular region of the image where each object is located in the image recognition result and the area threshold corresponding to the object, that is, the grid scale corresponding to each object in the target image is determined according to the target area ratio corresponding to each object, wherein the target area ratio is the ratio of the contour area corresponding to each object to the corresponding preset area threshold.

[0060] The above-mentioned determination of the target area ratio corresponding to each object based on the image recognition result includes: determining the contour area corresponding to each object according to the contour corresponding to each object in the image recognition result; determining the grid scale corresponding to each object in the target image by the ratio of the contour area corresponding to each object and the corresponding preset area threshold.

[0061] Optionally, the above-mentioned determining the grid scale corresponding to each object in the target image based on the target area ratio corresponding to each object includes: judging whether the target area ratio corresponding to each object is greater than 1; if the target area ratio corresponding to any object is greater than 1, determining the target area ratio as the grid scale of any object; if the target area ratio corresponding to any object is less than or equal to 1, setting the grid scale of any object to 1. This embodiment sets the lower limit of the grid scale to 1, avoids the generation of extremely fine grids due to too small objects, reduces the consumption of computing resources, improves processing efficiency, and shortens processing time.

[0062] The above-mentioned determination of the point unit volume corresponding to each object based on the image recognition result and the initial point cloud data includes: determining the point cloud space volume corresponding to each object according to each contour and the initial point cloud data; determining the point unit volume corresponding to each object according to the point cloud space volume corresponding to each object and the corresponding preset point count threshold.

[0063] Optionally, the above-mentioned determination of the downsampling distance threshold corresponding to each object based on the point unit volume includes: determining the side length of the point unit volume corresponding to each object as the downsampling distance threshold corresponding to each object. For example, if the object is a person, assuming that the point cloud space volume corresponding to the person is 1 cubic meter, and the preset upper limit of the number of points is 500, then the point unit volume of each point is 0.002 cubic meters, and calculated by the cube, the side length of the point unit volume corresponding to the person is 0.126 meters, and 0.126 meters will be used as the downsampling distance threshold within the grid when identifying people.

[0064] Optionally, the determining of the downsampling distance threshold corresponding to each object according to the point unit volume includes: determining the diagonal length of the point unit volume corresponding to each object as the downsampling distance threshold corresponding to each object.

[0065] 204. Determine candidate point cloud data corresponding to each grid in the target image based on the mapping matrix, the image recognition result, the initial point cloud data, and the grid scale corresponding to each object.

[0066] Based on the mapping matrix, the three-dimensional point cloud coordinates corresponding to each point in the initial point cloud data are transformed into the camera coordinate system to obtain the two-dimensional pixel coordinates corresponding to each point; the multiple grids corresponding to each object are determined according to the contours corresponding to each object and the grid scales corresponding to each object in the image recognition results; based on the two-dimensional pixel coordinates corresponding to each point, each point is assigned to the multiple grids corresponding to each object to obtain the candidate point cloud data corresponding to each grid.

[0067] Specifically, the initial pixel coordinates corresponding to each point are traversed, the coordinates of the upper left corner of the rectangle are subtracted, and divided by the corresponding grid scale to obtain the new coordinates under the new scale two-dimensional grid, that is, the two-dimensional pixel coordinates corresponding to each point in the target object, so as to facilitate the subsequent calculation of the Euclidean distance when downsampling the downsampling distance threshold of each object.

[0068] It is understandable that the target image may have a multi-scale grid, and the grid scale corresponding to each object in the target image is the grid scale determined in step 103. For example, the grid scale corresponding to object 1 is 502*502, and the grid scale corresponding to object 2 is 1000*1000. In this embodiment, the initial point cloud data is preliminarily processed through the target image to reduce the number of point clouds required for subsequent target recognition and improve the accuracy. The initial point cloud data is processed from a two-dimensional perspective through the target image, which can save the resources required for calculation and improve the processing speed compared to direct processing from a three-dimensional perspective. Furthermore, the processing of the initial point cloud data is constrained within the grid, which greatly reduces the traversal calculation in the process. At the same time, due to the constraints of the two-dimensional grid, a uniform distribution of point clouds based on the camera observation perspective can be achieved. When there are objects to be identified with large size differences in the target scene, multi-scale grids can downsample the point cloud according to the grid scales of different objects. This can improve the downsampling accuracy of the initial point clouds of different objects. For small and delicate objects, smaller grids are used to retain more details, while for large and simple-structured objects, larger grids are used to reduce processing time. In this way, a better balance can be found between speed and accuracy.

[0069] 205. Downsample the candidate point cloud data corresponding to each grid according to the downsampling distance threshold corresponding to each object to obtain target point cloud data.

[0070] Specifically, one target point is randomly selected from the candidate point cloud data corresponding to each grid and assigned to the target grid; when there is no grid point in the target grid, the target point is retained as a grid point; when there is a grid point in the target grid, it is determined whether the distance between the target point and the grid point is greater than the downsampling distance threshold; if so, the target point is deleted, otherwise the target point is retained as a grid point; the candidate point cloud data corresponding to each grid is traversed to obtain the target point cloud data.

[0071] This embodiment downsamples by downsampling the distance threshold so that a certain distance is maintained between any two points in each grid. Repeating the above steps for all newly added points is equivalent to downsampling the point cloud based on the first Euclidean distance in the grid. Each point in the grid is the cluster center of each cluster retained, which can significantly reduce the processing volume of subsequent clustering. And because the Euclidean distance traversal comparison between each point and other points is constrained within the grid, the traversal calculation in the process is greatly reduced. At the same time, due to the constraints of the two-dimensional grid, a uniform distribution of point clouds based on the camera observation perspective is achieved.

[0072] Optionally, after the above-mentioned downsampling through the downsampling distance threshold, it also includes: determining whether the remaining number of points in each grid is greater than the maximum point number threshold; if so, uniformly downsampling the remaining points in each grid; if not, retaining all the remaining points in each grid so that the remaining number of points in each grid is less than the maximum point number threshold.

[0073] Optionally, after the above-mentioned downsampling by the downsampling distance threshold, it also includes: determining whether the number of remaining points in each grid is within a preset allowable point range; if so, retaining all the remaining points in each grid; if not, adjusting the remaining points in each grid so that the number of remaining points in each grid is within the preset allowable point range.

[0074] If the above is not the case, the remaining points of each grid are adjusted so that the number of remaining points in each grid is within the preset allowable point range: if the remaining points of each grid are too few, all points in the grid are deleted; if the remaining points of each grid are too many, the remaining points in each grid are uniformly downsampled.

[0075] Optionally, after the above-mentioned downsampling by downsampling distance threshold, it also includes: downsampling the remaining points in each grid based on the angle threshold, so that only point pairs with angle differences below the set threshold are retained in each grid, which can effectively reduce the number of points that need further analysis and accelerate the subsequent clustering process.

[0076] 206. Cluster the areas to be extracted corresponding to each object in the target point cloud data based on the target constraint parameter and the preset clustering distance threshold to obtain clustering results corresponding to each object. The areas to be extracted are all target grids corresponding to each object.

[0077] Specifically, based on the grid constraint of the target image and a preset clustering distance threshold, the to-be-extracted regions corresponding to each object are clustered respectively to obtain a clustering result corresponding to each object.

[0078] Optionally, based on the grid constraint, direction constraint parameter and preset clustering distance threshold of the target image, the regions to be extracted corresponding to each object are clustered respectively to obtain clustering results corresponding to each object. This embodiment can reduce the search space in the clustering process by clustering the point cloud based on the two-dimensional grid constraint and direction constraint parameter, thereby improving the clustering speed.

[0079] The above cluster distance threshold is used to indicate the maximum distance between any point in each class and the cluster center. Similarly, the cluster distance threshold can also be expressed by Euclidean distance or Manhattan distance, or other methods, such as expressing the cluster distance threshold by the second Euclidean distance.

[0080] It can be understood that both the first Euclidean distance and the second Euclidean distance indicate the distance between two points, and the first Euclidean distance is used to downsample the points in each grid, and the second Euclidean distance is used to cluster the point clouds in each area to be extracted.

[0081] In this embodiment, "class" is used to indicate a point cloud set in which the distance between each point and the cluster center is within the cluster distance threshold range. The clustering results corresponding to each object include at least one class, each class includes multiple target points, and the distance between any target point in each class and the cluster center is less than the cluster distance threshold.

[0082] 207. Output the target object recognition result according to the clustering result of each area to be extracted.

[0083] Specifically, the class with the largest number of points in the clustering results of each area to be extracted is determined as the target class; if the number of points corresponding to the target class is greater than a preset point threshold, the target class is determined as the target object; the clustering results of each area to be extracted are traversed to obtain the target object recognition result.

[0084] For ease of understanding, an example is provided. Under the downsampling of the first Euclidean distance (such as set to 10 cm), the number of point clouds in the area to be extracted corresponding to each target object is controlled within a smaller range. Under the constraint of the two-dimensional grid, the remaining point clouds are clustered into multiple categories through the second Euclidean distance threshold (such as set to 20 cm), and the category with the largest number of points is selected. If the number of points in this category is greater than a certain point threshold (such as set to 10 points), it can be determined as the final target object point cloud, and then the three-dimensional spatial range of the target object is calculated. The point threshold set in this embodiment can eliminate the situation of misidentification.

[0085] In the embodiment of the present application, the target constraint parameters are determined by the contour features of each object in the image recognition results, and the constraint parameters can be set for each object in a targeted manner, so that the subsequent point cloud data processing process is constrained by the key features of the target object; by introducing a new grid scale and downsampling distance threshold of the target image, the calculation time of point cloud processing can be significantly reduced while ensuring the recognition accuracy; the grid scale and downsampling distance threshold are dynamically updated according to the actual observation conditions of the camera, so that the processing algorithm is insensitive to changes in the order of magnitude of the point cloud, thereby effectively reducing the computational complexity; the optimized method takes into account the advantages of small image data processing computational complexity and fast speed, as well as the advantage of rich three-dimensional spatial information of point cloud data, and can realize real-time spatial range recognition and extraction of multiple targets, meet application scenarios with high real-time requirements, and can also realize real-time multi-target recognition on low computing power platforms.

[0086] The above describes the target object recognition method in the embodiment of the present application. The following describes the target object recognition device in the embodiment of the present application. Figure 3 In the embodiment of the present application, an embodiment of the target object recognition device includes:

[0087] The acquisition module 301 is used to acquire initial point cloud data and original images.

[0088] The first recognition module 302 is used to perform image recognition on the original image to obtain an image recognition result.

[0089] A determination module 303 is used to determine target constraint parameters based on the contour features of each object in the image recognition result, where the target constraint parameters at least include a grid scale of the target image;

[0090] An allocation module 304 is used to allocate the initial point cloud data to each target grid of the target image according to a preset mapping matrix, image recognition results and target constraint parameters to obtain target point cloud data;

[0091] The second recognition module 305 is used to perform real-time target recognition on the target point cloud data based on the target constraint parameters to obtain a target object recognition result.

[0092] In the embodiment of the present application, the target constraint parameters are determined by the contour features of each object in the image recognition results, and the constraint parameters can be set for each object in a targeted manner, so that the subsequent point cloud data processing process is constrained by the key features of the target object; by introducing a new grid scale and other constraint parameters of the target image, the calculation time of point cloud processing can be significantly reduced while ensuring the recognition accuracy; the grid scale is dynamically updated according to the actual observation conditions of the camera, so that the processing algorithm is insensitive to changes in the order of magnitude of the point cloud, thereby effectively reducing the computational complexity; the optimized method takes into account the advantages of small image data processing calculation amount and fast speed, as well as the advantage of rich three-dimensional spatial information of point cloud data, and can realize real-time spatial range recognition and extraction of multiple targets to meet application scenarios with high real-time requirements.

[0093] See also Figure 4 Another embodiment of the target object recognition device in the embodiment of the present application includes:

[0094] The acquisition module 301 is used to acquire initial point cloud data and original images.

[0095] The first recognition module 302 is used to perform image recognition on the original image to obtain an image recognition result.

[0096] A determination module 303 is used to determine target constraint parameters based on the contour features of each object in the image recognition result, where the target constraint parameters at least include a grid scale of the target image;

[0097] An allocation module 304 is used to allocate the initial point cloud data to each target grid of the target image according to a preset mapping matrix, image recognition results and target constraint parameters to obtain target point cloud data;

[0098] The second recognition module 305 is used to perform real-time target recognition on the target point cloud data based on the target constraint parameters to obtain a target object recognition result.

[0099] Optionally, the determining module 303 includes:

[0100] A first determining unit 3031 is used to determine a target area ratio corresponding to each object based on the image recognition result, and determine a grid scale corresponding to each object in the target image based on the target area ratio corresponding to each object;

[0101] The second determining unit 3032 is used to determine the point unit volume corresponding to each object based on the image recognition result and the initial point cloud data, and determine the downsampling distance threshold corresponding to each object according to the point unit volume.

[0102] Optionally, the first determination unit 3031 is specifically configured to: determine the ratio of the preset area threshold corresponding to the contour area corresponding to each object according to the contour corresponding to each object in the image recognition result, and obtain the target area ratio corresponding to each object;

[0103] If the target area ratio corresponding to any object is greater than 1, the target area ratio is determined as the grid scale of any object;

[0104] If the target area ratio corresponding to any object is less than or equal to 1, the grid scale of any object is set to 1.

[0105] Optionally, the second determining unit 3032 is specifically configured to: determine the point cloud space volume corresponding to each object according to each contour and the initial point cloud data;

[0106] Determine the point unit volume corresponding to each object according to the point cloud space volume corresponding to each object and the corresponding preset point number threshold;

[0107] The side length of the point unit volume corresponding to each object is determined as the downsampling distance threshold corresponding to each object.

[0108] Optionally, the allocation module 304 includes:

[0109] An allocating unit 3041 is used to determine candidate point cloud data corresponding to each grid in the target image based on the mapping matrix, the image recognition result, the initial point cloud data, and the grid scale corresponding to each object;

[0110] The downsampling unit 3042 is used to downsample the candidate point cloud data corresponding to each grid according to the downsampling distance threshold corresponding to each object to obtain target point cloud data.

[0111] Optionally, the downsampling unit 3042 is specifically used to: select any one target point from the candidate point cloud data corresponding to each grid and assign it to the target grid;

[0112] When a grid point does not exist in the target grid, the target point is retained as a grid point;

[0113] When a grid point already exists in the target grid, determine whether the distance between the target point and the grid point is greater than the downsampling distance threshold;

[0114] If yes, delete the target point, otherwise keep the target point as a grid point;

[0115] Traverse the candidate point cloud data corresponding to each grid to obtain the target point cloud data.

[0116] Optionally, the second recognition module 305 is specifically used to: cluster the to-be-extracted areas corresponding to each object in the target point cloud data respectively based on the target constraint parameter and a preset clustering distance threshold, and obtain the clustering results corresponding to each object, where the to-be-extracted areas are all target grids corresponding to each object;

[0117] The target object recognition results are output according to the clustering results corresponding to each object.

[0118] In the embodiments of the present application, the target constraint parameters are determined by the contour features of each object in the image recognition results, and the constraint parameters can be set for each object in a targeted manner, so that the subsequent point cloud data processing process is constrained by the key features of the target object; by introducing a new grid scale and downsampling distance threshold of the target image, the calculation time of the point cloud processing can be significantly reduced while ensuring the recognition accuracy; the grid scale and downsampling distance threshold are dynamically updated according to the actual observation conditions of the camera, so that the processing algorithm is insensitive to changes in the order of magnitude of the point cloud, thereby effectively reducing the calculation complexity; the optimized method takes into account the advantages of small image data processing calculation amount and fast speed, as well as the advantage of rich three-dimensional spatial information of point cloud data, and can realize real-time spatial range recognition and extraction of multiple targets to meet application scenarios with high real-time requirements.

[0119] See also Figure 5 As shown, the system integrated with the laser radar includes an image acquisition device 500, a laser radar 501, a processor 502 and a memory 503. The image acquisition device 500 is used to acquire original images; the laser radar 501 is used to acquire initial point cloud data, and the memory 503 stores machine executable instructions that can be executed by the processor 502. The processor 502 executes the machine executable instructions to implement the above-mentioned target object recognition method.

[0120] Further, Figure 5 The shown integrated laser radar system further includes a bus 504 and a communication interface 505 , and the processor 502 , the communication interface 505 and the memory 503 are connected via the bus 504 .

[0121] The memory 503 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), for example, at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 505 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 504 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0122] The processor 502 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 502. The above processor 502 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 503 , and the processor 502 reads the information in the memory 503 and completes the method steps of the above-mentioned embodiment in combination with its hardware.

[0123] The present application also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are executed on a computer, the computer executes the steps of the target object recognition method.

[0124] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0125] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0126] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target object recognition method, characterized in that: The target object recognition method comprises: Obtain initial point cloud data and original images; Performing image recognition on the original image to obtain an image recognition result; Determining target constraint parameters based on contour features of each object in the image recognition result, wherein the target constraint parameters at least include a grid scale of the target image; Allocating the initial point cloud data to each target grid of the target image according to a preset mapping matrix, the image recognition result and the target constraint parameter to obtain target point cloud data; Based on the target constraint parameters, real-time target recognition is performed on the target point cloud data to obtain a target object recognition result.

2. The target object recognition method according to claim 1, characterized in that: The target constraint parameters also include a downsampling distance threshold corresponding to each object; The target constraint parameters are determined based on the contour features of each object in the image recognition result, and the target constraint parameters at least include a grid scale of the target image, including: Determine the target area ratio corresponding to each object based on the image recognition result, and determine the grid scale corresponding to each object in the target image based on the target area ratio corresponding to each object; The point unit volume corresponding to each object is determined based on the image recognition result and the initial point cloud data, and the downsampling distance threshold corresponding to each object is determined according to the point unit volume.

3. The target object recognition method according to claim 2, characterized in that: The determining of the target area ratio corresponding to each object based on the image recognition result, and determining the grid scale corresponding to each object in the target image based on the target area ratio corresponding to each object, includes: Determine the ratio of the preset area threshold corresponding to the contour area of ​​each object according to the contour corresponding to each object in the image recognition result, and obtain the target area ratio corresponding to each object; If the target area ratio corresponding to any one of the objects is greater than 1, the target area ratio is determined as the grid scale of the any one of the objects; If the target area ratio corresponding to any object is less than or equal to 1, the grid scale of the any object is set to 1.

4. The target object recognition method according to claim 2, characterized in that: The step of determining the point unit volume corresponding to each object based on the image recognition result and the initial point cloud data, and determining the downsampling distance threshold corresponding to each object according to the point unit volume, includes: Determine the point cloud space volume corresponding to each object according to each of the contours and the initial point cloud data; Determine the point unit volume corresponding to each object according to the point cloud space volume corresponding to each object and the corresponding preset point number threshold; The side length of the point unit volume corresponding to each object is determined as the downsampling distance threshold corresponding to each object.

5. The target object recognition method according to claim 1, characterized in that: The target constraint parameters include the grid scale and downsampling distance threshold corresponding to each object; The step of allocating the initial point cloud data to each target grid of the target image according to the preset mapping matrix, the image recognition result and the target constraint parameter to obtain the target point cloud data includes: Determine candidate point cloud data corresponding to each grid in the target image based on the mapping matrix, the image recognition result, the initial point cloud data, and the grid scale corresponding to each object; The candidate point cloud data corresponding to each grid is downsampled according to the downsampling distance threshold corresponding to each object to obtain the target point cloud data.

6. The target object recognition method according to claim 5, characterized in that: The step of downsampling the candidate point cloud data corresponding to each grid according to the downsampling distance threshold corresponding to each object to obtain the target point cloud data includes: Select a target point from the candidate point cloud data corresponding to each grid and assign it to the target grid; When the grid point does not exist in the target grid, retaining the target point as a grid point; When a grid point already exists in the target grid, determining whether the distance between the target point and the grid point is greater than the downsampling distance threshold; If yes, delete the target point, otherwise keep the target point as a grid point; Traverse the candidate point cloud data corresponding to each grid to obtain the target point cloud data.

7. The target object recognition method according to any one of claims 1 to 5, characterized in that: The performing real-time target recognition on the target point cloud data based on the target constraint parameters to obtain a target object recognition result includes: Based on the target constraint parameters and a preset clustering distance threshold, the regions to be extracted corresponding to each object in the target point cloud data are clustered respectively to obtain clustering results corresponding to each object, wherein the regions to be extracted are all target grids corresponding to each object; The target object recognition results are output according to the clustering results corresponding to each object.

8. A target object recognition device, characterized in that: The target object recognition device comprises: The acquisition module is used to obtain initial point cloud data and original images. The first recognition module is used to perform image recognition on the original image to obtain an image recognition result. A determination module, configured to determine target constraint parameters based on contour features of each object in the image recognition result, wherein the target constraint parameters at least include a grid scale of a target image; an allocation module, configured to allocate the initial point cloud data to each target grid of the target image according to a preset mapping matrix, the image recognition result and the target constraint parameter, so as to obtain target point cloud data; The second recognition module is used to perform real-time target recognition on the target point cloud data based on the target constraint parameters to obtain a target object recognition result.

9. A system integrating laser radar, characterized in that: The laser radar integrated system comprises: an image acquisition device, a laser radar, a memory and at least one processor, wherein instructions are stored in the memory; The image acquisition device is used to acquire original images; The laser radar is used to collect initial point cloud data; The at least one processor calls the instructions in the memory so that the system integrated with the laser radar executes the target object recognition method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instruction is read and executed, the target object recognition method according to any one of claims 1 to 7 is executed.

Images