Target object detection method, device, equipment and storage medium

Abstract

This application provides a method, apparatus, device, and storage medium for detecting a target object, relating to the field of data processing technology. The method includes: acquiring an image set and acquiring point cloud data collected by a lidar; the image set includes multiple images; the point cloud data includes multiple points, each point having feature data; determining the image corresponding to a point from the image set; the image corresponding to a point is an image of the projection point of the point within a preset area, and the projection point is the position point where the point is projected onto the image; determining a first feature vector of the point based on the feature data of the point and the image corresponding to the point; the first feature vector represents the data after fusing multi-scale image features of the point and the image corresponding to the point; and performing detection processing on the first feature vector of each point to obtain information about the target object. The method of this application reduces the amount of data computation during subsequent detection, improves detection efficiency, and also improves detection accuracy.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a method, apparatus, device, and storage medium for detecting a target object. Background Technology

[0002] In autonomous driving perception systems, 3D object detection plays a crucial role. Its main task is to detect objects of interest in the application scenario and determine their location, category, and other information in order to achieve true autonomous driving.

[0003] Existing 3D object detection algorithms utilize either cameras or LiDAR for environmental perception. Cameras provide rich texture and color information, but they cannot directly acquire depth information and struggle to make accurate depth estimates of target objects, resulting in low detection accuracy. While LiDAR can provide accurate distance and position information, it lacks the semantic information acquired by cameras, leading to numerous false detections and unsatisfactory results. Current camera-and-LiDAR fusion schemes fail to effectively address the continuity issue between the two systems during data fusion, resulting in excessive computational load, long processing times, and insufficient detection accuracy.

[0004] Therefore, there is an urgent need for a new method for detecting target objects to effectively solve the continuity problem between different modal data of cameras and lidar, thereby improving detection accuracy and efficiency. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for detecting a target object, in order to solve the problem of how to improve the detection accuracy and efficiency of the target object.

[0006] Firstly, this application provides a method for detecting a target object, the method comprising:

[0007] Acquire an image set and acquire point cloud data collected by the LiDAR; wherein, the image set includes multiple images, which are a collection of images of the target scene captured by cameras at multiple different positions on the vehicle, and the target scene includes a target object; the point cloud data is the data obtained by the LiDAR on the vehicle scanning the target scene, and the point cloud data includes multiple points, and the points have feature data;

[0008] From the image set, determine the image corresponding to the point; wherein, the image corresponding to the point is the image of the projection point of the point within a preset area, and the projection point is the position point of the point projected onto the image;

[0009] Based on the feature data of the point and the image corresponding to the point, a first feature vector of the point is determined; wherein, the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point;

[0010] The first feature vector of each point is processed to obtain information about the target object.

[0011] Optionally, determining the first feature vector of the point based on the feature data of the point and the image corresponding to the point includes:

[0012] The image corresponding to the point is processed to obtain the second feature vector corresponding to the point; wherein, the second feature vector represents the mean of the multi-scale image features of the projection point corresponding to the point in the image corresponding to the point;

[0013] The feature data of the point and the second feature vector corresponding to the point are concatenated to obtain the first feature vector of the point.

[0014] Optionally, the image corresponding to the point is processed to obtain a second feature vector corresponding to the point, including:

[0015] The projection points of the point on the image corresponding to the point are normalized to obtain the normalized projection points corresponding to the point.

[0016] The image corresponding to the point is subjected to feature extraction processing to obtain the multi-scale image features of the image corresponding to the point;

[0017] The second feature vector corresponding to the point is determined based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point.

[0018] Optionally, based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point, a second feature vector corresponding to the point is determined, including:

[0019] Based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point, the image features of the normalized projection point corresponding to the point in the image corresponding to each point at different scales are determined by using an interpolation function.

[0020] The image features of the normalized projection point corresponding to the point at different scales are averaged to determine the averaged image features as the second feature vector corresponding to the point.

[0021] Optionally, determining the image corresponding to the point from the image set includes:

[0022] Based on the feature data of the point and the preset transformation matrix of the LiDAR to the camera corresponding to each image, obtain the two-dimensional coordinates of the projection point of the point on each image;

[0023] The image corresponding to the two-dimensional coordinates of the projection point within a preset range is determined as the image corresponding to the point.

[0024] Optionally, the first feature vector of each of the points is processed to obtain information about the target object, including:

[0025] The first feature vector of each point is input into a preset point cloud detection network model, and the information of the target object is output; wherein, the preset point cloud detection network model is a pre-trained model used to determine the information of the target object.

[0026] Optionally, the information of the target object includes one or more of the following: the bounding box of the target object, the category of the target object, the position of the target object, the size of the target object, the posture information of the target object, the movement speed of the target object, and the number of target objects.

[0027] Secondly, this application provides a target object detection device, the device comprising:

[0028] An acquisition unit is used to acquire an image set and point cloud data collected by a lidar; wherein, the image set includes multiple images, which are a collection of images of a target scene captured by cameras at multiple different positions on the vehicle, and the target scene includes a target object; the point cloud data is data obtained by the lidar on the vehicle scanning the target scene, and the point cloud data includes multiple points, and the points have feature data;

[0029] A matching unit is used to determine the image corresponding to the point from the image set; wherein the image corresponding to the point is an image of the projection point of the point within a preset area, and the projection point is the position point of the point projected onto the image;

[0030] The processing unit is configured to determine a first feature vector of the point based on the feature data of the point and the image corresponding to the point; wherein the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point.

[0031] The detection unit is used to perform detection processing on the first feature vector of each point to obtain information about the target object.

[0032] Optionally, the processing unit includes a processing module and a splicing module;

[0033] The processing module is used to process the image corresponding to the point to obtain a second feature vector corresponding to the point; wherein, the second feature vector represents the mean of the multi-scale image features of the projection point corresponding to the point in the image corresponding to the point;

[0034] The splicing module is used to splice the feature data of the point and the second feature vector corresponding to the point to obtain the first feature vector of the point.

[0035] Optionally, the processing module includes a processing submodule, a feature extraction module, and a determination module;

[0036] The processing submodule is used to normalize the projection points of the point on the image corresponding to the point, so as to obtain the normalized projection points corresponding to the point.

[0037] The feature extraction module performs feature extraction processing on the image corresponding to the point to obtain multi-scale image features of the image corresponding to the point.

[0038] The determining module is used to determine the second feature vector corresponding to the point based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point.

[0039] Optionally, the determining module includes a first determining module and a second determining module;

[0040] The first determining module is used to determine the image features of the normalized projection point corresponding to the point in the image corresponding to each point at each different scale, based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point, using an interpolation function.

[0041] The second determining module is used to perform mean processing on the image features of the normalized projection point corresponding to the point in the images corresponding to the point at different scales, and determine the mean-processed image features as the second feature vector corresponding to the point.

[0042] Optionally, the matching unit includes a calculation module and a determination module;

[0043] The calculation module is used to obtain the two-dimensional coordinates of the projection point of the point on each image based on the feature data of the point and the preset transformation matrix of the LiDAR to the camera corresponding to each image.

[0044] The determination module is used to determine the image corresponding to the two-dimensional coordinates of the projection point within a preset range, which is the image corresponding to the point.

[0045] Optionally, the detection unit is specifically used to input the first feature vector of each point into a preset point cloud detection network model and output the information of the target object; wherein, the preset point cloud detection network model is a pre-trained model used to determine the information of the target object.

[0046] Optionally, the information of the target object includes one or more of the following: the bounding box of the target object, the category of the target object, the position of the target object, the size of the target object, the posture information of the target object, the movement speed of the target object, and the number of target objects.

[0047] Thirdly, this application provides an electronic device, the electronic device comprising: a processor, and a memory communicatively connected to the processor;

[0048] The memory stores computer-executed instructions;

[0049] The processor executes computer execution instructions stored in the memory to implement the method as described in the preceding one.

[0050] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method as described in any of the preceding claims.

[0051] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, is used to implement the method as described in any of the foregoing.

[0052] This application provides a method, apparatus, device, and storage medium for detecting a target object. The method includes: acquiring an image set and acquiring point cloud data collected by a lidar; wherein the image set includes multiple images, which are a collection of images of a target scene captured by cameras at multiple different positions on a vehicle, and the target scene includes a target object; the point cloud data is data obtained by the lidar on the vehicle scanning the target scene, and the point cloud data includes multiple points, each point having feature data; determining the image corresponding to the point from the image set; wherein the image corresponding to the point is an image of the projection point of the point within a preset area, and the projection point is the position point of the point projected onto the image; determining a first feature vector of the point based on the feature data of the point and the image corresponding to the point; wherein the first feature vector represents the data after fusing multi-scale image features of the point and the image corresponding to the point; and performing detection processing on the first feature vector of each point to obtain information about the target object. On the one hand, the processed image data includes images acquired by cameras from multiple different locations, covering a wider range and containing more image information, thus enabling the extraction of more image features. Furthermore, multi-scale feature extraction processing is performed on the images in the image set, providing a more robust data foundation for subsequent target object detection. On the other hand, this application fuses the image set acquired by the camera and the point cloud data acquired by the LiDAR, using the fused data for target object detection. This pre-fusion method not only reduces the computational load during subsequent detection and improves detection efficiency but also stably and effectively solves the continuity problem between the two different modal data from the camera and LiDAR, thereby improving detection accuracy. Attached Figure Description

[0053] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0054] Figure 1 A flowchart illustrating a target object detection method provided in an embodiment of this application;

[0055] Figure 2 This application provides a schematic diagram of the image distribution acquired by a camera on a vehicle.

[0056] Figure 3 An architectural block diagram illustrating an implementation of the solution provided in this application;

[0057] Figure 4 This is a schematic diagram of the structure of a target object detection device provided in an embodiment of this application;

[0058] Figure 5 A schematic diagram of the structure of another target object detection device provided in the embodiments of this application;

[0059] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0060] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0061] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0062] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] In autonomous driving perception systems, 3D object detection plays a crucial role. Its main task is to detect objects of interest in the application scenario and determine their location, category, and other information in order to achieve true autonomous driving.

[0064] The main idea behind the current implementation scheme is to unfold the features of the 2D (two-dimensional) image acquired by the camera into a 3D (three-dimensional) space. This mainly includes a 2D-3D scheme based on depth prediction and a 3D-2D scheme based on lidar to camera projection sampling.

[0065] 2D-3D depth prediction-based schemes, such as BevDepth, use LSS (Laser Sensor) to estimate depth in the camera coordinate system, supervise the depth estimation with LiDAR (LiDAR) information, and then transform the obtained depth distribution in the camera coordinate system to the LiDAR coordinate system using a camera-to-liDAR transform. Finally, existing LiDAR point cloud 3D detection algorithms are used to complete 3D target detection. This scheme mainly utilizes image information acquired by the camera, resulting in a relatively large prediction error for object position and a relatively low detection accuracy (mAP).

[0066] 3D-2D projection sampling schemes based on LiDAR to cameras, such as DETR3D, obtain 3D feature queries by projecting 3D reference points onto the camera's image features. This implicitly estimates object depth, and then uses classification and regression heads to achieve 3D target detection. This approach primarily utilizes LiDAR information. However, due to the loss of image depth information caused by perspective projection, the depth estimation error is relatively large. Furthermore, it lacks processing of color and texture information, resulting in significant detection position deviations and numerous false positives.

[0067] Although both of the above solutions use cameras and LiDAR, two complementary sensors, cameras can provide rich texture and color information, while LiDAR can provide accurate distance and position information. However, they do not effectively utilize the data provided by cameras and LiDAR, resulting in too much data computation and too long processing time, which does not meet the high efficiency requirements of autonomous driving. Moreover, their detection accuracy is not high enough.

[0068] To address the aforementioned issues, this application provides a target object detection method. First, multiple images are acquired using multiple cameras located at different positions on a vehicle, and LiDAR point cloud data is obtained using the vehicle's LiDAR. Then, multi-scale image features are extracted using an image feature extractor. Each point in the point cloud data is projected onto the image using a LiDAR-to-camera transform to obtain corresponding multi-scale image features. These features are then processed and concatenated with the point cloud data to obtain enhanced point cloud data. Finally, a point cloud detection network model is used to perform 3D target detection on the enhanced point cloud data. Because multiple cameras are used, the acquired images are comprehensive enough. Furthermore, the point cloud data and images are fused before target detection, which not only reduces the computational load during detection and improves detection efficiency but also enhances detection accuracy.

[0069] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0070] Figure 1 This is a flowchart illustrating a target object detection method provided in an embodiment of this application. The executing entity in this embodiment can be a target object detection device, which can be located on an electronic device such as a mobile terminal like a mobile phone, tablet, or computer. The detection device can also be located on a vehicle such as a car; this application does not impose any limitations. This embodiment uses a target object detection device as an example for detailed description, and it can be applied to vehicles.

[0071] like Figure 1 As shown, the target object detection method provided in this embodiment includes:

[0072] S101. Obtain an image set and the point cloud data collected by the LiDAR; wherein, the image set includes multiple images, which are a collection of images of the target scene captured by cameras at multiple different positions on the vehicle, and the target scene includes the target object; the point cloud data is the data obtained by the LiDAR on the vehicle scanning the target scene, and the point cloud data includes multiple points, and the points have feature data.

[0073] For example, cameras can be installed at multiple different locations on the vehicle according to the actual detection range requirements. The target object detection device of this application acquires images of the target scene captured by the cameras at multiple different locations to obtain an image set including multiple images.

[0074] This application does not impose any restrictions on the number or distribution of cameras. For example, Figure 2 This is a schematic diagram illustrating the image distribution acquired by a camera on a vehicle, as provided in an embodiment of this application. Figure 2 As shown, at least six cameras can be installed on the vehicle to acquire images around the vehicle, resulting in an image set including at least image 1 to image 6.

[0075] For example, the vehicle in this application is also equipped with a LiDAR. When multiple cameras simultaneously capture images of a target scene from different angles, the LiDAR also scans the target scene. If a target object exists within the target scene, the LiDAR can acquire point cloud data of the target object when the scene is captured. This point cloud data includes multiple points scanned by the LiDAR, each with corresponding feature data. For example, the feature data may include the point's three-dimensional coordinates, color information, reflection intensity information, echo count information, etc., which are not limited in this application. The target object detection device in this application, to perform target object detection, acquires not only the image set but also the point cloud data collected by the LiDAR.

[0076] 102. From the image set, determine the image corresponding to the point; where the image corresponding to the point is the image of the projection point of the point within a preset area, and the projection point is the position point of the point projected onto the image.

[0077] For example, after acquiring the image set and point cloud data, the object detection device of this application determines the image corresponding to each point from the image set including multiple images. The image corresponding to a point is the image where the projection point of the point is within a preset area, and the projection point is the position point where the point is projected onto the image. Therefore, each point can be sequentially projected onto the acquired multiple images, the position of the projection point of the point on each image can be determined, and the image corresponding to the position of the projection point within the preset area can be determined as the image corresponding to that point. If a point can be projected onto multiple images, the image whose projection point is closest to the center of the image can be selected as the image corresponding to the point.

[0078] In one example, determining the image corresponding to a point from a set of images could also include:

[0079] S1021. Based on the feature data of the point and the preset transformation matrix of the LiDAR to the camera corresponding to each image, obtain the two-dimensional coordinates of the projection point of the point on each image.

[0080] S1022. Determine the corresponding image of the two-dimensional coordinates of the projection point within a preset range, which is the image corresponding to the point.

[0081] For example, when determining the image corresponding to a point, the two-dimensional coordinates of the projection point of the point on each image can be obtained based on the point's feature data and the preset transformation matrix from LiDAR to the camera corresponding to each image. Then, the image corresponding to the point whose two-dimensional coordinates of the projection point are within a preset range is determined. For example, taking a point in the point cloud data as an example, the two-dimensional coordinates of the projection point of the point on image 1 can be calculated based on the point's feature data and the preset transformation matrix 1 from LiDAR to the camera corresponding to image 1. It is then determined whether the two-dimensional coordinates of the projection point of the point on image 1 are within a preset range. If they are within the preset range, image 1 is determined to be the image corresponding to the point. If they are not within the preset range, the two-dimensional coordinates of the projection point of the point on image 2 can be calculated based on the point's feature data and the preset transformation matrix 2 from LiDAR to the camera corresponding to image 2. It is then determined whether the two-dimensional coordinates of the projection point of the point on image 2 are within the preset range, and so on, until the image corresponding to the point is determined from the image set. The processing of each other point is similar to the above until the image corresponding to each point is determined.

[0082] S103. Based on the feature data of the point and the image corresponding to the point, determine the first feature vector of the point; wherein, the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point.

[0083] For example, after determining the image corresponding to each point, the detection device of the target object of this application performs feature fusion based on the feature data of each point and the image corresponding to the point to determine the first feature vector of each point, wherein the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point.

[0084] This application does not impose restrictions on how the first feature vector of a point is determined based on the point's feature data and the corresponding image. In one example, determining the first feature vector of a point based on its feature data and the corresponding image may include:

[0085] S1031. Process the image corresponding to the point to obtain the second feature vector corresponding to the point; wherein, the second feature vector represents the mean of the multi-scale image features of the projection point corresponding to the point in the image corresponding to the point.

[0086] S1032. The feature data of the point and the second feature vector corresponding to the point are concatenated to obtain the first feature vector of the point.

[0087] For example, the image corresponding to the point can be processed first to obtain the second feature vector corresponding to the point, that is, to determine the mean of the multi-scale image features of the projection point of the point in the image corresponding to the point. For example, the image corresponding to the point is processed to obtain the multi-scale image of the image corresponding to the point, and then the image features of the projection point of the point at different scales are determined. Then, the multi-scale image features are averaged to obtain the second feature vector. Then, the feature data of the point and the second feature vector corresponding to the point are concatenated. For example, the feature data of the point is concatenated before or after the second feature vector corresponding to the point to obtain the first feature vector of the point.

[0088] For example, processing the image corresponding to a point to obtain the second feature vector corresponding to the point may include:

[0089] S1. Normalize the projection points of the point onto the image corresponding to the point to obtain the normalized projection points corresponding to the point.

[0090] S2. Perform feature extraction processing on the image corresponding to the point to obtain the multi-scale image features of the image corresponding to the point.

[0091] S3. Determine the second feature vector corresponding to the point based on the two-dimensional coordinates of the normalized projection point and the multi-scale image features of the image corresponding to the point.

[0092] For example, when processing the image corresponding to a point, the projection point of the point on the corresponding image can be determined first, and then the projection point on the corresponding image can be normalized to obtain the normalized projection point. In practical applications, the two-dimensional coordinates of the projection point on the corresponding image of each point can be determined directly based on the feature data of each point and the preset transformation matrix of the LiDAR to the camera corresponding to the image of the point. Then, the two-dimensional coordinates of the projection point can be normalized to obtain the normalized two-dimensional coordinates of the projection point on the corresponding image of each point. At the same time, the image corresponding to the point is also input into an image feature extractor to perform feature extraction processing on the image corresponding to each point to obtain the multi-scale image features of the image corresponding to each point. Then, based on the two-dimensional coordinates of the normalized projection point and the multi-scale image features of the image corresponding to the point, the second feature vector corresponding to the point is determined.

[0093] The image feature extractor may include a backbone layer and a neck layer. The backbone layer is used to extract features, and commonly used ones include ResNet and VGG. The neck layer is placed between the backbone layer and the head layer to further improve the diversity and robustness of the features. This application does not limit the image feature extractor.

[0094] For example, determining the second feature vector corresponding to a point based on the two-dimensional coordinates of the normalized projected point and the multi-scale image features of the image corresponding to the point may include:

[0095] S31. Based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point, the image features of the normalized projection point corresponding to the point in the image corresponding to the point at each different scale are determined by using an interpolation function.

[0096] S32. Perform mean processing on the image features of the normalized projection points corresponding to the points in the images at different scales, and determine the mean-processed image features as the second feature vector corresponding to the points.

[0097] For example, the F.grid_sample interpolation function can be used to calculate the image features of projection points at different scales. That is, using the interpolation function, based on the normalized two-dimensional coordinates of the projection point corresponding to each point obtained above and the multi-scale image features of the image corresponding to the point, the image features of the normalized projection point corresponding to each point in the image corresponding to each point at each different scale are calculated. Then, the averaged image features of each point are processed, and the averaged image features of each point are determined as the second feature vector corresponding to that point. For example, the averaged image features of each point can be a 256-dimensional feature vector; this application does not impose any limitation on this.

[0098] Once the second feature vector corresponding to a point is determined, concatenating the point's feature data with the corresponding second feature vector yields the point's first feature vector. For example, if the point's feature data includes three-dimensional coordinate information x, y, z and reflection intensity information i, i.e., if the point's feature data is (x, y, z, i), concatenating the point's feature data (x, y, z, i) before or after the 256-dimensional second feature vector results in a 260-dimensional feature vector. This 260-dimensional feature vector is the point's first feature vector, representing the data obtained after fusing multi-scale image features of the point and its corresponding image.

[0099] S104. Detect and process the first feature vector of each point to obtain information about the target object.

[0100] For example, after obtaining the first feature vector of each point, the target object detection device of this application performs detection processing on the first feature vector of each point to obtain information about the target object. For example, the first feature vector of each point can first be voxelized, then processed by an encoder, and then the data processed by the encoder can be input into a Region Proposal Network (RPN) for processing. Finally, it can be detected by a 3D target detection head to obtain information about the target object.

[0101] For example, Figure 3 This is an architectural block diagram illustrating an embodiment of the solution provided in this application. Figure 3 As shown, the encoder in image processing processes images 1 to 6 acquired by the camera to obtain multi-scale image features; the point cloud data acquired by the LiDAR and images 1 to 6 acquired by the camera are processed according to this application. Figure 1 After the fusion processing described in the embodiment, a first feature vector of the data obtained by fusing the multi-scale image features representing the points and their corresponding images is obtained. The first feature vector is then used as follows: Figure 3 After sequentially inputting the voxelization model, encoder, region proposal network (RPN), and 3D detection head, the system outputs information about the target object to complete 3D target detection.

[0102] The target object information may include one or more of the following: the target object's bounding box, the target object's category, the target object's position, the target object's size, the target object's pose information, the target object's movement speed, and the number of target objects.

[0103] For example, after detecting and processing the first feature vector of each point, the target object can be circled on the image with a rectangle, and information such as the target object's category, location, size, pose, speed, and number can be marked. The target object's category can be pedestrians, cars, obstacles, etc.; its location can be represented by its distance from the vehicle and its orientation; its size can be represented by its length, width, and height; its pose information can be its rotation angle, orientation, etc.; if the target object is moving, its speed can also be marked; if there are multiple target objects, the total number of target objects collected can also be output. In practical applications, one or more of the above target object information can be output according to actual needs.

[0104] For example, the detection processing of the first feature vector of each point to obtain the information of the target object may also include: inputting the first feature vector of each point into a preset point cloud detection network model and outputting the information of the target object; wherein, the preset point cloud detection network model is a pre-trained model used to determine the information of the target object.

[0105] For example, the target object detection device of this application can input the first feature vector of each point into a pre-trained model for determining the information of the target object. The model performs detection processing on the first feature vector of each point and directly outputs the information of the target object.

[0106] The target object detection method provided in this application includes: acquiring an image set and acquiring point cloud data collected by a LiDAR; wherein the image set includes multiple images, and the image set is a collection of images of a target scene captured by cameras at multiple different positions on a vehicle, and the target scene includes a target object; the point cloud data is data obtained by the LiDAR on the vehicle scanning the target scene, and the point cloud data includes multiple points, each point having feature data; determining the image corresponding to a point from the image set; wherein the image corresponding to a point is an image of the projection point of the point within a preset area, and the projection point is the position point of the point projected onto the image; determining a first feature vector of the point based on the feature data of the point and the image corresponding to the point; wherein the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point; and performing detection processing on the first feature vector of each point to obtain information about the target object. The solution proposed in this application has two main aspects. First, the processed image data includes images acquired by cameras from multiple different locations, covering a wider range and containing more image information, thus enabling the extraction of more image features. Furthermore, multi-scale feature extraction processing is performed on the images in the image set, providing a more robust data foundation for subsequent target object detection. Second, this application fuses multi-scale image features with point cloud data acquired by LiDAR, and targets are detected based on the fused data. This pre-fusion method not only reduces the computational load during subsequent detection and improves detection efficiency, but also stably and effectively solves the continuity problem between the two different modal data sources (camera and LiDAR), thereby improving detection accuracy.

[0107] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0108] Figure 4 This is a schematic diagram of the structure of a target object detection device provided in an embodiment of this application. Figure 4As shown, the target object detection device 40 provided in this application embodiment includes: an acquisition unit 401, a matching unit 402, a processing unit 403, and a detection unit 404.

[0109] The acquisition unit 401 is used to acquire an image set and point cloud data collected by the LiDAR. The image set includes multiple images, which are a collection of images of the target scene captured by cameras at multiple different positions on the vehicle. The target scene includes the target object. The point cloud data is the data obtained by the LiDAR on the vehicle scanning the target scene. The point cloud data includes multiple points, and each point has feature data.

[0110] The matching unit 402 is used to determine the image corresponding to a point from the image set; wherein the image corresponding to the point is the image of the projection point of the point within a preset area, and the projection point is the position point of the point projected onto the image.

[0111] The processing unit 403 is used to determine the first feature vector of a point based on the feature data of the point and the image corresponding to the point; wherein the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point.

[0112] The detection unit 404 is used to detect and process the first feature vector of each point to obtain information about the target object.

[0113] The apparatus provided in this embodiment can be used to execute the methods of the above embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0114] Figure 5 This is a schematic diagram of the structure of another target object detection device provided in an embodiment of this application. (See attached diagram.) Figure 5 As shown, the target object detection device 50 provided in this application embodiment includes: an acquisition unit 501, a matching unit 502, a processing unit 503, and a detection unit 504.

[0115] The acquisition unit 501 is used to acquire an image set and point cloud data collected by the LiDAR. The image set includes multiple images, which are a collection of images of the target scene captured by cameras at multiple different positions on the vehicle. The target scene includes the target object. The point cloud data is the data obtained by the LiDAR on the vehicle scanning the target scene. The point cloud data includes multiple points, and each point has feature data.

[0116] The matching unit 502 is used to determine the image corresponding to a point from the image set; wherein the image corresponding to the point is the image of the projection point of the point within a preset area, and the projection point is the position point of the point projected onto the image.

[0117] The processing unit 503 is used to determine the first feature vector of a point based on the feature data of the point and the image corresponding to the point; wherein the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point.

[0118] The detection unit 504 is used to detect and process the first feature vector of each point to obtain information about the target object.

[0119] In one example, the processing unit 503 includes a processing module 5031 and a splicing module 5032.

[0120] The processing module 5031 is used to process the image corresponding to the point to obtain the second feature vector corresponding to the point; wherein, the second feature vector represents the mean of the multi-scale image features of the projection point corresponding to the point in the image corresponding to the point.

[0121] The splicing module 5032 is used to splice the feature data of a point and the second feature vector corresponding to the point to obtain the first feature vector of the point.

[0122] In one example, the processing module 5031 includes a processing submodule 50311, a feature extraction module 50312, and a determination module 50313.

[0123] The processing submodule 50311 is used to normalize the projection points of a point on the image corresponding to the point, so as to obtain the normalized projection points corresponding to the point.

[0124] The feature extraction module 50312 performs feature extraction processing on the image corresponding to the point to obtain multi-scale image features of the image corresponding to the point.

[0125] The determination module 50313 is used to determine the second feature vector corresponding to the point based on the two-dimensional coordinates of the normalized projection point and the multi-scale image features of the image corresponding to the point.

[0126] In one example, the determining module 50313 includes a first determining module 503131 and a second determining module 503132.

[0127] The first determining module 503131 is used to determine the image features of the normalized projection point in the image corresponding to the point at each different scale based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point, using an interpolation function.

[0128] The second determining module 503132 is used to perform mean processing on the image features of the normalized projection points corresponding to the points in the images corresponding to the points at different scales, and to determine the mean-processed image features as the second feature vector corresponding to the points.

[0129] In one example, the matching unit 502 includes a calculation module 5021 and a decision module 5022.

[0130] The calculation module 5021 is used to obtain the two-dimensional coordinates of the projection point of the point on each image based on the feature data of the point and the preset transformation matrix of the LiDAR to the camera corresponding to each image.

[0131] The determination module 5022 is used to determine the corresponding image of the two-dimensional coordinates of the projection point within a preset range, which is the image corresponding to the point.

[0132] In one example, the detection unit 504 is specifically used to input the first feature vector of each point into a preset point cloud detection network model and output the information of the target object; wherein, the preset point cloud detection network model is a pre-trained model used to determine the information of the target object.

[0133] In one example, the target object information includes one or more of the following: the target object's bounding box, the target object's category, the target object's position, the target object's size, the target object's pose information, the target object's movement speed, and the number of target objects.

[0134] The apparatus provided in this embodiment can be used to execute the methods of the above embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0135] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Furthermore, they can be stored as program code in the device's memory, and the data processing modules can be called and executed by a specific processing element. The implementation of other modules is similar. These modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0136] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 6 As shown, the electronic device 60 includes a processor 601 and a memory 602 communicatively connected to the processor.

[0137] The memory 602 stores computer-executable instructions; the processor 601 executes the computer-executable instructions stored in the memory 602 to implement the method as described in any of the preceding descriptions.

[0138] In the specific implementation of the aforementioned electronic device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The methods disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0139] This application also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the methods described in any of the foregoing embodiments.

[0140] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to computer instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0141] This application also provides a computer program product, including a computer program that, when executed by a processor, is used to implement the method as described in any of the foregoing embodiments.

[0142] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0143] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting a target object, characterized in that, The method includes: Acquire an image set and acquire point cloud data collected by the LiDAR; wherein, the image set includes multiple images, which are a collection of images of the target scene captured by cameras at multiple different positions on the vehicle, and the target scene includes a target object; the point cloud data is the data obtained by the LiDAR on the vehicle scanning the target scene, and the point cloud data includes multiple points, and the points have feature data; Each point is sequentially projected onto multiple acquired images. The position of the projection point of the point on each image is determined, and the image corresponding to the position of the projection point within a preset area is determined as the image corresponding to the point. The projection point is the position point of the point projected onto the image. If the point is projected onto multiple images, the image with the projection point closest to the middle position of the image is selected as the image corresponding to the point. Based on the feature data of the point and the image corresponding to the point, a first feature vector of the point is determined; wherein, the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point. The first feature vector of each point is processed to obtain information about the target object; Based on the feature data of the point and the image corresponding to the point, the first feature vector of the point is determined, including: The image corresponding to the point is processed to obtain a multi-scale image of the image corresponding to the point, the image features of the projection point corresponding to the point at different scales are determined, and the multi-scale image features are averaged to obtain a second feature vector corresponding to the point; wherein, the second feature vector represents the mean of the multi-scale image features of the projection point corresponding to the point in the image corresponding to the point. The feature data of the point and the second feature vector corresponding to the point are concatenated to obtain the first feature vector of the point.

2. The method according to claim 1, characterized in that, The image corresponding to the point is processed to obtain the second feature vector corresponding to the point, including: The projection points of the point on the image corresponding to the point are normalized to obtain the normalized projection points corresponding to the point. The image corresponding to the point is subjected to feature extraction processing to obtain the multi-scale image features of the image corresponding to the point; The second feature vector corresponding to the point is determined based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point.

3. The method according to claim 2, characterized in that, Based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point, a multi-scale image of the image corresponding to the point is obtained. The image features of the projection point corresponding to the point at different scales are determined. The multi-scale image features are averaged to determine the second feature vector corresponding to the point, including: Based on the two-dimensional coordinates of the normalized projection point corresponding to the point and the multi-scale image features of the image corresponding to the point, the image features of the normalized projection point corresponding to the point in the image corresponding to each point at different scales are determined by using an interpolation function. The image features of the normalized projection point corresponding to the point at different scales are averaged to determine the averaged image features as the second feature vector corresponding to the point.

4. The method according to claim 1, characterized in that, Each point is sequentially projected onto multiple acquired images. The position of the projection point of each point on each image is determined, and the image corresponding to the position of the projection point within a preset area is identified as the image corresponding to the point. This includes: Based on the feature data of the point and the preset transformation matrix of the LiDAR to the camera corresponding to each image, obtain the two-dimensional coordinates of the projection point of the point on each image; The image corresponding to the two-dimensional coordinates of the projection point within a preset range is determined as the image corresponding to the point.

5. The method according to any one of claims 1-4, characterized in that, The first feature vector of each of the points is processed to obtain information about the target object, including: The first feature vector of each point is input into a preset point cloud detection network model, and the information of the target object is output; wherein, the preset point cloud detection network model is a pre-trained model used to determine the information of the target object.

6. The method according to any one of claims 1-4, characterized in that, The information of the target object includes one or more of the following: the bounding box of the target object, the category of the target object, the position of the target object, the size of the target object, the posture information of the target object, the movement speed of the target object, and the number of target objects.

7. A target object detection device, characterized in that, The device includes: An acquisition unit is used to acquire an image set and point cloud data collected by a lidar; wherein, the image set includes multiple images, which are a collection of images of a target scene captured by cameras at multiple different positions on the vehicle, and the target scene includes a target object; the point cloud data is data obtained by the lidar on the vehicle scanning the target scene, and the point cloud data includes multiple points, and the points have feature data; The matching unit is used to sequentially project each point onto multiple acquired images, determine the position of the projection point of the point on each image, and determine the image corresponding to the point whose position of the projection point is within a preset area; wherein the projection point is the position point of the point projected onto the image; if the point is projected onto multiple images, the image whose position of the projection point is closest to the middle position of the image is selected as the image corresponding to the point. The processing unit is configured to determine a first feature vector of the point based on the feature data of the point and the image corresponding to the point; wherein the first feature vector represents the data after fusing the multi-scale image features of the point and the image corresponding to the point. The detection unit is used to perform detection processing on the first feature vector of each point to obtain information about the target object; The processing unit is used to process the image corresponding to the point, obtain a multi-scale image of the image corresponding to the point, determine the image features of the projection point of the point at different scales, average the multi-scale image features to obtain a second feature vector corresponding to the point; wherein, the second feature vector represents the mean of the multi-scale image features of the projection point of the point in the image corresponding to the point; and concatenate the feature data of the point and the second feature vector corresponding to the point to obtain a first feature vector of the point.

8. An electronic device, characterized in that, The electronic device includes: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

Images