Multi-modal object detection method based on augmented virtual points and adaptive distance hierarchy

Abstract

This invention relates to the field of image recognition, specifically a multimodal target detection method based on enhanced virtual points and adaptive distance grading. The method includes acquiring regularized image data, extracting semantic features from the regularized image data, and generating image BEV features; acquiring regularized point cloud data and performing distance grading to obtain near-range and far-range point clouds; voxelizing and extracting features from the near-range point cloud to obtain near-range point cloud BEV features, voxelizing and extracting features from the far-range point cloud to obtain far-range point cloud BEV features, and correcting the far-range point cloud BEV features using the image BEV features to obtain corrected far-range point cloud BEV features; fusing the image BEV features, near-range point cloud BEV features, and corrected far-range point cloud BEV features to obtain fused features, and performing target detection using a detection head based on the fused features to obtain the target detection result. This invention can improve the accuracy of multimodal target detection based on enhanced virtual points and adaptive distance grading.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and particularly relates to a multimodal target detection method based on enhanced virtual points and adaptive distance grading. Background Technology

[0002] With the rise of artificial intelligence and computer vision technologies, autonomous driving technology has entered the public eye and made significant progress. However, the safety and reliability of autonomous driving remain pressing issues, one key aspect of which is the vehicle's environmental perception. Cars need to accurately perceive their surroundings and make quick and accurate decisions based on the situation.

[0003] Currently, target perception methods in the field of autonomous driving can be divided into three categories based on the sensing device: The first is camera-based RGB image target detection methods. For example, the YOLO series algorithms, through a deep learning framework, can quickly locate the bounding boxes of target objects such as vehicles, pedestrians, and traffic signs in images and classify them. Its advantages lie in its high computational efficiency and good perception of the appearance features of targets (such as color and texture), making it more advantageous in small target detection, and the cameras are inexpensive. However, its disadvantages are also obvious. For example, in adverse weather conditions, the image quality of the camera will drop significantly, leading to a decrease in detection accuracy; in addition, RGB image detection methods are also difficult to accurately identify targets with severe occlusion. The second is target detection methods based on LiDAR (Light Detection and Ranging). LiDAR measures the distance to objects by emitting laser beams and receiving reflected light, thereby constructing a three-dimensional point cloud map of the surrounding environment. For example, PointPillars divides the point cloud data into columnar structures, then extracts features and performs target detection. The advantage of LiDAR lies in its extremely high accuracy in distance measurement, enabling it to accurately perceive the three-dimensional shape and position of objects. It is highly effective for obstacle detection (such as vehicles and pedestrians) in autonomous driving, and can operate normally even in low-light environments. However, LiDAR is relatively expensive and generates a large amount of point cloud data, placing high demands on computing resources. A third approach is a multimodal fusion target detection method based on images and point clouds. This method combines RGB images from a camera with point cloud data from LiDAR, aiming to fully utilize the advantages of both and overcome the limitations of single-modal perception. For example, RGB images from a camera provide rich visual information, such as color, texture, and semantic features, while point cloud data from LiDAR provides accurate depth information and three-dimensional spatial structure. By fusing these two modalities, target objects in the environment can be perceived more comprehensively.

[0004] In recent years, target detection technology using multimodal fusion of laser point clouds and RGB images has become a popular research direction in the field of autonomous driving. Multiple experiments have shown that fusing these two modalities for target detection is more accurate than single-modal target detection, thus better facilitating environmental perception in autonomous driving. However, due to the sparsity of LiDAR at long distances, a small amount of point cloud data at a distance is insufficient to characterize the target, leading to the loss of the target's shape and contour information. Furthermore, even when current technologies fuse image features, they only improve the accuracy of near-range target detection and may even increase noise for distant targets, resulting in relatively low target detection accuracy. Summary of the Invention

[0005] This invention provides a multimodal target detection method and apparatus based on enhanced virtual points and adaptive distance grading, which can improve the accuracy of multimodal target detection based on enhanced virtual points and adaptive distance grading.

[0006] To achieve the above objectives, this invention provides a multimodal target detection method based on enhanced virtual points and adaptive distance grading, comprising:

[0007] Acquire regularized image data, extract semantic features from the regularized image data, and generate image BEV features based on the semantic features;

[0008] Obtain regular point cloud data, and classify the regular point cloud data into near point cloud and far point cloud according to the distance classification threshold;

[0009] Voxelization and feature extraction are performed on near-range point clouds to obtain near-range point cloud BEV features, and voxelization and feature extraction are performed on far-range point clouds to obtain far-range point cloud BEV features. The image BEV features are then used to correct the far-range point cloud BEV features to obtain corrected far-range point cloud BEV features.

[0010] The image BEV features, near-range point cloud BEV features, and modified far-range point cloud BEV features are fused to obtain fused features. Based on the fused features, a preset detection head is used to perform target detection to obtain the target detection result.

[0011] To address the aforementioned problems, the present invention also provides a multimodal target detection device based on enhanced virtual points and adaptive distance grading, the device comprising:

[0012] The image data processing module is used to acquire regular image data, extract semantic features from the regular image data, and generate image BEV features based on the semantic features.

[0013] The point cloud data processing module is used to acquire regular point cloud data and classify the regular point cloud data into near point cloud and far point cloud according to the distance classification threshold. The near point cloud is voxelized and feature extracted to obtain near point cloud BEV features, and the far point cloud is voxelized and feature extracted to obtain far point cloud BEV features. The far point cloud BEV features are then corrected using image BEV features to obtain corrected far point cloud BEV features.

[0014] The target detection module is used to fuse the image BEV features, near-range point cloud BEV features, and modified far-range point cloud BEV features to obtain fused features, and then use a preset detection head to perform target detection based on the fused features to obtain the target detection result.

[0015] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0016] At least one processor; and,

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multimodal target detection method based on enhanced virtual points and adaptive distance grading described above.

[0019] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the multimodal target detection method based on enhanced virtual points and adaptive distance grading described above.

[0020] This invention acquires regularized image data, extracts semantic features, and generates image BEV features. Based on these semantic features, the perception capability for small targets and appearance features can be enhanced, and the generated BEV features provide global spatial context. Furthermore, it acquires regularized point cloud data and performs distance grading based on distance grading thresholds to obtain near-range and far-range point clouds. This distance grading distinguishes between near-range and far-range point clouds, enabling targeted processing of the point cloud data. This addresses the problem of target shape and contour information loss caused by the sparsity of LiDAR at long distances. Moreover, it utilizes… Image BEV features are used to correct the BEV features of distant point clouds, resulting in corrected distant point cloud BEV features. This can supplement the depth and shape information lost due to sparsity in distant point clouds, improve the accuracy of distant target detection, and reduce noise introduced by fusion. In addition, image BEV features, near-range point cloud BEV features, and corrected distant point cloud BEV features are fused, and target detection is performed using a preset detection head based on the fused features to obtain the target detection result. Through multimodal feature fusion, the advantages of image and point cloud data can be complemented, improving the accuracy of target detection. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multimodal target detection method based on enhanced virtual points and adaptive distance grading provided in an embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating an example of a multimodal target detection method based on enhanced virtual points and adaptive distance grading provided in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram illustrating the generation of a long-range virtual point set using a multimodal target detection method based on enhanced virtual points and adaptive distance grading, according to an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram illustrating the principle of distance-level feature extraction in a multimodal target detection method based on enhanced virtual points and adaptive distance-level grading, as provided in an embodiment of the present invention.

[0025] Figure 5 This is a functional block diagram of a multimodal target detection device based on enhanced virtual points and adaptive distance grading, provided in an embodiment of the present invention.

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the multimodal target detection method based on enhanced virtual points and adaptive distance grading, according to an embodiment of the present invention.

[0027] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0028] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0029] This application provides a multimodal target detection method based on enhanced virtual points and adaptive distance grading. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the multimodal target detection method based on enhanced virtual points and adaptive distance grading can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0030] Reference Figure 1 The diagram shown is a flowchart illustrating a multimodal target detection method based on enhanced virtual points and adaptive distance grading according to an embodiment of the present invention. In this embodiment, the multimodal target detection method based on enhanced virtual points and adaptive distance grading includes:

[0031] S1. Obtain regular image data, extract semantic features from the regular image data, and generate BEV features of the image based on the semantic features.

[0032] Understandably, regularized image data refers to preprocessed regularized image data obtained by using image data captured by an onboard camera, which undergoes operations such as denoising, enhancement, geometric correction, and rotation.

[0033] It is understood that the semantic features of the regular image data extracted in the embodiments of the present invention can be extracted using the CenterNet image backbone network. The CenterNet network is a target detection method based on center points, which completes the detection by predicting the center point of the target and its attributes.

[0034] For example, generating image BEV features and image instance masks based on semantic features can be achieved by the following implementation steps: using an image backbone network to extract the semantic features of the image; and feeding the extracted features into branch one, using the traditional LSS (Lift, Splat, Shoot) method to estimate pixel depth and obtain the image semantic BEV features; feeding the mask prediction head into branch two to predict the image instance mask of the target object.

[0035] Specifically, image BEV features are generated based on semantic features, including:

[0036] The depth distribution of each pixel in the regularized image data is estimated using a preset depth estimation algorithm, and the regularized image data is converted into a voxel probability volume based on the depth distribution.

[0037] The voxel probability volume is transformed into three-dimensional coordinates using the camera's intrinsic and extrinsic parameters, resulting in a three-dimensional voxel mesh;

[0038] Semantic features are embedded into a three-dimensional voxel grid to obtain a three-dimensional semantic voxel representation;

[0039] The image BEV features are obtained by pooling the three-dimensional semantic voxel representation along the height direction.

[0040] Understandably, image BEV features refer to the feature representation obtained by mapping the semantic or geometric information in a camera image to the bird's-eye-view (BEV) coordinate system through geometric transformation or depth estimation. Essentially, it is a feature expression that transforms the image perspective into a top-down plane perspective.

[0041] Understandably, the preset depth estimation algorithm refers to a model used to predict pixel depth from monocular or multi-view images, such as the traditional LSS (Lift, Splat, Shoot) method mentioned above.

[0042] Understandably, a voxel probability volume refers to the probability that each voxel cell in three-dimensional space stores a pixel falling within a certain depth range.

[0043] Understandably, camera intrinsic and extrinsic parameters are parameters that describe the geometric relationship of camera imaging (intrinsic parameters define the relationship between pixels and camera coordinates, while extrinsic parameters define the relationship between the camera and world / radar coordinates).

[0044] Understandably, a three-dimensional voxel mesh is a structure that discretizes three-dimensional space into regular cubic units.

[0045] Understandably, pooling refers to aggregating voxel features in the height direction to obtain two-dimensional bird's-eye view features. Aggregation includes average pooling and max pooling.

[0046] S2. Obtain regular point cloud data, and classify the regular point cloud data into near-distance point clouds and far-distance point clouds according to the distance classification threshold.

[0047] Understandably, well-organized point cloud data refers to point cloud data that is suitable for feature extraction and fusion after the initial point cloud data collected by the vehicle-mounted LiDAR has undergone denoising, filtering and other operations.

[0048] Specifically, regularized point cloud data is classified into near-range and far-range point clouds based on a distance classification threshold, including:

[0049] Obtain the deformation grading curve, and perform initial grading of the regular point cloud data based on the deformation grading curve and the distance grading threshold to obtain the initial point cloud division result;

[0050] Based on the initial point cloud segmentation results, the boundary of the deformation grading curve is adjusted by minimizing the energy function to obtain the optimal grading curve;

[0051] The optimal grading curve is used to divide regular point cloud data into near-range point cloud and far-range point cloud.

[0052] Furthermore, the boundaries of the deformation grading curve are adjusted by minimizing the energy function, including:

[0053] Minimize energy function Use the following formula:

[0054]

[0055] in, To the deformation grading curve Above, the gradient of the point cloud density function. Let be the point cloud density function, i.e., the point cloud in polar coordinates. Local density distribution under, The normal vector of the curve, i.e., the deformation grading curve. At angle The external normal vector at that location, For regularization weights, This is a transformation function.

[0056] Understandably, the distance classification threshold refers to a preset distance boundary value used to distinguish between near and far points in point cloud data.

[0057] Understandably, minimizing the energy function means constructing an energy objective function (containing data terms and smoothing terms) and adjusting the hierarchical curve during iterative optimization to make it conform to the true distribution of the point cloud while maintaining the continuity and smoothness of the boundary, thereby obtaining the optimal near / far distance partition.

[0058] For example, the method of classifying regular point cloud data into near-range and far-range point clouds based on a distance classification threshold can be implemented using the following steps:

[0059] First, the regularized point cloud data E is projected onto the horizontal plane and converted into polar coordinates:

[0060]

[0061]

[0062] in, For point clouds Two-dimensional projection coordinates on the horizontal plane, For point clouds The radial distance to the origin of the two-dimensional projection coordinate system. For point clouds The polar angle represents the point cloud. The direction on the horizontal plane.

[0063] Define density function , indicating that point clouds are in Local density at:

[0064]

[0065] Among them, bandwidth h(r) is the core parameter for kernel density estimation, which is used to control the smoothness of density calculation.

[0066] Define deformation grading curve Transformation function Represented using finite Fourier series:

[0067]

[0068]

[0069] in, Initializing to 0 indicates that the initial grading curve is circular. K is the truncation order of the series. When K is small, the curvature of the curve changes gently, avoiding jagged boundaries. Used to control symmetrical deformation (such as overall expansion / contraction or ellipticization). To control asymmetric deformations (such as local bulges or depressions), they require minimizing the energy function to obtain the optimal solution:

[0070]

[0071] in, To the deformation grading curve Above, the gradient of the point cloud density function. Let be the point cloud density function, i.e., the point cloud in polar coordinates. Local density distribution under, The normal vector of the curve, i.e., the deformation grading curve. At angle The external normal vector at that location, For regularization weights, The deformation function has two terms: the first is a density alignment term, which pushes the curve along the low-density gradient direction to prevent segmentation of point cloud instances; the second is a curvature regularization term. It can be set to 0.1 to suppress excessive curve bending, normal vector Represented as:

[0072]

[0073] The final solution This is the result of the adaptive grading curve.

[0074] S3. Perform voxelization and feature extraction on the near-range point cloud to obtain the near-range point cloud BEV features, perform voxelization and feature extraction on the far-range point cloud to obtain the far-range point cloud BEV features, and use the image BEV features to correct the far-range point cloud BEV features to obtain the corrected far-range point cloud BEV features.

[0075] Specifically, before performing voxelization and feature extraction on the distant point cloud to obtain the distant point cloud BEV features, the method also includes constructing a distant virtual point set.

[0076] Furthermore, a remote virtual point set is constructed, including:

[0077] Step 1: Project the distant point cloud onto the two-dimensional image plane using the projection formula to obtain a two-dimensional projected point set;

[0078] Step 2: Generate image instance masks based on semantic features. For each image instance mask, the set of two-dimensional projection points falling into the image instance mask is used as the initial generated point set, and the initial virtual point set is set to an empty set.

[0079] Step 3: When the image instance mask covers a preset number of pixel grids, the center point coordinates of all pixel grids covered by the image instance mask are used to form an initial candidate point set. When a pixel grid has fallen into the projection point, the candidate point corresponding to the center point of the pixel grid is deleted from the initial candidate point set.

[0080] Step 4: For each candidate point in the initial candidate point set, calculate the distance from each candidate point to the initial generated point set, and sort the distances from the candidate points to the initial generated point set.

[0081] Step 5: Select the candidate point with the largest distance in the distance sorting as the newly generated 2D point cloud, add the newly generated 2D point cloud to the initial generated point set and the initial virtual point set, and simultaneously delete the candidate point in the initial candidate point set;

[0082] Repeat steps 1-5 until the number of point clouds in the initial virtual point set reaches the preset value, thus obtaining the remote virtual point set.

[0083] Furthermore, voxelization and feature extraction are performed on the distant point cloud to obtain the distant point cloud BEV features, including:

[0084] For each point cloud in the distant virtual point set, query the nearest projected point of each point cloud in the image instance mask;

[0085] Using linear interpolation, the semantic features of the image of the nearest projection point are obtained, and the reflectance and depth values ​​of the nearest projection point are extracted.

[0086] The image semantic features, reflectance, and depth values ​​of the nearest projection point are associated with each point cloud corresponding to the distant virtual point set to obtain a distant two-dimensional point cloud with image semantic features, reflectance, and depth values.

[0087] The distant two-dimensional point cloud with image semantic features, reflectivity, and depth values ​​is back-projected back into the three-dimensional lidar coordinate system using the camera's intrinsic and extrinsic parameter matrix to obtain the distant point cloud.

[0088] Voxelization and feature extraction are performed on distant point clouds to obtain distant point cloud BEV features.

[0089] Understandably, the following projection formula can be used to project a distant point cloud onto a two-dimensional image plane:

[0090]

[0091] Where K is the intrinsic parameter matrix of the camera calibration, T is the extrinsic parameter matrix of the camera calibration, and P is the input point cloud to be projected. This is a 2D point cloud projected onto the image plane.

[0092] For example, constructing a remote virtual point set can be achieved using the following implementation steps:

[0093] (1) For each image instance mask , will fall into The 2D projection point set S={ } as the initial generated point set G={ }, and set the initial virtual point set V to an empty set.

[0094] (2) Representation of the initial candidate point set Q: If If it covers m pixels, then... The center coordinates of all the covered pixel grids are used as the initial candidate point set Q={ If a pixel has already fallen into the projection point, then the center point corresponding to that pixel is removed from the initial candidate points.

[0095] (3) For each candidate point Calculate the shortest distance from it to the currently generated point set G. :

[0096]

[0097] (4) Select the point that maximizes the above distance as the newly generated 2D point cloud and add it to the generated point set G. At the same time, add it to the initial virtual point set V and delete the point from the candidate point set Q.

[0098] (5) Repeat the above steps until the number of points in the virtual point set V is the preset value k, and obtain the remote virtual point set.

[0099] For example, the Voxelization and feature extraction of distant point clouds to obtain BEV features of distant point clouds can be achieved by the following implementation steps:

[0100] After generating the remote virtual point set, some virtual 2D point clouds V={ are generated within the instance mask. However, they do not yet contain any depth information. Therefore, for a point... Find the nearest projection point within the instance mask, and assign the depth value z of that projection point to the point. Thus, the virtual 2D point cloud... ,in These are the semantic features of the image obtained through linear interpolation. It is the aligned timestamp; projection point , It refers to reflectivity.

[0101] The virtual 2D point cloud carries image semantic features, while the projected points carry reflectance features. To unify the feature representations of the virtual 2D point cloud and the projected points, this method performs nearest neighbor matching again for the virtual 2D point cloud. In other words, find the nearest projection point within the instance mask and assign the reflectivity r of that projection point to the point. For the projection point In other words, the nearest virtual 2D point is found within the instance mask, and the image semantic feature f of that virtual 2D point is assigned to the point. Thus, the new representation of virtual 2D point clouds is... The new representation of the projection point is Finally, by using the camera's intrinsic and extrinsic parameter matrices, the virtual 2D points and the projected points are back-projected back into the 3D LiDAR coordinate system to obtain a long-range virtual 3D point cloud. This, along with the near-range real 3D point cloud, forms the enhanced point cloud E in the same coordinate system.

[0102] To perform subsequent point cloud feature extraction, the enhanced remote virtual 3D point cloud needs to be voxelized and feature extracted to obtain remote point cloud BEV features.

[0103] For example, the BEV features of a distant point cloud are corrected using the BEV features of the image to obtain the corrected BEV features of the distant point cloud. This can be achieved through the following implementation steps:

[0104] In the detection of distant targets, image features often outperform point cloud features. To make the representation of distant point cloud BEV features more accurate, an optical flow method is used, employing image BEV features as guiding features to correct distant point cloud BEV features.

[0105] Define the distortion function as:

[0106]

[0107] in, It is the long-range point cloud BEV feature output by multi-level point cloud feature extraction. These are the updated long-range point cloud BEV features. The flow field is used to extract BEV features from the original long-range point cloud. Twisted to align with the BEV features of the image F_cam The flow field is obtained by inputting the original long-range point cloud BEV features. The BEV feature F_cam of the image is obtained through a convolutional layer. , These correspond to the components of the flow field in two directions, respectively. In fact, It is through bilinear interpolation from The two maximal function terms, used in the mid-sampling, are used to calculate the weights of neighboring feature points in both directions during bilinear interpolation; these terms determine the... The contribution of each feature point (w', h') to the target location (w, h) is then weighted and summed.

[0108] S4. The image BEV features, near-range point cloud BEV features, and modified far-range point cloud BEV features are fused to obtain fused features. Based on the fused features, a preset detection head is used to perform target detection to obtain the target detection result.

[0109] Understandably, distant targets have fewer labels, while nearby targets have more. If all samples are treated equally, the model will prioritize fitting the "majority class" of nearby targets, causing the "minority class" of distant targets to be ignored, resulting in missed detections or inaccurate localization.

[0110] For example, before obtaining the target detection result by performing target detection using a preset detection head based on the fusion features, a strategy to alleviate the imbalance between near and far labels is also included, which weights the loss according to the number of labels after distance classification:

[0111]

[0112] Where N is the total number of labels for all categories in the dataset. It represents the number of labels within a certain distance-level interval b. This is a smoothing factor.

[0113] The loss function of this method is defined as:

[0114]

[0115]

[0116]

[0117] in, It is the loss of the image branch used to predict the two-dimensional instance mask; For the hierarchical weighted loss of the fusion branch, its For hierarchical 3D classification loss, This is a hierarchical 3D regression loss, where B is typically set to 2 (for far and near point clouds) or 3 (for far, mid, and near point clouds). In this embodiment of the invention, B is set to 2. As the first empirical parameter, As the second empirical parameter, This is the third empirical parameter.

[0118] This invention acquires regularized image data, extracts semantic features, and generates image BEV features. Based on these semantic features, the perception capability for small targets and appearance features can be enhanced, and the generated BEV features provide global spatial context. Furthermore, it acquires regularized point cloud data and performs distance grading based on distance grading thresholds to obtain near-range and far-range point clouds. This distance grading distinguishes between near-range and far-range point clouds, enabling targeted processing of the point cloud data. This addresses the problem of target shape and contour information loss caused by the sparsity of LiDAR at long distances. Moreover, it utilizes… Image BEV features are used to correct the BEV features of distant point clouds, resulting in corrected distant point cloud BEV features. This can supplement the depth and shape information lost due to sparsity in distant point clouds, improve the accuracy of distant target detection, and reduce noise introduced by fusion. In addition, image BEV features, near-range point cloud BEV features, and corrected distant point cloud BEV features are fused, and target detection is performed using a preset detection head based on the fused features to obtain the target detection result. Through multimodal feature fusion, the advantages of image and point cloud data can be complemented, improving the accuracy of target detection.

[0119] Reference Figure 2 The diagram shown is a structural flowchart of an example of a multimodal target detection method based on enhanced virtual points and adaptive distance grading provided in an embodiment of the present invention.

[0120] Reference Figure 3 The diagram shown is a schematic diagram of the generation of a long-range virtual point set in a multimodal target detection method based on enhanced virtual points and adaptive distance grading provided in an embodiment of the present invention.

[0121] Reference Figure 4 The diagram shown is a schematic of the distance-level feature extraction principle of a multimodal target detection method based on enhanced virtual points and adaptive distance grading provided in an embodiment of the present invention.

[0122] like Figure 5 The diagram shown is a functional block diagram of a multimodal target detection device based on enhanced virtual points and adaptive distance grading provided in an embodiment of the present invention.

[0123] The multimodal target detection device 100 based on enhanced virtual points and adaptive distance grading described in this invention can be installed in an electronic device. Depending on the functions implemented, the multimodal target detection device 100 may include an image data processing module 101, a point cloud data processing module 102, and a target detection module 103.

[0124] The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0125] In this embodiment, the functions of each module / unit are as follows:

[0126] The image data processing module 101 is used to acquire regular image data, extract semantic features from the regular image data, and generate image BEV features based on the semantic features.

[0127] The point cloud data processing module 102 is used to acquire regular point cloud data, and classify the regular point cloud data into near point cloud and far point cloud according to the distance classification threshold; perform voxelization and feature extraction on the near point cloud to obtain near point cloud BEV features, perform voxelization and feature extraction on the far point cloud to obtain far point cloud BEV features, and use image BEV features to correct the far point cloud BEV features to obtain corrected far point cloud BEV features.

[0128] The target detection module 103 is used to fuse the image BEV features, the near-range point cloud BEV features, and the modified far-range point cloud BEV features to obtain fused features, and to perform target detection using a preset detection head based on the fused features to obtain the target detection result.

[0129] like Figure 6 The diagram shown is a structural schematic of an electronic device that implements a multimodal target detection method based on enhanced virtual points and adaptive distance grading, according to an embodiment of the present invention.

[0130] The electronic device may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a multimodal target detection method program based on enhanced virtual points and adaptive distance grading.

[0131] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing a multimodal target detection method program based on enhanced virtual points and adaptive distance grading), and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0132] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 can include both internal and external storage units of the electronic device. The memory 11 can be used not only to store application software and various types of data installed on the electronic device, such as the code of a multimodal target detection method program based on enhanced virtual points and adaptive distance grading, but also to temporarily store data that has been output or will be output.

[0133] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0134] The communication interface 13 is used for communication between the aforementioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish communication connections between the electronic device and other electronic devices. The user interface may be a display, an input unit (such as a keyboard), or optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device and to display a visual user interface.

[0135] Figure 6 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 6The structure shown does not constitute a limitation on the electronic device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0136] For example, although not shown, the electronic device may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0137] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0138] The program for a multimodal target detection method based on enhanced virtual points and adaptive distance grading, stored in the memory 11 of the electronic device, is a combination of multiple instructions. When run in the processor 10, it can achieve the following:

[0139] Acquire regularized image data, extract semantic features from the regularized image data, and generate image BEV features based on the semantic features;

[0140] Obtain regular point cloud data, and classify the regular point cloud data into near point cloud and far point cloud according to the distance classification threshold;

[0141] Voxelization and feature extraction are performed on near-range point clouds to obtain near-range point cloud BEV features, and voxelization and feature extraction are performed on far-range point clouds to obtain far-range point cloud BEV features. The image BEV features are then used to correct the far-range point cloud BEV features to obtain corrected far-range point cloud BEV features.

[0142] The image BEV features, near-range point cloud BEV features, and modified far-range point cloud BEV features are fused to obtain fused features. Based on the fused features, a preset detection head is used to perform target detection to obtain the target detection result.

[0143] Specifically, the specific implementation method of the processor 10 for the above instructions can be referred to the description of the relevant steps in the corresponding embodiment of the accompanying drawings, and will not be repeated here.

[0144] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0145] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0146] Acquire regularized image data, extract semantic features from the regularized image data, and generate image BEV features based on the semantic features;

[0147] Obtain regular point cloud data, and classify the regular point cloud data into near point cloud and far point cloud according to the distance classification threshold;

[0148] Voxelization and feature extraction are performed on near-range point clouds to obtain near-range point cloud BEV features, and voxelization and feature extraction are performed on far-range point clouds to obtain far-range point cloud BEV features. The image BEV features are then used to correct the far-range point cloud BEV features to obtain corrected far-range point cloud BEV features.

[0149] The image BEV features, near-range point cloud BEV features, and modified far-range point cloud BEV features are fused to obtain fused features. Based on the fused features, a preset detection head is used to perform target detection to obtain the target detection result.

[0150] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0151] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0153] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0154] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0155] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0156] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0157] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal target detection method based on enhanced virtual points and adaptive distance grading, characterized in that, The method includes: Acquire regularized image data, extract semantic features from the regularized image data, and generate image BEV features based on the semantic features; Acquire regularized point cloud data, and classify the regularized point cloud data into near-range and far-range point clouds according to the distance classification threshold: Obtain the deformation grading curve, and perform initial grading of the regular point cloud data based on the deformation grading curve and the distance grading threshold to obtain the initial point cloud division result; Based on the initial point cloud partitioning results, the energy function is minimized. Adjusting the boundaries of the deformation grading curve yields the optimal grading curve, where, To the deformation grading curve Above, the gradient of the point cloud density function. Let be the point cloud density function, i.e., the point cloud in polar coordinates. Local density distribution under, The normal vector of the curve, i.e., the deformation grading curve. At angle The external normal vector at that location, For regularization weights, It is a transformation function; The optimal grading curve is used to divide regular point cloud data into near-range point cloud and far-range point cloud. Voxelization of near-range point clouds yields near-range BEV features, and voxelization of far-range point clouds yields far-range BEV features. The far-range point cloud BEV features are then corrected using the image BEV features to obtain corrected far-range point cloud BEV features, including the construction of a far-range virtual point set. Step 1: Project the distant point cloud onto the two-dimensional image plane to obtain a two-dimensional projected point set; Step 2: Generate image instance masks based on semantic features. For each image instance mask, the set of two-dimensional projection points falling into the image instance mask is used as the initial generated point set, and the initial virtual point set is set to an empty set. Step 3: When the image instance mask covers a preset number of pixel grids, use the center point coordinates of all pixel grids covered by the image instance mask to form an initial candidate point set. When a pixel grid has fallen into the projection point, delete the candidate point corresponding to the center point of the pixel grid from the initial candidate point set. Step 4: For each candidate point in the initial candidate point set, calculate the distance from each candidate point to the initial generated point set and sort them: Step 5: Select the candidate point with the largest distance in the distance sorting as the newly generated 2D point cloud, add the newly generated 2D point cloud to the initial generated point set and the initial virtual point set, and simultaneously delete the candidate point in the initial candidate point set; Repeat steps 1-5 until the number of point clouds in the initial virtual point set reaches the preset value to obtain the remote virtual point set; For each point cloud in the distant virtual point set, query the nearest projection point of each point cloud in the image instance mask, extract the reflectance of the nearest projection point, and associate the reflectance with each point cloud in the distant virtual point set to obtain a distant two-dimensional point cloud with depth information. The distant two-dimensional point cloud with depth information is back-projected back into the three-dimensional lidar coordinate system using the camera's intrinsic and extrinsic parameter matrix to obtain the distant point cloud. The image BEV features, near-range point cloud BEV features, and modified far-range point cloud BEV features are fused to obtain fused features. Based on the fused features, a preset detection head is used to perform target detection to obtain the target detection result.

2. The multimodal target detection method based on enhanced virtual points and adaptive distance grading as described in claim 1, characterized in that, The step of generating image BEV features based on semantic features includes: The depth distribution of each pixel in the regularized image data is estimated using a preset depth estimation algorithm, and the regularized image data is converted into a voxel probability volume based on the depth distribution. The voxel probability volume is transformed into three-dimensional coordinates using the camera's intrinsic and extrinsic parameters, resulting in a three-dimensional voxel mesh; Semantic features are embedded into a three-dimensional voxel grid to obtain a three-dimensional semantic voxel representation; The image BEV features are obtained by pooling the three-dimensional semantic voxel representation along the height direction.

3. A multimodal target detection device based on enhanced virtual points and adaptive distance grading, characterized in that, The apparatus is used to implement the multimodal target detection method based on enhanced virtual points and adaptive distance grading as described in any one of claims 1 to 2, the apparatus comprising: The image data processing module is used to acquire regular image data, extract semantic features from the regular image data, and generate image BEV features based on the semantic features. The point cloud data processing module is used to acquire regular point cloud data and classify the regular point cloud data into near point cloud and far point cloud according to the distance classification threshold. The near point cloud is voxelized to obtain near point cloud BEV features, and the far point cloud is voxelized to obtain far point cloud BEV features. The far point cloud BEV features are then corrected using image BEV features to obtain corrected far point cloud BEV features. The target detection module is used to fuse the image BEV features, near-range point cloud BEV features, and modified far-range point cloud BEV features to obtain fused features, and then use a preset detection head to perform target detection based on the fused features to obtain the target detection result.

4. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multimodal target detection method based on enhanced virtual points and adaptive distance grading as described in any one of claims 1 to 2.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal target detection method based on enhanced virtual points and adaptive distance grading as described in any one of claims 1 to 2.

Images