Target detection method, device and equipment and readable storage medium

Abstract

The invention provides a target detection method, device and equipment and a readable storage medium, and relates to the technical field of computer vision, and the method comprises the steps: receiving a laser radar point cloud and a false point cloud, carrying out the voxelization processing, carrying out the processing through a feature extraction network, obtaining a first multi-scale voxel feature and a second multi-scale voxel feature, and carrying out the detection of the first multi-scale voxel feature and the second multi-scale voxel feature; performing noise suppression and fusion on the second multi-scale voxel features to obtain multi-scale fusion voxel features; performing semantic enhancement on the multi-scale fusion voxel features to generate semantic enhanced interaction features; compressing the semantic enhanced interaction feature and the first voxel feature to a bird's-eye view space for fusion of a spatial domain feature and a frequency domain feature to obtain an enhanced bird's-eye view fusion feature; and performing three-dimensional target detection based on the enhanced aerial view fusion features to obtain a target detection result, thereby achieving the technical effect of remarkably improving the accuracy and robustness of the three-dimensional target detection in a complex scene and the perception ability of the three-dimensional target detection on shielded and small targets.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a target detection method, apparatus, device, and readable storage medium. Background Technology

[0002] Existing multimodal 3D target detection technologies improve perception performance by fusing laser point clouds with image-generated pseudo point clouds. However, several key limitations remain: First, pseudo point clouds lack structural semantic extraction capabilities. Existing methods often rely on standard sparse convolution or PointNet series networks, which have limited and fixed receptive fields. This makes it difficult to fully extract the rich structural information contained in pseudo points under controllable computational overhead, resulting in weak semantic representation and hindering overall detection performance improvement. Second, the pseudo point denoising process lacks geometric constraints. It mainly uses two-dimensional convolution or diffusion models to filter noise in the image space. Relying solely on image semantics can easily lead to false filtering, which weakens or even eliminates the true features of small targets such as pedestrians and cones while removing noise, severely affecting the detection accuracy of small targets. Third, spatial fusion mechanisms are prone to losing the semantics of occluded targets. Existing fusion methods are mostly limited to feature splicing or spatial attention. In this process, key semantic information such as high-frequency contours and edges of occluded targets are easily masked by low-frequency components of the background, leading to target feature degradation in occluded scenes and a significant decrease in detection performance.

[0003] Therefore, in view of the shortcomings of existing technical solutions, this application provides a target detection method. Summary of the Invention

[0004] Therefore, it is necessary to provide a target detection method, apparatus, device, and readable storage medium to address the aforementioned technical problems.

[0005] This application provides a target detection method, comprising: receiving a LiDAR point cloud and a pseudo-point cloud generated by image depth completion, and performing voxelization processing on them respectively to obtain a LiDAR voxel block and a pseudo-voxel block; processing the LiDAR voxel block through a first feature extraction network to obtain a first multi-scale voxel feature, and processing the pseudo-voxel block through a second feature extraction network to obtain a second multi-scale voxel feature; performing noise suppression on the second multi-scale voxel feature based on the geometric information in the first multi-scale voxel feature to obtain a denoised multi-scale pseudo-voxel feature; and fusing the first multi-scale voxel feature with the denoised multi-scale pseudo-voxel feature. The multi-scale fused voxel features are obtained by combining the three-dimensional coordinates and image calibration parameters corresponding to the multi-scale fused voxel features. The multi-scale fused voxel features are projected onto the two-dimensional image space to generate semantically enhanced interactive features. The semantically enhanced interactive features and the voxel features of the preset scale in the first multi-scale voxel features are compressed into the bird's-eye view space, and the spatial domain features and frequency domain features are extracted respectively. The spatial domain features and frequency domain features are fused through a dynamic attention weight mechanism to obtain the enhanced bird's-eye view fused features. Based on the enhanced bird's-eye view fused features, three-dimensional target detection is performed to obtain the three-dimensional bounding box and category information of the target.

[0006] This application also provides a target detection device, comprising: a first processing module for receiving a lidar point cloud and a pseudo point cloud generated by image depth completion, and performing voxelization processing on them respectively to obtain a lidar voxel block and a pseudo voxel block; a second processing module for processing the lidar voxel block through a first feature extraction network to obtain a first multi-scale voxel feature, and processing the pseudo voxel block through a second feature extraction network to obtain a second multi-scale voxel feature; a third processing module for performing noise suppression on the second multi-scale voxel feature based on the geometric information in the first multi-scale voxel feature to obtain a denoised multi-scale pseudo voxel feature; and a fourth processing module for processing the first multi-scale voxel feature... The first processing module fuses the multi-scale pseudo-voxel features with the denoised multi-scale voxel features to obtain multi-scale fused voxel features. The second processing module interacts the multi-scale fused voxel features with the semantic information of the two-dimensional image to generate semantically enhanced interactive features. The third processing module compresses the semantically enhanced interactive features and features of at least one scale from the first multi-scale voxel features into the bird's-eye view space, extracts spatial and frequency domain features respectively, and fuses the spatial and frequency domain features through a dynamic attention weight mechanism to obtain enhanced bird's-eye view fused features. The fourth processing module performs three-dimensional target detection based on the enhanced bird's-eye view fused features to obtain the target's three-dimensional bounding box and category information.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described target detection methods.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described target detection methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described target detection methods.

[0010] This application describes a method for obtaining laser voxel blocks and pseudo voxel blocks by voxelization of the received lidar point cloud and image depth completion-generated pseudo voxel cloud, respectively. The laser voxel blocks are processed by a first feature extraction network to obtain first multi-scale voxel features, and the pseudo voxel blocks are processed by a second feature extraction network to obtain second multi-scale voxel features. Based on the geometric information in the first multi-scale voxel features, noise suppression is applied to the second multi-scale voxel features to obtain denoised multi-scale pseudo voxel features. The first multi-scale voxel features and the denoised multi-scale pseudo voxel features are then fused to obtain multi-scale fused voxel features. Based on the three-dimensional coordinates and image calibration parameters corresponding to the multi-scale fused voxel features, the multi-scale fused voxel features are projected onto a two-dimensional image space to generate semantically enhanced interactive features. The semantically enhanced interactive features and the voxel features of the preset scale in the first multi-scale voxel features are compressed into the bird's-eye view space. Spatial and frequency domain features are extracted respectively, and the spatial and frequency domain features are fused through a dynamic attention weight mechanism to obtain the enhanced bird's-eye view fusion features. Based on the enhanced bird's-eye view fusion features, 3D target detection is performed to obtain the 3D bounding box and category information of the target. In this way, through deep fusion and complementarity, the accurate geometric information of the LiDAR point cloud and the rich semantic information of the image depth-complete pseudo point cloud are fully utilized, while effectively suppressing the noise of the pseudo point cloud. Through multi-scale feature extraction, cross-modal interaction, and joint feature enhancement in the spatial and frequency domains in the bird's-eye view space, the accuracy, robustness, and perception ability of 3D target detection in complex scenes are significantly improved. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a target detection method in one embodiment; Figure 2 (a) is a schematic diagram of the second multi-scale voxel feature extraction process of the target detection method in one embodiment; Figure 2(b) is a schematic diagram of the semantic enhancement convolution process of the object detection method in one embodiment; Figure 3 This is a schematic diagram of the noise suppression process of a target detection method in one embodiment; Figure 4 This is a schematic diagram of the voxel-pixel denoising interaction module of a target detection method in one embodiment; Figure 5 This is a schematic diagram of a voxel-pixel semantic enhancement module in a target detection method in one embodiment; Figure 6 This is a bird's-eye view fusion flowchart of a target detection method in one embodiment; Figure 7 This is a schematic diagram of the spatial multimodal semantic fusion module of a target detection method in one embodiment; Figure 8 This is a schematic diagram of the frequency domain semantic enhancement module of a target detection method in one embodiment; Figure 9 This is a schematic diagram of the spatial-frequency domain semantic refinement gating module of a target detection method in one embodiment; Figure 10 This is a schematic diagram of a three-dimensional target detection system according to one embodiment of the target detection method; Figure 11 This is a structural block diagram of a target detection device in one embodiment; Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] It should be understood that, in the description of this application, unless the context explicitly requires it, words such as "including" or "comprising" throughout the specification should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0015] It should also be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0016] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0017] In one embodiment, such as Figure 1 As shown, a target detection method is provided, including the following steps: S101: Receives the point cloud from the lidar and the pseudo point cloud generated by image depth completion, and performs voxelization processing on them respectively to obtain the lidar voxel block and the pseudo voxel block.

[0018] Here, the lidar point cloud refers to the LiDAR (Light Detection and Ranging) point cloud, which is a collection of a large number of discrete three-dimensional coordinate data points on the surface of a target object or environment obtained by using a lidar system to emit laser pulses and measure the time from emission to reflection back by an object.

[0019] Here, pseudo-point cloud refers to 3D point cloud data that is not directly scanned and measured by physical ranging sensors such as lidar, but is calculated and generated from other types of data through algorithms.

[0020] Depth completion can include sparse-to-dense depth completion, depth estimation of monocular / binocular images, and multi-task learning and feature fusion.

[0021] Specifically, it receives LiDAR point cloud data collected by multimodal sensors in autonomous driving scenarios, and pseudo point cloud data generated offline by monocular or binocular image depth completion.

[0022] Specifically, using one or more RGB (Red Green Blue) images, combined with sparse points from low-cost LiDAR or binocular matching, the algorithm predicts the depth value of each pixel in the image, and then back-projects the pixels with depth into three-dimensional space to generate a dense set of three-dimensional points similar to LiDAR point clouds.

[0023] Here, voxelization refers to the process of transforming and quantizing a continuous, irregular, and unstructured three-dimensional space (or its discrete point cloud data) into a structured representation composed of tiny, regular, and discrete three-dimensional cubic grids (i.e., voxels). A voxel, short for "volume pixel," is the smallest indivisible data unit in three-dimensional space. Each voxel has a fixed spatial size and occupies a definite position in space.

[0024] Voxelization can include binary placeholder voxelization, statistical feature voxelization, and intercept voxelization.

[0025] Specifically, lidar point clouds and pseudo point clouds are represented as follows: and All of these are voxelized into a G×W×H mesh (G typically represents the number of meshes in the depth direction, W represents the number of meshes in the width direction, and H represents the number of meshes in the height direction), where, and These represent the feature channels for lidar points and pseudo-points, respectively, where N is the number of lidar points and pseudo-points. Next, for each non-empty voxel, the feature average value of all lidar points or pseudo-points within it is calculated to generate a lidar voxel block. With pseudo-voxel blocks ,in and These are the feature channels for the laser voxel block and the pseudo voxel block, respectively.

[0026] S102: The laser voxel block is processed by the first feature extraction network to obtain the first multi-scale voxel feature, and the pseudo voxel block is processed by the second feature extraction network to obtain the second multi-scale voxel feature.

[0027] Here, the voxel block is the aggregated result of all point features within the container.

[0028] The first feature extraction network may include SPConv (Sparse Convolution).

[0029] Among them, multiple scales can include 1×, 2×, 4× and 8× scales.

[0030] Specifically, the laser voxel block is input into SPConv to generate LiDAR voxel features at scales of 1×, 2×, 4× and 8×, respectively.

[0031] Here, the second feature extraction network employs a convolutional structure with an expanded receptive field to enhance semantic extraction. The second feature extraction network can include semantically enhanced convolutions, traditional sparse convolutions, deformable convolutions, etc.

[0032] S103: Based on the geometric information in the first multi-scale voxel features, noise suppression is performed on the second multi-scale voxel features to obtain the denoised multi-scale pseudo voxel features.

[0033] Here, geometric information refers to the objective attributes that describe the shape, structure, surface, and spatial location of an object.

[0034] Geometric information can include spatial coordinates, spatial distribution and density, surface normal vectors, curvature, depth, local structural features, etc.

[0035] The denoised multi-scale pseudo-voxel features can include 4x denoised pseudo-voxel features and 8x denoised pseudo-voxel features.

[0036] Each scale corresponds to a denoised multi-scale pseudo-voxel feature, and the denoised multi-scale pseudo-voxel features may differ for different scales. For example, the denoised multi-scale pseudo-voxel features at 4x and 8x may be different.

[0037] Here, noise suppression refers to using reliable geometric information from real LiDAR to correct, filter, or reduce unrealistic, unreasonable, or unreliable parts of pseudo point clouds.

[0038] Noise can include shape distortion, floating points, uneven surfaces, and incorrect object outlines.

[0039] In one embodiment, the first and second multiscale voxel features use voxel features of 4× and 8× scales.

[0040] S104: The first multi-scale voxel feature is fused with the denoised multi-scale pseudo voxel feature to obtain the multi-scale fused voxel feature.

[0041] Among them, multiple scales can include 4x and 8x.

[0042] Fusion can include fusion between laser voxel features and pseudo voxel features of the same scale, as well as fusion between different scales.

[0043] For example, multi-scale fusion can be performed by fusing the centroid feature of a 4x-scale laser voxel with the denoised pseudo-voxel feature at the 4x scale to obtain a first feature, fusing the centroid feature of an 8x-scale laser voxel with the denoised pseudo-voxel feature at the 8x scale to obtain a second feature, and fusing the first feature and the second feature to obtain a multi-scale fused voxel feature.

[0044] One approach is to use dynamic weights for fusion.

[0045] Specifically, the centroid features of the first multi-scale voxel features are calculated through a fully connected layer. The weights of the centroid features of the first multi-scale voxel features and the denoised multi-scale pseudo voxel features are calculated through activation functions, concatenation, and fully connected layers. The features are then fused according to the weights.

[0046] For example, in a laser voxel block middle, For the 3D coordinates of the laser voxel, For laser voxel features, j is the voxel index, and N is the total number of voxels; while in pseudo voxel blocks... middle The 3D coordinates of the pseudo-voxel. This is a characteristic of pseudovoxels. (Through...) Calculate the centroid features of voxel features at the 4x scale. ,pass and The weights are calculated and then... The fusion between the centroid features at the 4x scale and the denoised 4x scale pseudo-voxel features is calculated. Similarly, the fusion between the centroid features at the 8x scale and the denoised 8x scale pseudo-voxel features is calculated using the same formula. The fusion results of 4x and 8x scales are combined to obtain multi-scale fused voxel features. Here, σ is the activation function, FC is the fully connected layer, CONACT is the concatenation operation, and (:,1) and (:,2) are indexing operations. The multi-scale pseudo-voxel features after denoising. These are denoised fusion voxel features at a 4× scale. This is a multi-scale fusion voxel feature.

[0047] S105: Based on the three-dimensional coordinates and image calibration parameters corresponding to the multi-scale fused voxel features, the multi-scale fused voxel features are projected onto the two-dimensional image space to generate semantically enhanced interactive features.

[0048] Here, the image calibration parameters are a set of precise mathematical parameters used to describe the mapping relationship between points in the three-dimensional world coordinate system and points in the two-dimensional image pixel coordinate system.

[0049] The image calibration parameters may include intrinsic parameters, extrinsic parameters, and distortion coefficients.

[0050] Here, the two-dimensional image space is a feature map space that is aligned one-to-one with the pixels of the original image and is used to carry three-dimensional feature information.

[0051] S106: Compress the semantically enhanced interactive features and the voxel features of the preset scale in the first multi-scale voxel features into the bird's-eye view space, extract the spatial domain features and frequency domain features respectively, and fuse the spatial domain features and frequency domain features through a dynamic attention weight mechanism to obtain the enhanced bird's-eye view fusion features.

[0052] Here, a bird's-eye view (BEV) is a view of the ground from a high vantage point, similar to a traditional two-dimensional map, which can clearly show the position, layout, and relative relationships of objects in a scene.

[0053] Here, spatial features refer to features directly observed or calculated within the spatial or temporal coordinate system of the original data itself. The focus is on the distribution, shape, statistical relationships, and location-dependent patterns of numerical values ​​within a local region.

[0054] Here, frequency domain characteristics refer to the features that emerge after data is transformed from the spatial domain to the frequency domain through mathematical transformations (such as Fourier transform, wavelet transform, and discrete cosine transform). The focus is on the periodicity of the signal, the rate of change (frequency), and the energy distribution of different frequency components.

[0055] Among them, compression can be performed along the voxel Z-axis.

[0056] Among them, the voxel features of the preset scale can be voxel features of the 8x scale.

[0057] Here, dynamic attention weighting is a computational method that enables neural networks to automatically and in real time calculate and allocate their internal processing resources to different features, locations, or channels based on the content of the current input data.

[0058] The weights are not static parameters that are pre-set or fixed through training, but are generated in real time based on different input samples during each forward propagation.

[0059] S107: Based on the enhanced bird's-eye view fusion features, perform 3D target detection to obtain the target's 3D bounding box and category information.

[0060] It should be noted that this application fully utilizes the precise geometric information of LiDAR point clouds and the rich semantic information of image depth-completed pseudo point clouds through deep fusion and complementation, while effectively suppressing the noise of pseudo point clouds. Through multi-scale feature extraction, cross-modal interaction, and joint feature enhancement in the spatial and frequency domains in the bird's-eye view space, it significantly improves the accuracy, robustness, and perception ability of occlusion and small targets in complex scenes for 3D target detection.

[0061] In some specific implementations, the pseudo-voxel blocks are processed by a second feature extraction network to obtain second multi-scale voxel features, including: The pseudo-voxel features at the initial scale are input into the first branch network to obtain the first intermediate features, wherein the first branch network includes two layers of sparse convolutional blocks. The first intermediate feature is input into the second branch network to obtain the second intermediate feature with enhanced semantics, wherein the second branch network includes sparse convolutional blocks; By fusing the first and second intermediate features, pseudo-voxel features at the next scale are obtained; The above steps are executed iteratively to generate a second multi-scale voxel feature containing multiple scales.

[0062] Here, the sparse convolutional block contains SpConv3d (sparse convolution), BatchNorm (normalization), and ReLU (activation function) operations, which can be called SBR (i.e., SpConv3d BatchNorm ReLU).

[0063] Here, a double-layer sparse convolutional block is a composite module consisting of two sparse convolutional blocks. It also includes SpConv3d (sparse convolution), BatchNorm (normalization), and ReLU (activation function) operations, and can be called DSBR (i.e., DoubleSpConv3d BatchNorm ReLU).

[0064] Specifically, the pseudo-voxel block The iterative input is fed into SEConv (SemanticEnhancement Convolution) to obtain pseudo-voxel features at scales of 1×, 2×, 4×, and 8×. .

[0065] For example, Figure 2 (a) is a schematic diagram of the second multi-scale voxel feature extraction process in an embodiment of this application, as shown below. Figure 2 As shown in (a), the second multi-scale voxel feature extraction process in this application includes: voxelizing the pseudo point cloud to obtain a 1x scale semantic enhancement convolution, obtaining a 2x scale semantic enhancement convolution from the 1x scale, obtaining a 4x scale semantic enhancement convolution from the 2x scale, obtaining an 8x scale semantic enhancement convolution from the 4x scale, and generating 8x scale pseudo voxel features.

[0066] Specifically, Figure 2 (b) is a schematic diagram of the semantically enhanced convolution process in the embodiments of this application, as shown below. Figure 2 As shown in (b), the semantic enhancement convolution process in this application includes: targeting 1× scale pseudo-voxel features ,pass Generate pseudo-sparse features at a scale of 2×. Where DSBR(·) represents a two-layer sparse convolutional block SBR(·), which includes SpConv3d, BatchNorm, and ReLU operations, targeting ,pass This yields pseudo-semantic features at a 2× scale. ,pass CONCAT(·) is a splicing operation that yields pseudo-voxel features at a 2× scale. .

[0067] In this way, through iterative fusion, pseudo-point cloud features can be extracted progressively and at multiple levels, enhancing the discriminative power of features while retaining effective information.

[0068] In some specific implementations, based on the geometric information in the first multi-scale voxel features, noise suppression is applied to the second multi-scale voxel features to obtain denoised multi-scale pseudo-voxel features, including: The laser centroid is calculated based on the spatial position of the points within the voxels in the first multi-scale voxel feature. Calculate the spatial distance between the laser centroid and the pseudopoints in the pseudo-voxel block; Filter pseudo-point features with spatial distances within a preset range, aggregate the filtered pseudo-point features, and combine them with laser centroid features to generate denoised multi-scale pseudo-voxel features.

[0069] The laser centroid is obtained by calculating the average value of the structured positions of M LiDAR points within the voxel block.

[0070] For example, by Calculate, where c j Let M be the three-dimensional coordinates of the centroid, M be the number of LiDAR points, and l be the number of points. i The position of each point.

[0071] Spatial distance can be Manhattan distance, Euclidean distance, Minkowski distance, Mahalanobis distance, cosine distance, etc.

[0072] Specifically, through Calculate the Manhattan distance between the laser centroid and the pseudopoint. Sample pseudo-points with Manhattan distances less than a preset value, through The denoised multi-scale pseudo-voxel features were calculated. Among them, (L x ,L y ,L z ) and (P x ,P y ,P z The coordinates of the laser centroid and the pseudopoint are shown below. and The coordinates and features of the sampled pseudopoints are given, MLP(·) is a multilayer perceptron, and MaxPool(·) is the max pooling operation.

[0073] In this way, outlier noise points generated by the depth completion algorithm that deviate significantly from the real geometric structure are effectively filtered out, thereby significantly improving the quality and credibility of pseudo point cloud features.

[0074] In some specific implementations, based on the three-dimensional coordinates corresponding to the multi-scale fused voxel features and image calibration parameters, the multi-scale fused voxel features are projected onto a two-dimensional image space to generate semantically enhanced interactive features, including: Based on the voxelization parameters, data augmentation parameters, and camera calibration parameters, the three-dimensional coordinates corresponding to the multi-scale fused voxel features are projected onto the two-dimensional image pixel coordinates. Based on the pixel coordinates of a two-dimensional image, multi-scale fused voxel features are processed using sparse convolution tensors and a two-dimensional convolutional network to generate the first two-dimensional associated features. A three-dimensional convolutional network is used to process multi-scale fused voxel features to generate a second two-dimensional associated feature. By integrating the first two-dimensional association features and the second two-dimensional association features, semantically enhanced interactive features are obtained.

[0075] Here, voxelization parameters are a set of configuration parameters used to control mesh size, resolution, and feature computation when discretizing a continuous 3D point cloud into a regular mesh (voxels). Examples include voxel size, point cloud extent, and mesh dimension.

[0076] Here, data augmentation parameters are a set of configurations used during training to randomly transform the original data (point cloud, image) to increase data diversity and improve the model's generalization ability. These can include point cloud augmentation parameters, image augmentation parameters, and simultaneous augmentation parameters.

[0077] Here, camera calibration parameters are mathematical parameters that describe the camera's imaging geometry model and are used to establish a precise mapping relationship between 3D world points and 2D image pixels. These can include intrinsic parameters, extrinsic parameters, and distortion parameters, etc.

[0078] Here, a sparse convolution tensor is a special data structure used for efficient storage and processing of sparse, regular data.

[0079] Among them, a two-dimensional convolutional network can be a two-dimensional sparse convolutional block.

[0080] Among them, the first two-dimensional association feature can be a voxel-pixel semantic feature, which is a dense two-dimensional feature map that is precisely aligned with the image pixels by mapping and aggregating the features in the three-dimensional voxel space onto the two-dimensional image pixel plane according to the geometric projection relationship. Its core is an image-level semantic feature that deeply integrates three-dimensional geometric context information.

[0081] Among them, the second two-dimensional association feature can be a voxel-pixel sparse feature, which is an intermediate, sparse data structure that exists before pixel position aggregation is completed during the process of projecting three-dimensional voxel features onto a two-dimensional image plane. It records the two-dimensional pixel position to which each three-dimensional voxel feature is projected, but these features projected to the same pixel have not yet been fused.

[0082] Specifically, the voxelization parameter φ(·), data augmentation parameter χ(·), and calibration parameter ψ(·) are obtained, and the three-dimensional coordinates of the denoised multi-scale pseudo-voxel features are... ,pass Projected onto two-dimensional image pixels, generating projected two-dimensional coordinates. .

[0083] Specifically, through voxel-pixel semantic features are obtained. (i.e., the first two-dimensional correlation feature). Among them, SPConvTensor(·) is a sparse convolution tensor, and SBR(·) is a two-dimensional sparse convolution block.

[0084] Specifically, through voxel-pixel sparse features are obtained. (i.e., the second two-dimensional correlation feature). Among them, SBR(·) is a three-dimensional sparse convolutional block.

[0085] Specifically, through This yields semantically enhanced interactive features. , where CONCAT(·) is the splicing operation.

[0086] In this way, establishing the relationship between three-dimensional voxel features and two-dimensional image pixels provides a channel for introducing image semantic information. By adopting a dual-path processing method that combines sparse convolution with standard three-dimensional / two-dimensional convolution, the information of the two modalities can be more fully explored and associated, so that the generated interactive features are deeply integrated with geometric structure and image semantics, enhancing the understanding of object appearance, texture and other attributes.

[0087] In some specific implementations, the semantically enhanced interactive features and the voxel features of a preset scale in the first multi-scale voxel features are compressed into the bird's-eye view space, and spatial and frequency domain features are extracted respectively. The spatial and frequency domain features are then fused using a dynamic attention weighting mechanism to obtain enhanced bird's-eye view fusion features, including: The semantically enhanced interactive features and the voxel features at a preset scale are compressed into the bird's-eye view space to obtain interactive bird's-eye view features and laser bird's-eye view features, respectively. Multi-scale spatial fusion and reconstruction of interactive bird's-eye view features and laser bird's-eye view features are performed to generate spatially fused bird's-eye view features. Frequency domain enhancement processing is performed on the interactive bird's-eye view features and the laser bird's-eye view features respectively to generate interactive frequency domain enhanced bird's-eye view features and laser frequency domain enhanced bird's-eye view features; By employing a dynamic attention weighting mechanism, the features of the spatial fusion bird's-eye view, the interactive frequency domain enhanced bird's-eye view, and the laser frequency domain enhanced bird's-eye view are weighted and fused to generate enhanced bird's-eye view fusion features.

[0088] Here, frequency domain enhancement processing is a signal processing technique that transforms a signal from the original domain (time domain / spatial domain) to the frequency domain, selectively enhances or suppresses specific frequency components of the signal in the frequency domain to improve signal quality or extract key features, and finally transforms it back to the original domain.

[0089] Frequency domain enhancement enhances the semantic representation of foreground targets by extracting frequency domain features (such as details and contours).

[0090] Specifically, semantic interaction features and laser voxel features at a preset scale are compressed into the bird's-eye view space to obtain interactive bird's-eye view features and laser bird's-eye view features; these features are then processed through downsampling convolutional blocks to obtain multi-scale bird's-eye view features; lightweight convolutional blocks are used to process these features to obtain multi-scale fused bird's-eye view features; lightweight convolutional blocks and a heavily parameterized network are used to process these features to obtain multi-scale fused semantic features; and finally, the multi-scale fused semantic features and multi-scale fused bird's-eye view features are aggregated. The aerial view features are used to determine the spatial features. Fourier transform is applied to the laser aerial view features and interactive aerial view features to obtain the laser frequency domain aerial view features and interactive frequency domain aerial view features. The laser aerial view features are calculated using activation functions and then fused to obtain the laser frequency domain features. Similarly, the interactive frequency domain aerial view features are calculated using activation functions and then fused to obtain the interactive frequency domain features. Finally, the spatial features, laser frequency domain features, and interactive frequency domain features are fused to determine the target aerial view features.

[0091] The downsampling convolutional block includes Conv (convolution), BatchNorm (normalization), and ReLU (activation function), which can be called CBR (i.e., Conv BatchNorm ReLU).

[0092] Specifically, LiDAR voxel features at the 8x scale Interactive features with semantic enhancement BEV features of LiDAR were obtained by voxel Z-axis compression. and interactive BEV features .

[0093] Specifically, through Multi-scale BEV features were obtained. In the spatial domain, semantic information of targets at different scales in BEV is expressed, where CBR(·) is a downsampled convolutional block and i is the size of the multi-scale BEV feature.

[0094] Specifically, through Multi-scale fused BEV features were obtained. This enhances the semantic relevance within the feature channels, where CBS(·) is a 1×1 lightweight convolutional block. The 1×1 lightweight convolutional block includes Conv, BatchNorm, and SiLU operations.

[0095] Specifically, through Multi-scale fused semantic features are obtained. To further enhance semantic expression capabilities, the convolutional receptive field for multi-scale BEV features is expanded, and RepVGG(·) is a reparameterized network.

[0096] Specifically, through The spatial multi-scale fusion of BEV features was obtained. ,in, For feature fusion.

[0097] Specifically, through The features of the airspace fusion bird's-eye view were obtained. UMBR(·) is a modulated deformable convolution, which includes MdeformConv, BatchNorm and ReLU operations, and CBR(·) is a downsampled convolution block.

[0098] Specifically, regarding the BEV feature B of LiDAR L ,pass The frequency domain BEV characteristics of LiDAR were obtained. , where FFT(·) represents Fourier transform and IFFT(·) represents inverse Fourier transform.

[0099] Specifically, through The laser frequency domain enhanced bird's-eye view features were obtained. ,in, For feature fusion, σ is the sigmoid function.

[0100] Specifically, the interactive BEV features are calculated using the same method to obtain the interactive frequency domain enhanced bird's-eye view features.

[0101] Thus, multi-scale spatial fusion and reconstruction enables the capture of targets of different sizes and contextual information; frequency domain enhancement processing can analyze and enhance features from the frequency dimension, which may help extract global structural patterns, suppress high-frequency noise, or enhance key frequency components; through the fusion and enhancement of semantic information in the spatial and frequency domains, the ability to perceive targets at multiple scales and the detection effect of occluded targets are improved.

[0102] In some specific implementations, a dynamic attention weighting mechanism is used to weight and fuse the spatial fusion bird's-eye view features, the interactive frequency domain enhanced bird's-eye view features, and the laser frequency domain enhanced bird's-eye view features to generate enhanced bird's-eye view fusion features, including: Attention weights were calculated for the features of the spatial fusion bird's-eye view, the interactive frequency domain enhanced bird's-eye view, and the laser frequency domain enhanced bird's-eye view, respectively, to obtain the spatial attention weight, the first frequency domain attention weight, and the second frequency domain attention weight. The spatial fusion bird's-eye view features, interactive frequency domain enhanced bird's-eye view features, and laser frequency domain enhanced bird's-eye view features are modulated by spatial domain attention weights, first frequency domain attention weights, and second frequency domain attention weights, respectively. The modulated features are then fused to obtain enhanced bird's-eye view fused features.

[0103] Specifically, for laser frequency domain enhanced bird's-eye view features Interactive frequency domain enhancement of bird's-eye view features The aerial view features (BS) are integrated with the airspace and are obtained through... , and Obtain attention features , and .

[0104] Specifically, through , and To obtain attention weights, , and .

[0105] Specifically, through , and , to obtain fused attention features , and .

[0106] Specifically, through This results in enhanced bird's-eye view fusion feature B.

[0107] In this way, by adaptively adjusting the importance weights between spatial domain features and frequency domain features from two sources, better feature fusion is achieved, thereby generating more comprehensive and discriminative BEV fusion features, more flexibly integrating information from different domains and data sources, and making the fusion strategy more adaptive and expressive.

[0108] In some specific implementations, 3D target detection is performed based on enhanced bird's-eye view fusion features to obtain the target's 3D bounding box and category information, including: 3D candidate regions are generated on enhanced bird's-eye view fusion features using a region proposal network; Multimodal features within a 3D candidate region are aggregated and refined using an attention-based neural network. Based on the refined features, predict the target's 3D bounding box and category information.

[0109] Here, the Region Proposal Network (RPN) is a deep learning network used to quickly generate candidate target regions. It directly predicts regions that may contain targets on the feature map and generates preliminary bounding boxes and confidence scores for each region.

[0110] The attention mechanism can be a Transformer.

[0111] Specifically, the enhanced bird's-eye view fusion feature B generates 3D candidate regions through RPN, and samples N grid points (G) in the 3D candidate regions. x G y G z The multimodal grid point features F are obtained by summing the multimodal BEV features within the three-dimensional candidate region. G ,pass Continuously refine to obtain the output feature F out In this context, NMLP(·) represents a nonlinear multilayer perceptron, and Attention(Q, K, V) is a Transformer.

[0112] Specifically, ,in, , , Where d is the scaling factor, W q W k and W v For the predefined initial weight vector, E pos This is a position-encoded vector.

[0113] Specifically, the output feature F out The input consists of two fully connected layers, which predict the 3D object category and bounding box, respectively.

[0114] This improves the accuracy of 3D bounding box regression and classification.

[0115] In one embodiment, Figure 3 This is a schematic diagram of the noise suppression process in the embodiments of this application, such as... Figure 3 As shown, the noise suppression process in this application includes: inputting 8x-scale pseudo-voxel features and 4x-scale LiDAR voxel features into the voxel-pixel denoising interaction module for processing, and then processing them through downsampling convolution blocks and sparse convolution blocks to obtain the first feature; inputting the 4x-scale pseudo-voxel features and 4x-scale LiDAR voxel features into the voxel-pixel denoising interaction module for processing, and then processing them through downsampling convolution blocks to obtain the second feature; adding the first feature and the second feature element by element, and inputting them into the voxel-pixel semantic enhancement module for processing to obtain the voxel-pixel semantic interaction feature (i.e., the semantically enhanced interaction feature).

[0116] In one embodiment, Figure 4 This is a schematic diagram of the voxel-pixel denoising interaction module in an embodiment of this application, as shown below. Figure 4 As shown, the voxel-pixel denoising interaction module in this application includes: inputting multimodal voxel features into a multimodal fusion module for voxel-level denoising fusion, inputting them into a fully connected layer for feature transformation, performing feature enhancement through a downsampling convolutional block, performing voxel-pixel sampling on the multimodal voxel features to achieve 3D to 2D projection alignment, performing post-projection feature adjustment through a fully connected layer, performing feature compression through a fully connected layer, performing downsampling through a max pooling layer, performing feature dimensionality reduction through a fully connected layer, performing further abstraction through a max pooling layer, and performing final feature extraction through a downsampling convolutional block to obtain denoised pseudo-voxel features.

[0117] In one embodiment, Figure 5 This is a schematic diagram of the voxel-pixel semantic enhancement module in an embodiment of this application, as shown below. Figure 5 As shown, the voxel-pixel semantic enhancement module in this application includes: performing SBR (i.e., sparse convolutional block) processing on the denoised multi-scale pseudo-voxel features, and simultaneously performing voxel-pixel projection matrix mapping and SBR processing on the denoised multi-scale pseudo-voxel features, and concatenating the two to obtain semantically enhanced interactive features.

[0118] In one embodiment, Figure 6 This is a schematic diagram of the bird's-eye view fusion process in the embodiments of this application, such as... Figure 6As shown, the bird's-eye view fusion process in this application includes: processing the 8x scale LiDAR voxel features and semantically enhanced interactive features through the frequency domain semantic enhancement module, and inputting both into the spatial domain multimodal semantic fusion module. The three processing results are then input into the spatial-frequency domain semantic refinement gating module to obtain the enhanced bird's-eye view fusion features.

[0119] In one embodiment, Figure 7 This is a schematic diagram of the spatial multimodal semantic fusion module in an embodiment of this application, as shown below. Figure 5 As shown, the spatial multimodal semantic fusion module in this application includes: processing LiDAR voxel features and semantically enhanced interactive features of multiple sizes through CBR, processing features of the same size through RepVGG, and then processing them separately through UMBR, concatenating the three processing results, and inputting them into CBR to obtain spatial fusion bird's-eye view features.

[0120] In one embodiment, Figure 8 This is a schematic diagram of the frequency domain semantic enhancement module in the embodiments of this application, as shown below. Figure 8 As shown, the frequency domain semantic enhancement module in this application includes: converting the input features from spatial domain features to frequency domain through Fourier transform, enhancing the frequency domain features through CBR, further processing the frequency domain features through CBR, converting the processed frequency domain features back to the spatial domain through inverse Fourier transform, and post-processing the spatial domain features through CBR.

[0121] In one embodiment, Figure 9 This is a schematic diagram of the spatial-frequency domain semantic refinement gating module in an embodiment of this application, as shown below. Figure 9 As shown, the spatial-frequency domain semantic refinement gating module in this application includes: a first branch performs CBR and MP processing (MaxPool, i.e., maximum pooling operation) on the input and outputs scale 1 features; a second branch performs CBR, MP, FC, and CBR processing on the input and outputs scale 2 features; a third branch performs CBR and MP processing on the input and outputs scale 3 features, and the features of the three scales are fused.

[0122] In one embodiment, Figure 10 This is a schematic diagram of the three-dimensional target detection system in the embodiments of this application, such as... Figure 9 As shown, the 3D target detection system in this application includes: input data, multimodal voxel feature extraction, voxel-pixel denoising and sparse semantic interaction, BEV semantic enhancement and thinning fusion, 3D detection head and result output.

[0123] Specifically, the multimodal voxel feature extraction module extracts multi-scale features of LiDAR voxels and pseudo-voxels through sparse convolution and semantic enhancement convolution, respectively; the voxel-pixel denoising and sparse semantic interaction module uses the accurate location of the LiDAR centroid to suppress pseudo-point noise and combines image pixel semantic information to enhance small target features; the BEV semantic enhancement and refinement fusion module refines semantic representation through joint spatial-frequency fusion to improve the detection effect of occluded targets; the 3D detection head generates and refines candidate regions based on fused features, and outputs target categories and 3D bounding boxes. Given a training dataset, the LiDAR point cloud data collected by the sensor and the pseudo-point cloud data generated by image completion can be preprocessed and used for model training. By expanding the scale of training data through data augmentation techniques, the detection accuracy and overall performance of the system for small targets and occluded targets can be further improved.

[0124] It should be understood that, although Figures 1-10 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-10 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0125] In one embodiment, such as Figure 11As shown, a target detection device is provided, comprising: a first processing module 1101, used to receive a laser radar point cloud and a pseudo point cloud generated by image depth completion, and perform voxelization processing on them respectively to obtain laser voxel blocks and pseudo voxel blocks; a second processing module 1102, used to process the laser voxel blocks through a first feature extraction network to obtain first multi-scale voxel features, and process the pseudo voxel blocks through a second feature extraction network to obtain second multi-scale voxel features; a third processing module 1103, used to perform noise suppression on the second multi-scale voxel features based on the geometric information in the first multi-scale voxel features to obtain denoised multi-scale pseudo voxel features; and a fourth processing module 1104, used to process the first multi-scale voxel... The first processing module 1105 is used to fuse the multi-scale pseudo-voxel features with the denoised multi-scale voxel features to obtain multi-scale fused voxel features; the second processing module 1106 is used to interact the multi-scale fused voxel features with the semantic information of the two-dimensional image to generate semantically enhanced interactive features; the third processing module 1106 is used to compress the semantically enhanced interactive features and features of at least one scale of the first multi-scale voxel features into the bird's-eye view space, extract the spatial domain features and frequency domain features respectively, and fuse the spatial domain features and frequency domain features through a dynamic attention weight mechanism to obtain enhanced bird's-eye view fused features; the fourth processing module 1107 is used to perform three-dimensional target detection based on the enhanced bird's-eye view fused features to obtain the three-dimensional bounding box and category information of the target.

[0126] As a preferred implementation, in this embodiment of the application, the second processing module 1102 is specifically used to: input the pseudo-voxel features of the initial scale into the first branch network to obtain the first intermediate features, wherein the first branch network includes a two-layer sparse convolution block; The first intermediate feature is input into the second branch network to obtain the second intermediate feature with enhanced semantics, wherein the second branch network includes sparse convolutional blocks; By fusing the first and second intermediate features, pseudo-voxel features at the next scale are obtained; The above steps are executed iteratively to generate a second multi-scale voxel feature containing multiple scales.

[0127] As a preferred implementation, in this embodiment of the application, the third processing module 1103 is specifically used to: calculate the laser centroid based on the spatial position of the voxel points in the first multi-scale voxel features; Calculate the spatial distance between the laser centroid and the pseudopoints in the pseudo-voxel block; Filter pseudo-point features with spatial distances within a preset range, aggregate the filtered pseudo-point features, and combine them with laser centroid features to generate denoised multi-scale pseudo-voxel features.

[0128] As a preferred implementation, in this embodiment of the application, the fifth processing module 1105 is specifically used to: project the three-dimensional coordinates corresponding to the multi-scale fused voxel features onto the two-dimensional image pixel coordinates according to the voxelization parameters, data augmentation parameters and camera calibration parameters; Based on the pixel coordinates of a two-dimensional image, multi-scale fused voxel features are processed using sparse convolution tensors and a two-dimensional convolutional network to generate the first two-dimensional associated features. A three-dimensional convolutional network is used to process multi-scale fused voxel features to generate a second two-dimensional associated feature. By integrating the first two-dimensional association features and the second two-dimensional association features, semantically enhanced interactive features are obtained.

[0129] As a preferred implementation, in this embodiment of the application, the sixth processing module 1106 is specifically used to: compress the semantically enhanced interactive features and the voxel features of the preset scale into the bird's-eye view space to obtain interactive bird's-eye view features and laser bird's-eye view features respectively. Multi-scale spatial fusion and reconstruction of interactive bird's-eye view features and laser bird's-eye view features are performed to generate spatially fused bird's-eye view features. Frequency domain enhancement processing is performed on the interactive bird's-eye view features and the laser bird's-eye view features respectively to generate interactive frequency domain enhanced bird's-eye view features and laser frequency domain enhanced bird's-eye view features; By employing a dynamic attention weighting mechanism, the features of the spatial fusion bird's-eye view, the interactive frequency domain enhanced bird's-eye view, and the laser frequency domain enhanced bird's-eye view are weighted and fused to generate enhanced bird's-eye view fusion features.

[0130] As a preferred implementation, in this embodiment of the application, the sixth processing module 1106 is further used to: calculate the attention weights of the spatial fusion bird's-eye view features, the interactive frequency domain enhanced bird's-eye view features, and the laser frequency domain enhanced bird's-eye view features respectively, to obtain the spatial attention weight, the first frequency domain attention weight, and the second frequency domain attention weight; The spatial fusion bird's-eye view features, interactive frequency domain enhanced bird's-eye view features, and laser frequency domain enhanced bird's-eye view features are modulated by spatial domain attention weights, first frequency domain attention weights, and second frequency domain attention weights, respectively. The modulated features are then fused to obtain enhanced bird's-eye view fused features.

[0131] As a preferred implementation, in this embodiment of the application, the seventh processing module 1107 is specifically used to: generate three-dimensional candidate regions on the enhanced bird's-eye view fusion features through a region proposal network; Multimodal features within a 3D candidate region are aggregated and refined using an attention-based neural network. Based on the refined features, predict the target's 3D bounding box and category information.

[0132] For specific limitations regarding the target detection device, please refer to the limitations of the target detection method above, which will not be repeated here. Each module in the aforementioned target detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0133] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a target detection method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0134] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0135] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described target detection method embodiments at runtime.

[0136] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0137] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described target detection method embodiments.

[0138] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described target detection method embodiments.

[0139] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] The target detection method, apparatus, table parsing system, and computer device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A target detection method, characterized in that, The method includes: The laser radar point cloud and the pseudo point cloud generated by image depth completion are received and voxelized respectively to obtain laser voxel blocks and pseudo voxel blocks. The laser voxel block is processed by a first feature extraction network to obtain a first multi-scale voxel feature, and the pseudo voxel block is processed by a second feature extraction network to obtain a second multi-scale voxel feature. Based on the geometric information in the first multi-scale voxel feature, noise suppression is performed on the second multi-scale voxel feature to obtain the denoised multi-scale pseudo voxel feature. The first multi-scale voxel feature is fused with the denoised multi-scale pseudo voxel feature to obtain the multi-scale fused voxel feature. Based on the three-dimensional coordinates and image calibration parameters corresponding to the multi-scale fused voxel features, the multi-scale fused voxel features are projected onto the two-dimensional image space to generate semantically enhanced interactive features. The semantically enhanced interactive features and the voxel features of the preset scale in the first multi-scale voxel features are compressed into the bird's-eye view space, and the spatial domain features and frequency domain features are extracted respectively. The spatial domain features and the frequency domain features are fused through a dynamic attention weight mechanism to obtain the enhanced bird's-eye view fusion features. Based on the enhanced bird's-eye view fusion features, 3D target detection is performed to obtain the target's 3D bounding box and category information.

2. The target detection method according to claim 1, characterized in that, The step of processing the pseudo-voxel block through a second feature extraction network to obtain second multi-scale voxel features includes: The pseudo-voxel features at the initial scale are input into the first branch network to obtain the first intermediate features, wherein the first branch network includes two layers of sparse convolutional blocks. The first intermediate feature is input into the second branch network to obtain a second intermediate feature with enhanced semantics, wherein the second branch network includes sparse convolutional blocks; By fusing the first intermediate feature and the second intermediate feature, the pseudo-voxel feature at the next scale is obtained; The above steps are executed iteratively to generate the second multi-scale voxel feature, which contains multiple scales.

3. The target detection method according to claim 1, characterized in that, The step of suppressing noise in the second multi-scale voxel features based on the geometric information in the first multi-scale voxel features to obtain denoised multi-scale pseudo-voxel features includes: The laser centroid is calculated based on the spatial position of the points within the voxels in the first multi-scale voxel feature. Calculate the spatial distance between the laser centroid and the pseudopoints in the pseudo voxel block; The pseudo-point features with spatial distances within a preset range are selected, aggregated, and combined with laser centroid features to generate denoised multi-scale pseudo-voxel features.

4. The target detection method according to claim 1, characterized in that, The step of projecting the multi-scale fused voxel features onto a two-dimensional image space based on the three-dimensional coordinates and image calibration parameters corresponding to the multi-scale fused voxel features to generate semantically enhanced interactive features includes: Based on the voxelization parameters, data augmentation parameters, and camera calibration parameters, the three-dimensional coordinates corresponding to the multi-scale fused voxel features are projected onto the two-dimensional image pixel coordinates. Based on the pixel coordinates of the two-dimensional image, the multi-scale fused voxel features are processed using sparse convolution tensors and a two-dimensional convolutional network to generate the first two-dimensional associated features. The multi-scale fused voxel features are processed using a three-dimensional convolutional network to generate a second two-dimensional associated feature. By fusing the first two-dimensional association features with the second two-dimensional association features, the semantically enhanced interaction features are obtained.

5. The target detection method according to claim 1, characterized in that, The step involves compressing the semantically enhanced interactive features and the voxel features of a preset scale from the first multi-scale voxel features into the bird's-eye view space, extracting spatial and frequency domain features respectively, and fusing the spatial and frequency domain features through a dynamic attention weight mechanism to obtain enhanced bird's-eye view fusion features, including: The semantically enhanced interactive features and the voxel features at the preset scale are compressed into the bird's-eye view space to obtain interactive bird's-eye view features and laser bird's-eye view features, respectively. Multi-scale spatial fusion and reconstruction are performed on the interactive bird's-eye view features and the laser bird's-eye view features to generate spatially fused bird's-eye view features. The interactive bird's-eye view features and the laser bird's-eye view features are respectively subjected to frequency domain enhancement processing to generate interactive frequency domain enhanced bird's-eye view features and laser frequency domain enhanced bird's-eye view features; The enhanced bird's-eye view features are generated by weighting and fusing the spatial fusion bird's-eye view features, the interactive frequency domain enhanced bird's-eye view features, and the laser frequency domain enhanced bird's-eye view features through the dynamic attention weighting mechanism.

6. The target detection method according to claim 5, characterized in that, The enhanced bird's-eye view fusion feature is generated by weighting and fusing the spatial fusion bird's-eye view feature, the interactive frequency domain enhanced bird's-eye view feature, and the laser frequency domain enhanced bird's-eye view feature through the dynamic attention weighting mechanism, including: Attention weights are calculated for the spatial fusion bird's-eye view features, the interactive frequency domain enhanced bird's-eye view features, and the laser frequency domain enhanced bird's-eye view features, respectively, to obtain spatial attention weights, first frequency domain attention weights, and second frequency domain attention weights. The spatial fusion bird's-eye view features, the interactive frequency domain enhanced bird's-eye view features, and the laser frequency domain enhanced bird's-eye view features are modulated using the spatial domain attention weight, the first frequency domain attention weight, and the second frequency domain attention weight, respectively. The modulated features are fused to obtain the enhanced bird's-eye view fused features.

7. The target detection method according to claim 1, characterized in that, The method of performing 3D target detection based on the enhanced bird's-eye view fusion features to obtain the target's 3D bounding box and category information includes: Three-dimensional candidate regions are generated on the enhanced bird's-eye view fusion features using a region proposal network; Multimodal features within the three-dimensional candidate region are aggregated and refined using an attention-based neural network. Based on the refined features, predict the target's 3D bounding box and category information.

8. A target detection device, characterized in that, The device includes: The first processing module is used to receive the point cloud of the lidar and the pseudo point cloud generated by image depth completion, and perform voxelization processing on them respectively to obtain the lidar voxel block and the pseudo voxel block. The second processing module is used to process the laser voxel block through the first feature extraction network to obtain the first multi-scale voxel features, and to process the pseudo voxel block through the second feature extraction network to obtain the second multi-scale voxel features. The third processing module is used to perform noise suppression on the second multi-scale voxel features based on the geometric information in the first multi-scale voxel features to obtain denoised multi-scale pseudo voxel features. The fourth processing module is used to fuse the first multi-scale voxel feature with the denoised multi-scale pseudo voxel feature to obtain a multi-scale fused voxel feature. The fifth processing module is used to interact the multi-scale fused voxel features with the semantic information of the two-dimensional image to generate semantically enhanced interactive features. The sixth processing module is used to compress at least one scale of the semantically enhanced interactive features and the first multi-scale voxel features into the bird's-eye view space, extract spatial domain features and frequency domain features respectively, and fuse the spatial domain features and the frequency domain features through a dynamic attention weight mechanism to obtain the enhanced bird's-eye view fusion features. The seventh processing module is used to perform three-dimensional target detection based on the enhanced bird's-eye view fusion features, and obtain the target's three-dimensional bounding box and category information.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images