A 3D target detection model, a detection method and device, and a vehicle

Abstract

The application discloses a 3D target detection model, a detection method and device and a vehicle, and the model comprises a self-learning size-independent supervision module and a feature enhancement module based on block query. The self-learning size-independent supervision module automatically explores the weight distribution law of foreground and background labels by learning the feature information of point clouds, so that the foreground and background division in a complex scene is adapted, and the occluded targets and small targets are efficiently processed. The feature enhancement module based on block sampling divides the extracted bird's-eye view features into blocks, dynamically selects initial queries according to target distribution, and enhances the attention to small targets. The self-learning size-independent module and the feature enhancement module based on block query are proposed, and the detection capability of the model for small targets is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a 3D target detection model, detection method, device, and vehicle. Background Technology

[0002] 3D target detection data is widely used in autonomous driving, smart logistics, and security monitoring due to its rich spatial information and accurate target location. However, in real-world applications, targets exhibit varying scales and are susceptible to scene interference, making detection tasks challenging. To address this issue, researchers have proposed various improvement schemes, primarily focusing on multimodal information fusion and enhanced feature representation, aiming to improve the accuracy and robustness of target detection in complex scenes.

[0003] Numerous studies have focused on feature fusion between images and point clouds in multimodal 3D object detection to improve model detection performance and robustness. Feature projection fusion methods include PointFusion, PointPainting, EPNET, and EPNET++. PointFusion projects points from a LiDAR sensor onto the image plane, extracts image features from a pre-trained 2D detector, and uses a voxel network architecture to stitch image features with corresponding points. PointPainting projects LiDAR points onto the output of an image semantic segmentation network to obtain semantic features of the image location corresponding to the LiDAR points, resulting in a colored point cloud for subsequent detection. EPNET establishes a fine correspondence between point cloud and image features on a point-by-point basis, adaptively estimating the importance of image features to enhance point features and suppress interference. EPNET++ uses the CB-Fsuion module to bidirectionally fuse the results of image and point cloud and introduces a multimodal consistency loss to ensure consistency between the prediction scores of the two modalities. Attention-based fusion methods include Bridged Transformer, IGNet, and LidarIG. Bridged Transformer achieves attention bridging by applying the same positional encoding to image and point cloud data, fusing point cloud and image information at the feature level. IGNet utilizes an image-guided radar cross-modal channel attention module and a radar-guided image cross-modal spatial attention module for bidirectional fusion to address the potential loss of positional information in the unidirectional information flow of traditional FPN. LidarIG uses the Vision Transformer module to enhance the semantic features of images and point clouds, and alternates between self-attention and cross-attention to promote cross-modal information exchange, obtaining enhanced features with global contextual semantics.

[0004] In addition, some studies explore enhancing feature representation. The SEED method proposes a deformable mesh attention module, dividing the reference box into a mesh and predicting offsets to obtain a more flexible receptive field, thereby capturing more informative features. PIPC-3Ddet explicitly models the correlation between 3D detection box proposals, obtaining proposal features that perceive global context information, thus promoting more accurate proposal optimization. MS-A uses low-score features and upsampled new features as keys and values ​​in different groups of attention heads, achieving an attention mechanism for learning multi-scale, finer geometric details. GE-FSOD uses a fusion mechanism that dynamically generates spatially adaptive weights to balance contributions from high-level semantic features, mid-level features, and current-level detail features, aiming to generate richer and more accurate multi-scale feature representations. DSPDet3D proposes a dynamic spatial pruning strategy, dynamically pruning redundant voxels in scene features based on object distribution. Relation-DETR explicitly models the positional relationship between detection boxes using normalized relative geometric features and integrates this relationship as a learnable bias into the self-attention mechanism of the Transformer decoder.

[0005] Existing methods have made progress in feature fusion and feature representation enhancement for multimodal 3D object detection, effectively improving the detection performance and robustness of the models. However, these methods still have certain limitations: feature projection fusion methods may not perform well when dealing with complex scenes and point cloud sparsity; attention-based fusion methods, while promoting cross-modal information exchange, may have high computational complexity; and methods for enhancing feature representation focus more on improving the overall feature quality, lacking additional attention to small targets. These methods often aim to generate richer and more accurate multi-scale feature representations or optimize and integrate features, but when faced with small targets with indistinct features, it is difficult to specifically enhance their features, resulting in room for improvement in the detection accuracy and recall of small targets. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a 3D target detection model, detection method, device, and vehicle, aiming to solve the difficulty of traditional models in accurately capturing the features of small targets and occluded targets, thereby improving detection accuracy. It can also avoid the problem of the model biasing towards large objects when selecting queries, and make full use of the effective information in point cloud data.

[0007] According to a first aspect of the present disclosure, a 3D object detection model is provided, including a backbone feature extraction network, a self-learning size-independent supervised module, a block-based feature enhancement module, a first object detection module, and a second object detection module, wherein... The backbone feature extraction network extracts features from the image to be detected and its point cloud data to generate bird's-eye view features; The self-learning size-independent supervision module performs self-learning matching on the bird's-eye view features to generate a supervised heatmap, and combines the supervised heatmap with the bounding boxes to generate a supervision heatmap, thus forming a size-independent supervision signal. During the generation of the supervised heatmap, for queries at each coordinate position in the supervised heatmap, the target correlation is defined as a self-learning mapping related to the ground bounding box. When the query point is within the ground bounding box, the supervised value of the query point is obtained based on the distance from the query point to the center of the ground bounding box. The feature enhancement module based on block query divides the bird's-eye view features into blocks, calculates the average value of the foreground prediction result in each block based on the foreground and background prediction results of each block, dynamically adjusts the number of query samples based on the average value of the foreground prediction result, and performs query initialization by sequentially inputting the initial query into the self-attention layer and the cross-attention layer for processing, obtaining the updated query sequence, and thus obtaining the enhanced features. The first target detection module will enhance the features input to the decoding head and the detection head, and output the initial target detection result; The second target detection module adjusts the initial target detection result by fusing multi-scale features of the image to be detected, and obtains the final detection result.

[0008] In some embodiments, the supervision value of the query point is measured using the target association metric, specifically expressed as: , in, Indicates the location in the monitored heatmap The corresponding target relevance, Represents the true bounding box. This is a self-learning mapping; the self-learning process is based on scoring the distance from a point to the center of the bounding box. This represents the activation function. It represents the relative distance between the centers of objects.

[0009] In some embodiments, the number of query samples is dynamically adjusted based on the average value of the prospect prediction results. The specific expression is: , in, This represents the average of the prospect prediction results within each block. N The number of subgraphs, The minimum value of the foreground / background prediction result. The maximum value of the foreground / background prediction result. This is used as a bias to ensure the minimum number of queries required for each block. This represents the maximum number of queries. The number of query centers for each block.

[0010] In some embodiments, the initial query is specifically a concatenation of the query feature vector and the location encoding vector, wherein the query feature vector is the bird's-eye view feature corresponding to the query center position of each block, and the location encoding vector is a fixed-length location encoding vector generated for each position of the input feature.

[0011] In some embodiments, the initial query is sequentially input into the self-attention layer and the cross-attention layer for processing to obtain an updated query sequence, specifically including the following steps: The initial query sequence is fused with the location encoding vector to obtain the query features with fused location information; Self-attention is calculated on the query features to obtain self-attention enhanced features; The self-attention enhancement features and query features are joined by a residual connection, and then summed and normalized to obtain the self-attention output features. The self-attention output features are cross-attentioned with the overall view features to obtain cross-attention enhanced features; The cross-attention enhancement features and self-attention output features are residually connected, and then summed and normalized to obtain the updated query sequence.

[0012] In some embodiments, the loss function of the 3D object detection model includes the detection result loss of the auxiliary branches and main branches of the dense detection head, the contrast loss, the loss of the region of interest detection head, and the size-independent supervision loss. The detection result loss of the auxiliary branches and main branches of the dense detection head and the loss of the region of interest detection head are both composed of classification loss and regression loss. The contrast loss is used to enhance the model's ability to distinguish target features.

[0013] According to a second aspect of the present disclosure, a 3D target detection method is provided, comprising: Acquire the image to be detected and its point cloud data; Train the 3D object detection model; The image to be detected and its point cloud data are input into a trained 3D target detection model for detection processing, and the detection results are output; the 3D target detection model is the 3D target detection model described above.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the 3D target detection method described above.

[0015] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein computer instructions are stored on the storage medium, and when executed by a processor, the instructions implement the steps of the above-described 3D target detection method.

[0016] According to a fifth aspect of the present disclosure, a vehicle is provided, including the 3D target detection model described above.

[0017] This disclosure provides a 3D target detection model, method, device, and vehicle, which is a block-based size-independent dynamic query selection 3D target detection model. It consists of two key modules that jointly enhance the detection capability for small targets and occluded targets. First, a self-learning size-independent supervision module learns the feature information of point clouds to automatically explore the weight distribution pattern of foreground and background labels, adapting to foreground and background segmentation in complex scenes and efficiently handling occluded and small targets. Second, a feature enhancement module based on block sampling divides the extracted bird's-eye view features into blocks and dynamically selects the initial query based on the target distribution, enhancing the focus on small targets. Its beneficial effects specifically include: 1) A block-based size-independent dynamic query selection 3D object detection model is proposed. Through the collaborative work of a self-learning size-independent supervision module and a block-based sampling feature extraction module, the model can accurately capture the features of small and occluded targets in complex scenes, significantly improving the detection accuracy for these types of targets. Compared with traditional 3D object detection models that rely solely on fixed query selection or struggle to handle targets with limited information, this method fully leverages the advantages of point cloud feature information and dynamic query selection, particularly enhancing the detection capability for small and occluded targets. Therefore, it is more suitable for 3D object detection tasks in complex scenarios such as autonomous driving and intelligent robot navigation. 2) The self-learning size-independent supervision module generates size-independent foreground and background prediction masks, guiding the model to give relatively equal attention to all targets. This effectively alleviates the problem of over-focusing on large targets and neglecting small targets in 3D object detection tasks. The block-based feature enhancement module utilizes the foreground and background prediction masks obtained from the self-learning size-independent supervision module. It selects the initial query in each block according to block division, further enhancing the focus on the region where small targets are located, enabling the model to give more attention to small targets when selecting the initial query.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 This is an overall architecture diagram of the 3D target detection model in an embodiment of the present invention; Figure 2 This is the intent of size-independent supervision in the embodiments of the present invention; Figure 3 This is a schematic diagram of block division and sampling in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the implementation principle of the block-based query feature extraction module in this embodiment of the invention. Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the drawings, not the entire structure.

[0022] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0023] This invention addresses the challenges of 3D object detection in scenarios such as autonomous driving, including the difficulty in capturing small object features and the ease with which objects are occluded. It proposes the BQ-SIFNet method, which effectively improves the 3D detection accuracy of small and occluded objects in complex scenes through self-learning size-independent supervision and a block-based query-based feature enhancement strategy. 3D object detection technology demonstrates broad application prospects in various practical scenarios, including autonomous driving, intelligent transportation systems, robot navigation and logistics, and security monitoring.

[0024] (1) Autonomous driving This is the most direct application area of ​​this technology. In autonomous driving scenarios, it is necessary to accurately identify various targets, including small vehicles at a distance and vehicles that may be obscured. By improving the detection accuracy of these targets, the safety and reliability of autonomous driving systems can be enhanced, ensuring that vehicles make accurate decisions in complex traffic environments, such as obstacle avoidance, lane keeping, and overtaking.

[0025] (2) Intelligent Transportation Systems In urban traffic management and traffic flow monitoring, this technology can be used to accurately detect various vehicles and targets on the road. For traffic management departments, obtaining real-time and accurate traffic information helps optimize traffic signal control, plan traffic flow, and improve the operational efficiency of urban traffic.

[0026] (3) Robot navigation and logistics In industrial logistics scenarios, robots need to autonomously navigate and transport goods in environments such as warehouses. This target detection technology can help robots identify surrounding obstacles, goods, and other equipment, enabling them to complete tasks more flexibly and accurately, and improving the efficiency and safety of logistics automation.

[0027] (4) Security monitoring In the field of security monitoring, this technology can be used to accurately detect and track people and vehicles within the monitored area. It is particularly effective for distant or partially obscured targets, improving detection accuracy, enabling timely detection of anomalies, and ensuring public safety.

[0028] With the continuous advancements in point cloud processing technology, deep learning optimization algorithms, and multimodal fusion technology, the application scenarios for 3D target detection will be further expanded. It will not only enhance the detection performance and reliability in existing application areas but will also demonstrate greater value in emerging scenarios such as intelligent building construction monitoring, aerospace target recognition, and underwater detection and operations, providing strong technical support and decision-making basis for the intelligent upgrading and efficient development of various industries.

[0029] A novel 3D object detection model is proposed to overcome the limitations of traditional 3D object detection models in detecting small objects and vehicles with incomplete information in complex autonomous driving scenarios. This model employs a block-based size-independent dynamic query selection approach, which not only solves the problem of traditional models failing to accurately capture the features of small and occluded objects, thus improving detection accuracy, but also avoids the bias towards large objects during query selection, fully utilizing the effective information in point cloud data. Specifically, a self-learning size-independent supervised module is designed to automatically explore the weight distribution patterns of foreground and background labels by learning point cloud feature information, adapting to foreground and background segmentation in complex scenes. Building upon this, a block-based feature extraction module divides the bird's-eye view features into blocks and dynamically selects the initial query based on the target distribution, enhancing the focus on small objects. The organic combination of these two key modules provides a more accurate, efficient, and robust solution for 3D object detection in complex autonomous driving scenarios.

[0030] The 3D object detection model includes a backbone feature extraction network, a self-learning size-independent supervised module, a block-based query-based feature enhancement module, a first object detection module, and a second object detection module. The backbone feature extraction network extracts features from the image to be detected and its point cloud data to generate a bird's-eye view feature. The self-learning size-independent supervised module performs self-learning matching on the bird's-eye view features to generate a supervised heatmap. This supervised heatmap is then combined with bounding boxes to generate a supervised heatmap, forming a size-independent supervision signal. During the generation of the supervised heatmap, for queries at each coordinate position in the supervised heatmap, the object correlation is defined as a self-learning mapping related to the ground truth bounding box. When the query point is within the ground truth bounding box, the object correlation is determined based on the query point's coordinates. The distance from the center of the ground truth bounding box is used to obtain the supervised value of the query point. The feature enhancement module based on block query divides the bird's-eye view features into blocks, calculates the average value of the foreground prediction result in each block based on the foreground and background prediction results of each block, and dynamically adjusts the number of query samples based on the average value of the foreground prediction result. At the same time, query initialization is performed, and the initial query is sequentially input into the self-attention layer and the cross-attention layer for processing to obtain the updated query sequence and obtain the enhanced features. The first object detection module inputs the enhanced features into the decoding head and the detection head and outputs the initial object detection result. The second object detection module adjusts the initial object detection result by fusing the multi-scale features of the image to be detected to obtain the final detection result.

[0031] Specifically, such as Figure 1As shown, the self-learning size-independent supervised module can generate size-independent foreground / background prediction masks using the weight distribution obtained through self-learning. The feature enhancement module based on block queries changes the focus of the query, guiding it from the position of the global prediction maximum to the center of each block. In this way, the module can focus more on the feature information within each block, especially small target features that are easily ignored by traditional methods.

[0032] In target detection methods based on Detection Transformer (DETR), the initial query selection depends on the prediction confidence. The inventors of this application construct a size-independent target correlation as the supervision target, instead of simply using discrete labels [0, 1] to distinguish foreground and background. This avoids the model's tendency to assign higher confidence to regions containing large targets while ignoring smaller targets, thus overcoming size bias. Compared to direct supervision using distance saliency values, in actual detection, issues such as occlusion and distance often result in missing point clouds in the center of the object to be detected, creating a hollow area. In such cases, it is usually necessary to use the point cloud around the center point to assist in determining the center point of the object. Directly using distance-based saliency supervision, which provides higher weights at the center and lower weights at the edges, performs poorly for objects with occlusion and distance.

[0033] The supervised values ​​of the query points are quantified using the target association metric; specifically, for location... Corresponding query Its target relevance Defined as: , in, Indicates the location in the monitored heatmap The corresponding target relevance, This represents the true bounding box, where x, y, and z are the coordinates of the center of the true bounding box on the x, y, and z axes, l is the length of the true bounding box, w is the width of the true bounding box, h is the height of the true bounding box, and yaw is the rotation angle of the true bounding box. This is a self-learning mapping. The self-learning process is based on scoring the distance from a point to the center of the bounding box. In a specific embodiment, the distances from the point to the center of the bounding box are divided into several groups: 0~0.2, 0.2~0.4, 0.4~0.6, 0.4~0.6, 0.6~0.8, and 0.8~1.0. Each group utilizes the above... Mapping retrieves the supervision value at the current position. This represents the activation function. The relative distance to the center of the object is represented by the target correlation degree, which is calculated using a self-learning mapping based on the relative distance to the center of the object.

[0034] , in, and This indicates the distance between the query location and the center of the corresponding bounding box. l and w These are the length and width of the bounding box.

[0035] Figure 2 The comparison between target relevance and discrete supervision is shown. Unlike labels that have no clear distinction and are favorable to large objects, the target relevance supervision of this invention ensures fine-grained confidence for objects of different sizes, resulting in more stable filtering results.

[0036] Query selection based on detection transformers often relies directly on the foreground / background prediction results of a global query to select initial features. However, this approach has limitations when dealing with complex and varied feature maps. It fails to fully consider the uniqueness and differences of each region in the feature map, potentially leading to insufficient information mining in some key areas or wasting excessive computational resources in unnecessary regions. To overcome these problems, this invention proposes a more refined and efficient method: dividing the bird's-eye view into blocks and dynamically adjusting the number of query samples based on the foreground / background prediction results of each block, while simultaneously performing reasonable query initialization, to achieve more accurate target detection and feature utilization.

[0037] like Figure 3 As shown, in order to select the query center from each region of the feature, the bird's-eye view is first... Divided into multiple blocks , Where H and W are the sizes of the BEV feature maps, h, w N represents the size of each sub-map after the feature map is divided, and N is the number of sub-maps.

[0038] After dividing the feature map into multiple sub-maps, the foreground and background prediction results are based on the prediction results of each sub-map. Calculate the average value of the foreground prediction results within each block. , When the average foreground prediction value is high, it indicates a higher probability that the block contains an object to be detected, and a larger number of such objects may exist. Queries sampling more data from this block are recommended. Conversely, when the average foreground prediction value is low, it indicates a lower probability that the block contains an object to be detected, and a smaller number of such objects may exist. Queries sampling less data from this block are recommended. The specific expression is as follows: , in, , This represents the average of the prospect prediction results within each block. N The number of subgraphs, The minimum value of the foreground / background prediction result. The maximum value of the foreground / background prediction result. This is used as a bias to ensure the minimum number of queries required for each block. This represents the maximum number of queries. The number of query centers for each block.

[0039] The initial query is specifically a concatenation of the query feature vector and the location encoding vector. The query feature vector is the bird's-eye view feature corresponding to the query center position of each block, and the location encoding vector is a fixed-length location encoding vector generated for each position of the input feature.

[0040] Specifically, after obtaining different blocks, each block... Query center location The corresponding bird's-eye view features are used as the feature vector for this query. , Furthermore, by calculating sine and cosine function values ​​at different frequencies, a fixed-length position encoding vector is generated for each location of the input feature. , In this way, subsequent models can know the precise location information of the query center, and concatenate the feature vector and the location encoding vector to obtain the initial query. In this way, the query includes both the object's feature information and its location information, which is beneficial for query decoding.

[0041] After obtaining the initial query, a block-based query feature enhancement module was designed, such as... Figure 4 As shown, it consists of a self-attention layer and a cross-attention layer. In object detection tasks, the query center often corresponds to the location where a potential object to be detected may exist. Considering that the objects to be detected usually differ from the surrounding background in features, and that objects of the same category often share some common features, a self-attention layer is first introduced to enable interaction between different queries. This cross-query interaction helps to uncover the correlation information between potential objects corresponding to different queries, and thus infer the correspondence between different candidate objects. At the same time, this interaction method can also effectively reduce the interference of background information, enhance the model's ability to abstract and integrate object features, enable the model to represent objects from multiple perspectives, and improve the accuracy of object features.

[0042] In some embodiments, the initial query is sequentially input into the self-attention layer and the cross-attention layer for processing to obtain an updated query sequence, specifically including the following steps: The initial query sequence is fused with the location encoding vector to obtain the query features with fused location information; Self-attention is calculated on the query features to obtain self-attention enhanced features; The self-attention enhancement features and query features are joined by a residual connection, and then summed and normalized to obtain the self-attention output features. The self-attention output features are cross-attentioned with the overall view features to obtain cross-attention enhanced features; The cross-attention enhancement features and self-attention output features are residually connected, and then summed and normalized to obtain the updated query sequence.

[0043] After interaction through the self-attention layer, the model acquires richer and more representative feature information, laying a solid foundation for subsequent detection and classification tasks. However, simply using self-attention primarily focuses on the relationships between queries, lacking effective utilization of the contextual information surrounding objects. To compensate for this deficiency, a cross-attention mechanism is introduced after the self-attention layer, enabling queries to interact with the overall view. The overall view contains rich background information, and cross-attention can filter out important information relevant to the query itself from the overall view. For example, for a query corresponding to an occluded target object, cross-attention can extract relevant clues about the occluded part and information about the object and its surrounding environment from the overall view. Utilizing this auxiliary information for more accurate feature extraction and localization of target objects helps the model better cope with object diversity, occlusion, and complex backgrounds, improving detection accuracy and robustness.

[0044] After obtaining query features using a block-based query feature extraction network, an encoder module consisting of six encoder layers is employed. This model predicts 3D detection results based on the query features and iteratively refines them between layers. Each encoder layer comprises a multi-head attention layer and a cross-attention layer. Through this layer-by-layer iterative approach, the position and size of the predicted bounding box are continuously refined, making it more closely resemble the actual target object, thereby improving the accuracy and reliability of the predicted bounding box.

[0045] The loss function of the 3D object detection model includes the detection result loss of the auxiliary branches and main branches of the dense detection head, the contrast loss, the loss of the region of interest detection head, and the size-independent supervision loss. The detection result loss of the auxiliary branches and main branches of the dense detection head and the loss of the region of interest detection head are composed of classification loss and regression loss, while the contrast loss is used to enhance the model's ability to distinguish target features.

[0046] Specifically, the overall loss function can be expressed as: , in, and The loss for the dense detection head consists of classification loss and regression loss, with the detection results of the auxiliary branch and the detection results of the main branch as their inputs, respectively. Contrastive loss is used to enhance the model's ability to distinguish target features. It is the loss of the region of interest detection head, which also consists of classification loss and regression loss. This is the proposed size-independent supervised loss. The expression for this loss function is as follows: , in, and These represent the weights of the two losses, respectively. For size-independent supervision labels, For predicting foreground background labels. This represents the weights of each region within the ground truth bounding box in the size-independent supervised prediction. These weights reflect the importance of different locations within the ground truth bounding box in the detection task. It is a focus loss based on the Sigmoid activation function, used to address the problem of imbalanced samples in object detection tasks. This is the adjacent position difference loss, whose main function is to ensure that the weight distribution within the ground truth bounding box exhibits a pattern of high weights in the center and gradually decreasing towards the sides. This constraint of the loss function prevents the model from setting the weight distribution to 0 in most areas and only setting it to 1 at the edges in order to minimize the loss function; such an unreasonable weight distribution would severely impact the model's performance. By constraining the weight differences between adjacent positions, this undesirable situation is avoided, enabling the model to learn more reasonable and effective feature representations.

[0047] To more clearly illustrate the training process, pseudocode is provided in Algorithm 1. The overall process is as follows: After the input image is processed by a pre-trained virtual point cloud generation network, it is concatenated with the radar point cloud, and then processed by a backbone network to extract multi-scale features to obtain the multi-scale features. The top-level features After passing through the bird's-eye view feature extraction network, the Z-axis is compressed to obtain bird's-eye view features, which are then input into a self-learning, size-independent supervised module to generate a size-based foreground / background prediction mask. This foreground / background prediction mask will serve as a reference for selecting the initial query for feature extraction in subsequent block query feature enhancement modules. These features are then further input into a deformable decoder and a dense detection head to obtain the first-stage prediction results. The results from the dense detection head are subsequently refined and adjusted using multi-scale features in the region of interest detection head to generate the final detection results.

[0048]

[0049] In addition to the modules described above, the 3D object detection model may also include other components; however, since these components are not relevant to the embodiments of this disclosure, their illustrations and descriptions are omitted here.

[0050] Another embodiment provides a 3D object detection method, including: acquiring an image to be detected and its point cloud data; training a 3D object detection model; inputting the image to be detected and its point cloud data into the trained 3D object detection model for detection processing, and outputting detection results; wherein the 3D object detection model is the 3D object detection model described above.

[0051] Other specific working processes of the 3D object detection method are described in the above-described 3D object detection model embodiment and will not be repeated here.

[0052] Another embodiment illustrating that the system of the present invention can also be achieved by means of... Figure 5 The architecture of the computing device shown is used to implement this. Figure 5 The architecture of the computing device is shown. For example... Figure 5 As shown, the computer system 510 includes a system bus 530, one or more CPUs 540, input / output 520, and memory 550. The memory 550 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including 3D target detection methods. Figure 5 The architecture shown is merely exemplary and should be adjusted according to actual needs when implementing different devices. Figure 5 One or more components are included. The memory 550, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the 3D target detection method in this embodiment of the invention. One or more CPUs 540 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions, and modules stored in the memory 550.

[0053] Of course, the processor of the server provided in the embodiments of the present invention is not limited to performing the method operations described above, but can also perform related operations in the 3D target detection method provided in any embodiment of the present invention.

[0054] The memory 550 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 550 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 550 may further include memory remotely configured relative to one or more CPUs 540, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0055] Input / output 520 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Input / output 520 may also include a display device such as a display screen.

[0056] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the 3D target detection method described in the above embodiments. The computer-readable storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0057] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0058] The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0059] Furthermore, other specific working processes of a non-transitory computer-readable storage medium are described in the above-described embodiments of the 3D target detection method, and will not be repeated here.

[0060] Another embodiment provides a vehicle including the aforementioned 3D object detection model. Its specific working process is described in the above-described 3D object detection model embodiment and will not be repeated here.

[0061] To verify the effectiveness of the model of this invention, as shown in Table 1, the ablation experiment results on the KITTI autonomous driving dataset show that the introduction of the self-learning size-independent supervision module and the feature enhancement module based on block query in the 3D target detection model can improve the detection performance, especially in the difficult categories, namely occluded and small targets, which verifies their effectiveness in improving the recognition of small targets.

[0062] Table 1. Ablation experimental results on the KITTI dataset (AP_R40@0.70, 0.50, 0.50)

[0063] To further verify the effectiveness of this invention, as shown in Tables 2 and 3, experimental results on two autonomous driving datasets, KITTI and PANDASET, demonstrate that our method achieves optimal performance across multiple evaluation metrics. Compared to existing object detection methods, our method exhibits significant advantages in detecting small objects, i.e., hard-class samples. This result fully validates the effectiveness of our invention in improving detection accuracy and robustness in 3D object detection tasks.

[0064] Table 2. KITTI dataset detection results (AP_R40@0.70, 0.50, 0.50)

[0065] Table 3. Detection results on the PANDASET dataset (AP_R40@0.70, 0.50, 0.50)

[0066] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such a step or method.

[0067] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A 3D target detection model, characterized in that, It includes a backbone feature extraction network, a self-learning size-independent supervised module, a block-based feature enhancement module, a first object detection module, and a second object detection module. The backbone feature extraction network extracts features from the image to be detected and its point cloud data to generate bird's-eye view features. The self-learning size-independent supervision module performs self-learning matching on the bird's-eye view features to generate a supervised heatmap, and combines the supervised heatmap with the bounding boxes to generate a supervision heatmap, thus forming a size-independent supervision signal. During the generation of the supervised heatmap, for queries at each coordinate position in the supervised heatmap, the target correlation is defined as a self-learning mapping related to the ground bounding box. When the query point is within the ground bounding box, the supervised value of the query point is obtained based on the distance from the query point to the center of the ground bounding box. The feature enhancement module based on block query divides the bird's-eye view features into blocks, calculates the average value of the foreground prediction result in each block based on the foreground and background prediction results of each block, dynamically adjusts the number of query samples based on the average value of the foreground prediction result, and performs query initialization by sequentially inputting the initial query into the self-attention layer and the cross-attention layer for processing, obtaining the updated query sequence, and thus obtaining the enhanced features. The first target detection module will enhance the input features to the decoding head and the detection head, and output the initial target detection result; The second target detection module adjusts the initial target detection result by fusing multi-scale features of the image to be detected, and obtains the final detection result.

2. The 3D target detection model according to claim 1, characterized in that, The supervised value of the query point is quantified using the target association metric, specifically expressed as: ， in, Indicates the location in the monitored heatmap The corresponding target relevance, Represents the true bounding box. This is a self-learning mapping; the self-learning process is based on scoring the distance from a point to the center of the bounding box. This represents the activation function. It represents the relative distance between the centers of objects.

3. The 3D target detection model according to claim 1, characterized in that, The number of query samples is dynamically adjusted based on the average value of the prospect prediction results. The specific expression is: ， in, This represents the average of the prospect prediction results within each block. N The number of subgraphs, The minimum value of the foreground / background prediction result. The maximum value of the foreground / background prediction result. This is used as a bias to ensure the minimum number of queries required for each block. This represents the maximum number of queries. The number of query centers for each block.

4. The 3D target detection model according to claim 1, characterized in that, The initial query is specifically a concatenation of the query feature vector and the location encoding vector. The query feature vector is the bird's-eye view feature corresponding to the query center position of each block, and the location encoding vector is a fixed-length location encoding vector generated for each position of the input feature.

5. The 3D target detection model according to claim 1, characterized in that, The initial query is sequentially input into the self-attention layer and the cross-attention layer for processing to obtain the updated query sequence. The specific steps include: The initial query sequence is fused with the location encoding vector to obtain the query features with fused location information; Self-attention is calculated on the query features to obtain self-attention enhanced features; The self-attention enhancement features and query features are joined by a residual connection, and then summed and normalized to obtain the self-attention output features. The self-attention output features are cross-attentioned with the overall view features to obtain cross-attention enhanced features; The cross-attention enhancement features and self-attention output features are residually connected, and then summed and normalized to obtain the updated query sequence.

6. The 3D target detection model according to claim 1, characterized in that, The loss function of the 3D object detection model includes the detection result loss of the auxiliary branches and main branches of the dense detection head, the contrast loss, the loss of the region of interest detection head, and the size-independent supervision loss. The detection result loss of the auxiliary branches and main branches of the dense detection head and the loss of the region of interest detection head are both composed of classification loss and regression loss. The contrast loss is used to enhance the model's ability to distinguish target features.

7. A 3D target detection method, characterized in that, include: Acquire the image to be detected and its point cloud data; Train the 3D object detection model; The image to be detected and its point cloud data are input into a trained 3D target detection model for detection processing, and the detection result is output; the 3D target detection model is the 3D target detection model according to any one of claims 1-6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the 3D target detection method as described in claim 7.

9. A non-transitory computer-readable storage medium, wherein computer instructions are stored on the storage medium, characterized in that, When the instructions are executed by the processor, they implement the steps of the 3D target detection method as described in claim 7.

10. A vehicle, characterized in that, Including the 3D target detection model as described in any one of claims 1-6.

Images