An example unified three-dimensional object detection and segmentation method and apparatus

Abstract

The application provides a kind of example perceived unified three-dimensional target detection segmentation method and device, it is related to three-dimensional target detection and instance segmentation technical field.The method comprises: first, initial query vector is obtained by initial query.Then the initial query vector is input into the multi-head output structure to generate the interactive enhanced feature vector.Subsequently, the interactive enhanced feature vector is input into the multi-head output structure again, so as to obtain the panoramic segmentation result and three-dimensional detection bounding box, and input into the spatial perception self-attention model.Finally, based on the local model module of vertex guided cross attention mechanism local modeling, the segmentation mask is refined.The application can synchronously and accurately predict the detection frame and the segmentation mask, thereby significantly improving the overall three-dimensional scene understanding ability.

Description

Technical Field

[0001] This invention relates to the field of 3D target detection and instance segmentation technology, and in particular to an instance-aware unified 3D target detection and segmentation method and apparatus. Background Technology

[0002] 3D object detection is a technique for locating and identifying objects in 3D space. The output typically includes the object category and its 3D bounding box (center, size, and orientation). This technology is widely used in autonomous driving, robotics, warehousing and logistics, AR / VR perception, and digital twins. In a typical workflow, multi-source sensor data (such as LiDAR point clouds, RGB-D, and single / multi-view cameras) is first calibrated and preprocessed. Then, feature extraction and fusion are performed using spatial representations such as points, voxels, and bird's-eye views. Finally, candidate objects are generated, and 3D bounding box parameters and confidence scores are regressed. High-quality spatial representation, multi-scale feature aggregation, and multi-sensor fusion are crucial for 3D detection performance. 3D detection faces various sources of uncertainty, including long-range sparse point clouds, occlusion and truncation, sensor noise and calibration errors, long-tailed categories, and domain offsets. Effective uncertainty modeling (e.g., variance estimation of position and orientation, confidence score calibration) helps assess the reliability of detection results and supports downstream planning and decision-making.

[0003] 3D scene segmentation aims to assign semantic labels to each point or voxel in a 3D scene, and, when necessary, distinguish instances to form semantic / instance / panoramic segmentation results. This task plays a crucial role in high-precision map construction, robot navigation, indoor and outdoor scene understanding, security patrol, and digital twin modeling. Common workflows include: acquiring and aligning 3D data (point cloud, RGB-D, reconstructed mesh, or multi-frame fusion), performing preprocessing such as downsampling and denoising, using multi-scale and long-range context modeling for feature learning, and combining time series and multimodal information to improve robustness. Efficient geometric and semantic context modeling, class imbalance handling, and memory / computation optimization for large-scale scenes are crucial for segmentation results. 3D segmentation also presents significant uncertainties, such as incomplete observations due to occlusion, boundary confusion, dynamic target interference, sensor noise, and label errors. Reasonable uncertainty modeling can quantify label credibility, indicate potential misclassified areas, and provide a basis for map maintenance, proactive perception, and security decision-making.

[0004] Existing research has explored various prediction strategies along two main lines: 3D object detection and 3D segmentation. In 3D object detection, point-based voting methods (e.g., VoteNet) achieve high-fidelity candidate generation through the fusion of center proposal and geometric priors. Sparse convolutional detectors (e.g., GSPN) excel in efficiency and scale generalization, achieving robust performance on densely packed small objects indoors through free anchor point design, fine label allocation, and two-stage refinement. Transformer-based end-to-end detectors (e.g., 3DETR) utilize global attention and matching mechanisms to improve performance under complex layouts and occlusion conditions. In 3D segmentation, semantic segmentation often employs UNet-style encoder-decoder structures (e.g., PointNeXt), or directly performs feature aggregation and Transformer modeling at points (e.g., PointTransformer), or voxels the point cloud and efficiently processes it with dense / sparse 3D convolutions (e.g., MinkUNet). Instance segmentation typically involves point-by-point aggregation of semantic features (e.g., 3dsis), employing both top-down proposal-based methods (e.g., TD3D) and bottom-up aggregation methods (e.g., Dyco3d). Recently, Transformer has achieved leading performance in both accuracy and inference speed (e.g., Mask3D). Panoramic segmentation, on the other hand, outputs both instances and semantics in a unified manner. Methods such as PanopticFusion use input RGB to generate panoramic results and map and aggregate them into 3D space to compensate for the lack of appearance information in pure point clouds.

[0005] Existing methods typically separate 3D object detection and 3D segmentation into two independent tasks, using 3D bounding boxes and point-by-point semantic / instance labels as supervision respectively. Only the box parameters and point-by-point labels constrain the output of the corresponding branch and update the model parameters. However, relying solely on this fragmented, task-specific supervision often only prompts the model to fit "evaluable boxes or masks," without explicitly enabling the model to understand the geometric structure and semantic composition of the target, how instance boundaries support the scale and pose of the box, how occlusion and physical consistency constrain feasible solutions, and why detection and segmentation should corroborate each other. The result is redundant architecture and training signals, difficulty in sharing information across tasks, and a limited overall and comprehensive understanding of the 3D scene. Summary of the Invention

[0006] To address the technical problems of redundant architecture and training signals in existing 3D scene understanding technologies, and the difficulty in sharing information across tasks, this invention provides an instance-aware unified 3D object detection and segmentation method and apparatus. The technical solution is as follows:

[0007] On the one hand, an instance-aware unified 3D object detection and segmentation method is provided, which is implemented by an instance-aware unified 3D object detection and segmentation device. The method includes:

[0008] S1: Initialize the query to obtain the initial query vector. The initial query includes initializing the query based on the superpoint mapping.

[0009] S2: Input the initial query vector into the multi-head output structure to obtain the interactively enhanced feature vector;

[0010] S3: Input the feature vector based on interactive enhancement into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch.

[0011] S4: Input the 3D detection bounding box, interactively enhanced feature vector and panoramic segmentation result into the spatially perceptive self-attention model to obtain a feature representation that incorporates spatial location information. The spatially perceptive self-attention model is obtained by global modeling using a spatially perceptive self-attention mechanism. The spatially perceptive self-attention model will directly incorporate the relative position of the instance center into the attention calculation part of the spatially perceptive self-attention model. The attention calculation part has a center-aware attention representation.

[0012] S5: Input the feature representation incorporating spatial location information and the three-dimensional detection bounding box into the local model module to obtain the enhanced feature representation and refined segmentation mask. The local model module is obtained by local modeling based on the vertex-guided cross-attention mechanism.

[0013] Preferably, the initial query in S1 obtains an initial query vector, and the initial query includes initializing the query based on the superpoint mapping, including:

[0014] S11: Receive a query group, which contains multiple queries;

[0015] S12: Map the query group to the same superpoint according to the superpoint mapping to obtain the initial query vector.

[0016] Preferably, step S2, which inputs the initial query vector into a multi-head output structure to obtain an interactively enhanced feature vector, includes:

[0017] S21: Receive the initial query vector;

[0018] S22: Based on the initial query vector, the multi-head output structure performs feature fusion and information interaction to obtain an interactively enhanced feature vector. The multi-head output structure includes a joint loop iterative decoding mechanism. The joint loop iterative decoding mechanism is based on the classification loss, mask loss and regression loss to jointly train the network. The classification loss is used for supervised detection prediction, the mask loss is used for supervised segmentation, and the regression loss is used for supervised regression of the 3D detection bounding box.

[0019] Preferably, in step S3, the feature vector based on interactive enhancement is input into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch, including:

[0020] S31: The multi-head output structure processes the interactively enhanced feature vector through attention bridging and shared channel feature compression to generate a shared feature representation that can be used by multiple heads;

[0021] S32: In the detection task branch, based on the shared feature representation, calculate the bounding box parameters to obtain the bounding box information, and form a three-dimensional detection bounding box based on the bounding box information;

[0022] S33: In the segmentation task branch, based on the shared feature representation, semantic and instance analysis are performed to obtain semantic segmentation and instance mask series;

[0023] S34: Based on the semantic segmentation and instance mask series, sum and complete the foreground and background results to obtain the panoramic segmentation result.

[0024] Preferably, in step S4, the 3D detection bounding box, interactively enhanced feature vector, and panoramic segmentation result are input into the spatially perceptive self-attention model to obtain a feature representation incorporating spatial location information. The spatially perceptive self-attention model is obtained through global modeling using a spatially perceptive self-attention mechanism. This model directly incorporates the relative position of the instance center into its attention calculation part, which possesses a center-aware attention representation, including:

[0025] S41: Based on the interactively enhanced feature vector and the panoramic segmentation results, the 3D detection bounding box is used as the reference for the instance center. The spatial attention feature is obtained by introducing the position encoding of the normalized coordinates relative to the instance center through spatial attention calculation.

[0026] S42: The spatial attention features are sine feature encoding, convolution and dot product, and then softmax to obtain a feature representation that incorporates spatial location information. The attention calculation part has a center-aware attention representation.

[0027] Preferably, in step S5, the feature representation incorporating spatial location information and the 3D detection bounding box are input into the local model module to obtain an enhanced feature representation and a refined segmentation mask. The local model module is obtained by local modeling based on a vertex-guided cross-attention mechanism, including:

[0028] S51: Based on the feature representation incorporating spatial location information and the panoramic segmentation results, a masking domain is formed. The relative geometric guidance vectors of the vertices of the three-dimensional detection bounding box are used as modulation factors. Cross-attention calculation is performed according to the vertex-guided cross-attention response mechanism. The features within the masking domain are weighted and converged to output an enhanced feature representation.

[0029] S52: Update the pixel-level weight distribution of the enhanced feature representation to obtain a refined segmentation mask.

[0030] Preferably, step S34, based on the semantic segmentation and instance mask series, sums and completes the foreground and background results to obtain a panoramic segmentation result, including:

[0031] S341: Fill the corresponding regions of all instance mask series from high to low confidence by sorting all instance mask series by confidence level;

[0032] S342: Assign values ​​to the semantic segmentation results in the boundary regions, which are regions not covered by the instance mask series;

[0033] S343: For the boundary region, the semantic mask and the instance mask are fused, that is, the weighted combination of the two is taken in the boundary region, or the one with higher probability is selected first to obtain the panoramic segmentation result.

[0034] On the other hand, an instance-aware unified 3D object detection and segmentation apparatus is provided, which is applied to the instance-aware unified 3D object detection and segmentation method. The apparatus includes:

[0035] Initialize query module: used to initialize the query to obtain an initial query vector. The initialization of the query includes initializing the query based on the superpoint mapping.

[0036] Interaction enhancement module: used to input the initial query vector into the multi-head output structure to obtain the interaction-enhanced feature vector;

[0037] Multi-head output structure module: used to input the feature vector based on interaction enhancement into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch.

[0038] Spatial Aware Self-Attention Module: This module is used to input the 3D detection bounding box, interactive enhancement feature vector, and panoramic segmentation result into the spatial awareness self-attention model to obtain a feature representation that incorporates spatial location information. The spatial awareness self-attention model is obtained by global modeling using a spatial awareness self-attention mechanism. The spatial awareness self-attention model will directly incorporate the relative position of the instance center into the attention calculation part of the spatial awareness self-attention model. The attention calculation part has a center-aware attention representation.

[0039] Local model module: It is used to input the feature representation incorporating spatial location information and the 3D detection bounding box into the local model module to obtain enhanced feature representation and refined segmentation mask. The local model module is obtained by local modeling based on vertex-guided cross-attention mechanism.

[0040] On the other hand, an instance-aware unified 3D target detection and segmentation device is provided, the instance-aware unified 3D target detection and segmentation device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement the method described in any of the above-described instance-aware unified 3D target detection and segmentation methods.

[0041] On the other hand, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores program code, which can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

[0042] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0043] A unified architecture is proposed to simultaneously perform 3D object detection and 3D scene segmentation. By abandoning restrictive mask attention and introducing instance relative position encoding, the model can directly output segmentation masks and detection bounding boxes without cross-task result transformation and post-processing. Through this global-to-local perception process, the model bridges the gap between context and instance details while maintaining computational efficiency. It can synchronously and accurately predict detection boxes and segmentation masks, reducing system complexity and engineering redundancy. Unified modeling enables detection and segmentation to support and enhance each other, significantly improving the overall 3D scene understanding capability, forming a scene-level global understanding, and achieving fine-grained instance-level perception. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of an instance-aware unified 3D target detection and segmentation method provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a multi-head output structure provided in an embodiment of the present invention;

[0047] Figure 3 This is a block diagram of an instance-aware unified 3D target detection and segmentation device provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the structure of an instance-aware unified three-dimensional target detection and segmentation device provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0050] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0051] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0052] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0053] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0054] This invention provides an instance-aware unified 3D object detection and segmentation method, which can be implemented by an instance-aware unified 3D object detection and segmentation device, which can be a terminal or a server. Figure 1 The flowchart shown is for an instance-aware unified 3D object detection and segmentation method. The processing flow of this method may include the following steps:

[0055] S1: Initialize the query to obtain the initial query vector. The initial query includes initializing the query based on the superpoint mapping.

[0056] Preferably, S1 includes:

[0057] S11: Receive a query group, which contains multiple queries;

[0058] S12: Map the query group to the same superpoint according to the superpoint mapping to obtain the initial query vector.

[0059] In some embodiments, query initialization is performed using superpoint pooling, enabling the query vector to accurately capture the target instance indicated by the language.

[0060] It should be noted that superpoint pooling transforms high-dimensional unordered point clouds into structured superpoint representations through semantic aggregation, geometric preservation, and modality alignment, enabling query vectors to possess high efficiency, discriminative power, and scalability.

[0061] S2: Input the initial query vector into the multi-head output structure to obtain the interactively enhanced feature vector;

[0062] Preferably, S2 includes:

[0063] S21: Receive the initial query vector;

[0064] S22: Based on the initial query vector, the multi-head output structure performs feature fusion and information interaction to obtain an interactively enhanced feature vector. The multi-head output structure includes a joint loop iterative decoding mechanism. The joint loop iterative decoding mechanism is based on the classification loss, mask loss and regression loss to jointly train the network. The classification loss is used for supervised detection prediction, the mask loss is used for supervised segmentation, and the regression loss is used for supervised regression of the 3D detection bounding box.

[0065] In some embodiments, multi-head output is used: segmentation masks and detection bounding boxes are predicted simultaneously, avoiding post-processing of cross-task result transformation. The detection head regresses the 3D detection box parameters, and the segmentation head generates masks through interaction with scene features, achieving mutually beneficial learning under shared representation.

[0066] It should be noted that the loss function is designed as follows:

[0067] a. Classification loss:

[0068] L_cls = CE(p, y)

[0069] CE is the cross-entropy loss function, p is the classification probability output by the network, and y is the class label.

[0070] b. Masking loss:

[0071] L_mask = BCE(m, y) + 1 – 2 (m y + 1) / (|m| + |y| + 1)

[0072] BCE represents the binary cross-entropy loss function, y represents the binary mask label, and m represents the predicted mask probability.

[0073] c. Regression loss

[0074] IoU = area(b_p ∩ b_gt) / area(b_p ∪ b_gt)

[0075] L_reg = 1 – IoU

[0076] IoU represents the intersection-union ratio between the predicted bounding box b_p and the labeled bounding box b_gt. Since a larger ratio indicates greater similarity, the upper bound of 1 minus IoU is used as the loss function. area() represents the area of ​​the total region.

[0077] S3: Input the feature vector based on interactive enhancement into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch.

[0078] Preferably, S3 includes:

[0079] S31: The multi-head output structure processes the interactively enhanced feature vector through attention bridging and shared channel feature compression to generate a shared feature representation that can be used by multiple heads;

[0080] S32: In the detection task branch, based on the shared feature representation, calculate the bounding box parameters to obtain the bounding box information, and form a three-dimensional detection bounding box based on the bounding box information;

[0081] S33: In the segmentation task branch, based on the shared feature representation, semantic and instance analysis are performed to obtain semantic segmentation and instance mask series;

[0082] S34: Based on the semantic segmentation and instance mask series, sum and complete the foreground and background results to obtain the panoramic segmentation result.

[0083] like Figure 2 As shown, preferably, S34 includes:

[0084] S341: Fill the corresponding regions of all instance mask series from high to low confidence by sorting all instance mask series by confidence level;

[0085] S342: Assign values ​​to the semantic segmentation results in the boundary regions, which are regions not covered by the instance mask series;

[0086] S343: For the boundary region, the semantic mask and the instance mask are fused, that is, the weighted combination of the two is taken in the boundary region, or the one with higher probability is selected first to obtain the panoramic segmentation result.

[0087] S4: Input the 3D detection bounding box, interactively enhanced feature vector and panoramic segmentation result into the spatially perceptive self-attention model to obtain a feature representation that incorporates spatial location information. The spatially perceptive self-attention model is obtained by global modeling using a spatially perceptive self-attention mechanism. The spatially perceptive self-attention model will directly incorporate the relative position of the instance center into the attention calculation part of the spatially perceptive self-attention model. The attention calculation part has a center-aware attention representation.

[0088] Preferably, S4 includes:

[0089] S41: Based on the interactively enhanced feature vector and the panoramic segmentation results, the 3D detection bounding box is used as the reference for the instance center. The spatial attention feature is obtained by introducing the position encoding of the normalized coordinates relative to the instance center through spatial attention calculation.

[0090] S42: The spatial attention features are sine feature encoding, convolution and dot product, and then softmax to obtain a feature representation that incorporates spatial location information. The attention calculation part has a center-aware attention representation.

[0091] In some embodiments, traditional self-attention is based solely on feature similarity, making it difficult to explicitly inject 3D geometric relationships. Spatial Aware Self Attention (SASA) directly incorporates the relative position of the instance center into the attention calculation:

[0092] p = MLP(Tranfrom(c_box - c_sp))

[0093] SASA = SoftMax((q + p) (k + p)T) v

[0094] In this design, MLP stands for Multilayer Perceptron, which introduces learnable parameters for adaptive spatial representation. Transform represents a transformation function (such as a sine curve) for dimension alignment. c_box and c_sp represent the center point of the predicted bounding box and the current query reference point, respectively. SoftMax is the normalization function, q, k, and v are the same-dimensional representations of the query obtained through different MLPs, and T represents transpose. This design enables each query to have a "center-aware" global context, which is beneficial for global layout understanding and reliable target localization, while providing a coherent scene semantic background for segmentation.

[0095] S5: Input the feature representation incorporating spatial location information and the three-dimensional detection bounding box into the local model module to obtain the enhanced feature representation and refined segmentation mask. The local model module is obtained by local modeling based on the vertex-guided cross-attention mechanism.

[0096] Preferably, S5 includes:

[0097] S51: Based on the feature representation incorporating spatial location information and the panoramic segmentation results, a masking domain is formed. The relative geometric guidance vectors of the vertices of the three-dimensional detection bounding box are used as modulation factors. Cross-attention calculation is performed according to the vertex-guided cross-attention response mechanism. The features within the masking domain are weighted and converged to output an enhanced feature representation.

[0098] S52: Update the pixel-level weight distribution of the enhanced feature representation to obtain a refined segmentation mask.

[0099] In some embodiments, local modeling (vertex-guided cross attention) has certain drawbacks. For example, traditional cross attention suffers from an architectural disconnect between detection and segmentation, characterized by "positional encoding vs. hard masking." Vertex-guided cross attention (VGCA) provides soft-masking geometric guidance based on vertex relationships:

[0100] p_vg = MLP(Tranfrom(cat(sum_{i=1,2…,8} (c_sp - vi))))

[0101] VGCA = SoftMax((q + p_vg) * k T ) * v

[0102] Where p_vg is the vertex-guided geometry vector, vi is the vertex corresponding to the predicted bounding box, and k TIt calculates the dot product. Compared to the binary hard mask used in the original masked attention, it provides continuous, fine-grained spatial guidance. For detection, it preserves the geometric flexibility and differentiability constraints of the bounding box; for segmentation, it refines instance boundaries and local details. This design eliminates the conflict between the attention mechanisms of the two tasks and unifies the geometric inductive bias across tasks.

[0103] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0104] Figure 3 This is a block diagram illustrating an instance-aware unified 3D object detection and segmentation apparatus according to an exemplary embodiment. The apparatus is used in an instance-aware unified 3D object detection and segmentation method. (Refer to...) Figure 3 The device includes an initialization query module, an interaction enhancement module, a multi-head output structure module, a spatial awareness self-attention module, and a local model module.

[0105] Initialize query module: used to initialize the query to obtain an initial query vector. The initialization of the query includes initializing the query based on the superpoint mapping.

[0106] Interaction enhancement module: used to input the initial query vector into the multi-head output structure to obtain the interaction-enhanced feature vector;

[0107] Multi-head output structure module: used to input the feature vector based on interaction enhancement into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch.

[0108] Spatial Aware Self-Attention Module: This module is used to input the 3D detection bounding box, interactive enhancement feature vector, and panoramic segmentation result into the spatial awareness self-attention model to obtain a feature representation that incorporates spatial location information. The spatial awareness self-attention model is obtained by global modeling using a spatial awareness self-attention mechanism. The spatial awareness self-attention model will directly incorporate the relative position of the instance center into the attention calculation part of the spatial awareness self-attention model. The attention calculation part has a center-aware attention representation.

[0109] Local model module: It is used to input the feature representation incorporating spatial location information and the 3D detection bounding box into the local model module to obtain enhanced feature representation and refined segmentation mask. The local model module is obtained by local modeling based on vertex-guided cross-attention mechanism.

[0110] Figure 4 This is a schematic diagram of the structure of an instance-aware unified 3D target detection and segmentation device provided in an embodiment of the present invention, as shown below. Figure 4As shown, an instance-aware unified 3D object detection and segmentation device may include the above-mentioned Figure 3 The example shown is an instance-aware unified 3D object detection and segmentation device. Optionally, the instance-aware unified 3D object detection and segmentation device 410 may include a first processor 2001.

[0111] Optionally, the instance-aware unified 3D target detection and segmentation device 410 may also include a memory 2002 and a transceiver 2003.

[0112] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0113] The following is combined Figure 4 The following is a detailed introduction to the various components of the instance-aware unified 3D target detection and segmentation device 410:

[0114] The first processor 2001 is the control center of the instance-aware unified 3D target detection and segmentation device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0115] Optionally, the first processor 2001 can perform various functions of the instance-aware unified 3D target detection and segmentation device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0116] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0117] In a specific implementation, as one example, the instance-aware unified 3D target detection and segmentation device 410 may also include multiple processors, for example... Figure 4The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0118] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0119] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the instance-aware unified three-dimensional target detection and segmentation device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0120] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0121] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0122] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the instance-aware unified 3D target detection and segmentation device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0123] It should be noted that, Figure 4 The structure of the instance-aware unified 3D target detection and segmentation device 410 shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0124] Furthermore, the technical effect of the instance-aware unified 3D target detection and segmentation device 410 can be referred to the technical effect of the instance-aware unified 3D target detection and segmentation method described in the above method embodiments, and will not be repeated here.

[0125] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0126] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0127] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0128] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0129] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0130] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0131] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0133] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0136] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An instance-aware unified 3D object detection and segmentation method, characterized in that, The method includes: S1: Initialize the query to obtain the initial query vector. The initialization query includes initializing the query based on the superpoint mapping. S2: Input the initial query vector into the multi-head output structure to obtain the interactively enhanced feature vector; S3: Input the feature vector based on interactive enhancement into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch. S4: Input the 3D detection bounding box, interactively enhanced feature vector and panoramic segmentation result into the spatially perceptive self-attention model to obtain a feature representation that incorporates spatial location information. The spatially perceptive self-attention model is obtained by global modeling using a spatially perceptive self-attention mechanism. The spatially perceptive self-attention model will directly incorporate the relative position of the instance center into the attention calculation part of the spatially perceptive self-attention model. The attention calculation part has a center-aware attention representation. S5: Input the feature representation incorporating spatial location information and the 3D detection bounding box into the local model module to obtain enhanced feature representation and refined segmentation mask. The local model module is obtained by local modeling based on a vertex-guided cross-attention mechanism, specifically including: S51: Based on the feature representation incorporating spatial location information and the panoramic segmentation results, a masking domain is formed. The relative geometric guidance vectors of the vertices of the three-dimensional detection bounding box are used as modulation factors. Cross-attention calculation is performed according to the vertex-guided cross-attention response mechanism. The features within the masking domain are weighted and converged to output an enhanced feature representation. The vertex-guided cross-attention response mechanism provides soft mask-style geometric guidance based on the relative relationship of the bounding box vertices. S52: Update the pixel-level weight distribution of the enhanced feature representation to obtain a refined segmentation mask.

2. The instance-aware unified 3D target detection and segmentation method according to claim 1, characterized in that, The initial query in S1 obtains an initial query vector. The initial query includes initializing the query based on the superpoint mapping, including: S11: Receive a query group, which contains multiple queries; S12: Map the query group to the same superpoint according to the superpoint mapping to obtain the initial query vector.

3. The instance-aware unified 3D target detection and segmentation method according to claim 1, characterized in that, The process S2 inputs the initial query vector into a multi-head output structure to obtain an interactively enhanced feature vector, including: S21: Receive the initial query vector; S22: Based on the initial query vector, the multi-head output structure performs feature fusion and information interaction to obtain an interactively enhanced feature vector. The multi-head output structure includes a joint loop iterative decoding mechanism. The joint loop iterative decoding mechanism is based on the classification loss, mask loss and regression loss to jointly train the network. The classification loss is used for supervised detection prediction, the mask loss is used for supervised segmentation, and the regression loss is used for supervised regression of the 3D detection bounding box.

4. The instance-aware unified 3D target detection and segmentation method according to claim 1, characterized in that, S3 inputs the feature vector based on interactive enhancement into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch, including: S31: The multi-head output structure processes the interactively enhanced feature vector through attention bridging and shared channel feature compression to generate a shared feature representation that can be used by multiple heads; S32: In the detection task branch, based on the shared feature representation, calculate the bounding box parameters to obtain the bounding box information, and form a three-dimensional detection bounding box based on the bounding box information; S33: In the segmentation task branch, based on the shared feature representation, semantic and instance analysis are performed to obtain semantic segmentation and instance mask series; S34: Based on the semantic segmentation and instance mask series, sum and complete the foreground and background results to obtain the panoramic segmentation result.

5. The instance-aware unified 3D target detection and segmentation method according to claim 1, characterized in that, The S4 inputs the 3D detection bounding box, interactively enhanced feature vectors, and panoramic segmentation results into the spatially aware self-attention model to obtain a feature representation incorporating spatial location information. This spatially aware self-attention model is obtained through global modeling using a spatially aware self-attention mechanism. The model directly integrates the relative position of the instance center into its attention calculation part, which possesses a center-aware attention representation, including: S41: Based on the interactively enhanced feature vector and the panoramic segmentation results, the 3D detection bounding box is used as the reference for the instance center. The spatial attention feature is obtained by introducing the position encoding of the normalized coordinates relative to the instance center through spatial attention calculation. S42: The spatial attention features are sinusoidally encoded, convolved, and dot-producted, and then processed by the softmax function to obtain a feature representation that incorporates spatial location information. The attention calculation part has a center-aware attention representation.

6. The instance-aware unified 3D target detection and segmentation method according to claim 4, characterized in that, S34, based on the semantic segmentation and instance mask series, sums and completes the foreground and background results to obtain a panoramic segmentation result, including: S341: Fill the corresponding regions of all instance mask series from high to low confidence by sorting all instance mask series by confidence level; S342: Assign values ​​to the semantic segmentation results in the boundary regions, which are regions not covered by the instance mask series; S343: For the boundary region, the semantic mask and the instance mask are fused, that is, the weighted combination of the two is taken in the boundary region, or the one with higher probability is selected first to obtain the panoramic segmentation result.

7. An instance-aware unified 3D target detection and segmentation device, wherein the instance-aware unified 3D target detection and segmentation device is used to implement the instance-aware unified 3D target detection and segmentation method as described in any one of claims 1-6, characterized in that, The device includes: Initialize query module: used to initialize the query to obtain an initial query vector. The initialization of the query includes initializing the query based on the superpoint mapping. Interaction enhancement module: used to input the initial query vector into the multi-head output structure to obtain the interaction-enhanced feature vector; Multi-head output structure module: used to input the interactively enhanced feature vector into the multi-head output structure to obtain the panoramic segmentation result and the 3D detection bounding box. The multi-head output structure includes a segmentation task branch and a detection task branch. Spatial Aware Self-Attention Module: This module is used to input the 3D detection bounding box, interactive enhancement feature vector, and panoramic segmentation result into the spatial awareness self-attention model to obtain a feature representation that incorporates spatial location information. The spatial awareness self-attention model is obtained by global modeling using a spatial awareness self-attention mechanism. The spatial awareness self-attention model will directly incorporate the relative position of the instance center into the attention calculation part of the spatial awareness self-attention model. The attention calculation part has a center-aware attention representation. Local Model Module: This module takes the feature representation incorporating spatial location information and the 3D detection bounding box as input to obtain enhanced feature representations and refined segmentation masks. The local model module is based on a vertex-guided cross-attention mechanism for local modeling and specifically includes: S51: Based on the feature representation incorporating spatial location information and the panoramic segmentation results, a masking domain is formed. The relative geometric guidance vectors of the vertices of the three-dimensional detection bounding box are used as modulation factors. Cross-attention calculation is performed according to the vertex-guided cross-attention response mechanism. The features within the masking domain are weighted and converged to output an enhanced feature representation. The vertex-guided cross-attention response mechanism provides soft mask-style geometric guidance based on the relative relationship of the bounding box vertices. S52: Update the pixel-level weight distribution of the enhanced feature representation to obtain a refined segmentation mask.

8. An instance-aware unified 3D target detection and segmentation device, characterized in that, The instance-aware unified 3D object detection and segmentation processor; a memory storing computer-readable instructions, which, when executed by the processor, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images