Three-dimensional directivity target segmentation method of image enhancement prompt decoding network

Abstract

The invention discloses a three-dimensional directivity target segmentation method of an image enhancement prompt decoding network. The method comprises the following steps: S1, feature extraction; s2, generating a multi-view semantic embedding strategy and query; s3, prompting the perception decoder; s4, training loss: training loss comprises basic loss, probability loss and comparative learning loss; according to the method, the reasoning ability of the model is released by using multi-view images and task-driven information from top to bottom; the MSE uses CLIP to extract two-dimensional image features, and then the two-dimensional image features are fused with three-dimensional point cloud features to significantly enhance visual features; a PAD uses a task-driven prompt signal to guide a decoding process, through a task-driven prompt module, a prompt emphasizing correlation between each query and a text is generated, task-specific information is effectively injected into a model, and learning complexity is remarkably reduced.

Description

Technical Field

[0001] The invention relates to the field of computer processing technology, and in particular to a three-dimensional directional target segmentation method of an image enhancement prompt decoding network. Background Art

[0002] 3D Referring Expression Segmentation (3D-RES) has important potential applications in virtual reality, augmented reality, robot navigation, and human-computer interaction. The goal of this task is to segment 3D objects in a 3D point cloud scene based on a given directional representation.

[0003] Traditional 3D-RES methods adopt a two-stage paradigm: first, they use an instance segmentation network to generate candidate targets, then match these candidate targets with text, calculate the matching score, and obtain the final segmentation result. However, this paradigm is lacking in both efficiency and effectiveness. Therefore, recent research has turned to a query-based single-stage paradigm. For example, 3D-STMN achieves effective segmentation by directly matching text with superpoints, while MCLN and some other works promote joint improvement by jointly training 3D-RES and other tasks.

[0004] However, although existing methods have achieved good results, they still have certain limitations: (1) Feature ambiguity: They only use point cloud data for visual feature extraction. Due to the influence of factors such as lighting, angle, sampling rate, etc. on data acquisition, point cloud data often cannot reproduce the real-world scenes well and there is information loss. Therefore, it is difficult to extract excellent features from point cloud data alone. In addition, the scale of point cloud datasets is usually small, which limits the development of large-scale visual language pre-training models in the three-dimensional field. Features extracted using a purely visual three-dimensional backbone network are more difficult to align with text features in subsequent processing. (2) Intent ambiguity: For all queries, they are given equal importance, similar to instance segmentation. However, in the 3D-RES task, only the target object described in the text needs to be segmented. Ideally, queries related to the text should be given priority. However, existing methods do not emphasize these relevant queries, resulting in the model having to implicitly learn the distinction between relevant and irrelevant queries, which significantly increases the difficulty of the learning process. Summary of the invention

[0005] To solve the above problems, the present invention proposes a three-dimensional directional target segmentation method of an image enhancement hint decoding network.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A three-dimensional directional target segmentation method of an image enhancement hint decoding network comprises the following steps:

[0008] S1. Feature extraction: For the target object description text, the pre-trained RoBERTa model is used to extract word-level features; for point cloud data, a segmenter is first used to segment superpoints, and the superpoint positions are obtained by average pooling, and then the pre-trained sparse 3DU-Net network and superpoint pooling layer are used to obtain superpoint features; for multi-view 2D images of point cloud scenes, the CLIP visual encoder and interpolation are used to obtain pixel-level features of the images;

[0009] S2. Multi-view semantic embedding strategy and query generation: The pixel-level features of the two-dimensional image are projected into the three-dimensional point cloud space and super-point pooling is performed, and then the visual features are fused with the super-point features extracted from the point cloud data; the farthest point sampling is used for the super-points to reduce the subsequent calculation overhead, and the spatial perception attention is used to perform multiple rounds of iterations to compensate for the lack of spatial position relationship caused by the introduction of two-dimensional image features, and the text interaction sampling is used to obtain the decoder query from the visual features;

[0010] S3, cue-aware decoder: By using text features as the query in the attention mechanism and the decoder query as the key and value for cross-attention operation, the text features are updated and the correlation between the decoder query and the text features is obtained; the probability filtering correlation of the decoder query corresponding to the target object output by the prediction head is then used to obtain the task-driven cue signal to guide the decoding process;

[0011] S4. Training loss: The training loss includes base loss, probability loss and contrastive learning loss. The base loss includes binary cross entropy loss and Dice loss applied on the prediction mask. The probability loss is the binary cross entropy loss applied on the probability of the decoder query corresponding to the target object output by the prediction head. The contrastive learning loss is the dense semantic alignment loss applied on the query and text features.

[0012] Preferably, the specific process of step S1 is:

[0013] S11. Given a text description of the target object, use the pre-trained RoBERTa model to extract word-level features E, and then convert the word-level features E into text features T through linear projection, so that there is a unified feature dimension d in the decoder. The conversion formula is:

[0014] T=EW t ,

[0015] Among them, E represents word-level features, T represents text features, and W t represents randomly initialized learnable parameters, Nt Indicates the number of words, C t represents the dimension of the feature, d represents the unified feature dimension, N t ×C t dimensional vector space, N t ×d-dimensional vector space, Represents C t ×d-dimensional vector space;

[0016] S12. Given a point cloud scene Among them, N p represents the number of points, each with a three-dimensional coordinate and an f-dimensional auxiliary feature vector, N p ×(3+f)-dimensional vector space; use a pre-trained sparse 3D U-Net network to extract point-level features Among them, C p represents the feature dimension, N p ×C p dimensional vector space; segment the point cloud and obtain N s Super Point And for the point-level feature F 3d Perform super-point pooling operation to obtain three-dimensional super-point features Among them, N s Indicates the number of super points, N s ×C p dimensional vector space; then use a multi-layer perceptron to change the dimension to d to obtain three-dimensional visual features N s ×d-dimensional vector space, the calculation formula is:

[0017] V 3d =MLP(SPPool(F 3d )),

[0018] Among them, MLP(·) is a learnable multi-layer perceptron, and SPPool(·) is a super point pooling operation;

[0019] S13, N corresponding to the given point cloud scene I Pictures from different angles The CLIP visual encoder pre-aligned with the visual language is used to extract block-level 2D features; the 2D features are then upsampled back to the original image resolution through interpolation to obtain pixel-level 2D features. Among them, C I Represents the feature dimension, H represents the height of the image, and W represents the width of the image. Indicates H×W×C I dimensional vector space.

[0020] Preferably, the specific process of step S2 is:

[0021] S21, project the two-dimensional pixel coordinates into the three-dimensional point cloud space according to the camera parameters. For a pixel coordinate (u, v), given the camera intrinsic parameters External Reference And depth represents a 3×3 dimensional vector space, represents a 3×1 dimensional vector space, Represents a real number set, and obtains the three-dimensional coordinates (x, y, z) through projection. The calculation formula for projection is:

[0022]

[0023] Among them, the superscript T represents the matrix transpose, and the superscript -1 represents the matrix inversion;

[0024] S22. After projection, all two-dimensional pixel features are assigned a three-dimensional coordinate In the point cloud scene, the three-dimensional coordinate p 3d A sphere query operation is performed to inject multi-view semantics into the point cloud and assign each pixel feature to a point inside a sphere with the corresponding three-dimensional coordinate of the pixel as the center. For points existing inside multiple spheres, the final multi-view feature is the average of all multi-view features obtained, and the multi-view semantic feature of each point is obtained. N p ×C I dimensional vector space; then segment the point cloud, perform super point pooling operations and multi-layer perceptron processing to obtain two-dimensional visual features N s ×d-dimensional vector space; then V 3d and V 2d Add up the visual features

[0025] S23, perform farthest point sampling on the super points to reduce the number of super points to 2m. The specific calculation formula is:

[0026] Q seed =v[FPS(p sp )],

[0027] Among them, FPS(·) represents the farthest point sampling algorithm; V[·] represents obtaining elements according to the index; P sp represents the coordinates of the superpoint, N s×3-dimensional vector space; Q seed It is a seed query. Represents a 2m×d dimensional vector space, m<<N s ;

[0028] S24, using spatially aware attention to introduce explicit 3D spatial relationships for enhanced spatial position modeling, for seed query Q seed Perform spatial perception attention modeling. The modeling process is: construct a k-nearest neighbor matrix Among them, the element M in the i-th row and j-th column ij Indicates whether the j-th query is the k-most recent query before the ith query. If yes, it is True, otherwise it is False. k is a hyperparameter and is set to 8. represents a 2m×2m dimensional vector space; the query coordinates are the corresponding superpoint coordinates, and then the k nearest neighbor matrix M is used as a mask in the seed query Q seed Perform self-attention operation on it to generate output As the input of the text interaction sampling module, Represents a 2m×d dimensional vector space:

[0029] Q=SPA(Q seed )=Masked_Self(Q seed , M),

[0030] Among them, Q represents the k nearest neighbor matrix M as a mask in the seed query Q seed The output of the self-attention operation on ; Masked_Self(·) represents the masked self-attention operation; SPA(·) represents the spatially aware attention operation;

[0031] S25. Resample the output Q using the text interaction sampling module, followed by a multi-layer perceptron to generate queries for input to the decoder:

[0032] Q0=MLP(Sample(Q,T)),

[0033] Among them, Q0 is the initial query, represents an m×d-dimensional vector space; Sample(·) represents the text interaction sampling module; Q represents the k-nearest neighbor matrix M as a mask in the seed query Q seed The output of the self-attention operation on ; T represents the text feature.

[0034] Preferably, the specific process of step S3 is:

[0035] S31. Use the text feature T as the Query in the attention mechanism, query Q lAs Key and Value for cross attention operation, it is used to measure the relevance between text and query. The calculation formula is:

[0036]

[0037] in, represents the text features of the lth layer, N t ×d-dimensional vector space; A l represents the attention score of the lth layer, N t ×m-dimensional vector space; Q l represents the query at level l, represents the m×d dimensional vector space; Cross(·) represents the cross attention operation; T represents the text feature; by converting the attention score A l Sum along the first dimension to get the relevance score representing the closeness of the connection between the query and the text represents an m-dimensional vector space;

[0038] S32, using the probability of the output of the previous layer prediction head Perform a filter query with probability Prob l-1 Indicates the possibility that the query corresponds to the target instance; for queries with probabilities lower than the threshold r, their relevance scores are set to negative infinity, indicating that their values ​​will be 0 after applying the Softmax function; the Softmax function is then applied to the relevance score and the result is multiplied by the query to generate a hint to guide the model to distinguish between relevant and irrelevant queries. The guidance process is:

[0039]

[0040]

[0041]

[0042] in, Represents the jth element of the relevance score after filtering at the lth layer; Represents the jth element of the probability of the output of the l-1th layer; r is a hyperparameter, set to 0.75; represents the jth element of the lth level relevance score; Pt l Indicates a prompt, represents an m×d-dimensional vector space; Q l represents the query at level l; represents the correlation score after filtering at the lth layer; Softmax(·) represents the softmax function; represents the query with hints at level l, represents a 2m×d dimensional vector space; Concat(·) represents a concatenation operation;

[0043] The decoding process uses queries with hints Integrate text features And the visual features V:

[0044]

[0045] in, represents the query after the integration of the l-th layer features; Abandon(·) represents discarding the prompt; Cross(·) represents the cross attention operation; Self(·) represents the self-attention operation; represents the query with hints at level l; represents the text features of the lth layer; V represents the visual features;

[0046] At the end of each layer, spatially aware attention is used again to enhance features and generate the query Q for the next layer. l+1 , the calculation formula is:

[0047]

[0048] Among them, Ql+1 represents the query of the next layer; represents the query after the integration of the l-th layer features; SPA(·) represents the spatial perception attention operation;

[0049] Before entering the next level, Q l+1 After a prediction head is used to generate a prediction mask Mask l and the predicted probability Prob l , the calculation formula is:

[0050] Mask l =Q l+1 (VW mask ) T ,

[0051] Prob l =Q l+1 W prob ,

[0052] Among them, W mask is the learning parameter, represents a d×d dimensional vector space; W prob is the learning parameter, Represents a d×1 dimensional vector space; Mask l represents the mask for each query prediction, Represents m×N s dimensional vector space; Prob lrepresents the probability that each query corresponds to the target instance, Represents an m-dimensional vector space; the mask generated by the query with the highest Prob value is used as the final result.

[0053] Preferably, in step S4,

[0054] The base loss is only applied to the query corresponding to the target object, and the calculation formula is:

[0055]

[0056] in, Indicates basic loss; M + The mask representing the query output corresponding to the target object; M tgt represents the true mask; BCE(·) represents the binary cross entropy loss function; DICE(·) represents the Dice loss function;

[0057] The probability loss is used to supervise Prob, and the calculation formula is:

[0058]

[0059] in, Represents probability loss; label L tgt Indicates whether the query corresponds to the target instance, L tgt ∈{0,1} m , the corresponding target instance is 1, otherwise it is 0;

[0060] The contrastive learning loss Used to align text features and corresponding queries;

[0061] The final loss is the weighted sum of the three:

[0062]

[0063] in, represents the final loss; λ b ,λ p and λ c are all hyperparameters and are set to 1.0, 0.1, and 0.1 respectively.

[0064] After adopting the above technical solution, the present invention has the following beneficial effects: the present invention uses multi-view images and task-driven information in a top-down approach to release the reasoning ability of the model. In order to solve the problem of feature ambiguity, the present invention proposes a multi-view semantic embedding (MSE) strategy. MSE uses CLIP to extract two-dimensional image features and then fuses them with three-dimensional point cloud features to significantly enhance visual features; in addition, spatial-aware attention is used to solve the problem of lack of spatial position relationship in two-dimensional features; this method produces visual features with superior representational capabilities, contains text prior knowledge from CLIP, and can be better aligned with text features. In order to solve the problem of intent ambiguity, the present invention designs a prompt-aware decoder (PAD), which uses task-driven prompt signals to guide the decoding process. Through the task-driven prompt module, prompts that emphasize the relevance of each query to the text are generated, effectively injecting task-specific information into the model and significantly reducing the complexity of learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a flow chart of the present invention;

[0066] Figure 2 This is a diagram of the image enhancement hint decoding network model architecture of the present invention;

[0067] Figure 3 This is a quantitative comparison result diagram of the present invention and the existing methods on the Multi3DRefer dataset;

[0068] Figure 4 This is a quantitative comparison result diagram of the present invention and the existing method on the ScanRefer dataset;

[0069] Figure 5 This is a qualitative comparison result diagram between the present invention and MDIN on the ScanRefer dataset. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0071] like Figures 1 to 5 As shown, a three-dimensional directional target segmentation method of an image enhancement hint decoding network includes the following steps:

[0072] S1. Feature extraction: For the target object description text, the pre-trained RoBERTa model is used to extract word-level features; for point cloud data, a segmenter is first used to segment superpoints, and the superpoint positions are obtained by average pooling. Then, the pre-trained sparse 3D U-Net network and superpoint pooling layer are used to obtain superpoint features; for multi-view 2D images of point cloud scenes, the CLIP visual encoder and interpolation are used to obtain pixel-level features of the images;

[0073] The specific process of step S1 is:

[0074] S11. Given a text description of the target object, use the pre-trained RoBERTa model to extract word-level features E, and then convert the word-level features E into text features T through linear projection, so that there is a unified feature dimension d in the decoder. The conversion formula is:

[0075] T=EW t ,

[0076] Among them, E represents word-level features, T represents text features, and W t represents randomly initialized learnable parameters, N t Indicates the number of words, C t represents the dimension of the feature, d represents the unified feature dimension, N t ×C t dimensional vector space, N t ×d-dimensional vector space, Represents C t ×d-dimensional vector space;

[0077] S12. Given a point cloud scene Among them, N p represents the number of points, each with a three-dimensional coordinate and an f-dimensional auxiliary feature vector, N p ×(3+f)-dimensional vector space; use a pre-trained sparse 3D U-Net network to extract point-level features Among them, C p represents the feature dimension, N p ×C p dimensional vector space; segment the point cloud and obtain N s Super Point And for the point-level feature F 3d Perform super-point pooling operation to obtain three-dimensional super-point features Among them, N s Indicates the number of super points, Ns ×C p dimensional vector space; then use a multi-layer perceptron to change the dimension to d to obtain three-dimensional visual features N s ×d-dimensional vector space, the calculation formula is:

[0078] V 3d =MLP(SPPool(F 3d )),

[0079] Among them, MLP(·) is a learnable multi-layer perceptron, and SPPool(·) is a super point pooling operation;

[0080] S13, N corresponding to the given point cloud scene I Pictures from different angles The CLIP visual encoder pre-aligned with the visual language is used to extract block-level 2D features; the 2D features are then upsampled back to the original image resolution through interpolation to obtain pixel-level 2D features. Among them, C i Represents the feature dimension, H represents the height of the image, and W represents the width of the image. Indicates H×W×C I dimensional vector space;

[0081] S2. Multi-view semantic embedding strategy and query generation: The pixel-level features of the two-dimensional image are projected into the three-dimensional point cloud space and super-point pooling is performed, and then the visual features are fused with the super-point features extracted from the point cloud data; the farthest point sampling is used for the super-points to reduce the subsequent calculation overhead, and the spatial perception attention is used to perform multiple rounds of iterations to compensate for the lack of spatial position relationship caused by the introduction of two-dimensional image features, and the text interaction sampling is used to obtain the decoder query from the visual features;

[0082] The specific process of step S2 is:

[0083] S21, project the two-dimensional pixel coordinates into the three-dimensional point cloud space according to the camera parameters. For a pixel coordinate (u, v), given the camera intrinsic parameters External Reference And depth represents a 3×3 dimensional vector space, represents a 3×1 dimensional vector space, Represents a real number set, and obtains the three-dimensional coordinates (x, y, z) through projection. The calculation formula for projection is:

[0084]

[0085] Among them, the superscript T represents the matrix transpose, and the superscript -1 represents the matrix inversion;

[0086] S22. After projection, all two-dimensional pixel features are assigned a three-dimensional coordinate In the point cloud scene, the three-dimensional coordinate p 3d A sphere query operation is performed to inject multi-view semantics into the point cloud and assign each pixel feature to a point inside a sphere with the corresponding three-dimensional coordinate of the pixel as the center. For points existing inside multiple spheres, the final multi-view feature is the average of all multi-view features obtained, and the multi-view semantic feature of each point is obtained. N p ×C I dimensional vector space; then segment the point cloud, perform super point pooling operations and multi-layer perceptron processing to obtain two-dimensional visual features N s ×d-dimensional vector space; then V 3d and V 2d Add up the visual features

[0087] S23, perform farthest point sampling on the super points to reduce the number of super points to 2m. The specific calculation formula is:

[0088] Q seed =V[FPS(p sp )],

[0089] Among them, FPS(·) represents the farthest point sampling algorithm; V[·] represents obtaining elements according to the index; p sp represents the coordinates of the superpoint, N s ×3-dimensional vector space; Q seed It is a seed query. Represents a 2m×d dimensional vector space, m<<N s ;

[0090] S24, using spatially aware attention to introduce explicit 3D spatial relationships for enhanced spatial position modeling, for seed query Q seed Perform spatial perception attention modeling. The modeling process is: construct a k-nearest neighbor matrix Among them, the element M in the i-th row and j-th column ij Indicates whether the j-th query is the k-most recent query before the ith query. If yes, it is True, otherwise it is False. k is a hyperparameter and is set to 8. represents a 2m×2m dimensional vector space; the query coordinates are the corresponding superpoint coordinates, and then the k nearest neighbor matrix M is used as a mask in the seed query Q seed Perform self-attention operation on it to generate output As the input of the text interaction sampling module, Represents a 2m×d dimensional vector space:

[0091] Q=SPA(Q seed )=Masked_Self(Q seed , M),

[0092] Among them, Q represents the k nearest neighbor matrix M as a mask in the seed query Q seed The output of the self-attention operation on ; Masked_Self(·) represents the masked self-attention operation; SPA(·) represents the spatially aware attention operation;

[0093] S25. Resample the output Q using the text interaction sampling module, followed by a multi-layer perceptron to generate queries for input to the decoder:

[0094] Q0=MLP(Sample(Q,T)),

[0095] Among them, Q0 is the initial query, represents an m×d-dimensional vector space; Sample(·) represents the text interaction sampling module; Q represents the k-nearest neighbor matrix M as a mask in the seed query Q seed The output generated by the self-attention operation on ; T represents the text feature;

[0096] S3, cue-aware decoder: By using text features as the query in the attention mechanism and the decoder query as the key and value for cross-attention operation, the text features are updated and the correlation between the decoder query and the text features is obtained; the probability filtering correlation of the decoder query corresponding to the target object output by the prediction head is then used to obtain the task-driven cue signal to guide the decoding process;

[0097] The specific process of step S3 is:

[0098] S31. Use the text feature T as the Query in the attention mechanism, query Q l As Key and Value for cross attention operation, it is used to measure the relevance between text and query. The calculation formula is:

[0099]

[0100] in, represents the text features of the lth layer, N t ×d-dimensional vector space; A l represents the attention score of the lth layer, N t ×m-dimensional vector space; Q lrepresents the query at level l, represents the m×d dimensional vector space; Cross(·) represents the cross attention operation; T represents the text feature; by converting the attention score A l Sum along the first dimension to get the relevance score representing the closeness of the connection between the query and the text represents an m-dimensional vector space;

[0101] S32, using the probability of the output of the previous layer prediction head Perform a filter query with probability Prob l-1 Indicates the possibility that the query corresponds to the target instance; for queries with probabilities lower than the threshold r, their relevance scores are set to negative infinity, indicating that their values ​​will be 0 after applying the Softmax function; the Softmax function is then applied to the relevance score and the result is multiplied by the query to generate a hint to guide the model to distinguish between relevant and irrelevant queries. The guidance process is:

[0102]

[0103]

[0104]

[0105] in, Represents the jth element of the relevance score after filtering at the lth layer; Represents the jth element of the probability of the output of the l-1th layer; r is a hyperparameter, set to 0.75; represents the jth element of the lth level relevance score; Pt l Indicates a prompt, represents an m×d-dimensional vector space; Q l represents the query at level l; represents the correlation score after filtering at the lth layer; Softmax(·) represents the softmax function; represents the query with hints at level l, represents a 2m×d dimensional vector space; Concat(·) represents a concatenation operation;

[0106] The decoding process uses queries with hints Integrate text features And the visual features V:

[0107]

[0108] in, represents the query after the integration of the l-th layer features; Abandon(·) represents discarding the prompt; Cross(·) represents the cross attention operation; Self(·) represents the self-attention operation; represents the query with hints at level l; represents the text features of the lth layer; V represents the visual features;

[0109] At the end of each layer, spatially aware attention is used again to enhance features and generate the query Q for the next layer. l+1 , the calculation formula is:

[0110]

[0111] Among them, Q l+1 Indicates the query of the next layer; represents the query after the integration of the l-th layer features; SPA(·) represents the spatial perception attention operation;

[0112] Before entering the next level, Q l+1 After a prediction head is used to generate a prediction mask Mask l and the predicted probability Prob l , the calculation formula is:

[0113] Mask l =Q l+1 (VW mask ) T ,

[0114] Prob l =Q l+1 W prob ,

[0115] Among them, W mask is the learning parameter, represents a d×d dimensional vector space; W prob is the learning parameter, Represents a d×1 dimensional vector space; Mask l represents the mask for each query prediction, Represents m×N s dimensional vector space; Prob l represents the probability that each query corresponds to the target instance, Represents an m-dimensional vector space; the mask generated by the query with the highest Prob value is used as the final result;

[0116] S4, training loss: training loss includes base loss, probability loss and contrastive learning loss; base loss includes binary cross entropy loss and Dice loss applied on the prediction mask; probability loss is the binary cross entropy loss applied on the probability of the decoder query corresponding to the target object output by the prediction head; contrastive learning loss is the dense semantic alignment loss applied on the query and text features;

[0117] In step S4,

[0118] The base loss is only applied to the query corresponding to the target object, and the calculation formula is:

[0119]

[0120] in, Indicates basic loss; M + The mask representing the query output corresponding to the target object; M tgt represents the true mask; BCE(·) represents the binary cross entropy loss function; DICE(·) represents the Dice loss function;

[0121] The probability loss is used to supervise Prob, and the calculation formula is:

[0122]

[0123] in, Represents probability loss; label L tgt Indicates whether the query corresponds to the target instance, L tgt ∈{0, 1} m , the corresponding target instance is 1, otherwise it is 0;

[0124] The contrastive learning loss Used to align text features and corresponding queries;

[0125] The final loss is the weighted sum of the three:

[0126]

[0127] in, represents the final loss; λ b ,λ p and λ c are all hyperparameters and are set to 1.0, 0.1, and 0.1 respectively.

[0128] Specific experimental results:

[0129] The present invention is trained and tested on the ScanRefer dataset and the Multi3DRefer dataset respectively. The ScanRefer dataset is a test dataset for the 3D-RES task, containing 51,583 text descriptions pointing to a single object; the Multi3DRefer dataset is a test dataset for the 3D-GRES task, containing 61,926 text descriptions, of which 6,688 descriptions do not contain any qualified targets, and 13,178 descriptions contain multiple qualified targets. All experiments are implemented with PyTorch and trained on a single NVIDIA GeForce RTX3090 GPU.

[0130] The mean intersection over union (mIoU) and the accuracy of intersection over union over k (Acc@kIoU) are used as evaluation indicators. The accuracy of intersection over union over k indicates the proportion of description texts whose IoU between the overlapped part of the predicted mask and the true annotation is greater than k, where k∈0.25,0.5. In the 3D-GRES task, when the target pointed to by the text does not exist, if the model outputs a mask of all zeros, the mIoU is 1, otherwise the mIoU is 0.

[0131] like Figure 3 As shown (where the left and right sides of " / " represent the cases with and without distractors, respectively), the model of the present invention is significantly better than other methods in the 3D-GRES task, achieving an improvement of 5.4 points in mIoU and even 7.5 points in Acc@0.5. It can be observed that the model of the present invention significantly outperforms MDIN in all other cases except for the zero target case with distractors, which performs slightly worse. In particular, in single-target scenarios, the model of the present invention is nearly 10 points higher than MDIN in the Acc@0.5 indicator, regardless of whether there are distractors or not. This shows that the task-driven prompts of the present invention effectively guide the model to focus on more important queries, thereby achieving more accurate positioning of key targets.

[0132] Experiments were conducted on the 3D-RES task, such as Figure 4 As shown. On the ScanRefer dataset, the model of the present invention achieves advanced performance overall. Specifically, the model of the present invention outperforms the previous best model MDIN by 2.6, 1.8, and 1.9 points in Acc@0.25, Acc@0.5, and mIoU, respectively. It is worth noting that more significant improvements are observed in the model of the present invention in challenging scenes with multiple interfering objects. This suggests that the model of the present invention benefits from more robust multi-view semantic embedding and task-driven prompts, which effectively guide the model to focus on more critical information, thereby enhancing its discriminative ability to accurately identify target objects in multiple instances of the same category.

[0133] A set of representative examples are visualized in the ScanRefer dataset, such as Figure 5 As shown. Figure 5 It can be seen that the model of the present invention exhibits stronger reasoning ability than MDIN. Specifically, in case (a), there are no interfering objects in the scene, only one vending machine, but MDIN still cannot recognize it. This is because, in the ScanRefer training dataset with 36,665 samples, only 10 samples have vending machines as their targets, which is not enough for the model to recognize such objects. However, the large-scale pre-trained CLIP model does not suffer from this problem and can recognize vending machines well, allowing the model of the present invention to accurately locate the target. In case (b), the concept of "left" is involved, which depends on the perspective. Since the three-dimensional space theoretically contains an infinite number of perspectives, it is difficult for a pure three-dimensional model to distinguish between left and right. In contrast, the perspective in a two-dimensional image is fixed, which provides important help in dealing with this situation. Finally, in case (c), due to the powerful prompting ability of the task-driven prompt of the present invention, the model of the present invention can still accurately locate the target object even if there are nearly ten interfering objects.

[0134] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A three-dimensional directional target segmentation method using an image enhancement hint decoding network, characterized in that: The following steps are involved: S1. Feature extraction: For the target object description text, the pre-trained RoBERTa model is used to extract word-level features; for point cloud data, a segmenter is first used to segment superpoints, and the superpoint positions are obtained by average pooling. Then, the pre-trained sparse 3D U-Net network and superpoint pooling layer are used to obtain superpoint features; for multi-view 2D images of point cloud scenes, the CLIP visual encoder and interpolation are used to obtain pixel-level features of the images; S2, multi-view semantic embedding strategy and query generation: the pixel-level features of the two-dimensional image are projected into the three-dimensional point cloud space and super-point pooling is performed, and then fused with the super-point features extracted from the point cloud data to obtain visual features; The farthest point sampling is used for super points to reduce the subsequent calculation overhead. The spatial perception attention is used to make up for the lack of spatial position relationship caused by the introduction of two-dimensional image features through multiple rounds of iterations. The text interaction sampling is used to obtain the decoder query from the visual features. S3, cue-aware decoder: By using text features as the query in the attention mechanism and the decoder query as the key and value for cross-attention operation, the text features are updated and the correlation between the decoder query and the text features is obtained; the probability filtering correlation of the decoder query corresponding to the target object output by the prediction head is then used to obtain the task-driven cue signal to guide the decoding process; S4. Training loss: The training loss includes base loss, probability loss and contrastive learning loss. The base loss includes binary cross entropy loss and Dice loss applied on the prediction mask. The probability loss is the binary cross entropy loss applied on the probability of the decoder query corresponding to the target object output by the prediction head. The contrastive learning loss is the dense semantic alignment loss applied on the query and text features.

2. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 1, characterized in that: The specific process of step S1 is: S11. Given a text description of the target object, use the pre-trained RoBERTa model to extract word-level features E, and then convert the word-level features E into text features T through linear projection, so that there is a unified feature dimension d in the decoder. The conversion formula is: T=EW t , Among them, E represents word-level features, T represents text features, and W t represents randomly initialized learnable parameters, N t Indicates the number of words, C t represents the dimension of the feature, d represents the unified feature dimension, N t ×C t dimensional vector space, N t ×d-dimensional vector space, Represents C t ×d-dimensional vector space; S12. Given a point cloud scene Among them, N p represents the number of points, each with a three-dimensional coordinate and an f-dimensional auxiliary feature vector, N p ×(3+f)-dimensional vector space; use a pre-trained sparse 3D U-Net network to extract point-level features Among them, C p represents the feature dimension, N p ×C p dimensional vector space; segment the point cloud and obtain N s Super Point And for the point-level feature F 3d Perform super-point pooling operation to obtain three-dimensional super-point features Among them, N s Indicates the number of super points, N s ×C p dimensional vector space; then use a multi-layer perceptron to change the dimension to d to obtain three-dimensional visual features N s ×d-dimensional vector space, the calculation formula is: V 3d =MLP(SPPool(F 3d )), Among them, MLP(·) is a learnable multi-layer perceptron, and SPPool(·) is a super point pooling operation; S13, N corresponding to the given point cloud scene I Pictures from different angles The CLIP visual encoder pre-aligned with the visual language is used to extract block-level 2D features; the 2D features are then upsampled back to the original image resolution through interpolation to obtain pixel-level 2D features. Among them, C I Represents the feature dimension, H represents the height of the image, and W represents the width of the image. Indicates H×W×C I dimensional vector space.

3. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 2, characterized in that: The specific process of step S2 is: S21, project the two-dimensional pixel coordinates into the three-dimensional point cloud space according to the camera parameters. For a pixel coordinate (u, v), given the camera intrinsic parameters External Reference And depth represents a 3×3 dimensional vector space, represents a 3×1 dimensional vector space, Represents a set of real numbers, and obtains three-dimensional coordinates (x, y, z) through projection. The calculation formula for projection is: Among them, the superscript T represents the matrix transpose, and the superscript -1 represents the matrix inversion; S22. After projection, all two-dimensional pixel features are assigned a three-dimensional coordinate In the point cloud scene, the three-dimensional coordinate p 3d A sphere query operation is performed to inject multi-view semantics into the point cloud and assign each pixel feature to a point inside a sphere with the corresponding three-dimensional coordinate of the pixel as the center. For points existing inside multiple spheres, the final multi-view feature is the average of all multi-view features obtained, and the multi-view semantic feature of each point is obtained. N p ×C I dimensional vector space; then segment the point cloud, perform super point pooling operations and multi-layer perceptron processing to obtain two-dimensional visual features N s ×d-dimensional vector space; then V 3d and V 2d Add up the visual features S23, perform farthest point sampling on the super points to reduce the number of super points to 2m. The specific calculation formula is: Q seed =V[FPS(p sp )], Among them, FPS(·) represents the farthest point sampling algorithm; V[·] represents obtaining elements according to the index; p sp represents the coordinates of the superpoint, N s ×3-dimensional vector space; Q seed It is a seed query. Represents a 2m×d dimensional vector space, m<<N s ; S24, using spatially aware attention to introduce explicit 3D spatial relationships for enhanced spatial position modeling, for seed query Q seed Perform spatial perception attention modeling. The modeling process is: construct a k-nearest neighbor matrix Among them, the element M in the i-th row and j-th column ij Indicates whether the j-th query is the k-most recent query before the ith query. If yes, it is True, otherwise it is False. k is a hyperparameter and is set to 8. represents a 2m×2m dimensional vector space; the query coordinates are the corresponding superpoint coordinates, and then the k nearest neighbor matrix M is used as a mask in the seed query Q seed Perform self-attention operation on it to generate output As the input of the text interaction sampling module, Represents a 2m×d dimensional vector space: Q=SPA(Q seed )=Masked_Self(Q seed ,M), Among them, Q represents the k nearest neighbor matrix M as a mask in the seed query Q seed The output of the self-attention operation on ; Masked_Self(·) represents the masked self-attention operation; SPA(·) represents the spatially aware attention operation; S25. Resample the output Q using the text interaction sampling module, followed by a multi-layer perceptron to generate queries for input to the decoder: Q0=MLP(Sample(Q,T)), Among them, Q0 is the initial query, , represents an m×d-dimensional vector space; Sample(·) represents the text interaction sampling module; Q represents the k-nearest neighbor matrix M as a mask in the seed query Q seed The output of the self-attention operation on ; T represents the text feature.

4. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 3, characterized in that: The specific process of step S3 is: S31. Use text feature T as the Query in the attention mechanism, query Q l As Key and Value for cross attention operation, it is used to measure the relevance between text and query. The calculation formula is: in, represents the text features of the lth layer, N t ×d-dimensional vector space; A l represents the attention score of the lth layer, N t ×m-dimensional vector space; Q l represents the query at level l, represents the m×d dimensional vector space; Cross(·) represents the cross attention operation; T represents the text feature; by converting the attention score A l Sum along the first dimension to get the relevance score representing the closeness of the connection between the query and the text represents an m-dimensional vector space; S32, using the probability of the output of the previous layer prediction head Perform a filter query with probability Prob l-1 Indicates the possibility that the query corresponds to the target instance; for queries with probabilities lower than the threshold r, their relevance scores are set to negative infinity, indicating that their values ​​will be 0 after applying the Softmax function; the Softmax function is then applied to the relevance score and the result is multiplied by the query to generate a hint to guide the model to distinguish between relevant and irrelevant queries. The guidance process is: in, Represents the jth element of the relevance score after filtering at the lth layer; Represents the jth element of the probability of the output of the l-1th layer; r is a hyperparameter, set to 0.75; represents the jth element of the lth level relevance score; Pt l Indicates a prompt, represents an m×d-dimensional vector space; Q l represents the query at level l; represents the correlation score after filtering at the lth layer; Softmax(·) represents the softmax function; represents the query with hints at level l, represents a 2m×d dimensional vector space; Concat(·) represents a concatenation operation; The decoding process uses queries with hints Integrate text features And the visual features V: in, represents the query after the integration of the l-th layer features; Abandon(·) represents discarding the prompt; Cross(·) represents the cross attention operation; Self(·) represents the self-attention operation; represents the query with hints at level l; represents the text features of the lth layer; V represents the visual features; At the end of each layer, spatially aware attention is used again to enhance features and generate the query Q for the next layer. l+1 , the calculation formula is: Among them, Q l+1 Indicates the query of the next layer; represents the query after the integration of the l-th layer features; SPA(·) represents the spatial perception attention operation; Before entering the next level, Q l+1 After a prediction head is used to generate a prediction mask Mask l and the predicted probability Prob l , the calculation formula is: Mask l =Q l+1 (VW mask ) T , Prob l =Q l+1 W prob , Among them, W mask is the learning parameter, represents a d×d dimensional vector space; W prob is the learning parameter, Represents a d×1 dimensional vector space; Mask l represents the mask for each query prediction, Represents m×N s dimensional vector space; Prob l represents the probability that each query corresponds to the target instance, Represents an m-dimensional vector space; the mask generated by the query with the highest Prob value is used as the final result.

5. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 4, characterized in that: In step S4, The base loss is only applied to the query corresponding to the target object, and the calculation formula is: in, Indicates basic loss; M + The mask representing the query output corresponding to the target object; M tgt represents the true mask; BCE(·) represents the binary cross entropy loss function; DICE(·) represents the Dice loss function; The probability loss is used to supervise Prob, and the calculation formula is: in, Represents probability loss; label L tgt Indicates whether the query corresponds to the target instance, L tgt ∈{0,1} m , the corresponding target instance is 1, otherwise it is 0; The contrastive learning loss Used to align text features and corresponding queries; The final loss is the weighted sum of the three: in, represents the final loss; λ b ,λ p and λ c are all hyperparameters and are set to 1.0, 0.1, and 0.1 respectively.

Images