Three-dimensional directional target segmentation method of image enhancement prompt decoding network

Abstract

The application discloses a three-dimensional directivity target segmentation method of an image enhancement prompt decoding network, comprising the following steps: S1, feature extraction; S2, multi-view semantic embedding strategy and query generation; S3, prompt perception decoder; S4, training loss: the training loss comprises a basic loss, a probability loss and a contrast learning loss; the application releases the inference ability of the model by using multi-view images and task-driven information in a top-down method; the MSE uses CLIP to extract two-dimensional image features, and then the two-dimensional image features are fused with three-dimensional point cloud features, so that the visual features are significantly enhanced; the PAD uses a task-driven prompt signal to guide the decoding process, generates a prompt that emphasizes the relevance of each query and text through a task-driven prompt module, effectively injects task-specific information into the model, and significantly reduces the complexity of learning.

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to a three-dimensional directional target segmentation method of an image enhancement hint decoding network. Background Art

[0002] 3D Referring Expression Segmentation (3D-RES) has important potential applications in virtual reality, augmented reality, robotic navigation, and human-computer interaction. The goal of this task is to segment 3D objects in a 3D point cloud scene based on a given directional representation.

[0003] Traditional 3D-RES methods adopt a two-stage paradigm: first, they use an instance segmentation network to generate candidate objects, then match these candidates with text, calculate the matching score, and obtain the final segmentation result. However, this paradigm lacks in efficiency and effectiveness. Therefore, recent research has shifted to a single-stage query-based paradigm. For example, 3D-STMN achieves effective segmentation by directly matching text with superpoints, while MCLN and some other works promote joint improvement by jointly training 3D-RES with other tasks.

[0004] However, although existing methods have achieved good results, they still have certain limitations: (1) Feature ambiguity: They only use point cloud data for visual feature extraction. Due to the influence of factors such as lighting, angle, and sampling rate on data acquisition, point cloud data often cannot reproduce real-world scenes well and there is information loss. Therefore, it is difficult to extract excellent features from point cloud data alone. In addition, the scale of point cloud datasets is usually small, which limits the development of large-scale visual language pre-training models in the 3D field. Features extracted using a purely visual 3D backbone network are more difficult to align with text features in subsequent processing. (2) Intent ambiguity: For all queries, they are given equal importance, similar to instance segmentation. However, in the 3D-RES task, only the target object described in the text needs to be segmented. Ideally, queries related to the text should be given priority. However, existing methods do not emphasize these relevant queries, resulting in the model having to implicitly learn the distinction between relevant and irrelevant queries, which significantly increases the difficulty of the learning process. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a three-dimensional directional target segmentation method based on an image enhancement hint decoding network.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A three-dimensional directional target segmentation method using an image enhancement hint decoding network comprises the following steps:

[0008] S1. Feature Extraction: For the target object description text, the pre-trained RoBERTa model is used to extract word-level features. For point cloud data, a segmenter is first used to segment superpoints, and average pooling is used to obtain superpoint locations. Then, a pre-trained sparse 3DU-Net network and superpoint pooling layer are used to obtain superpoint features. For multi-view 2D images of point cloud scenes, the CLIP visual encoder and interpolation are used to obtain pixel-level features of the image.

[0009] S2. Multi-view semantic embedding strategy and query generation: Pixel-level features of 2D images are projected into 3D point cloud space and super-point pooling is performed. Visual features are then fused with super-point features extracted from the point cloud data. Farthest point sampling is used for super-points to reduce subsequent computational overhead. Spatial-aware attention is used to perform multiple rounds of iterations to compensate for the lack of spatial positional relationships introduced by the 2D image features. Interactive text sampling is used to obtain the decoder's query from the visual features.

[0010] S3, Cue-aware decoder: This method uses the text features as the query in the attention mechanism and the decoder query as the key and value to perform a cross-attention operation to update the text features and obtain the correlation between the decoder query and the text features. The decoder query output from the prediction head is then used to filter the correlation by the probability of the target object to obtain a task-driven cue signal to guide the decoding process.

[0011] S4. Training loss: Training loss includes base loss, probability loss, and contrastive learning loss. Base loss includes binary cross entropy loss and Dice loss applied to the predicted mask. Probability loss is the binary cross entropy loss applied to the probability of the decoder query corresponding to the target object output by the prediction head. Contrastive learning loss is the dense semantic alignment loss applied to the query and text features.

[0012] Preferably, the specific process of step S1 is:

[0013] S11. Given a text description of the target object, use the pre-trained RoBERTa model to extract word-level features E, and then convert the word-level features E into text features T through linear projection, so that there is a unified feature dimension d in the decoder. The conversion formula is:

[0014] T=EW t ,

[0015] Among them, E represents word-level features, T represents text features, and W t represents randomly initialized learnable parameters, Nt Indicates the number of words, C t represents the dimension of the feature, d represents the unified feature dimension, Indicates N t ×C t dimensional vector space, Indicates N t ×d-dimensional vector space, Indicates C t ×d-dimensional vector space;

[0016] S12. Given a point cloud scene Among them, N p represents the number of points, each with a three-dimensional coordinate and an f-dimensional auxiliary feature vector, Indicates N p ×(3+f)-dimensional vector space; use pre-trained sparse 3D U-Net network to extract point-level features Among them, C p represents the feature dimension, Indicates N p ×C p dimensional vector space; segment the point cloud and obtain N s Super point And for the point-level feature F 3d Perform superpoint pooling operation to obtain three-dimensional superpoint features Among them, N s Indicates the number of super points, Indicates N s ×C p dimensional vector space; then use a multi-layer perceptron to change the dimension to d to obtain three-dimensional visual features Indicates N s ×d-dimensional vector space, the calculation formula is:

[0017] V 3d =MLP(SPPool(F 3d )),

[0018] Among them, MLP(·) is a learnable multi-layer perceptron, and SPPool(·) is a super point pooling operation;

[0019] S13, N corresponding to the given point cloud scene I Pictures from different perspectives The CLIP visual encoder pre-aligned with the visual language is used to extract block-level 2D features; the 2D features are then upsampled back to the original image resolution through interpolation to obtain pixel-level 2D features. Among them, C I Represents the feature dimension, H represents the height of the image, and W represents the width of the image. Indicates H×W×C I dimensional vector space.

[0020] Preferably, the specific process of step S2 is:

[0021] S21, project the two-dimensional pixel coordinates into the three-dimensional point cloud space according to the camera parameters. For a pixel coordinate (u, v), given the camera intrinsic parameters External Reference and depth represents a 3×3 dimensional vector space, represents a 3×1 dimensional vector space, Represents a set of real numbers, and obtains the three-dimensional coordinates (x, y, z) through projection. The projection calculation formula is:

[0022]

[0023] Among them, the superscript T represents the matrix transpose, and the superscript -1 represents the matrix inversion;

[0024] S22. After projection, all two-dimensional pixel features are assigned a three-dimensional coordinate In the point cloud scene, the three-dimensional coordinate p 3d Perform a sphere query operation to inject multi-view semantics into the point cloud and assign each pixel feature to a point inside a sphere with the corresponding three-dimensional coordinate of the pixel as the center. For points existing inside multiple spheres, the final multi-view feature is the average of all the multi-view features obtained, and the multi-view semantic feature of each point is obtained. Indicates N p ×C I dimensional vector space; then segment the point cloud, perform super point pooling operations and multi-layer perceptron processing to obtain two-dimensional visual features Indicates N s ×d-dimensional vector space; then V 3d and V 2d Add up to get visual features

[0025] S23. Sampling the farthest point of the super points to reduce the number of super points to 2m. The specific calculation formula is:

[0026] Q seed =v[FPS(p sp )],

[0027] Among them, FPS(·) represents the farthest point sampling algorithm; V[·] represents obtaining elements according to the index; P sp represents the coordinates of the superpoint, Indicates N sX3-dimensional vector space; Q seed is a seed query, represents a 2mxd-dimensional vector space, m<<N s ;

[0028] S24, use spatial-aware attention to introduce explicit three-dimensional spatial relationship for enhancing spatial position modeling, for the seed query Q seed spatial-aware attention modeling, the modeling process is: construct a k-nearest neighbor matrix Where, the i-th row and the j-th column element M ij Indicates whether the jth query is the kth nearest query of the ith query, True if yes, otherwise False; k is a hyperparameter, set to 8; represents a 2m x 2m-dimensional vector space; the coordinates of the query are the corresponding hyperpoint coordinates, and then perform self-attention operation on the seed query Q seed with the k-nearest neighbor matrix M as a mask to generate the output as the input of the text interaction sampling module, represents a 2m x d-dimensional vector space:

[0029] Q=SPA(Q seed )=Masked_Self(Q seed , M),

[0030] Where, Q represents the output generated by performing self-attention operation on the seed query Q seed with the k-nearest neighbor matrix M as a mask; Masked_Self(·) represents the masked self-attention operation; SPA(·) represents the spatial-aware attention operation.

[0031] S25, use the text interaction sampling module to resample the output Q, and follow a multi-layer perception to generate a query for inputting the decoder:

[0032] Q0=MLP(Sample(Q, T)),

[0033] Where, Q0 is the initial query, represents an m x d-dimensional vector space; Sample(·) represents the text interaction sampling module; Q represents the output generated by performing self-attention operation on the seed query Q seed with the k-nearest neighbor matrix M as a mask; T represents the text feature.

[0034] Preferably, the specific process of step S3 is:

[0035] S31, use the text feature T as Query in the attention mechanism, query Q lAs Key and Value, cross attention operation is performed to measure the relevance between text and query. The calculation formula is:

[0036]

[0037] in, represents the text features of the lth layer, Indicates N t ×d-dimensional vector space; A l represents the attention score of the lth layer, Indicates N t ×m-dimensional vector space; Q l represents the query at level l, represents the m×d dimensional vector space; Cross(·) represents the cross attention operation; T represents the text feature; by adding the attention score A l Sum along the first dimension to get the relevance score representing the closeness of the connection between the query and the text represents an m-dimensional vector space;

[0038] S32, using the probability output of the previous layer prediction head Perform filtering query, probability Prob l-1 Represents the probability that the query corresponds to the target instance; for queries with probabilities lower than the threshold r, their relevance scores are set to negative infinity, indicating that their values ​​will be 0 after applying the Softmax function; the Softmax function is then applied to the relevance score and the result is multiplied by the query to generate a hint that guides the model to distinguish between relevant and irrelevant queries. The guidance process is:

[0039]

[0040]

[0041]

[0042] in, Represents the jth element of the relevance score after filtering at the lth layer; The jth element represents the probability of the output of the l-1th layer; r is a hyperparameter, set to 0.75; represents the jth element of the lth level relevance score; Pt l Indicates a prompt, represents an m×d-dimensional vector space; Q l represents the query at level l; represents the correlation score after filtering at the lth layer; Softmax(·) represents the softmax function; represents the query with hints at level l, denotes a 2m x d dimensional vector space; Concat(·) denotes the concatenation operation;

[0043] The decoding process utilizes the query with hints Integrating text features and visual features V:

[0044]

[0045] where, denotes the integrated query after the l-th layer feature; Abandon(·) denotes the abandon hint; Cross(·) denotes the cross-attention operation; Self(·) denotes the self-attention operation; denotes the query with hints in the l-th layer; denotes the text feature in the l-th layer; V denotes the visual feature;

[0046] At the end of each layer, spatial-aware attention is used again to enhance the features and generate the query Q l+1 , the calculation formula is:

[0047]

[0048] where, Ql+1denotes the query of the next layer; denotes the integrated query after the l-th layer feature; SPA(·) denotes the spatial-aware attention operation;

[0049] Before entering the next layer, Q l+1 goes through a prediction head to generate a prediction mask Mask l and a prediction probability Prob l , the calculation formula is:

[0050] Mask l = Q l+1 (VW mask ) T ,

[0051] Prob l = Q l+1 W prob ,

[0052] where, W mask is a learning parameter, denotes a d x d dimensional vector space; W prob is a learning parameter, denotes a d x 1 dimensional vector space; Mask l denotes the mask predicted for each query, denotes an m x N s dimensional vector space; Prob lrepresents the probability that each query corresponds to the target instance, Represents an m-dimensional vector space; the mask generated by the query with the highest Prob value is used as the final result.

[0053] Preferably, in step S4,

[0054] The base loss is only applied to queries corresponding to the target object, and the calculation formula is:

[0055]

[0056] in, Indicates basic loss; M + The mask representing the query output corresponding to the target object; M tgt represents the true mask; BCE(·) represents the binary cross entropy loss function; DICE(·) represents the Dice loss function;

[0057] The probability loss is used to supervise Prob, and the calculation formula is:

[0058]

[0059] in, Represents probability loss; label L tgt Indicates whether the query corresponds to the target instance, L tgt ∈{0,1} m , the corresponding target instance is 1, otherwise it is 0;

[0060] The contrastive learning loss Used to align text features and corresponding queries;

[0061] The final loss is the weighted sum of the three:

[0062]

[0063] in, represents the final loss; λ b ,λ p and λ c are all hyperparameters and are set to 1.0, 0.1, and 0.1 respectively.

[0064] The technical scheme has the following beneficial effects: the method uses a top-down method to release the inference ability of the model by using multi-view images and task-driven information. In order to solve the problem of feature ambiguity, the method proposes a multi-view semantic embedding (MSE) strategy, which uses CLIP to extract two-dimensional image features, and then fuses them with three-dimensional point cloud features to significantly enhance the visual features. In addition, spatial perception attention is used to solve the problem of lack of spatial position relationship in two-dimensional features. This method produces visual features with superior representation ability, which contains text prior knowledge from CLIP and can be better aligned with text features. In order to solve the problem of intention ambiguity, the method designs a prompt perception decoder (PAD), which uses task-driven prompt signals to guide the decoding process. Through the task-driven prompt module, the method generates prompts that emphasize the relevance of each query to the text, effectively injects task-specific information into the model, and significantly reduces the complexity of learning. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The flowchart of the present application is shown in the figure;

[0066] Figure 2 The image enhancement prompt decoding network model architecture of the present application is shown in the figure;

[0067] Figure 3 The quantitative comparison result graph of the present application and the existing method on the Multi3DRefer dataset is shown in the figure;

[0068] Figure 4 The quantitative comparison result graph of the present application and the existing method on the ScanRefer dataset is shown in the figure;

[0069] Figure 5 The qualitative comparison result graph of the present application and MDIN on the ScanRefer dataset is shown in the figure. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the present application clearer, the following embodiments are used to further illustrate the present application. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0071] As shown in the figure, a three-dimensional directional target segmentation method of an image enhancement prompt decoding network includes the following steps: Figures 1 to 5

[0072] ​S1, feature extraction: for the target object description text, a pre-trained RoBERTa model is used to extract word-level features; for point cloud data, a segmenter is first used to segment super points, and average pooling is used to obtain super point positions, and then a pre-trained sparse 3D U-Net network and a super point pooling layer are used to obtain super point features; for multi-view two-dimensional pictures of point cloud scenes, a CLIP visual encoder and interpolation are used to obtain pixel-level features of the pictures;

[0073] The specific process of step S1 is as follows:

[0074] S11, given the text description of the target object, the pre-trained RoBERTa model is used to extract the word-level feature E, and then the word-level feature E is converted into the text feature T through linear projection, so that there is a unified feature dimension d in the decoder, and the conversion formula is:

[0075] T=EW t ,

[0076] Wherein, E represents the word-level feature, T represents the text feature, W t represents a randomly initialized learnable parameter, N t represents the number of words, C t represents the dimension of the feature, d represents the unified feature dimension, represents the N t ×C t vector space, represents the N t ×d vector space, represents the C t ×d vector space;

[0077] S12, given a point cloud scene Wherein, N p represents the number of points, each point has a three-dimensional coordinate and a f-dimensional auxiliary feature vector, represents the N p ×(3+f) vector space; a pre-trained sparse 3D U-Net network is used to extract point-level features Wherein, C p represents the feature dimension, represents the N p ×C p vector space; the point cloud is segmented to obtain N s super points and the point-level feature F 3d is super point pooled to obtain three-dimensional super point features Wherein, N s represents the number of super points, represents the Ns ×C p Vector space; Reuse a multi-layer perception to change the dimension to d, and obtain three-dimensional visual features N s ×d-dimensional vector space, and the calculation formula is:

[0078] V 3d = MLP(SPPool(F 3d )),

[0079] wherein MLP(·) is a learnable multi-layer perception, and SPPool(·) is a super point pooling operation;

[0080] S13, given the N I different view pictures corresponding to the point cloud scene Use the CLIP visual encoder of visual language pre-alignment to extract block-level two-dimensional features; then interpolate the two-dimensional features to upsample them back to the original image resolution to obtain pixel-level two-dimensional features wherein C i represents the feature dimension, H represents the height of the picture, and W represents the width of the picture, H×W×C I dimensional vector space;

[0081] S2, multi-view semantic embedding strategy and query generation: project the pixel-level features of the two-dimensional picture to the three-dimensional point cloud space and perform super point pooling, and then fuse the super point features extracted from the point cloud data to obtain visual features; use farthest point sampling to reduce the subsequent calculation overhead, use spatial perception attention to iteratively compensate for the lack of spatial location relationship caused by the introduction of two-dimensional picture features, and use text interaction sampling to obtain the decoder query from the visual features;

[0082] The specific process of step S2 is:

[0083] S21, project the two-dimensional pixel coordinates to the three-dimensional point cloud space according to the camera parameters, for a pixel coordinate (u, v), given the camera intrinsic parameters extrinsic parameters and depth 3×3-dimensional vector space, 3×1-dimensional vector space, real set, obtain three-dimensional coordinates (x, y, z) through projection, and the calculation formula of projection is:

[0084]

[0085] wherein the superscript T represents matrix transposition, and the superscript -1 represents matrix inversion;

[0086] S22. After projection, all two-dimensional pixel features are assigned a three-dimensional coordinate In the point cloud scene, the three-dimensional coordinate p 3d Perform a sphere query operation to inject multi-view semantics into the point cloud and assign each pixel feature to a point inside a sphere with the corresponding three-dimensional coordinate of the pixel as the center. For points existing inside multiple spheres, the final multi-view feature is the average of all the multi-view features obtained, and the multi-view semantic feature of each point is obtained. Indicates N p ×C I dimensional vector space; then segment the point cloud, perform super point pooling operations and multi-layer perceptron processing to obtain two-dimensional visual features Indicates N s ×d-dimensional vector space; then V 3d and V 2d Add up to get visual features

[0087] S23. Sampling the farthest point of the super points to reduce the number of super points to 2m. The specific calculation formula is:

[0088] Q seed =V[FPS(p sp )],

[0089] Among them, FPS(·) represents the farthest point sampling algorithm; V[·] represents obtaining elements according to the index; p sp represents the coordinates of the superpoint, Indicates N s ×3-dimensional vector space; Q seed It is a seed query. Represents a 2m×d dimensional vector space, m<<N s ;

[0090] S24, using spatial awareness attention to introduce explicit three-dimensional spatial relationships to enhance spatial position modeling, for seed query Q seed Perform spatial perception attention modeling. The modeling process is: construct a k-nearest neighbor matrix Among them, the element M in the i-th row and j-th column ij Indicates whether the j-th query is the k-most recent query before the i-th query. If so, it is True; otherwise, it is False. k is a hyperparameter and is set to 8. Represents a 2m×2m dimensional vector space; the query coordinates are the corresponding superpoint coordinates, and then the k nearest neighbor matrix M is used as a mask in the seed query Q seed Perform self-attention operation on it to generate output As the input of the text interaction sampling module, denotes a 2mxd-dimensional vector space:

[0091] Q = SPA(Q seed , M), seed

[0092] where Q denotes the output of performing self-attention operation on the seed query Q seed with the k-neighborhood matrix M as the mask; Masked_Self(·) denotes the masked self-attention operation; SPA(·) denotes the space-aware attention operation;

[0093] S25, resample the output Q using the text interaction sampling module, followed by a multi-layer perceptron to generate a query for inputting the decoder:

[0094] Q0= MLP(Sample(Q, T)),

[0095] where Q0is the initial query, denotes an mxd-dimensional vector space; Sample(·) represents the text interaction sampling module; Q denotes the output of performing self-attention operation on the seed query Q seed with the k-neighborhood matrix M as the mask; T denotes the text feature;

[0096] S3, prompt-aware decoder: by taking the text feature as Query in the attention mechanism, the decoder query as Key and Value to perform cross-attention operation, the text feature is updated, and the relevance between the decoder query and the text feature is obtained. Then, the probability of the decoder query output by the prediction head corresponding to the target object is used to filter the relevance to obtain a task-driven prompt signal for guiding the decoding process.

[0097] The specific process of step S3 is as follows:

[0098] S31, use the text feature T as Query in the attention mechanism, and use the query Q l as Key and Value to perform cross-attention operation, which is used to measure the relevance between the text and the query, and the calculation formula is as follows:

[0099]

[0100] wherein, denotes the text feature of the l-th layer, denotes an N t xd-dimensional vector space; A l denotes the attention score of the l-th layer, denotes an N t xm-dimensional vector space; Q l ​represents the query at level l, represents the m×d dimensional vector space; Cross(·) represents the cross attention operation; T represents the text feature; by adding the attention score A l Sum along the first dimension to get the relevance score representing the closeness of the connection between the query and the text represents an m-dimensional vector space;

[0101] S32, using the probability output of the previous layer prediction head Perform filtering query, probability Prob l-1 Represents the probability that the query corresponds to the target instance; for queries with probabilities lower than the threshold r, their relevance scores are set to negative infinity, indicating that their values ​​will be 0 after applying the Softmax function; the Softmax function is then applied to the relevance score and the result is multiplied by the query to generate a hint that guides the model to distinguish between relevant and irrelevant queries. The guidance process is:

[0102]

[0103]

[0104]

[0105] in, Represents the jth element of the relevance score after filtering at the lth layer; The jth element represents the probability of the output of the l-1th layer; r is a hyperparameter, set to 0.75; represents the jth element of the lth level relevance score; Pt l Indicates a prompt, represents an m×d-dimensional vector space; Q l represents the query at level l; represents the correlation score after filtering at the lth layer; Softmax(·) represents the softmax function; represents the query with hints at level l, represents a 2m×d-dimensional vector space; Concat(·) represents a concatenation operation;

[0106] The decoding process uses queries with hints Integrating text features And visual features V:

[0107]

[0108] in, represents the query after the integration of the l-th layer features; Abandon(·) represents discarding the prompt; Cross(·) represents the cross attention operation; Self(·) represents the self-attention operation; represents the query with hints at level l; represents the text features of the lth layer; V represents the visual features;

[0109] At the end of each layer, spatially aware attention is used again to enhance features and generate the query Q for the next layer. l+1 , the calculation formula is:

[0110]

[0111] Among them, Q l+1 Indicates the query of the next layer; represents the query after the integration of the l-th layer features; SPA(·) represents the spatial perception attention operation;

[0112] Before entering the next level, Q l+1 After a prediction head is used to generate a prediction mask Mask l and predicted probability Prob l , the calculation formula is:

[0113] Mask l =Q l+1 (VW mask ) T ,

[0114] Prob l =Q l+1 W prob ,

[0115] Among them, W mask is the learning parameter, represents a d×d dimensional vector space; W prob is the learning parameter, Represents a d×1 dimensional vector space; Mask l represents the mask for each query prediction, Represents m×N s dimensional vector space; Prob l represents the probability that each query corresponds to the target instance, Represents an m-dimensional vector space; the mask generated by the query with the highest Prob value is used as the final result;

[0116] S4, training loss: the training loss includes a base loss, a probability loss, and a contrastive learning loss; wherein the base loss includes a binary cross-entropy loss applied on the predicted mask and a Dice loss; the probability loss is a binary cross-entropy loss applied on the probability of the target object corresponding to the decoder query output by the prediction head; and the contrastive learning loss is a dense semantic alignment loss applied on the query and the text feature;

[0117] In step S4,

[0118] The base loss is only applied to the query corresponding to the target object, and the calculation formula is:

[0119]

[0120] wherein, denotes the base loss; M + denotes the mask output by the query corresponding to the target object; M tgt denotes the real mask; BCE(·) denotes a binary cross-entropy loss function; and DICE(·) denotes a Dice loss function.

[0121] The probability loss is used to supervise Prob, and the calculation formula is:

[0122]

[0123] wherein, denotes the probability loss; and label L tgt denotes whether the query corresponds to the target instance, L tgt ∈{0,1} m 1 if the target instance corresponds to the target instance, and 0 otherwise.

[0124] The contrastive learning loss is used to align the text feature and the corresponding query.

[0125] The final loss is a weighted sum of the three:

[0126]

[0127] wherein, denotes the final loss; and λ b , λ p , and λ c are hyperparameters, and are respectively set to 1.0, 0.1, and 0.1.

[0128] Specific experimental results:

[0129] The present application is trained and tested on the ScanRefer dataset and the Multi3DRefer dataset respectively. The ScanRefer dataset is a 3D-RES task test dataset, containing 51583 text descriptions pointing to a single object; the Multi3DRefer dataset is a 3D-GRES task test dataset, containing 61926 text descriptions, of which 6688 descriptions do not describe the target object, and 13178 descriptions describe multiple target objects. All experiments are implemented using PyTorch and trained on a single NVIDIA GeForce RTX3090 GPU.

[0130] The mean intersection over union (mIoU) and the accuracy of intersection over union greater than k (Acc@kIoU) are used as evaluation indicators. The accuracy of intersection over union greater than k represents the proportion of text descriptions whose overlapping part between the predicted mask and the real label has an IoU greater than k, where k∈0.25, 0.5. In the 3D-GRES task, when the target pointed by the text does not exist, if the model outputs a full 0 mask, the mIoU is 1, otherwise the mIoU is 0.

[0131] As shown in Figure 3 , the model of the present application is significantly better than other methods in the 3D-GRES task, achieving a 5.4-point improvement in mIoU and even a 7.5-point improvement in Acc@0.5. It can be observed that the model of the present application significantly outperforms MDIN in all other cases except in the case of zero target with interference. In particular, in the single target scene, whether there is interference or not, the model of the present application is nearly 10 points higher than MDIN in the Acc@0.5 indicator. This indicates that the task-driven prompt of the present application effectively guides the model to focus on more important queries, thereby achieving more accurate positioning of the key target.

[0132] Experiments are conducted on the 3D-RES task, as shown in Figure 4 . On the ScanRefer dataset, the model of the present application generally achieves advanced performance. Specifically, the model of the present application is 2.6, 1.8 and 1.9 points higher than the previous best model MDIN in Acc@0.25, Acc@0.5 and mIoU, respectively. Notably, the model of the present application observes more significant improvements in challenging scenes with multiple interfering objects. This indicates that the model of the present application benefits from more robust multi-view semantic embeddings and task-driven prompts, which effectively guide the model to focus on more critical information, thereby enhancing its ability to accurately identify target objects among multiple instances of the same category.

[0133] A representative set of examples are visualized in the ScanRefer dataset, as shown in Figure 5 Figure 5 As can be seen from Figure 5 , the model of the present application exhibits stronger reasoning ability than MDIN. Specifically, in case (a), there is no interfering object in the scene, only a vending machine, but MDIN still cannot identify it. This is because, in the ScanRefer training dataset with 36665 samples, only 10 samples have the target as a vending machine, which is not enough for the model to identify such an object. However, the large-scale pre-trained CLIP model does not suffer from this problem and can well identify the vending machine, thus allowing the model of the present application to accurately locate the target. In case (b), the concept of "left" is involved, which depends on the perspective. Since three-dimensional space theoretically contains an infinite number of perspectives, it is difficult for a pure three-dimensional model to distinguish left and right. In contrast, the perspective in a two-dimensional image is fixed, which provides important help in handling such cases. Finally, in case (c), due to the strong prompting ability of the task-driven prompt of the present application, even with nearly ten interfering object objects, the model of the present application can still accurately locate the target object.

[0134] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A three-dimensional directional target segmentation method based on an image enhancement hint decoding network, characterized in that: The following steps are involved: S1. Feature Extraction: For the target object description text, the pre-trained RoBERTa model is used to extract word-level features. For point cloud data, a segmenter is first used to segment superpoints, and average pooling is used to obtain superpoint locations. Then, a pre-trained sparse 3D U-Net network and superpoint pooling layer are used to obtain superpoint features. For multi-view 2D images of point cloud scenes, the CLIP visual encoder and interpolation are used to obtain pixel-level features of the image. S2. Multi-view semantic embedding strategy and query generation: Project the pixel-level features of the 2D image into the 3D point cloud space and perform super-point pooling, which is then fused with the super-point features extracted from the point cloud data to obtain visual features; The farthest point sampling is used for superpoints to reduce subsequent computational overhead. Spatial-aware attention is used to perform multiple rounds of iterations to compensate for the lack of spatial position relationships caused by the introduction of two-dimensional image features. Text interaction sampling is used to obtain the decoder query from visual features. S3, Cue-aware decoder: This method uses the text features as the query in the attention mechanism and the decoder query as the key and value to perform a cross-attention operation to update the text features and obtain the correlation between the decoder query and the text features. The decoder query output from the prediction head is then used to filter the correlation by the probability of the target object to obtain a task-driven cue signal to guide the decoding process. S4. Training loss: Training loss includes base loss, probability loss, and contrastive learning loss. Base loss includes binary cross entropy loss and Dice loss applied to the predicted mask. Probability loss is the binary cross entropy loss applied to the probability of the decoder query corresponding to the target object output by the prediction head. Contrastive learning loss is the dense semantic alignment loss applied to the query and text features.

2. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 1, characterized in that: The specific process of step S1 is: S11. Given a text description of the target object, use the pre-trained RoBERTa model to extract word-level features E, and then convert the word-level features E into text features T through linear projection, so that there is a unified feature dimension d in the decoder. The conversion formula is: T=EW t , Among them, E represents word-level features, T represents text features, and W t represents randomly initialized learnable parameters, N t Indicates the number of words, C t represents the dimension of the feature, d represents the unified feature dimension, Indicates N t ×C t dimensional vector space, Indicates N t ×d-dimensional vector space, Indicates C t ×d-dimensional vector space; S12. Given a point cloud scene Among them, N p represents the number of points, each with a three-dimensional coordinate and an f-dimensional auxiliary feature vector, Indicates N p ×(3+f)-dimensional vector space; use pre-trained sparse 3D U-Net network to extract point-level features Among them, C p represents the feature dimension, Indicates N p ×C p dimensional vector space; segment the point cloud and obtain N s Super point And for the point-level feature F 3d Perform superpoint pooling operation to obtain three-dimensional superpoint features Among them, N s Indicates the number of super points, Indicates N s ×C p dimensional vector space; then use a multi-layer perceptron to change the dimension to d to obtain three-dimensional visual features Indicates N s ×d-dimensional vector space, the calculation formula is: V 3d =MLP(SPPool(F 3d )), Among them, MLP(·) is a learnable multi-layer perceptron, and SPPool(·) is a super point pooling operation; S13, N corresponding to the given point cloud scene I Pictures from different perspectives The CLIP visual encoder pre-aligned with the visual language is used to extract block-level 2D features; the 2D features are then upsampled back to the original image resolution through interpolation to obtain pixel-level 2D features. Among them, C I Represents the feature dimension, H represents the height of the image, and W represents the width of the image. Indicates H×W×C I dimensional vector space.

3. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 2, characterized in that: The specific process of step S2 is: S21, project the two-dimensional pixel coordinates into the three-dimensional point cloud space according to the camera parameters. For a pixel coordinate (u, v), given the camera intrinsic parameters External Reference and depth represents a 3×3 dimensional vector space, represents a 3×1 dimensional vector space, Represents a set of real numbers, and obtains the three-dimensional coordinates (x, y, z) through projection. The projection calculation formula is: Among them, the superscript T represents the matrix transpose, and the superscript -1 represents the matrix inversion; S22. After projection, all two-dimensional pixel features are assigned a three-dimensional coordinate In the point cloud scene, the three-dimensional coordinate p 3d Perform a sphere query operation to inject multi-view semantics into the point cloud and assign each pixel feature to a point inside a sphere with the corresponding three-dimensional coordinate of the pixel as the center. For points existing inside multiple spheres, the final multi-view feature is the average of all the multi-view features obtained, and the multi-view semantic feature of each point is obtained. Indicates N p ×C I dimensional vector space; then segment the point cloud, perform super point pooling operations and multi-layer perceptron processing to obtain two-dimensional visual features Indicates N s ×d-dimensional vector space; then V 3d and V 2d Add up to get visual features S23, perform farthest point sampling on the super points to reduce the number of super points to 2m. The specific calculation formula is: Q seed =V[FPS(p sp )], Among them, FPS(·) represents the farthest point sampling algorithm; V[·] represents obtaining elements according to the index; p sp represents the coordinates of the superpoint, Indicates N s ×3-dimensional vector space; Q seed It is a seed query. Represents a 2m×d dimensional vector space, m<<N s ; S24, using spatial awareness attention to introduce explicit three-dimensional spatial relationships to enhance spatial position modeling, for seed query Q seed Perform spatial perception attention modeling. The modeling process is: construct a k-nearest neighbor matrix Among them, the element M in the i-th row and j-th column ij Indicates whether the j-th query is the k-most recent query before the i-th query. If so, it is True; otherwise, it is False. k is a hyperparameter and is set to 8. Represents a 2m×2m dimensional vector space; the query coordinates are the corresponding superpoint coordinates, and then the k nearest neighbor matrix M is used as a mask in the seed query Q seed Perform self-attention operation on it to generate output As the input of the text interaction sampling module, Represents a 2m×d dimensional vector space: Q=SPA(Q seed )=Masked_Self(Q seed ,M), Among them, Q represents the k-nearest neighbor matrix M as a mask in the seed query Q seed The output of the self-attention operation on ; Masked_Self(·) represents the masked self-attention operation; SPA(·) represents the spatial perception attention operation; S25. Use the text interaction sampling module to resample the output Q, and follow it with a multi-layer perceptron to generate a query for input to the decoder: Q0=MLP(Sample(Q,T)), Among them, Q0 is the initial query, , represents an m×d-dimensional vector space; Sample(·) represents the text interaction sampling module; Q represents the k-nearest neighbor matrix M as a mask in the seed query Q seed The output of the self-attention operation on ; T represents the text feature.

4. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 3, characterized in that: The specific process of step S3 is: S31. Use text feature T as Query in the attention mechanism, query Q l As Key and Value, cross attention operation is performed to measure the relevance between text and query. The calculation formula is: in, represents the text features of the lth layer, Indicates N t ×d-dimensional vector space; A l represents the attention score of the lth layer, Indicates N t ×m-dimensional vector space; Q l represents the query at level l, represents the m×d dimensional vector space; Cross(·) represents the cross attention operation; T represents the text feature; by converting the attention score A l Sum along the first dimension to get the relevance score representing the closeness of the connection between the query and the text represents an m-dimensional vector space; S32, using the probability output of the previous layer prediction head Perform filtering query, probability Prob l-1 Represents the probability that the query corresponds to the target instance; for queries with probabilities lower than the threshold r, their relevance scores are set to negative infinity, indicating that their values ​​will be 0 after applying the Softmax function; the Softmax function is then applied to the relevance score and the result is multiplied by the query to generate a hint that guides the model to distinguish between relevant and irrelevant queries. The guidance process is: in, Represents the jth element of the relevance score after filtering at the lth layer; The jth element represents the probability of the output of the l-1th layer; r is a hyperparameter, set to 0.75; represents the jth element of the lth level relevance score; Pt l Indicates a prompt, represents an m×d-dimensional vector space; Q l represents the query at level l; represents the correlation score after filtering at the lth layer; Softmax(·) represents the softmax function; represents the query with hints at level l, represents a 2m×d-dimensional vector space; Concat(·) represents a concatenation operation; The decoding process uses queries with hints Integrating text features And visual features V: in, represents the query after the integration of the l-th layer features; Abandon(·) represents discarding the prompt; Cross(·) represents the cross attention operation; Self(·) represents the self-attention operation; represents the query with hints at level l; represents the text features of the lth layer; V represents the visual features; At the end of each layer, spatially aware attention is used again to enhance features and generate the query Q for the next layer. l+1 , the calculation formula is: Among them, Q l+1 Indicates the query of the next layer; represents the query after the integration of the l-th layer features; SPA(·) represents the spatial perception attention operation; Before entering the next level, Q l+1 After a prediction head is used to generate a prediction mask Mask l and predicted probability Prob l , the calculation formula is: Mask l =Q l+1 (VW mask ) T , Prob l =Q l+1 W prob , Among them, W mask is the learning parameter, represents a d×d dimensional vector space; W prob is the learning parameter, Represents a d×1 dimensional vector space; Mask l represents the mask for each query prediction, Represents m×N s dimensional vector space; Prob l represents the probability that each query corresponds to the target instance, Represents an m-dimensional vector space; the mask generated by the query with the highest Prob value is used as the final result.

5. The three-dimensional directional target segmentation method of the image enhancement hint decoding network according to claim 4, characterized in that: In step S4, The base loss is only applied to queries corresponding to the target object, and the calculation formula is: in, Indicates basic loss; M + The mask representing the query output corresponding to the target object; M tgt represents the true mask; BCE(·) represents the binary cross entropy loss function; DICE(·) represents the Dice loss function; The probability loss is used to supervise Prob, and the calculation formula is: in, Represents probability loss; label L tgt Indicates whether the query corresponds to the target instance, L tgt ∈{0,1} m , the corresponding target instance is 1, otherwise it is 0; The contrastive learning loss Used to align text features and corresponding queries; The final loss is the weighted sum of the three: in, represents the final loss; λ b ,λ p and λ c are all hyperparameters and are set to 1.0, 0.1, and 0.1 respectively.

Images