Human-vehicle interaction three-dimensional segmentation method based on cross-modal spatial superpoint

Abstract

The application discloses a kind of based on cross-modal space superpoint human-vehicle interaction three-dimensional segmentation method, comprising the following steps: step S1, text feature encoder encodes text language information;Step 2, point cloud feature coding and superpoint generation;Step 3, superpoint geometric feature fusion;Step 4, hierarchical feature matching;Step 5, loss calculation and training.The three-dimensional segmentation method of human-vehicle interaction based on cross-modal space superpoint of the application, in human-vehicle interaction and intelligent driving, through the multimodal fusion of language text and point cloud, the matching of visual feature information and text feature information is realized.The superpoint set with semantic and geometric consistency is output after the normalization of text and point cloud encoder, and the object described by user language in scene is segmented and understood, so as to realize human-vehicle interaction, intelligent navigation, safety obstacle avoidance and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, in particular to a three-dimensional interactive segmentation method based on cross-modal spatial super point for vehicle-human interaction. BACKGROUND

[0002] In modern automatic driving and intelligent vehicle-human interaction, accurate understanding and efficient response of environmental information according to user instructions are crucial. Three-dimensional interactive segmentation, which inputs three-dimensional point cloud data of the vehicle environment and the user's language description of an object in the environment, outputs the segmentation mask of the object. This technology is the basis for realizing complex functions such as vehicle-human interaction, intelligent navigation, and safe obstacle avoidance.

[0003] Currently, three-dimensional interactive segmentation technology mainly falls into three categories: single-stage prediction method, instance matching-based prediction method, and super point-based prediction method. The single-stage prediction method is a method that directly inputs three-dimensional point cloud and user interaction information and directly outputs segmentation prediction results. Its advantages are simple structure and fast speed, and its disadvantages are poor generalization ability, low precision, and instability for complex targets. The instance matching-based prediction method is a method that uses instance-level segmentation to segment different objects in the point cloud, and matches them one by one with the text to get the highest matching result. Its advantage is higher precision than single-stage prediction, and its disadvantage is complex matching calculation and slow speed, making it difficult to apply in intelligent driving that requires high response speed. The super point-based prediction method is a method that divides point cloud into super point units, matches super points with input text, and selects high-matching super points for combination to obtain three-dimensional point cloud interactive segmentation results. This method balances speed and precision and is the most practical and expandable solution in current three-dimensional point cloud interactive segmentation, but it still has limitations due to its dependence on super point division quality and limited detail segmentation capability. First, the generated super point information is limited to a single modality and cannot be well segmented based on text information. Second, existing methods mostly ignore the spatial structure information between key points, leading to insufficient understanding of object shape, layout, and other high-level semantics, resulting in inaccurate segmentation results. Third, when matching visual super points with text features, traditional methods often have a huge amount of calculation, making it difficult to meet the real-time application requirements. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and provide a three-dimensional interactive segmentation method based on cross-modal spatial super point for vehicle-human interaction. This method innovatively introduces text information in the super point generation process and constructs cross-modal super points that integrate point cloud and text. Moreover, this method adopts a two-stage hierarchical matching strategy from coarse to fine, which can effectively preserve the spatial characteristics of key points, reduce computational complexity, and achieve more efficient feature matching, providing a new technical path to solve the above challenges.

[0005] To achieve the above object, the application provides a three-dimensional segmentation method for human-vehicle interaction based on cross-modal spatial superpoint, comprising the following steps:

[0006] Step S1, encoding text language information by using a text feature encoder;

[0007] Step S2, guiding the text language information based on three-dimensional point cloud data to generate superpoint and superpoint cross-modal features;

[0008] Step S3, fusing KPConv superpoint geometric feature enhancement based on spatial information cross-modal superpoint interaction;

[0009] Step S4, performing coarse-to-fine layered cross-modal matching on the cross-modal superpoint, and screening out superpoints with high cosine similarity as three-dimensional point cloud interactive segmentation results;

[0010] Step S5, calculating the loss function between the three-dimensional point cloud interactive segmentation results and the artificial annotation, adopting a ternary joint loss supervision training, and performing back propagation fine-tuning through the loss function, calculating the gradient and locally updating the network parameters.

[0011] Further, the step S1 specifically comprises:

[0012] Step S1.1, inputting natural language description, first using BERT to encode to generate word features and sentence features , performing word segmentation and embedding on the text;

[0013] Step S1.2, modeling the word vector sequence by using a Transformer encoder to capture the semantic association between words, focusing on key semantic words in the text through a multi-layer attention mechanism, so as to generate a text feature vector containing global semantics .

[0014] Further, the step S2 specifically comprises:

[0015] Step S2.1, inputting original three-dimensional point cloud data, using a three-dimensional UNet model to generate a basic feature, so as to output a basic feature vector containing 3D geometry ;

[0016] Step S2.2, performing multi-modal interaction and fusion between the text feature vector and the basic feature vector of 3D geometry through an attention model, and generating a fused cross-modal fusion feature through attention weight calculation;

[0017] Step S2.3, based on the cross-modal fusion feature obtained in step S2.2 The SPG superpoint prediction module is used to generate a set of superpoints with semantic and geometric consistency. It uses a single convolutional layer to extract features from each superpoint in the superpoint set. .

[0018] Furthermore, step S3 specifically involves: based on the set of superpoints obtained in step S2.3 KPConv is used to extract and generate geometric spatial information, and generate hyperpoint spatial features. Superpoint features With superpoint space features Cascaded to output fused superpoint features that incorporate geometric information. .

[0019] Furthermore, step S4 specifically includes:

[0020] Step S4.1, Coarse Matching: Input the fused super-point features obtained in step S3. and the text feature vector of step S1 The 3D space is coarsely meshed, and the fused superpoint features in each coarse mesh are... Aggregated into coarse grid features based on mean. Then add all coarse mesh features With text feature vectors L2 normalization is used to normalize to the same space, and then the cosine similarity between the two is calculated. Coarse grids with high similarity to the text are selected as candidate regions, and the selected candidate coarse grids and candidate super points within the candidate coarse grids are output. These candidate super points are then used for the next step of refined matching.

[0021] Step S4.2, Refine Matching: For each candidate superpoint in the coarse grid region retained in step S4.1, calculate its fused superpoint features. With text feature vectors The similarity is used to select superpoints with high cosine similarity as the interactive segmentation result of the 3D point cloud.

[0022] Furthermore, in step S4.2, coarse grids with a cosine similarity of not less than 0.5 with the text are selected as candidate regions.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The three-dimensional segmentation method for human-vehicle interaction based on cross-modal spatial super points can realize matching of visual feature information and text feature information through multi-modal fusion of language text and point cloud in human-vehicle interaction and intelligent driving. The text and point cloud encoder are normalized to unify the feature scale, and a super point set with semantic and geometric consistency is output, the object described by the user in the scene is segmented and understood, and human-vehicle interaction, intelligent navigation, safe obstacle avoidance and the like are realized.

[0025] In addition to the purposes, features and advantages described above, the present application has other purposes, features and advantages. The present application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings are included to provide a further understanding of embodiments of the application, and constitute a part of the specification, and are used to explain the embodiments of the application together with the specific embodiments described below, but do not constitute a limitation on the embodiments of the application. In the drawings:

[0027] Figure 1 is a flowchart of a three-dimensional segmentation method for human-vehicle interaction based on cross-modal spatial super points. DETAILED DESCRIPTION

[0028] The present application will be described in detail below with reference to the embodiments shown in the drawings, but it should be noted that these embodiments are not a limitation on the present application, and equivalent transformations or substitutions of function, method or structure made by those skilled in the art based on these embodiments are within the scope of protection of the present application.

[0029] Please refer to Figure 1 The present embodiment provides a three-dimensional segmentation method for human-vehicle interaction based on cross-modal spatial super points, comprising the following steps:

[0030] Step S1: encoding text language information using a text feature encoder. Specifically, it includes:

[0031] Step S1.1: converting text semantics into a computable feature vector to lay a semantic foundation for subsequent cross-modal fusion. Input natural language description, first use BERT encoding to generate word features and sentence features , and perform word segmentation and embedding on the text;

[0032] Step S1.2: modeling the word vector sequence with a Transformer encoder to capture the semantic association between words, focusing on key semantic words in the text through a multi-layer attention mechanism, thereby generating a text feature vector containing global semantics .

[0033] Step S2: Cross-modal text information guidance is performed on the text language information based on the three-dimensional point cloud data, and super points and super point cross-modal features are generated. Specifically, it includes:

[0034] Step S2.1: Input the original three-dimensional point cloud data, use a three-dimensional UNet model to generate a basic feature, and output a basic feature vector containing 3D geometry . Wherein, the expression of three-dimensional point cloud data is:

[0035] ;

[0036] In the above formula, Pcloud represents three-dimensional point cloud data, which is a set or tensor form of data representation; R represents the real number field, indicating that the elements of the point cloud data take values in the real number range; represents the number of points in the point cloud, that is, how many points in the three-dimensional space the point cloud contains; 3 corresponds to the coordinates (x, y, z) of the three-dimensional space; F represents other feature dimensions that the point cloud may carry in addition to the coordinates.

[0037] Step S2.2: Since the super points generated by pure vision do not take into account the text information, the generated super points may have errors, while the super points generated based on text information can directly inject user semantic intent into the point cloud structure modeling process, thereby improving the accuracy, controllability and generalization ability of segmentation. The features of the text are interacted and fused with the basic feature vector of 3D geometry through an attention model, and a cross-modal feature is generated through attention weight calculation; the purpose of this step is to better generate super point information based on text.

[0038] Step S2.3: Based on the above cross-modal fusion features , a SPG super point prediction module is used to generate a super point set with semantic and geometric consistency , and a convolution layer is used to extract features for each super point .

[0039] Step S3: Fusion of KPConv super point geometric features enhances cross-modal super point interaction based on spatial information. Specifically:

[0040] Based on the super point set obtained in step S2.3 , the KPConv method is used to extract and generate information of the super point geometric space, i.e. object shape, layout, etc., and generate super point spatial features . The super point features and the super point spatial features are concatenated, thereby outputting super point features fused with geometric information The purpose of this step is to better describe the three-dimensional structure, strengthen the association between the 3D structure features and the text semantics, and provide more accurate feature representation for subsequent matching.

[0041] Step S4, cross-modal hyper-point is matched from coarse to fine, and hyper-points with high cosine similarity are screened out as the interactive segmentation result of the three-dimensional point cloud. In order to realize the matching between the final feature information of vision and the text feature information, and thus obtain the final accurate segmentation result, considering that the single-stage matching directly predicts the target, the accuracy is relatively low, therefore, a two-stage hierarchical matching from coarse to fine is designed in this embodiment.

[0042] Step S4.1: coarse matching is performed first, and the fusion hyper-point features obtained in step S3 are input and the text features of step S1 , the 3D space is roughly gridded, and the hyper-point features in each coarse grid are aggregated into coarse grid features by averaging . These coarse grid features and the text features are normalized to the same space using L2, and then the similarity between the two is calculated, and the coarse grid with high similarity to the text is screened out as the candidate region, thereby outputting the screened coarse grid and the candidate hyper-points in the candidate coarse grid, and using these candidate hyper-points for the next step of fine matching; the purpose of this step is to find the region with high confidence in a large area, filter out 90% of irrelevant coarse grid spaces, and retain 10% of coarse grids for the next step of fine matching.

[0043] Step S4.2: based on step S4.1, the similarity between the features of each candidate hyper-point in the coarse grid region retained in step S4.1 and the text features is calculated; hyper-points with high similarity are screened out as the interactive segmentation result of the three-dimensional point cloud. Among them, the coarse grid with a cosine similarity to the text not less than 0.5 is screened out as the candidate region.

[0044] Step S5: calculate the loss function between the interactive segmentation result of the three-dimensional point cloud and the artificial annotation, adopt three loss functions for joint loss supervision training, and comprehensively train the model; through the loss function, the gradient is calculated and the network parameters are locally updated. The three loss functions are respectively the function for supervising the pixel-level classification of the hyper-point level segmentation mask, the function for balancing the foreground and the sample, and the function for strengthening the relevance between the text description and the hyper-point semantics. The expressions of the three loss functions are respectively:

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] In the above formula, is the final response map predicted by the model (predicted probability for each super point); is the binary real super point mask (aggregated from point-level real masks); is the total number of super points; is the real label (0 or 1) of the th super point; is the predicted probability value of the th super point; is the relevance score of the th super point to the text description; is the binary label (1 means the super point belongs to the object category mentioned in the expression, otherwise 0).

[0050] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional segmentation method for human-vehicle interaction based on cross-modal spatial hyper-points, characterized in that, The method comprises the following steps: Step S1, encoding text language information by using a text feature encoder; Step S2, performing cross-modal text information guidance on the text language information based on three-dimensional point cloud data to generate hyperpoints and cross-modal features of the hyperpoints; The step specifically comprises: Step S2.1, input the original three-dimensional point cloud data, use a three-dimensional UNet model to generate a basic feature, thereby output a basic feature vector containing 3D geometry ; Step S2.2, obtaining the text feature vector by attention model and the basic feature vector of 3D geometry Carrying out multi-modal interaction and fusion, and generating the fused cross-modal fusion feature through attention weight calculation ; Step S2.3, generating a cross-modal fusion feature based on the cross-modal fusion feature obtained in step S2.2 , using the SPG super point prediction module to generate a super point set with semantic and geometric consistency , and using a convolutional layer to extract features of each super point in the super point set ; Step S3, fusing KPConv super point geometry feature enhances cross-modal super point interaction based on spatial information; the step is specifically: based on the super point set obtained in the step S2.3 , the geometry spatial information is extracted and generated by using KPConv, and super point spatial features are generated ; the super point features and the super point spatial features are cascaded, so as to output the fusion super point features fused with geometry information ; Step S4, performing coarse-to-fine hierarchical cross-modal matching on the cross-modal hyperpoints to screen out hyperpoints with high cosine similarity as three-dimensional point cloud interactive segmentation results; the step specifically comprises: Step S4.1, rough matching: input the fusion super point features in step S3 and the text feature vector in step S1 , roughly grid the 3D space, aggregate the fusion super point features in each coarse grid by mean value , and normalize all the coarse grid features and the text feature vector to the same space using L2 norm, then calculate the cosine similarity between them, and select the coarse grids with high similarity to the text as candidate regions, thereby output the selected coarse grids and the candidate super points in the candidate coarse grids, and use the candidate super points for the next step of refined matching ​ Step S4.2, Refine matching: for each candidate superpoint in the coarse mesh region reserved in the step S4.1, calculate its fused superpoint feature Similarity with the text feature vector ; filter out the superpoints with high cosine similarity as the interactive segmentation result of the three-dimensional point cloud; Step S5, calculating a loss function between the three-dimensional point cloud interactive segmentation results and artificial labeling, performing three-element joint loss supervision training, performing back propagation fine-tuning through the loss function, calculating gradients and locally updating network parameters.

2. The method of claim 1, wherein, The step S1 specifically comprises: Step S1.1, input natural language description, first use BERT encoding to generate word features and sentence features segment the text and embed it; Step S1.2, model the word vector sequence with a Transformer encoder to capture semantic associations between words and words, focus the model on key semantic words in the text through multiple layers of attention mechanisms, and thus generate a text feature vector containing global semantics .

3. The method of claim 1, wherein, In the step S4.2, coarse grids with a cosine similarity with the text not less than 0.5 are screened out as candidate regions.

Images