A model training method and point cloud segmentation method based on attention mechanism

Abstract

The present specification discloses a model training method and a point cloud segmentation method based on an attention mechanism. The method comprises: obtaining a training sample, wherein the training sample includes support set point cloud data and query set point cloud data; inputting the training sample into a semantic segmentation model, determining feature data corresponding to the training sample through a feature extraction network; transposing the feature data to obtain transposed feature data, weighting the transposed feature data through a first attention network to obtain a first weighted feature, and weighting the feature data through a second attention network to obtain a second weighted feature; determining a target feature based on the first weighted feature and the second weighted feature; semantically segmenting the query set point cloud data based on the target feature corresponding to the support set point cloud data and the target feature corresponding to the query set point cloud data, determining a loss value based on the deviation between the semantic segmentation result and its label, and performing model training.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model training method and a point cloud segmentation method based on an attention mechanism. Background Art

[0002] Point cloud segmentation technology is a key technology in 3D data processing and is widely used in many fields such as robotics, autonomous driving, medical treatment, virtual reality, etc. This technology effectively divides the unstructured 3D image data (point cloud) in the real world so that the data can be better used in subsequent tasks.

[0003] However, it is difficult to accurately train the point cloud image segmentation model based on the small amount of existing sample data, resulting in low performance of the trained semantic segmentation model and making it difficult to accurately segment the point cloud data during actual point cloud image segmentation tasks.

[0004] Therefore, how to improve the model performance of the semantic segmentation model and further improve the accuracy of image segmentation is an urgent problem to be solved. Summary of the invention

[0005] This specification provides a model training and task execution method based on the attention mechanism to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This manual provides a model training method based on the attention mechanism, including:

[0008] Acquire a training sample, wherein the training sample includes support set point cloud data and query set point cloud data, the query set point cloud data includes point clouds of several categories, and the several categories at least include a target category corresponding to the support set point cloud data;

[0009] Inputting the training sample into the semantic segmentation model to be trained, so as to determine feature data corresponding to the training sample through a feature extraction network of the semantic segmentation model;

[0010] Transposing the feature data to obtain transposed feature data, performing weighted processing on the transposed feature data through a first attention network of the semantic segmentation model to obtain a first weighted feature, and performing weighted processing on the feature data through a second attention network of the semantic segmentation model to obtain a second weighted feature;

[0011] Determining a target feature according to the feature data, the first weighted feature, and the second weighted feature;

[0012] According to the target features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data as a target point cloud, and a semantic segmentation result is determined according to the target point cloud;

[0013] According to the deviation between the semantic segmentation result and the label corresponding to the target point cloud, the loss value of the semantic segmentation model is determined, and the semantic segmentation model is trained according to the loss value.

[0014] Optionally, weighted processing is performed on the transposed feature data through a first attention network of the semantic segmentation model to obtain a first weighted feature, specifically including:

[0015] Performing convolution processing on the transposed feature data through a convolution layer set in the first attention network to obtain a first convolution feature;

[0016] Transpose the first convolution feature, and determine a weight corresponding to the first convolution feature according to the first convolution feature and the first convolution feature after the transposition process, as a first weight;

[0017] The first convolution feature is weighted according to the first weight to obtain the first weighted feature.

[0018] Optionally, weighted processing is performed on the feature data through a second attention network of the semantic segmentation model to obtain a second weighted feature, specifically including:

[0019] Performing convolution processing on the feature data through a convolution layer set in the second attention network to obtain a second convolution feature;

[0020] Transpose the second convolution feature, and determine a weight corresponding to the second convolution feature according to the second convolution feature and the transposed second convolution feature as a second weight;

[0021] The second convolution feature is weighted according to the second weight to obtain the second weighted feature.

[0022] Optionally, segmenting a point cloud matching the target category in the query set point cloud data according to the target feature corresponding to the support set point cloud data and the target feature corresponding to each point included in the query set point cloud data specifically includes:

[0023] Dividing the support set point cloud data into a plurality of sub-regions, and for each sub-region, determining features corresponding to points of the target category in the sub-region from target features corresponding to the support set point cloud data;

[0024] Determine the prototype features corresponding to the support set point cloud data according to the features corresponding to the points of the target category in each sub-region;

[0025] According to the prototype features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data.

[0026] Optionally, segmenting a point cloud matching the target category in the query set point cloud data according to a prototype feature corresponding to the support set point cloud data and a target feature corresponding to each point included in the query set point cloud data specifically includes:

[0027] Determining a word embedding feature corresponding to the target category according to category information corresponding to the target category;

[0028] Based on the word embedding feature, the prototype feature is enhanced to obtain a word embedding enhanced feature;

[0029] According to the word embedding enhancement feature and the target feature corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented in the query set point cloud data.

[0030] Optionally, according to the word embedding enhancement feature and the target feature corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data as a target point cloud, specifically including:

[0031] Taking each point included in the support set point cloud data and each point in the query set point cloud data as a vertex, constructing a graph neural network according to the word embedding enhancement feature corresponding to each point included in the support set point cloud data and the target feature corresponding to each point in the query set point cloud data;

[0032] Determine, according to the association relationship between different vertices in the graph neural network, features corresponding to points matching the target category as output features;

[0033] The target point cloud is determined according to the output features.

[0034] This specification provides a point cloud segmentation method based on an attention mechanism, including:

[0035] Obtain point cloud data to be segmented;

[0036] The point cloud data to be segmented is input into a pre-trained semantic segmentation model to determine a segmentation result for the point cloud data to be segmented through the semantic segmentation model, wherein the semantic segmentation model is trained by the above-mentioned model training method.

[0037] This specification provides a model training device based on an attention mechanism, including:

[0038] An acquisition module, used for acquiring training samples, wherein the training samples include support set point cloud data and query set point cloud data, the query set point cloud data includes point clouds of several categories, and the several categories at least include the target category corresponding to the support set point cloud data;

[0039] An input module, used for inputting the training sample into the semantic segmentation model to be trained, so as to determine feature data corresponding to the training sample through a feature extraction network of the semantic segmentation model;

[0040] A weighting module, configured to perform a transposition process on the feature data to obtain transposed feature data, and perform a weighted process on the transposed feature data through a first attention network of the semantic segmentation model to obtain a first weighted feature, and perform a weighted process on the feature data through a second attention network of the semantic segmentation model to obtain a second weighted feature;

[0041] A fusion module, used for determining a target feature according to the feature data, the first weighted feature and the second weighted feature;

[0042] A determination module, configured to segment a point cloud matching the target category in the query set point cloud data as a target point cloud according to the target features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, and determine a semantic segmentation result according to the target point cloud;

[0043] A training module is used to determine the loss value of the semantic segmentation model according to the deviation between the semantic segmentation result and the label corresponding to the target point cloud, and train the semantic segmentation model according to the loss value.

[0044] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model training and task execution method based on the attention mechanism.

[0045] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned model training and task execution method based on the attention mechanism is implemented.

[0046] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0047] In the model training method provided in the present specification, a training sample is obtained, wherein the training sample includes support set point cloud data and query set point cloud data; the training sample is input into a semantic segmentation model, and feature data corresponding to the training sample is determined through a feature extraction network; the feature data is transposed to obtain transposed feature data, and the transposed feature data is weighted through a first attention network to obtain a first weighted feature, and the feature data is weighted through a second attention network to obtain a second weighted feature; a target feature is determined based on the first weighted feature and the second weighted feature; semantic segmentation is performed on the query set point cloud data based on the target features corresponding to the support set point cloud data and the target features corresponding to the query set point cloud data, and a loss value is determined based on the deviation between the semantic segmentation result and its label, and model training is performed.

[0048] It can be seen from the above method that in the process of training the semantic segmentation model, this scheme can obtain attention features based on long-distance dependencies by weighting the feature data and the transposed feature data respectively, and fuse the weighted features, so as to fully extract the semantic information of the sample data. Then, the query set point cloud data is semantically segmented based on the target features corresponding to the support set point cloud data, and the model is trained based on the segmentation results, which significantly improves the model performance of the semantic segmentation model and further improves the accuracy of the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0050] Figure 1 A flowchart of a model training method based on an attention mechanism provided in this specification;

[0051] Figure 2 A schematic diagram of a target feature determination process provided in this specification;

[0052] Figure 3 A schematic diagram of a process of word embedding enhancement feature provided in this specification;

[0053] Figure 4 A schematic diagram of the overall process of model training provided in this specification;

[0054] Figure 5A schematic diagram of a process of a point cloud segmentation method provided in this specification;

[0055] Figure 6 A schematic diagram of a model training device based on an attention mechanism provided in this specification;

[0056] Figure 7 A method corresponding to the Figure 1 or Figure 5 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0058] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0059] Figure 1 A flowchart of a model training method based on an attention mechanism provided in this specification includes the following steps:

[0060] S101: Acquire training samples, wherein the training samples include support set point cloud data and query set point cloud data, the query set point cloud data includes point clouds of several categories, and the several categories at least include a target category corresponding to the support set point cloud data.

[0061] In this specification, the execution entity for implementing a model training method and a point cloud segmentation method based on an attention mechanism can be a designated device such as a server. For the sake of ease of description, the following will only take the server as an example of the execution entity to illustrate a model training method and a point cloud segmentation method based on an attention mechanism provided in this specification.

[0062] The server can obtain a point cloud data set, and then use the point cloud data corresponding to the category with a larger number of samples as training samples, and use the point cloud data corresponding to the category with a smaller number of samples as test samples.

[0063] In this specification, the training samples include support set point cloud data and query set point cloud data, wherein the category corresponding to the instances in the support set point cloud data is the target category, and the query set point cloud data may include instances of other categories in addition to instances of the target category.

[0064] For point cloud data, the instances contained therein are composed of points in the point cloud, and the category labels corresponding to the instances and the points constituting the instances are the same, that is, the query set point cloud data contains points of several categories, and these categories include at least the target category.

[0065] In one embodiment provided in this specification, the point cloud dataset may use the S3DIS dataset, which contains point cloud data of rooms of different styles, and the number of target categories may be set to 13 (12 semantic categories plus one messy category). The semantic categories are then evenly divided into two non-overlapping subsets, each with 6 categories, and the two subsets are a training set and a test set. In the above scenario, the target categories may include categories corresponding to different items in the room, such as tables, chairs, sofas, etc.

[0066] The server can pre-process the data in each subset, divide the overall point cloud data into 7547 blocks using a non-overlapping sliding window strategy, and randomly extract M = 2048 points from each block.

[0067] The server can select blocks containing at least 100 points of any training set semantic category, and all these blocks constitute training samples. Then the server can randomly select N categories from the training set semantic categories as target categories, and N must be less than the total number of training semantic categories. Then the server can randomly extract the above support set point cloud data S and query set point cloud data Q from the training samples based on the selected N categories.

[0068] For the test samples, the server may exhaustively test all combinations of N categories in the semantic category, extract 100 segments for each combination, and use them as test samples.

[0069] S102: Input the training sample into the semantic segmentation model to be trained, so as to determine feature data corresponding to the training sample through a feature extraction network of the semantic segmentation model.

[0070] In this specification, a feature extraction network, a self-attention network, and a word embedding enhancement network may be provided in the semantic segmentation model. The feature extraction network may be the backbone network of the semantic segmentation model, and the self-attention network may be a dual attention mechanism, which includes two sub-networks, namely a first attention network and a second attention network.

[0071] The server can input the training samples into the semantic segmentation model and extract features from the training samples through the feature extraction network to obtain the feature data corresponding to the training samples. .

[0072] It should be noted that the training samples in this specification include support set point cloud data and query set point cloud data. Therefore, the feature data extracted by the feature extraction network may include: feature data corresponding to the support set point cloud data and feature data corresponding to the query set point cloud data.

[0073] S103: Transpose the feature data to obtain transposed feature data, and weight the transposed feature data through the first attention network of the semantic segmentation model to obtain a first weighted feature, and weight the feature data through the second attention network of the semantic segmentation model to obtain a second weighted feature.

[0074] S104: Determine a target feature according to the feature data, the first weighted feature, and the second weighted feature.

[0075] The server can perform transposition processing on the feature data to obtain transposed feature data , and then input it into the first attention network of the semantic segmentation model for weighted processing to obtain the first weighted feature.

[0076] Specifically, the server can use the convolutional layer set in the first attention network to transpose the feature data Perform convolution processing to obtain a first convolution feature, then transpose the first convolution feature, and determine the weight corresponding to the first convolution feature according to the first convolution feature and the first convolution feature after the transposition processing, as the first weight, and then weight the first convolution feature according to the first weight to obtain a first weighted feature .

[0077] At the same time, the server can use the convolutional layer set in the second attention network to analyze the feature data. Perform convolution processing to obtain a second convolution feature, then transpose the second convolution feature, and determine the weight corresponding to the second convolution feature according to the second convolution feature and the transposed second convolution feature as the second weight, and then weight the second convolution feature according to the second weight to obtain a second weighted feature .

[0078] The server can then combine the feature data , the first weighted feature after transposition , the second weighted feature Fusion to obtain target features Among them, the target feature F can be expressed as:

[0079]

[0080] In this specification, the first attention network and the second attention network may respectively include three identical convolutional layers, so that the input feature data and the transposed feature data are respectively convoluted through these convolutional layers. For ease of understanding, this specification provides a schematic diagram of the process of determining the target feature, such as Figure 2 shown.

[0081] Figure 2 A schematic diagram of a target feature determination process provided in this specification.

[0082] Among them, for the transposed feature data , the server can pass it to the three convolutional layers in the first attention network respectively. Each convolutional layer will output the first convolutional feature. Then the server can transpose the feature output by the first convolutional layer, multiply it by the feature output by the second convolutional layer, and pass the result to the Softmax function to calculate the attention weight to obtain the first attention weight ( ). Finally, the feature output by the third convolutional layer is multiplied by the first attention weight to obtain the first weighted feature enhanced based on the self-attention mechanism. .

[0083] For feature data , the server can pass it to the three convolutional layers in the second attention network respectively. Each convolutional layer will output the second convolutional feature. Then the server can transpose the feature output by the first convolutional layer, and then multiply it by the feature output by the second convolutional layer. The result is passed to the Softmax function to calculate the attention weight and obtain the second attention weight ( ). Finally, the features output by the third convolutional layer are multiplied by the second attention weight to obtain the second weighted features enhanced based on the self-attention mechanism. .

[0084] The server can then concatenate the feature data, the transposed first weighted feature, and the second weighted feature to obtain the target feature .

[0085] from Figure 2 It can be seen that due to the first attention weight ( ) and the second attention weight ( ) corresponds to different feature channels. Therefore, by fusing the two weighted features, richer semantic information can be obtained, thereby extracting more accurate feature representations.

[0086] Of course, only one convolution layer can be set in each attention network. The server can back up the convolution features output by the convolution layer, determine the attention weight according to the transposed convolution features and the backed-up convolution features, and then weight the backed-up convolution features by the attention weight.

[0087] It should also be pointed out that since the feature data extracted by the feature extraction network includes feature data corresponding to the support set point cloud data and feature data corresponding to the query set point cloud data, the target features corresponding to the training samples may include: target features corresponding to the support set point cloud data and target features corresponding to the query set point cloud data.

[0088] S105: According to the target features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data as a target point cloud, and a semantic segmentation result is determined based on the target point cloud.

[0089] The server can segment a point cloud matching the target category in the query set point cloud data as the target point cloud based on the target features corresponding to the support set point cloud data and the target features corresponding to each point included in the query set point cloud data.

[0090] In order to further enhance the semantic information expressed by the above features, the server can divide the support set point cloud data into several sub-areas, and for each sub-area, determine the features corresponding to the points of the target category in the sub-area from the target features corresponding to the support set point cloud data, and then determine the prototype features corresponding to the support set point cloud data based on the features corresponding to the points of the target category in each sub-area.

[0091] In practical applications, the support set point cloud data may include point clouds of several target categories. For each target category, the server can divide the support set point cloud data into n=100 sub-areas by sampling. Each sub-area corresponds to a seed point. The server can calculate the features corresponding to the seed points in the sub-area. , this feature can be expressed as:

[0092]

[0093] Among them, c represents a target category (the total number of categories is N), Indicates the first sub-regions, Indicates sub-area The feature corresponding to point j in . When dividing the sub-areas, the following constraints must be met:

[0094]

[0095] in, represents the first The features corresponding to the seed points of the sub-regions. The prototype features of the support set point cloud data finally generated can be expressed as FS, with a size of (N+1, 100, 384).

[0096] The server may segment a point cloud matching the target category from the query set point cloud data according to the prototype features corresponding to the support set point cloud data and the target features corresponding to each point included in the query set point cloud data.

[0097] In addition, the server can further enhance the semantics expressed by the above-mentioned prototype features through a word embedding enhancement network. The server can determine the word embedding features corresponding to the target category based on the category information corresponding to the target category, and based on the word embedding features, perform feature enhancement on the prototype features to obtain word embedding enhancement features. Then, based on the word embedding enhancement features and the target features corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented in the query set point cloud data. For ease of understanding, this specification provides a schematic diagram of the process of word embedding enhancement features, such as Figure 3 shown.

[0098] Figure 3 A schematic diagram of a process of word embedding enhancement features provided in this specification.

[0099] The server can use the pre-trained language model GloVe to obtain the word embedding feature EM corresponding to the detection target category. Then the weight is calculated based on the word embedding feature EM. In this process, the server can pass the word embedding feature into a fully connected layer-Relu-fully connected layer structure, and then pass the result into the Softmax function to obtain the attention weight ATT, which can be expressed as:

[0100]

[0101] The server can then perform feature enhancement on the prototype feature FS according to the attention weight ATT to obtain the support set prototype feature enhanced by word embedding. (i.e. word embedding enhanced feature), which can be expressed as:

[0102]

[0103] The server can then segment a target point cloud matching the target category in the query set point cloud data based on the word embedding enhancement features and the target features corresponding to each point contained in the query set point cloud data, and then determine the semantic segmentation result based on each target point cloud segmented from the query set point cloud data.

[0104] Specifically, the server can take each point included in the support set point cloud data and each point in the query set point cloud data as a vertex, and construct a graph neural network according to the word embedding enhancement feature corresponding to each point included in the support set point cloud data and the target feature corresponding to each point in the query set point cloud data, wherein for each vertex, if the vertex is determined by a point in the support set point cloud data, its attribute value can be the word embedding enhancement feature corresponding to the point, and if the vertex is determined by a point in the query set point cloud data, its attribute value can be the target feature corresponding to the point. The graph neural network can be expressed as:

[0105]

[0106] Where X represents the input feature, with a size of V×384, V = n × (N + 1) + T × M, T represents the number of point clouds in the query set point cloud data, and M represents the number of points in each point cloud of the query set point cloud data. A represents the adjacency matrix, , W represents the learnable parameter matrix.

[0107] In the embedding space, the server can calculate the Gaussian similarity between each vertex and its k nearest neighbor vertices to construct the adjacency matrix A, which can be calculated by the following formula:

[0108] ,

[0109] in, and represents the vertices in the graph neural network, and Represents the features corresponding to the above two vertices. Represents the variance of the distance between two vertices.

[0110] Afterwards, the server can determine the features corresponding to the points matching the target category based on the above graph neural network and the association relationship between different vertices in the graph neural network as output features.

[0111] The server can pass the above output features into a fully connected layer (384, N+1) to get the prediction result , thereby determining the above target point cloud.

[0112] It should be noted that in practical applications, considering the computational efficiency of the model, the server may only determine the prototype features, and determine the target point cloud based on the prototype features corresponding to the support set point cloud data and the target features corresponding to the query set point cloud data.

[0113] S106: Determine a loss value of the semantic segmentation model according to a deviation between the semantic segmentation result and a label corresponding to the target point cloud, and train the semantic segmentation model according to the loss value.

[0114] In practical applications, the support set point cloud data may contain several target categories. Based on the word embedding enhancement features corresponding to these target categories, the points corresponding to each target category in the query set point cloud data can be segmented to obtain semantic segmentation results. In this way, the server can determine the loss value of the semantic segmentation model based on the deviation between the semantic segmentation result and the actual label corresponding to the target point cloud, and train the semantic segmentation model with minimizing the loss value as the optimization goal. For ease of understanding, this manual provides a schematic diagram of the overall process of model training, such as Figure 4 shown.

[0115] Figure 4 The figure is a schematic diagram of the overall process of model training provided in this specification.

[0116] Among them, the server can input the training samples into the backbone network, and input the feature data output by the backbone network and the transposed feature data into the attention network for weighted processing, so as to obtain the target features corresponding to the support set point cloud data (support set features) and the target features corresponding to the query set point cloud data (query set features), and then enhance the target features corresponding to the support set point cloud data through the prototype generation network and the word embedding enhancement network to obtain the word embedding enhanced features, and perform image segmentation based on the word embedding enhanced features and the target features corresponding to the query set point cloud data, and then calculate the segmentation loss based on the image segmentation results and the labels.

[0117] In one embodiment provided in this specification, the server can Take out the predictions corresponding to the T query set point cloud data, expressed as ,in ∈ R M>×>(N+1) . Then use the Softmax function to The prediction results of each point in are normalized to a probability distribution, which can be expressed as:

[0118]

[0119] The server can then calculate and tags The cross entropy loss function between , the loss value of this loss function can be expressed as:

[0120]

[0121] After the training conditions for the above-mentioned semantic segmentation model are met (such as reaching a preset number of training times or the model converges to a preset range), the server can test the semantic segmentation model through the test samples described in step S101, and deploy the semantic segmentation model after the test passes.

[0122] The following will explain an image segmentation method provided in this specification from the perspective of practical application. Figure 5 shown.

[0123] Figure 5 A schematic diagram of a point cloud segmentation method provided in this specification includes the following steps:

[0124] S501: Obtain point cloud data to be segmented.

[0125] S502: Inputting the point cloud data to be segmented into a pre-trained semantic segmentation model to determine a segmentation result for the point cloud data to be segmented through the semantic segmentation model, wherein the semantic segmentation model is trained by the above-mentioned model training method.

[0126] In actual applications, the server can obtain the point cloud data to be segmented, and then input the point cloud data into the above-mentioned semantic segmentation model. Since the semantic segmentation model has learned the feature knowledge corresponding to multiple target categories during the training process (that is, the word embedding enhanced features corresponding to multiple target categories), after the semantic segmentation model determines the target features corresponding to the point cloud data to be segmented, it can perform image segmentation on the point cloud data based on the word embedding enhanced features of the above-mentioned known categories and the target features corresponding to the point cloud data to be segmented, thereby obtaining the semantic segmentation results of the point cloud data.

[0127] After obtaining the semantic segmentation results of the point cloud data, the server can perform subsequent tasks based on the semantic segmentation results, such as performing practical tasks such as target recognition, unmanned driving, virtual reality, mixed reality, etc. based on the semantic segmentation results of the point cloud data.

[0128] From the above methods, we can see that this solution enhances the prototype features in the segmentation model through the attention mechanism, introduces text embedding into the semantic segmentation model, and also provides additional information for the model. In order to improve the expressiveness of the support set features and query set features, this solution explores the long-distance dependencies in the features in the form of self-attention, thereby obtaining richer features, which greatly improves the model's 3D semantic segmentation accuracy for few-sample targets.

[0129] The above is one or more implementations of the model training method and point cloud segmentation method based on the attention mechanism of this specification. Based on the same idea, this specification also provides corresponding devices, such as Figure 6 shown.

[0130] Figure 6 A schematic diagram of a model training device based on an attention mechanism provided for this specification includes:

[0131] An acquisition module 601 is used to acquire training samples, wherein the training samples include support set point cloud data and query set point cloud data, and the query set point cloud data includes point clouds of several categories, and the several categories at least include the target category corresponding to the support set point cloud data;

[0132] An input module 602 is used to input the training sample into the semantic segmentation model to be trained, so as to determine feature data corresponding to the training sample through a feature extraction network of the semantic segmentation model;

[0133] A weighting module 603 is used to perform a transposition process on the feature data to obtain transposed feature data, and to perform a weighted process on the transposed feature data through a first attention network of the semantic segmentation model to obtain a first weighted feature, and to perform a weighted process on the feature data through a second attention network of the semantic segmentation model to obtain a second weighted feature;

[0134] A fusion module 604 is used to determine a target feature according to the feature data, the first weighted feature and the second weighted feature;

[0135] A determination module 605, based on the target features corresponding to the support set point cloud data and the target features corresponding to each point included in the query set point cloud data, segments a point cloud matching the target category in the query set point cloud data as a target point cloud, and determines a semantic segmentation result based on the target point cloud;

[0136] The training module 606 is used to determine the loss value of the semantic segmentation model according to the deviation between the semantic segmentation result and the label corresponding to the target point cloud, and train the semantic segmentation model according to the loss value.

[0137] Optionally, the weighting module 603 is specifically used to perform convolution processing on the transposed feature data through a convolution layer set in the first attention network to obtain a first convolution feature; perform transposition processing on the first convolution feature, and determine the weight corresponding to the first convolution feature as the first weight based on the first convolution feature and the first convolution feature after the transposition processing; and weight the first convolution feature according to the first weight to obtain the first weighted feature.

[0138] Optionally, the weighting module 603 is specifically used to perform convolution processing on the feature data through a convolution layer set in the second attention network to obtain a second convolution feature; transpose the second convolution feature, and determine the weight corresponding to the second convolution feature as the second weight based on the second convolution feature and the transposed second convolution feature; weight the second convolution feature according to the second weight to obtain the second weighted feature.

[0139] Optionally, the determination module 605 is specifically used to divide the support set point cloud data into a number of sub-regions, and for each sub-region, determine the features corresponding to the points of the target category in the sub-region from the target features corresponding to the support set point cloud data; determine the prototype features corresponding to the support set point cloud data based on the features corresponding to the points of the target category in each sub-region; and segment a point cloud matching the target category in the query set point cloud data based on the prototype features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data.

[0140] Optionally, the determination module 605 is specifically used to determine the word embedding features corresponding to the target category based on the category information corresponding to the target category; based on the word embedding features, perform feature enhancement on the prototype features to obtain word embedding enhanced features; and segment a point cloud matching the target category in the query set point cloud data based on the word embedding enhanced features and the target features corresponding to each point contained in the query set point cloud data.

[0141] Optionally, the determination module 605 is specifically used to construct a graph neural network with each point contained in the support set point cloud data and each point in the query set point cloud data as a vertex, according to the word embedding enhancement features corresponding to each point contained in the support set point cloud data and the target features corresponding to each point in the query set point cloud data; determine the features corresponding to the points matching the target category as output features according to the association relationship between different vertices in the graph neural network; and determine the target point cloud based on the output features.

[0142] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A model training method based on the attention mechanism provided or Figure 5 A point cloud segmentation method based on attention mechanism is provided.

[0143] This manual also provides Figure 7 The one shown corresponds to Figure 1 A schematic diagram of the electronic device. Figure 7 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The model training method based on the attention mechanism and Figure 5 The point cloud segmentation method based on the attention mechanism. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0144] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0145] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0146] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0147] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0148] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0156] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0158] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0159] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A model training method based on attention mechanism, characterized in that: include: Acquire a training sample, wherein the training sample includes support set point cloud data and query set point cloud data, the query set point cloud data includes point clouds of several categories, and the several categories at least include a target category corresponding to the support set point cloud data; Inputting the training sample into the semantic segmentation model to be trained, so as to determine feature data corresponding to the training sample through a feature extraction network of the semantic segmentation model; Transposing the feature data to obtain transposed feature data, performing weighted processing on the transposed feature data through a first attention network of the semantic segmentation model to obtain a first weighted feature, and performing weighted processing on the feature data through a second attention network of the semantic segmentation model to obtain a second weighted feature; Determining a target feature according to the feature data, the first weighted feature, and the second weighted feature; According to the target features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data as a target point cloud, and a semantic segmentation result is determined according to the target point cloud; According to the deviation between the semantic segmentation result and the label corresponding to the target point cloud, the loss value of the semantic segmentation model is determined, and the semantic segmentation model is trained according to the loss value.

2. The method according to claim 1, characterized in that The transposed feature data is weighted by the first attention network of the semantic segmentation model to obtain a first weighted feature, specifically including: Performing convolution processing on the transposed feature data through a convolution layer set in the first attention network to obtain a first convolution feature; Transpose the first convolution feature, and determine a weight corresponding to the first convolution feature according to the first convolution feature and the first convolution feature after the transposition process, as a first weight; The first convolution feature is weighted according to the first weight to obtain the first weighted feature.

3. The method according to claim 1, characterized in that The feature data is weighted by the second attention network of the semantic segmentation model to obtain a second weighted feature, which specifically includes: Performing convolution processing on the feature data through a convolution layer set in the second attention network to obtain a second convolution feature; Transpose the second convolution feature, and determine a weight corresponding to the second convolution feature according to the second convolution feature and the transposed second convolution feature as a second weight; The second convolution feature is weighted according to the second weight to obtain the second weighted feature.

4. The method according to claim 1, characterized in that According to the target features corresponding to the support set point cloud data and the target features corresponding to each point included in the query set point cloud data, segmenting a point cloud matching the target category in the query set point cloud data, specifically comprising: Dividing the support set point cloud data into a plurality of sub-regions, and for each sub-region, determining features corresponding to points of the target category in the sub-region from target features corresponding to the support set point cloud data; Determine the prototype features corresponding to the support set point cloud data according to the features corresponding to the points of the target category in each sub-region; According to the prototype features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data.

5. The method according to claim 4, characterized in that According to the prototype features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data, specifically comprising: Determining a word embedding feature corresponding to the target category according to category information corresponding to the target category; Based on the word embedding feature, the prototype feature is enhanced to obtain a word embedding enhanced feature; According to the word embedding enhancement feature and the target feature corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented in the query set point cloud data.

6. The method according to claim 5, characterized in that According to the word embedding enhancement feature and the target feature corresponding to each point contained in the query set point cloud data, a point cloud matching the target category is segmented from the query set point cloud data as a target point cloud, specifically including: Taking each point included in the support set point cloud data and each point in the query set point cloud data as a vertex, constructing a graph neural network according to the word embedding enhancement feature corresponding to each point included in the support set point cloud data and the target feature corresponding to each point in the query set point cloud data; Determine, according to the association relationship between different vertices in the graph neural network, features corresponding to points matching the target category as output features; The target point cloud is determined according to the output features.

7. A point cloud segmentation method based on attention mechanism, characterized in that: include: Obtain point cloud data to be segmented; The point cloud data to be segmented is input into a pre-trained semantic segmentation model to determine a segmentation result for the point cloud data to be segmented through the semantic segmentation model, wherein the semantic segmentation model is trained by the method described in any one of claims 1 to 6 above.

8. A model training device based on an attention mechanism, characterized in that: include: An acquisition module, used for acquiring training samples, wherein the training samples include support set point cloud data and query set point cloud data, the query set point cloud data includes point clouds of several categories, and the several categories at least include the target category corresponding to the support set point cloud data; An input module, used for inputting the training sample into the semantic segmentation model to be trained, so as to determine feature data corresponding to the training sample through a feature extraction network of the semantic segmentation model; A weighting module, configured to perform a transposition process on the feature data to obtain transposed feature data, and perform a weighted process on the transposed feature data through a first attention network of the semantic segmentation model to obtain a first weighted feature, and perform a weighted process on the feature data through a second attention network of the semantic segmentation model to obtain a second weighted feature; A fusion module, used for determining a target feature according to the feature data, the first weighted feature and the second weighted feature; A determination module, configured to segment a point cloud matching the target category in the query set point cloud data as a target point cloud according to the target features corresponding to the support set point cloud data and the target features corresponding to each point contained in the query set point cloud data, and determine a semantic segmentation result according to the target point cloud; A training module is used to determine the loss value of the semantic segmentation model according to the deviation between the semantic segmentation result and the label corresponding to the target point cloud, and train the semantic segmentation model according to the loss value.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images