A three-dimensional target recognition method and system based on attention mechanism

Abstract

The embodiment of the present invention provides a three-dimensional target recognition method and system based on an attention mechanism, the method comprising: obtaining initial three-dimensional point cloud data of a target object; inputting the initial three-dimensional point cloud data into a preset recognition network to obtain a recognition result of the target object, the preset recognition network is obtained by training sample data and sample labels, and the preset recognition network samples the initial three-dimensional point cloud data based on the attention mechanism. The embodiment of the present invention uses an attention mechanism to sample the initial three-dimensional point cloud data, and finds points that need attention in the disordered initial three-dimensional point cloud data. These points are mainly relatively important points in the initial three-dimensional point cloud data, and are rich in more key information. Therefore, these points can replace the original point cloud data without losing point cloud information as much as possible, and reduce the amount of data and calculation, thereby improving the efficiency of target recognition.

Description

Technical Field

[0001] The present invention relates to the field of robotics and computer vision technology, and in particular to a three-dimensional target recognition method and system based on an attention mechanism. Background Art

[0002] Recently, the recognition of 3D objects has attracted more and more attention. Unlike two-dimensional data, three-dimensional data is a spatial stereo image composed of points, which contains not only color information but also depth and distance information. It has been widely used in many fields such as map surveying, real-time three-dimensional map reconstruction, and robots. In recent years, with the rich achievements and excellent recognition effects achieved by deep learning-based methods in the field of two-dimensional visual recognition, many researchers have tried to use deep learning-related algorithms in the field of three-dimensional vision to replace traditional three-dimensional object recognition and achieve efficient and high recognition accuracy of three-dimensional environment perception.

[0003] At present, deep learning-based 3D object recognition methods can be mainly divided into: multi-view based methods, volume based methods and methods that directly process point clouds. Among them, the method based on direct processing of point clouds has attracted much attention in recent years since it was first proposed in 2017. The construction mode of this type of network includes three main modules: sampling, grouping and fusion. The sampling module is the first step of the entire network, which determines which valuable information can be retained so that the entire network can have better accuracy; it determines which information needs to be discarded to improve the time efficiency of the entire network.

[0004] There are two algorithms for the existing sampling module: farthest point sampling and random sampling. The advantage of the farthest point sampling algorithm is that it can cover all points in the space as much as possible, and it can have a good effect on CAD models with a complete point cloud structure and relatively uniform point cloud distribution. However, in actual applications, the distribution of point cloud data collected by ordinary sensors is mostly uneven. At the same time, due to the difference in sensors, the difference in ambient light, and the existence of mirror reflection on the surface of objects, the point cloud data actually collected contains a large number of invalid points (NaN points); there is also a problem that the points close to the sensor position are relatively dense and the points far away from the sensor position are relatively sparse. As a result, the existing method will lose more points when downsampling in dense point clouds, and lose a lot of key information in the point cloud. In addition, the time efficiency of the farthest point sampling is relatively low, especially the sampling efficiency in large scenes cannot meet the real-time requirements of the network. For example: the data points collected by a single frame of a laser radar can be as many as 100k. It takes about 3.8879 seconds to sample from 100,000 points to 1024 points using the farthest point sampling method. However, if it is implemented through random sampling, it only takes about 0.0016 seconds. The time efficiency of random sampling is 10,000 times higher than that of the farthest point sampling.

[0005] The random sampling algorithm is to collect every element (point) in the sequence (point cloud) with equal probability. Compared with the farthest point sampling algorithm, although the random sampling speed is very fast, because the points at any position are equal in the random sampling process, the probability of points being sampled is exactly the same, so the sampling results will contain a large number of outliers and points with non-critical features, that is, the information of key points in the point cloud cannot be obtained, resulting in feature loss, and ultimately making the accuracy of the entire network poor. Summary of the invention

[0006] The embodiment of the present invention provides a three-dimensional target recognition method based on an attention mechanism, so as to solve the defect that efficiency and accuracy cannot be taken into account at the same time in the prior art, and achieve a balance between efficiency and accuracy.

[0007] The embodiment of the present invention provides a three-dimensional object recognition method based on an attention mechanism, comprising:

[0008] Obtain initial three-dimensional point cloud data of the target object;

[0009] The initial three-dimensional point cloud data is input into a preset recognition network to obtain a recognition result of the target object. The preset recognition network is obtained by training sample data and sample labels. The preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism.

[0010] According to a three-dimensional target recognition method based on an attention mechanism according to an embodiment of the present invention, the inputting of the initial three-dimensional point cloud data into a preset recognition network to obtain the recognition result of the target object specifically includes:

[0011] Inputting the initial three-dimensional point cloud data into the abstract module of the preset recognition network to obtain a target feature vector, wherein the abstract module samples the initial three-dimensional point cloud data based on an attention mechanism;

[0012] The target feature vector is input into the recognition module of the preset recognition network to obtain the recognition result of the target object.

[0013] According to a three-dimensional target recognition method based on an attention mechanism according to an embodiment of the present invention, the inputting of the initial three-dimensional point cloud data into the abstract module of the preset recognition network to obtain the target feature vector specifically includes:

[0014] For any abstraction layer in the abstraction module, the abstraction layer includes a sampling layer, a grouping layer and a fusion layer, accordingly:

[0015] First, the initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data;

[0016] Then, the key three-dimensional point cloud data is input into the grouping layer, and the included neighbor points are used as the key point local area point set by using the divided spherical area;

[0017] Finally, the local area point set is input into the fusion layer, feature mapping is performed on each local area point set, and a high-dimensional feature vector is obtained, and the high-dimensional feature vector is the target feature vector.

[0018] According to a three-dimensional target recognition method based on an attention mechanism according to an embodiment of the present invention, the initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data. The specific calculation formula is:

[0019] P t =AS(P t-1 ,H t-1 ,C t-1 ,N),

[0020] Among them, P t represents the key three-dimensional point cloud data, P t-1 represents the initial 3D point cloud data or the point cloud data sampled in the previous abstraction layer, H t-1 and C t-1 They represent the hidden layer features and state features of the previous abstract layer respectively, and N represents the number of sampling points.

[0021] According to a three-dimensional target recognition method based on an attention mechanism according to an embodiment of the present invention, the sampling layer includes a first convolution module, a second convolution module, a first long short-term memory module and a second long short-term memory module, the input end of the first convolution module is input data, the output end of the first convolution module is connected to the input end of the first long short-term memory module, the output end of the first long short-term memory module is respectively connected to the input end of the second convolution module and the input end of the second long short-term memory module, the output end of the second convolution module is connected to the input end of the second long short-term memory module, and the output end of the second long short-term memory module is the output data end of the sampling layer.

[0022] According to the three-dimensional object recognition method based on the attention mechanism according to an embodiment of the present invention, the value of N is 2.

[0023] The embodiment of the present invention further provides a three-dimensional object recognition system based on an attention mechanism, comprising:

[0024] An acquisition module, used to obtain initial three-dimensional point cloud data of the target object;

[0025] The recognition module is used to input the initial three-dimensional point cloud data into a preset recognition network to obtain the recognition result of the target object. The preset recognition network is obtained by training sample data and sample labels. The preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism.

[0026] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described three-dimensional target recognition methods based on the attention mechanism are implemented.

[0027] An embodiment of the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any of the above-described three-dimensional target recognition methods based on the attention mechanism are implemented.

[0028] An embodiment of the present invention provides a three-dimensional target recognition method and system based on an attention mechanism, which adopts the attention mechanism to sample the initial three-dimensional point cloud data, and finds points that need attention in the unnecessary initial three-dimensional point cloud data. These points are mainly relatively important points in the initial three-dimensional point cloud data and are rich in more key information. Therefore, these points can replace the original point cloud data without losing point cloud information as much as possible, and reduce the amount of data and calculation, thereby improving the target recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flowchart of a three-dimensional target recognition method based on an attention mechanism provided by an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the structure of a sampling layer provided in an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of the structure of a preset identification network provided by an embodiment of the present invention;

[0033] Figure 4 A schematic diagram showing the comparison of the robustness of different sampling methods to data loss;

[0034] Figure 5A schematic diagram of the structure of a three-dimensional object recognition system based on an attention mechanism provided by an embodiment of the present invention;

[0035] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] For practical applications in the field of 3D vision, it is necessary to process not only CAD dataset models like ModelNet40, but also large-scale actual point cloud data. PointNet++ and RSCNN can handle CAD dataset models like ModelNet40 very well, but it is difficult to process large-scale actual point clouds because the farthest point sampling algorithm used for sampling, grouping, and fusion can guarantee a certain degree of accuracy, but it consumes a lot of time; secondly, for dense points, the farthest point sampling algorithm will lose a lot of useful key information. When the input point cloud is a large scene with uneven distribution, the problem will become worse.

[0038] In order to solve this problem, the embodiment of the present invention uses a preset recognition network to generate an attention map, which focuses on the key parts of the initial three-dimensional point cloud data. It points to the relatively important points in the initial three-dimensional point cloud and guides the attention in the grouping layer and fusion layer in the preset recognition network to focus on the key points. The attention map is generated by a recurrent neural network composed of a long short-term memory convolutional network (LSTM) and some standard convolutional layers.

[0039] Figure 1 A flowchart of a three-dimensional target recognition method based on an attention mechanism is provided in an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0040] S1, obtaining the initial three-dimensional point cloud data of the target object;

[0041] First, the initial three-dimensional point cloud data of the target object is obtained. The initial three-dimensional point cloud data is the three-dimensional point cloud data obtained using relevant equipment. Generally speaking, the initial three-dimensional point cloud data will contain a particularly large number of noise points, outliers, etc., and the data volume of the three-dimensional point cloud data is generally large. Therefore, it is necessary to sample the initial three-dimensional point cloud data to improve the efficiency of subsequent data processing.

[0042] S2, inputting the initial three-dimensional point cloud data into a preset recognition network to obtain a recognition result of the target object, wherein the preset recognition network is obtained by training sample data and sample labels, and the preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism.

[0043] The collected initial three-dimensional point cloud data is input into the preset recognition network to obtain the recognition result of the target object. In the embodiment of the present invention, the preset recognition network is obtained after training based on the sample data and sample labels. Among them, the preset recognition network samples the initial three-dimensional point cloud data based on the attention sampling mechanism. Here, the attention sampling mechanism refers to only focusing on the key points in the initial three-dimensional point cloud data, extracting the sampling points carrying key information, and putting them into the subsequent calculations.

[0044] In the implementation of the present invention, an attention mechanism is used to sample the initial three-dimensional point cloud data, and points that need attention are found in the unnecessary initial three-dimensional point cloud data. These points are mainly relatively important points in the initial three-dimensional point cloud data and are rich in more key information. Therefore, these points can replace the original point cloud data without losing point cloud information as much as possible, and reduce the amount of data and calculation, thereby improving the target recognition efficiency.

[0045] Based on the above embodiment, preferably, the initial three-dimensional point cloud data is input into a preset recognition network to obtain the recognition result of the target object, which specifically includes:

[0046] Inputting the initial three-dimensional point cloud data into the abstract module of the preset recognition network to obtain a target feature vector, wherein the abstract module samples the initial three-dimensional point cloud data based on an attention mechanism;

[0047] The target feature vector is input into the recognition module of the preset recognition network to obtain the recognition result of the target object.

[0048] Specifically, the preset recognition network is a neural network, which is composed of an abstract module and a recognition module. The abstract module and the recognition module are connected in sequence. First, the initial three-dimensional point cloud data is input into the abstract module of the preset recognition network to obtain the target feature vector, and then the target feature vector is input into the recognition module to obtain the recognition result of the target object.

[0049] Among them, the abstract module samples the initial three-dimensional point cloud data based on the attention mechanism.

[0050] Based on the above embodiment, preferably, the initial three-dimensional point cloud data is input into the abstract module of the preset recognition network to obtain the target feature vector, which specifically includes:

[0051] For any abstract layer in the abstract module, the abstract layer includes a sampling layer, a grouping layer and a fusion layer. Accordingly, firstly, the initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data;

[0052] Then, the key three-dimensional point cloud data is input into the grouping layer, and the included neighbor points are used as the key point local area point set by using the divided spherical area;

[0053] Finally, the local area point set is input into the fusion layer, feature mapping is performed on each local area point set, and a high-dimensional feature vector is obtained, and the high-dimensional feature vector is the target feature vector.

[0054] Specifically, the abstract module in the embodiment of the present invention is composed of multiple abstract layers. In each abstract layer, the data generated by the previous abstract layer is processed to obtain new data. Compared with the data generated by the previous abstract layer, the new data is smaller in quantity. Each abstract layer has the function of reducing the amount of data, and each abstract layer is composed of a sampling layer, a grouping layer and a fusion layer.

[0055] Take any abstract layer as an example for explanation. If the abstract layer is the frontmost abstract layer, then the data input to the abstract layer is the initial 3D point cloud data. If the abstract layer is not the frontmost abstract layer, then the data input to the abstract layer is the data obtained from the previous abstract layer. The frontmost abstract layer is taken as an example for explanation.

[0056] The initial three-dimensional point cloud data is input into the sampling layer of the abstract layer, and the sampling layer selects sampling points carrying key information, namely key three-dimensional point cloud data, from the input original three-dimensional point cloud data.

[0057] Then the key 3D point cloud data is input into the grouping layer, and the adjacent points around the key points are found through the grouping layer to construct the local area point set. Finally, the local area point set is mapped into the target feature vector through the fusion layer.

[0058] On the basis of the above embodiment, preferably, the initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data, and the specific calculation formula is:

[0059] P t =AS(Pt-1 ,H t-1 ,C t-1 ,N), (1)

[0060] Among them, P t represents the key three-dimensional point cloud data, P t-1 represents the initial 3D point cloud data or the point cloud data sampled in the previous abstraction layer, H t-1 and C t-1 Respectively represent the hidden layer features and state features of the previous abstract layer, and N represents the preset number of times. In the training process, the embodiment of the present invention sets H t-1 and C t-1 The initial value of is 0, the value of N is manually designed, and the value of N is generally preset based on experience to determine the value. In the embodiments of the present invention, after a large number of experiments, it is verified that when the value of N is 2, the experimental results obtained are better.

[0061] On the basis of the above embodiments, preferably, the sampling layer includes a first convolution module, a second convolution module, a first long short-term memory module and a second long short-term memory module, the input end of the first convolution module is input data, the output end of the first convolution module is connected to the input end of the first long short-term memory module, the output end of the first long short-term memory module is respectively connected to the input end of the second convolution module and the input end of the second long short-term memory module, the output end of the second convolution module is connected to the input end of the second long short-term memory module, and the output end of the second long short-term memory module is the output data end of the sampling layer.

[0062] The long-short time-based module is the LSTM neural network, which selects three-dimensional point cloud data through a carefully designed "gate" structure. The generated attention feature map has the same dimension as the input point cloud and has 1 channel. After the sigmoid activation function, it represents the importance of each point relative to all points for the entire input data point cloud, that is, the attention intensity.

[0063] Figure 2 A schematic diagram of the structure of the sampling layer provided in an embodiment of the present invention is shown in FIG. Figure 2 As shown in the figure, nx3 represents input data, Conv_B1 represents the first convolution module, Conv_B2 represents the second convolution module, LSTM_B1 represents the first long short-term memory module, LSTM_B2 represents the second long short-term memory module, and 1 / 2nx3 represents output data. It can be seen that the amount of output data is smaller than that of input data.

[0064] The sampling layer is implemented through the following five steps, as follows:

[0065] The first step is to decide what information to discard from the data. This decision is implemented using the "forget gate" as shown in formula (2):

[0066] f t =σ(W f [H t-1 ,FP t-1 ]+b f ), (2)

[0067] Among them, H t-1 and FP t-1 It is the hidden layer feature of the previous layer and the feature of the input point cloud, W is the weight, and b is the bias.

[0068] The second step is to decide what information to keep from the data and what to update the status. This is divided into two parts.

[0069] First, use an “input gate” to implement which data is retained and which values ​​are to be updated (as in formula (3));

[0070] Secondly, use the state of the previous layer and the input-output gate to update the state (as shown in formula (4)), multiply the "old state" and the "forget gate" to discard unnecessary information, such as outliers and NaN points, and then add the "input gate" multiplied by a new candidate value vector created using the tanh activation layer to finally update the state;

[0071] i t =σ(W i [H t-1 ,FP t-1 ]+b i ), (3)

[0072] C t =f t *C t-1 +i t *tanh(W c [H t-1 ,P t-1 ]+b c ), (4)

[0073] The third step is to determine the output value, which is implemented using the "output gate" as shown in formula (5), and then processed through the tanh activation layer and multiplied by the "output gate" to finally determine the output part as shown in formula (6);

[0074] o t =σ(W o [H t-1 ,FP t-1 ]+b o ), (5)

[0075] H t =ot *tanh(C t ), (6)

[0076] The fourth step is to select the determined output value as shown in formula (7), and map it to (0-1) after activation. The larger the value, the higher the attention level. Finally, an attention map of the global point cloud information is obtained.

[0077] AS_map t =sigmoid t (W t H t +b t ), (7)

[0078] Finally, the number of samples of the generated attention map is manually selected to obtain the sampling key points. The attention sampling method is used to find the key information points that are more important to the global point cloud in the disordered point cloud data with translation and rotation, and complete the sampling.

[0079] Figure 3 A schematic diagram of a structure of a preset identification network provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, AS1 and AS2 both represent sampling layers, GP represents grouping layer, and fusion represents fusion layer. This preset identifies the sampling layer structure and Figure 2 The same as in.

[0080] In order to solve the time efficiency problem of farthest point sampling and the problem of key feature loss of random sampling algorithm, the embodiment of the present invention refers to the selective attention mechanism of the human visual system, such as the principle that people are interested in large, close objects with large color contrast, and proposes an attention mechanism sampling network that takes into account both sampling accuracy and speed to replace the existing farthest point sampling and random sampling algorithms, so as to achieve fast and effective downsampling of sparse point clouds in large scenes.

[0081] In an embodiment of the present invention, an attention mechanism is used to replace the existing sampling algorithm, and an attention sampling method is proposed to directly downsample the disordered point cloud data to retain the key features of the point cloud data. The sampling method of the embodiment of the present invention can be applied to different three-dimensional neural networks based on direct point cloud processing to construct an attention sampling network based on the present invention. Due to the superiority of the sampling method of the present invention, compared with the original three-dimensional network, the corresponding replaced and transformed three-dimensional network can not only ensure time efficiency but also take into account its accuracy when processing highly complex and challenging large-scene point cloud data.

[0082] In summary, the embodiments of the present invention propose an attention sampling method for point cloud data, which solves the problem that the current farthest point sampling method will cause some key information to be lost in places where point cloud data is dense, and also solves the problem that the sampling process is time-consuming and not suitable for processing large-scale point clouds.

[0083] The embodiments of the present invention can achieve good robustness in maintaining key information regardless of data loss or data noise; in terms of time efficiency, it has obvious advantages, especially when processing large-scene point cloud data.

[0084] The existing 3D network structures based on direct point cloud processing all use sampling, grouping, and fusion strategies to process point cloud data and achieve good results (such as PointNet++ and the RSCNN network with the best overall effect at present). The sampling method selected by these 3D networks is FPS (Farthest Point Sampling) to achieve downsampling of the original point cloud data and extract relatively critical information from it.

[0085] However, the FPS algorithm may cause some key information to be lost in areas where point cloud data is dense, and is not suitable for processing large-scale point clouds due to the time-consuming sampling process.

[0086] In order to solve these two problems, firstly, inspired by the human visual attention mechanism, the present invention introduces the attention mechanism into the three-dimensional network and constructs an attention sampling method. This method can actively perceive and sample more important points in the point cloud data that carry a lot of key information by analyzing the disordered point cloud data. Compared with the original three-dimensional network, the corresponding replaced and transformed three-dimensional network can not only ensure time efficiency but also take into account its accuracy when processing highly complex and challenging large-scale scene point cloud data.

[0087] (1) Robustness test

[0088] During the robustness test, the results obtained by different sampling methods are input into a benchmark network based on direct point cloud processing (here the first network PointNet that directly performs convolution on point clouds is used as the benchmark network) to test the sampling method’s robustness to point cloud information loss and noise. The evaluation index is the accuracy of object classification under different sampling strategies.

[0089] The test dataset is ModelNet40. The ModelNet40 dataset contains 12,311 CAD models of 40 man-made object categories, of which 9,843 are used for training and 2,468 are used for testing. The final metric is the overall classification accuracy of the ModelNet40 test set.

[0090] (i) Robustness to data loss

[0091] Table 1 is a comparison table of the robustness of different sampling methods to data loss. Figure 4 The robustness comparison diagram of different sampling methods for data loss is shown in Table 1 and Figure 4 As shown:

[0092] Table 1

[0093]

[0094] When the data loss rate reaches 75% (that is, when the number of points after sampling is 256), the accuracy of the farthest point sampling method drops to 82%; the accuracy of the random sampling method is only 75%, while the accuracy of the attention sampling method of the embodiment of the present invention is still 87.1%.

[0095] When the loss rate is 93.8%, the accuracies of the farthest point sampling and random sampling methods are only 50% and 33%, respectively, while the method of the embodiment of the present invention can still reach 85.1%.

[0096] Moreover, even when the data loss rate reaches 96.9%, the network using the sampling method of the embodiment of the present invention still has strong robustness, still 81.8%. Therefore, it is clear that the attention sampling method of the embodiment of the present invention can accurately find truly useful sampling points in the disordered point cloud, that is, the key information in the point cloud, and its robustness to different degrees of data loss is better than the existing farthest point sampling and random sampling methods.

[0097] (ii) Robustness to Gaussian noise

[0098] In order to verify the robustness of the attention sampling method proposed in this paper to point perturbations, a Gaussian noise with a mean of 0 and a standard deviation of 0.05 or 0.1 is added to each point; the effects of the attention sampling algorithm, the farthest point sampling algorithm and the random sampling algorithm on the entire network are compared. Table 2 is a schematic comparison of the robustness of different sampling methods to interference noise. The results are shown in Table 2.

[0099] Table 2

[0100]

[0101] Among them, RS (Random sampling), FPS (Farthest Point Sampling), and AS (Attentive Sampling).

[0102] After adding Gaussian noise, the original point cloud data is sampled to 512 and 256 points respectively. The results show that the attention sampling method of the embodiment of the present invention achieves better object classification accuracy than the farthest point sampling algorithm and the random sampling algorithm, regardless of whether the Gaussian noise has a standard deviation of 0.05 or 0.1.

[0103] The above two robustness experiments show that the attention sampling method proposed in the embodiment of the present invention can well select key points containing more information from the original point cloud data, and provide a reliable guarantee for high-precision object classification of the entire three-dimensional network.

[0104] (2) Complexity analysis in large scenarios

[0105] In this section, an embodiment of the present invention performs semantic segmentation prediction on the virtual scan data synthesized in the ScanNet dataset scene, and counts the usage of time and space costs in the network, and evaluates the differences in parameter quantity (#params), inference time (Forward time) and floating-point operations (FLOPs) between PointNet++ (which uses the farthest point sampling) and the network based on the attention sampling method of the present invention.

[0106] ScanNet has 1513 scanned and reconstructed indoor scenes. According to the experimental design in the PointNet++ paper, 1201 scenes were used for training and 312 scenes were used for testing and evaluation. In order to make a fair comparison, all experimental data in the embodiments of the present invention only use coordinate information, not RGB information, and then convert the point cloud label prediction into voxels for experiments.

[0107] During the test, the embodiment of the present invention used TensorFlow 1.12 and a single GTX 1080Ti to record the number of parameters, inference time, and floating-point operations for a batch of 8 to evaluate the complexity of the network. Since the preparation of the GPU takes time, the first batch of test time is ignored, and the statistics start from the second batch of data. Table 3 is a comparison table of different networks in terms of parameter number, inference time, and floating-point operations, as shown in Table 3:

[0108] Table 3

[0109]

[0110] Since the embodiment of the present invention uses the attention sampling method to replace the farthest point sampling algorithm in PointNet++, and the subsequent grouping and fusion modules are not changed, the parameters and floating point operations are comparable to PointNet++ (slightly higher by about 1%), but in terms of reasoning time, the network based on attention sampling only takes up 50% of the time of PointNet++. Therefore, the attention sampling method proposed in the present invention has obvious advantages in time efficiency when processing large-scale point cloud data.

[0111] Figure 5 A schematic diagram of the structure of a three-dimensional target recognition system based on an attention mechanism provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the system includes a collection module 501 and an identification module 502, wherein:

[0112] The acquisition module 501 is used to obtain initial three-dimensional point cloud data of the target object;

[0113] The recognition module 502 is used to input the initial three-dimensional point cloud data into a preset recognition network to obtain the recognition result of the target object. The preset recognition network is obtained by training sample data and sample labels. The preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism.

[0114] This embodiment is a system embodiment corresponding to the above method. For details, please refer to the above method embodiment. This system embodiment will not be described in detail here.

[0115] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a three-dimensional target recognition method based on an attention mechanism, the method comprising:

[0116] Obtain initial three-dimensional point cloud data of the target object;

[0117] The initial three-dimensional point cloud data is input into a preset recognition network to obtain a recognition result of the target object. The preset recognition network is obtained by training sample data and sample labels. The preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism.

[0118] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0119] On the other hand, an embodiment of the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute a three-dimensional object recognition method based on an attention mechanism provided by the above-mentioned method embodiments, the method comprising:

[0120] Obtain initial three-dimensional point cloud data of the target object;

[0121] The initial three-dimensional point cloud data is input into a preset recognition network to obtain a recognition result of the target object. The preset recognition network is obtained by training sample data and sample labels. The preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism.

[0122] In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program is implemented to execute a three-dimensional object recognition method based on an attention mechanism provided in the above embodiments, the method comprising:

[0123] Obtain initial three-dimensional point cloud data of the target object;

[0124] The initial three-dimensional point cloud data is input into a preset recognition network to obtain a recognition result of the target object. The preset recognition network is obtained by training sample data and sample labels. The preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional object recognition method based on attention mechanism, characterized in that: include: Obtain initial three-dimensional point cloud data of the target object; Inputting the initial three-dimensional point cloud data into a preset recognition network to obtain a recognition result of the target object, wherein the preset recognition network is obtained by training sample data and sample labels, and the preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism; The step of inputting the initial three-dimensional point cloud data into a preset recognition network to obtain the recognition result of the target object specifically includes: Inputting the initial three-dimensional point cloud data into the abstract module of the preset recognition network to obtain a target feature vector, wherein the abstract module samples the initial three-dimensional point cloud data based on an attention mechanism; Inputting the target feature vector into the recognition module of the preset recognition network to obtain the recognition result of the target object; The step of inputting the initial three-dimensional point cloud data into the abstract module of the preset recognition network to obtain the target feature vector specifically includes: For any abstraction layer in the abstraction module, the abstraction layer includes a sampling layer, a grouping layer and a fusion layer, accordingly: First, the initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data; Then, the key three-dimensional point cloud data is input into the grouping layer, and the included neighbor points are used as the key point local area point set by using the divided spherical area; Finally, the key point local area point set is input into the fusion layer, and feature mapping is performed on each key point local area point set to obtain a high-dimensional feature vector, which is the target feature vector; The initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data. The specific calculation formula is: P t =AS(P t-1 ,H t-1 ,C t-1 ,N), Among them, P t represents the key three-dimensional point cloud data, P t-1 represents the initial 3D point cloud data or the point cloud data sampled in the previous abstraction layer, H t-1 and C t-1 They represent the hidden layer features and state features of the previous abstract layer respectively, N represents the number of sampling points; AS represents attention sampling.

2. The three-dimensional target recognition method based on the attention mechanism according to claim 1, characterized in that: The sampling layer includes a first convolution module, a second convolution module, a first long short-term memory module and a second long short-term memory module. The input end of the first convolution module is input data, the output end of the first convolution module is connected to the input end of the first long short-term memory module, the output end of the first long short-term memory module is respectively connected to the input end of the second convolution module and the input end of the second long short-term memory module, the output end of the second convolution module is connected to the input end of the second long short-term memory module, and the output end of the second long short-term memory module is the output data end of the sampling layer.

3. The three-dimensional target recognition method based on the attention mechanism according to claim 1, characterized in that: The value of N is manually designed.

4. A three-dimensional object recognition system based on attention mechanism, characterized in that: include: An acquisition module, used to obtain initial three-dimensional point cloud data of the target object; A recognition module, used for inputting the initial three-dimensional point cloud data into a preset recognition network to obtain a recognition result of the target object, wherein the preset recognition network is obtained by training sample data and sample labels, and the preset recognition network samples the initial three-dimensional point cloud data based on an attention mechanism; The step of inputting the initial three-dimensional point cloud data into a preset recognition network to obtain the recognition result of the target object specifically includes: Inputting the initial three-dimensional point cloud data into the abstract module of the preset recognition network to obtain a target feature vector, wherein the abstract module samples the initial three-dimensional point cloud data based on an attention mechanism; Inputting the target feature vector into the recognition module of the preset recognition network to obtain the recognition result of the target object; The step of inputting the initial three-dimensional point cloud data into the abstract module of the preset recognition network to obtain the target feature vector specifically includes: For any abstraction layer in the abstraction module, the abstraction layer includes a sampling layer, a grouping layer and a fusion layer, accordingly: First, the initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data; Then, the key three-dimensional point cloud data is input into the grouping layer, and the included neighbor points are used as the key point local area point set by using the divided spherical area; Finally, the key point local area point set is input into the fusion layer, and feature mapping is performed on each key point local area point set to obtain a high-dimensional feature vector, which is the target feature vector; The initial three-dimensional point cloud data is input into the sampling layer to obtain key three-dimensional point cloud data. The specific calculation formula is: P t =AS(P t-1 ,H t-1 ,C t-1 ,N), Among them, P t represents the key three-dimensional point cloud data, P t-1 represents the initial 3D point cloud data or the point cloud data sampled in the previous abstraction layer, H t-1 and C t-1 They represent the hidden layer features and state features of the previous abstract layer respectively, N represents the number of sampling points; AS represents attention sampling.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the three-dimensional target recognition method based on the attention mechanism as described in any one of claims 1 to 3 are implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional target recognition method based on the attention mechanism as described in any one of claims 1 to 3 are implemented.

Images