An arbitrary scale point cloud upsampling method based on key semantic correction

Abstract

The present invention discloses a method for upsampling point clouds of arbitrary scale based on key semantic correction, comprising the following steps: S1: obtaining an interpolated point cloud of a preset scale using an intermediate point interpolation method; S2: performing feature extraction on the interpolated point cloud; S3: calculating the attention weight of the key semantics of each point in the interpolated point cloud; S4: correcting the extracted features using the attention weights, then performing channel reduction and residual connection to obtain an upsampled point cloud; S5: processing the upsampled point cloud using a point selection strategy and a farthest point sampling strategy to obtain an upsampled point cloud of a target scale. By using trigonometric function encoding, a point selection strategy, and a farthest point sampling strategy, the present invention reduces network learning parameters, achieves point cloud generation of arbitrary scale, and improves the uniformity of the upsampled point cloud.

Description

Technical Field

[0001] The present invention relates to the field of point cloud upsampling, and more specifically, to a method, system and storage medium for upsampling point clouds of arbitrary scale based on key semantic correction. Background Art

[0002] With the popularity of commercial 3D scanners, capturing point clouds from real-world scenes has become convenient and affordable. As a result, point clouds have been widely used in fields such as autonomous driving, robotics, and remote sensing. However, the raw point clouds produced by 3D scanners or depth cameras are usually sparse and noisy, sometimes with small holes and vacancies, which greatly affects the performance of downstream tasks such as semantic classification, rendering, and surface reconstruction. Therefore, it is crucial to upsample the raw point cloud into a dense, clean, and complete point cloud. The upsampled point cloud has more complex collective information and can serve downstream tasks well. However, many existing point cloud upsampling methods usually face two key problems: a fixed upsampling rate after one-time training, because the feature expansion unit is customized for each upsampling rate; secondly, outliers or shrinkage artifacts caused by the difficulty in accurately predicting the 3D coordinates or residuals of the upsampled points. Summary of the Invention

[0003] In order to overcome the defects of the above-mentioned prior art point cloud upsampling methods in which the upsampling rate is fixed after one-time training, and the outliers or shrinkage artifacts caused by the difficulty in accurately predicting the 3D coordinates or residuals of the upsampling points, the present invention provides a point cloud upsampling method, system and storage medium for arbitrary scale based on key semantic correction.

[0004] The primary purpose of the present invention is to solve the above technical problems, and the technical solutions of the present invention are as follows:

[0005] A first aspect of the present invention provides an arbitrary scale point cloud upsampling method based on key semantic correction, comprising:

[0006] S1: Use the intermediate point interpolation method to obtain the interpolated point cloud of the preset scale;

[0007] S2: Feature extraction on the interpolated point cloud;

[0008] S3: Calculate the attention weight of the semantic information of each point in the interpolated point cloud;

[0009] S4: Use attention weights to modify the extracted features, then perform channel reduction and residual connection to obtain the upsampled point cloud;

[0010] S5: The upsampled point cloud is processed by using the point selection strategy and the farthest point sampling strategy to obtain an upsampled point cloud of the target scale.

[0011] Furthermore, in step S1, the intermediate point interpolation method is used to obtain an interpolated point cloud of a preset scale. The specific process is as follows:

[0012] First, generate interpolation points. For each sparse point , first find the point Neighbors , then use As the newly generated point. Then use the farthest point sampling method to remove the overlapping generated points and control the number of points to times, , The required upsampling multiple is used to obtain the final coarse upsampling interpolation point cloud. ; Among them, the upsampled interpolated point cloud The ratio is greater than the upsampling ratio of;

[0013] Represents a point cloud.

[0014] Furthermore, in step S2, feature extraction is performed on the interpolated point cloud. The specific process is as follows:

[0015] First, perform feature encoding of the point cloud to increase the feature dimension of the point cloud. For any interpolated point , using trigonometric function encoding, the specific implementation is:

[0016]

[0017] Among them, PosEmb() means to put the point Encode and get the point The encoded features, here means converting the point position from three dimensions to C dimensions for position encoding; () means Put together, similar to the (x, y, z) form; Indicates that the three coordinates along each point are encoded separately, is the feature channel size of each point after encoding;

[0018] The specific trigonometric function details are as follows:

[0019]

[0020]

[0021] Among them, each position index C / 6], Control amplitude and wavelength separately;

[0022] Then, based on the encoded features, a four-stage structure is used to hierarchically aggregate local features; then feature extension is performed, first finding Neighbor features , for any central point Features , perform the following processing along each feature dimension:

[0023]

[0024] in, Indicates that each center point extends the features according to each neighboring point; Indicates the center point The number of k nearest neighbors, in this case k is equal to ;

[0025] After feature extension, for each feature Perform relative position encoding and assign weights; first standardize the point coordinates of each point's neighbors:

[0026]

[0027] The x-position direction is normalized as shown above, where Indicates a very small amount; represents the mean x position of the neighboring points, represents the standard deviation, Indicates the x-axis point after this normalization process;

[0028] The same operation is performed on other position directions as on the x position direction;

[0029] Then the characteristics of the k nearest neighbors change with the weight, expressed as:

[0030]

[0031] in, Represents the center point after the above processing position standardization The position of the jth neighbor point after transformation; Represents each center point The local features between the neighboring points are aggregated with the features between the neighboring points; the encoding method adopts the trigonometric function encoding method mentioned above to encode, and the weights are assigned to the neighboring points of each center point. Indicates the multiplication operation of each element;

[0032] After the features of each local area are extended, the features are aggregated to obtain the local feature information represented by each center point. :

[0033]

[0034] in Respectively represent the aggregation of neighbor feature information from the perspective of maximum pooling and average pooling;

[0035] Finally, the first stage will get the response characteristics , where the characteristics are , represents the features of the first stage, where N represents the number of points in the sparse point cloud;

[0036] Through four stages of different extraction, each stage focuses on different feature spaces and local information representation, and the local information of the upsampled point cloud can be obtained. , where the local features of the point cloud are .

[0037] Furthermore, the attention weight of the semantic information of each point in the interpolated point cloud is calculated in step S3. The specific process is:

[0038] Assign different bias weights to each interpolation point based on the semantic information of the point in the point cloud; consider the sensitivity of each point in the entire point cloud, and then assign different feature weights and coordinate increments to different points; first, perform feature encoding through interpolation points:

[0039]

[0040] in, represents all points of the sparse point cloud, Indicates the geometric information of the point cloud composed of these points. EdgeConvMLP represents a deep network with edge relationships, which can learn the geometric information of the point cloud and the interaction information between each position.

[0041] Assign the characteristics of the space:

[0042]

[0043] Among them, i represents the index of each point in the point cloud, represents a collection of point cloud points, Indicates processing along each feature channel, represents the attention weight.

[0044] Furthermore, in step S4, the extracted features are corrected using the attention weights, and then channel reduction and residual connection are performed to obtain the upsampled point cloud. The specific process is as follows:

[0045] Through S2 and S3, we get the local information of the point cloud in space and the attention weight of each point, and then get the entire upsampled point cloud. The local features are transformed into:

[0046]

[0047] in, It means that the features extracted by point cloud features are connected with the features extracted by key point feature correction to correct the feature extraction in S2; Represents the local features of the point cloud obtained in step S2 The feature of the j-th channel of each point c in; c represents each point, j represents each feature channel;

[0048] Through the above processing, we can obtain the local information of the semantic information constraint of the upsampled point cloud; then we can perform channel reduction and residual connection to obtain a more refined upsampled point cloud:

[0049]

[0050] Among them, F represents the feature extracted from the point cloud obtained in the previous step, express The position increment of each point in MLP is used to reduce the features from Reduce to 3D space and get the position increment of the interpolated point cloud; then correct the position of each point:

[0051]

[0052] get:

[0053]

[0054] in, Indicates the corrected x position of the i-th point, It represents the x position of the i-th point obtained by the first interpolation step, and Q represents the point cloud Q' generated by interpolation after the position of each point is corrected.

[0055] Furthermore, in step S5, the upsampled point cloud is processed by using the point selection strategy and the farthest point sampling strategy to obtain an upsampled point cloud of the target ratio. The specific process is as follows:

[0056] First, the point selection strategy is used. Here, the point selection strategy uses the outlier removal strategy. For each intermediate point , and its k nearest neighbors , calculate the mean deviation of each midpoint :

[0057]

[0058] For each midpoint, use the same method to get the mean deviation of each point :

[0059]

[0060] represents the i-th adjacent point of each midpoint; N represents the number of points in the original sparse point cloud; Indicates using the farthest point sampling method, removing overlapping generated points and controlling the multiple of the number of points;

[0061] Then determine the selection of each point. If a point is an outlier, it is expressed as:

[0062]

[0063] in, It represents a constant ratio;

[0064] After the outlier removal strategy, the farthest point sampling is used Strategy, get the corresponding multiples; and then get the final upsampled point cloud ; Select Earth Mover's Distance (EMD) distance for constraint during training:

[0065]

[0066] in, Indicates the EMD distance between two point clouds. It is a regular constraint term, a constant that changes with the number of training times to ensure the stability of the generated point cloud. This article uses and with The sum of the distances of change, is the generated upsampled point cloud, is the reference point cloud of the target, Q1 represents the point cloud that needs to be moved, and Q2 represents the distribution of the point cloud that you want to obtain, that is, the target point cloud. represents the i-th point in Q1, Represents the function that maps from Q1 to Q2.

[0067] A second aspect of the present invention provides a system for upsampling a point cloud at any scale based on key semantic correction. The system includes: a memory and a processor. The memory includes a program for upsampling a point cloud at any scale based on key semantic correction. When the program is executed by the processor, the following steps are implemented:

[0068] S1: Use the intermediate point interpolation method to obtain the interpolated point cloud of the preset scale;

[0069] S2: Feature extraction on the interpolated point cloud;

[0070] S3: Calculate the attention weight of the semantic information of each point in the interpolated point cloud;

[0071] S4: Use attention weights to modify the extracted features, then perform channel reduction and residual connection to obtain the upsampled point cloud;

[0072] S5: The upsampled point cloud is processed by using the point selection strategy and the farthest point sampling strategy to obtain an upsampled point cloud of the target scale.

[0073] Furthermore, in step S1, the intermediate point interpolation method is used to obtain an interpolated point cloud of a preset scale. The specific process is as follows:

[0074] First, generate interpolation points. For each sparse point , first find the point Neighbors , then use As the newly generated point. Then use the farthest point sampling method to remove the overlapping generated points and control the number of points to times, , The required upsampling multiple is used to obtain the final coarse upsampling interpolation point cloud. ; Among them, the upsampled interpolated point cloud The ratio is greater than the upsampling ratio of.

[0075] Furthermore, in step S2, feature extraction is performed on the interpolated point cloud. The specific process is as follows:

[0076] First, perform feature encoding of the point cloud to increase the feature dimension of the point cloud. For any interpolated point , using trigonometric function encoding, the specific implementation is:

[0077]

[0078] Among them, PosEmb() means to put the point Encode and get the point The encoded features, here means converting the point position from three dimensions to C dimensions for position encoding; () means Put together, similar to the (x, y, z) form; Indicates that the three coordinates along each point are encoded separately, is the feature channel size of each point after encoding;

[0079] The specific trigonometric function details are as follows:

[0080]

[0081]

[0082] Among them, each position index C / 6], Control amplitude and wavelength separately;

[0083] Then, based on the encoded features, a four-stage structure is used to hierarchically aggregate local features; then feature extension is performed, first finding Neighbor features , for any central point Features , perform the following processing along each feature dimension:

[0084]

[0085] in, Indicates that each center point extends the features according to each neighboring point; Indicates the center point The number of k nearest neighbors, in this case k is equal to ;

[0086] After feature extension, for each feature Perform relative position encoding and assign weights; first standardize the point coordinates of each point's neighbors:

[0087]

[0088] The x-position direction is normalized as shown above, where Indicates a very small amount; represents the mean x position of the neighboring points, represents the standard deviation, Indicates the x-axis point after this normalization process;

[0089] The same operation is performed on other position directions as on the x position direction;

[0090] Then the characteristics of the k nearest neighbors change with the weight, expressed as:

[0091]

[0092] in, Represents the center point after the above processing position standardization The position of the jth neighbor point after transformation; Represents each center point The local features between the neighboring points are aggregated with the features between the neighboring points; the encoding method adopts the trigonometric function encoding method mentioned above to encode, and the weights are assigned to the neighboring points of each center point. Indicates the multiplication operation of each element;

[0093] After the features of each local area are extended, the features are aggregated to obtain the local feature information represented by each center point. :

[0094]

[0095] in Respectively represent the aggregation of neighbor feature information from the perspective of maximum pooling and average pooling;

[0096] Finally, the first stage will get the response characteristics , where the characteristics are , represents the features of the first stage, where N represents the number of points in the sparse point cloud;

[0097] Through four stages of different extraction, each stage focuses on different feature spaces and local information representation, and the local information of the upsampled point cloud can be obtained. , where the local features of the point cloud are .

[0098] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for a method for upsampling a point cloud of any scale based on key semantic correction. When the program for upsampling a point cloud of any scale based on key semantic correction is executed by a processor, the steps of the method for upsampling a point cloud of any scale based on key semantic correction are implemented.

[0099] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0100] The present invention uses an intermediate point interpolation method to obtain an interpolated point cloud of a preset proportion; performs feature extraction on the interpolated point cloud, and uses trigonometric function encoding in the feature extraction part to map the features to a high-dimensional space, which can capture high-frequency 3D geometric information, and through the changes of sine and cosine, the absolute and relative position representation between different points can be realized, thereby better learning the geometric information of the point cloud; calculates the attention weight of the semantic information of each point in the interpolated point cloud; uses the attention weight to correct the extracted features, and then performs channel reduction and residual connection to obtain an upsampled point cloud, which can obtain a more refined upsampled point cloud; processes the upsampled point cloud by using a point selection strategy and a farthest point sampling strategy to obtain an upsampled point cloud of a target proportion, so that the upsampled point cloud is more uniform and can achieve any proportion. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 A flowchart of a method for upsampling point clouds at any scale based on key semantic correction is provided in an embodiment of the present invention.

[0102] Figure 2 A network architecture diagram of an arbitrary-scale point cloud upsampling method based on key semantic correction provided by an embodiment of the present invention.

[0103] Figure 3 A schematic diagram of key point position processing in an arbitrary scale point cloud upsampling method based on key semantic correction provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0104] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0105] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0106] Example 1

[0107] like Figure 1 As shown, the present invention provides an arbitrary scale point cloud upsampling method based on key semantic correction, comprising the following steps:

[0108] S1: Use the intermediate point interpolation method to obtain the interpolated point cloud of the preset scale.

[0109] More specifically, the intermediate point interpolation method is used to perform point cloud upsampling of any scale. By correcting the point cloud upsampling after interpolation, the network learning and point cloud generation can be decoupled, thereby better achieving point cloud upsampling of any scale. The intermediate point interpolation point cloud includes the following steps: first, the interpolation point is generated. For each sparse point , first find of Neighbors , then use As the newly generated points. Since the task ratio is too large, it is easy to cause point overlap or uneven distribution of some dense points. Secondly, the farthest point sampling method is used to remove the overlapping generated points and control the number of points to times, , The required upsampling multiple is used to obtain the final coarse upsampling interpolation point cloud. . The upsampled interpolated point cloud The ratio is greater than the upsampling ratio By using a ratio greater than the upsampling ratio The sampling can learn better geometric information later, and finally obtain an upsampled point cloud of any proportion through the point selection strategy and the farthest point sampling strategy, which can achieve a more uniform and more geometrically informative upsampled dense point cloud of a certain proportion, and better serve downstream tasks.

[0110] S2: Perform feature extraction on the interpolated point cloud.

[0111] More specifically, the process details are as follows Figure 2 Network structure diagram.

[0112] After the intermediate points are interpolated, a rough upsampled point cloud is obtained for feature extraction. First, feature encoding of the point cloud is performed to increase the feature dimension of the point cloud. For any interpolated point

[0113] , using trigonometric function encoding, the specific implementation is:

[0114]

[0115] Among them, PosEmb() means to put the point Encode and get the point Encoded features. Here, it means converting the point position from three dimensions to C dimensions for position encoding. () means Put together, similar to the (x, y, z) form; , which means that the three coordinates along each point are encoded separately. is the feature channel size of each point after encoding. The specific trigonometric function details are as follows:

[0116]

[0117]

[0118] Among them, the encoding (x, y, z) is three-dimensional, but after encoding, it is necessary to obtain C dimensions. Therefore, the average dimension encoded on the original x, y, and z dimensions is C / 3, which means that each channel is encoded in C / 3 dimensions. The encoding uses trigonometric functions, so the sine and cosine components are used alternately. Sine is used at odd index positions and cosine functions are used at even index positions. Therefore, each dimension is given sine and cosine from C / 3, which is C / 6. Therefore, each position index is C / 6]. By controlling the amplitude and wavelength respectively and encoding them through trigonometric functions, the relative position relationship between the encoded points can be better represented, and the geometric information of the object can be better represented and obtained.

[0119] Then, based on the encoded features, a four-stage structure is used to hierarchically aggregate local features. As the network structure continues to deepen, large-dimensional features are required to better learn the semantic information of the point cloud. Therefore, feature extension is first performed to find Neighbor features , for any central point Features , perform the following processing along each feature dimension:

[0120]

[0121] in, Indicates that each center point extends the features according to each neighboring point; Indicates the center point The number of k nearest neighbors, in this case k is equal to .

[0122] After feature extension, in order to learn the contribution of the distribution of k neighboring points of each center point in the space to the local space, each feature Perform relative position encoding and assign weights. First, standardize the coordinates of each point's neighbors:

[0123]

[0124] The x-position direction is normalized as shown above, where Indicates a very small amount; represents the mean x position of the neighboring points, represents the standard deviation, Represents the x-axis point after this normalization process.

[0125] The same operation is performed on other position directions as on the x position direction.

[0126] Then the characteristics of the k nearest neighbors change with the weight, expressed as:

[0127]

[0128] in, Represents the center point after the above processing position standardization The position of the jth neighbor point after transformation; Represents each center point The local features between the neighboring points are aggregated with the features between the neighboring points; the encoding method still adopts the trigonometric function encoding method mentioned above, and the weights are assigned to the neighboring points of each center point, and Indicates the multiplication operation for each element.

[0129] After the features of each local area are extended, the features are aggregated to obtain the local feature information of each center point. :

[0130]

[0131] in They represent the aggregation of neighboring feature information from the aspects of maximum pooling and average pooling, respectively, to better represent local information.

[0132] Finally, the first stage will get the response characteristics , where the characteristics are , represents the features of the first stage, where N represents the number of points in the sparse point cloud. For example, if an object point cloud is upsampled r times from N sparse points, the number of upsampled points generated is rN, where N represents the number of points in the sparse point cloud.

[0133] Through four stages of different extraction, for example, the local features obtained in the first stage , as the input of the second stage of local geometric aggregation, each stage focuses on different feature spaces and local information representations to obtain local information of the upsampled point cloud , where the local features of the point cloud are .

[0134] S3: Calculate the attention weight of the semantic information of each point in the interpolated point cloud.

[0135] More specifically, since each point in the point cloud has a different position, it has different geometric information in the object, which has a different impact on the aggregation of the entire point cloud set. Therefore, different bias weights are assigned to each interpolated point based on the semantic information of these points. Considering the sensitivity of each point in the entire point cloud, different feature weights and coordinate increments are assigned to different points. First, feature encoding is performed using interpolated points:

[0136]

[0137] in, represents all points of the sparse point cloud, It represents the geometric information of the point cloud composed of these points. EdgeConvMLP represents a deep network with edge relationships, which can learn the geometric information of the point cloud and the interaction information between each position.

[0138] like Figure 3 , the selection weight constraint of the key points should be biased towards the side close to the edge rather than the opposite side. Raising the interpolation points to a higher dimensional space can better learn the contribution of different points to the geometric space, and in the high-dimensional space, it is possible to learn the semantic feature information of high-frequency points. These points are mostly distributed at the edge of the object, so it is possible to control and give very small incremental information. In high-dimensional space, high-frequency information dominates, but in actual position increments, high-frequency points are the points that need attention. Therefore, we want to converge locally at the points on the edge of the object, rather than adding them across the domain space to the opposite position of the blank area, and distribute the features of the space:

[0139]

[0140] Among them, i represents the index of each point in the point cloud, represents a collection of point cloud points, It means that processing is performed along each feature channel, and then in the high-dimensional space, high-frequency information is processed as above to obtain a smaller incremental weight.

[0141] S4: Use attention weights to modify the extracted features, then perform channel reduction and residual connection to obtain the upsampled point cloud.

[0142] More specifically, through S2 and S3, we get the local information of the point cloud in space and the attention weight of each point, and then get the entire upsampled point cloud. The local features are transformed into:

[0143]

[0144] in, It means that the features extracted by point cloud features are connected with the features extracted by key point feature correction to correct the feature extraction in S2; Represents the local features of the point cloud obtained in step S2 The feature of the j-th channel of each point c in ; c represents each point and j represents each feature channel.

[0145] Through the above processing, the semantically constrained local information of the upsampled point cloud is obtained. Then, channel reduction and residual connection are performed to obtain a more refined upsampled point cloud:

[0146]

[0147] Among them, F represents the feature extracted from the point cloud obtained in the previous step, express The position increment of each point in MLP is used to reduce the features from Reduce to 3D space and get the position increment of the interpolated point cloud. Correct the position of each point:

[0148]

[0149] get:

[0150]

[0151] in, Indicates the corrected x position of the i-th point, It represents the x position of the i-th point obtained by the first interpolation step, and Q represents the point cloud Q' generated by the interpolation after the position of each point is corrected. Since the upsampling ratio is , and what we get is Therefore, a sampling point selection strategy is designed to make the upsampled point cloud more uniform and achieve arbitrary scale.

[0152] First, the point selection strategy is used. Here, the point selection strategy uses the outlier removal strategy. For each intermediate point , and its k nearest neighbors , calculate the mean deviation of each midpoint :

[0153]

[0154] For each midpoint, use the same method to get the mean deviation of each point :

[0155]

[0156] represents the i-th adjacent point of each midpoint; N represents the number of points in the original sparse point cloud; Indicates using the farthest point sampling method, removing overlapping generated points and controlling the multiple of the number of points;

[0157] Then determine the selection of each point. If a point is an outlier, it is expressed as:

[0158]

[0159] in, It represents a constant ratio, such as 1.5, which means that the average distance between the point and its neighboring points is greater than 1.5 times the average distance between all points and their neighboring points. It is considered an outlier. After the outlier removal strategy is used, in order to achieve a fixed ratio of point cloud upsampling , and then use the farthest point sampling strategy and get the corresponding multiples.

[0160] Then we get the final upsampled point cloud .

[0161] During training, select Earth Mover's Distance (EMD) distance for constraint:

[0162]

[0163] in, Indicates the EMD distance between two point clouds. It is a regular constraint term, a constant that changes with the number of training times to ensure the stability of the generated point cloud. This article uses and with The sum of the distances of change, is the generated upsampled point cloud, It is the reference point cloud of the target, Q1 represents the point cloud that needs to be moved, which can be understood as the generated point cloud. At this time, it is different from the target point cloud. Q2 represents the distribution of the point cloud that you want to obtain, which can be understood as the target point cloud. represents the i-th point in Q1, Represents the function that maps from Q1 to Q2.

[0164] During feature extraction, an encoding module of trigonometric functions that does not require learning is used to reduce the number of network parameters. However, MLP, a learnable method for feature encoding, can also be used to achieve this task.

[0165] A second aspect of the present invention provides a system for upsampling a point cloud at any scale based on key semantic correction. The system includes: a memory and a processor. The memory includes a program for upsampling a point cloud at any scale based on key semantic correction. When the program is executed by the processor, the following steps are implemented:

[0166] S1: Use the intermediate point interpolation method to obtain the interpolated point cloud of the preset scale;

[0167] S2: Feature extraction on the interpolated point cloud;

[0168] S3: Calculate the attention weight of the semantic information of each point in the interpolated point cloud;

[0169] S4: Use attention weights to modify the extracted features, then perform channel reduction and residual connection to obtain the upsampled point cloud;

[0170] S5: The upsampled point cloud is processed by using the point selection strategy and the farthest point sampling strategy to obtain an upsampled point cloud of the target scale.

[0171] The third aspect of the present invention provides a computer-readable storage medium, which includes a program for a method for upsampling a point cloud of any scale based on key semantic correction. When the program for upsampling a point cloud of any scale based on key semantic correction is executed by a processor, the steps of the method for upsampling a point cloud of any scale based on key semantic correction are implemented.

[0172] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A method for upsampling point clouds at any scale based on key semantic correction, characterized in that: The following steps are involved: S1: Use the intermediate point interpolation method to obtain the interpolated point cloud of the preset scale; S2: Feature extraction on the interpolated point cloud; S3: Calculate the attention weight of the semantic information of each point in the interpolated point cloud; S4: Use attention weights to modify the extracted features, then perform channel reduction and residual connection to obtain the upsampled point cloud; S5: Processing the upsampled point cloud by using the point selection strategy and the farthest point sampling strategy to obtain the upsampled point cloud of the target scale; Step S5 processes the upsampled point cloud by using the point selection strategy and the farthest point sampling strategy to obtain the upsampled point cloud of the target ratio. The specific process is as follows: First, the point selection strategy is used. Here, the point selection strategy uses the outlier removal strategy. For each intermediate point , and its k nearest neighbors , calculate the mean deviation of each midpoint : For each midpoint, use the same method to get the mean deviation of each point : represents the i-th adjacent point of each midpoint; N represents the number of points in the original sparse point cloud; Indicates using the farthest point sampling method, removing overlapping generated points and controlling the multiple of the number of points; Then determine the selection of each point. If a point is an outlier, it is expressed as: in, It represents a constant ratio; After the outlier removal strategy, the farthest point sampling is used Strategy, get the corresponding multiples; and then get the final upsampled point cloud ; Select Earth Mover's Distance (EMD) distance for constraint during training: in, Indicates the EMD distance between two point clouds. It is a regular constraint term, a constant that changes with the number of training times to ensure the stability of the generated point cloud. express and with The sum of the distances of change, is the generated upsampled point cloud, is the reference point cloud of the target, Q1 represents the point cloud that needs to be moved, and Q2 represents the distribution of the point cloud that you want to obtain, that is, the target point cloud. represents the i-th point in Q1, Represents the function that maps from Q1 to Q2.

2. The method for upsampling point clouds at any scale based on key semantic correction according to claim 1, characterized in that: In step S1, the intermediate point interpolation method is used to obtain an interpolated point cloud of a preset scale. The specific process is as follows: First, generate interpolation points. For each sparse point , first find the point Neighbors , then use As the newly generated points; secondly, use the farthest point sampling method to remove overlapping generated points and control the number of points to times, , The required upsampling multiple is used to obtain the final coarse upsampling interpolation point cloud. ; Among them, the upsampled interpolated point cloud The ratio is greater than the upsampling ratio of; Represents a point cloud.

3. The arbitrary scale point cloud upsampling method based on key semantic correction according to claim 2, characterized in that: Step S2 performs feature extraction on the interpolated point cloud. The specific process is as follows: First, perform feature encoding of the point cloud to increase the feature dimension of the point cloud. For any interpolated point , using trigonometric function encoding, the specific implementation is: Among them, PosEmb() means to put the point Encode and get the point The encoded features, here means converting the point position from three dimensions to C dimensions for position encoding; () means put together; Indicates that the three coordinates along each point are encoded separately, is the feature channel size of each point after encoding; The specific trigonometric function details are as follows: Among them, each position index C / 6], Control amplitude and wavelength separately; Then, based on the encoded features, local features are aggregated hierarchically; then feature extension is performed, first finding Neighbor features , for any central point Features , perform the following processing along each feature dimension: in, Indicates that each center point extends the features according to each neighboring point; Indicates the center point The number of k nearest neighbors, in this case k is equal to ; After feature extension, for each feature Perform relative position encoding and assign weights; first standardize the point coordinates of each point's neighbors: The x-position direction is normalized as shown above, where Indicates a very small amount; represents the mean x position of the neighboring points, represents the standard deviation, Indicates the x-axis point after this normalization process; The same operation is performed on other position directions as on the x position direction; Then the characteristics of the k nearest neighbors change with the weight, expressed as: in, Represents the center point after the above processing position standardization The position of the jth neighbor point after transformation; Represents each center point The local features between the neighboring points are aggregated with the features between the neighboring points; the encoding method adopts the trigonometric function encoding method mentioned above to encode, and the weights are assigned to the neighboring points of each center point. Indicates the multiplication operation of each element; After the features of each local area are extended, the features are aggregated to obtain the local feature information represented by each center point. : in Respectively represent the aggregation of neighbor feature information from the perspective of maximum pooling and average pooling; Finally, the first stage will get the response characteristics , where the characteristics are , represents the features of the first stage, where N represents the number of points in the sparse point cloud; Through four stages of different extraction, each stage focuses on different feature spaces and local information representation, and the local information of the upsampled point cloud can be obtained. , where the local features of the point cloud are .

4. The arbitrary scale point cloud upsampling method based on key semantic correction according to claim 3 is characterized in that: Step S3 calculates the attention weight of the semantic information of each point in the interpolated point cloud. The specific process is: Assign different bias weights to each interpolation point based on the semantic information of the point in the point cloud; assign different feature weights and their coordinate increments to different points; first perform feature encoding through the interpolation points: in, represents all points of the sparse point cloud, Indicates the geometric information of the point cloud composed of these points. EdgeConvMLP represents a deep network with edge relationships, which can learn the geometric information of the point cloud and the interaction information between each position. Assign the characteristics of the space: Among them, i represents the index of each point in the point cloud, represents a collection of point cloud points, Indicates processing along each feature channel, represents the attention weight.

5. The method for upsampling point clouds at any scale based on key semantic correction according to claim 4, characterized in that: In step S4, the extracted features are corrected using the attention weights, and then channel reduction and residual connection are performed to obtain the upsampled point cloud. The specific process is as follows: Through S2 and S3, we get the local information of the point cloud in space and the attention weight of each point, and then get the entire upsampled point cloud. The local features are transformed into: in, It means that the features extracted by point cloud features are connected with the features extracted by key point feature correction to correct the feature extraction in S2; Represents the local features of the point cloud obtained in step S2 The feature of the j-th channel of each point c in; c represents each point, j represents each feature channel; Through the above processing, we can obtain the local information of the semantic constraints of the upsampled point cloud; then we can perform channel reduction and residual connection to obtain a more refined upsampled point cloud: Among them, F represents the feature extracted from the point cloud obtained in the previous step, express The position increment of each point in MLP is used to reduce the features from Reduce to 3D space and get the position increment of the interpolated point cloud; then correct the position of each point: get: in, Indicates the corrected x position of the i-th point, It represents the x position of the i-th point obtained by the first interpolation step, and Q represents the point cloud Q' generated by interpolation after the position of each point is corrected.

6. An arbitrary scale point cloud upsampling system based on key semantic correction, characterized in that: The system includes: a memory and a processor, wherein the memory includes a program for a point cloud upsampling method for any scale based on key semantic correction, and when the program is executed by the processor, the following steps are implemented: S1: Use the intermediate point interpolation method to obtain the interpolated point cloud of the preset scale; S2: Feature extraction on the interpolated point cloud; S3: Calculate the attention weight of the semantic information of each point in the interpolated point cloud; S4: Use attention weights to modify the extracted features, then perform channel reduction and residual connection to obtain the upsampled point cloud; S5: Processing the upsampled point cloud by using the point selection strategy and the farthest point sampling strategy to obtain the upsampled point cloud of the target scale; Step S5 processes the upsampled point cloud by using the point selection strategy and the farthest point sampling strategy to obtain the upsampled point cloud of the target ratio. The specific process is as follows: First, the point selection strategy is used. Here, the point selection strategy uses the outlier removal strategy. For each intermediate point , and its k nearest neighbors , calculate the mean deviation of each midpoint : For each midpoint, use the same method to get the mean deviation of each point : represents the i-th adjacent point of each midpoint; N represents the number of points in the original sparse point cloud; Indicates using the farthest point sampling method, removing overlapping generated points and controlling the multiple of the number of points; Then determine the selection of each point. If a point is an outlier, it is expressed as: in, It represents a constant ratio; After the outlier removal strategy, the farthest point sampling is used Strategy, get the corresponding multiples; and then get the final upsampled point cloud ; Select Earth Mover's Distance (EMD) distance for constraint during training: in, Indicates the EMD distance between two point clouds. It is a regular constraint term, a constant that changes with the number of training times to ensure the stability of the generated point cloud. express and with The sum of the distances of change, is the generated upsampled point cloud, is the reference point cloud of the target, Q1 represents the point cloud that needs to be moved, and Q2 represents the distribution of the desired point cloud, that is, the target point cloud. represents the i-th point in Q1, Represents the function that maps from Q1 to Q2.

7. The arbitrary scale point cloud upsampling system based on key semantic correction according to claim 6 is characterized in that: In step S1, the intermediate point interpolation method is used to obtain an interpolated point cloud of a preset scale. The specific process is as follows: First, generate interpolation points. For each sparse point , first find the point Neighbors , then use As the newly generated points; secondly, use the farthest point sampling method to remove overlapping generated points and control the number of points to times, , The required upsampling multiple is used to obtain the final coarse upsampling interpolation point cloud. ; Among them, the upsampled interpolated point cloud The ratio is greater than the upsampling ratio of; Represents a point cloud.

8. The arbitrary scale point cloud upsampling system based on key semantic correction according to claim 7 is characterized in that: Step S2 performs feature extraction on the interpolated point cloud. The specific process is as follows: First, perform feature encoding of the point cloud to increase the feature dimension of the point cloud. For any interpolated point , using trigonometric function encoding, the specific implementation is: Among them, PosEmb() means to put the point Encode and get the point The encoded features, here means converting the point position from three dimensions to C dimensions for position encoding; () means put together; Indicates that the three coordinates along each point are encoded separately, is the feature channel size of each point after encoding; The specific trigonometric function details are as follows: Among them, each position index C / 6], Control amplitude and wavelength separately; Then, based on the encoded features, local features are aggregated hierarchically; then feature extension is performed, first finding Neighbor features , for any central point Features , perform the following processing along each feature dimension: in, Indicates that each center point extends the features according to each neighboring point; Indicates the center point The number of k nearest neighbors, in this case k is equal to ; After feature extension, for each feature Perform relative position encoding and assign weights; first standardize the point coordinates of each point's neighbors: The x-position direction is normalized as shown above, where Indicates a very small amount; represents the mean x position of the neighboring points, represents the standard deviation, Indicates the x-axis point after this normalization process; The same operation is performed on other position directions as on the x position direction; Then the characteristics of the k nearest neighbors change with the weight, expressed as: in, Represents the center point after the above processing position standardization The position of the jth neighbor point after transformation; Represents each center point The local features between the neighboring points are aggregated with the features between the neighboring points; the encoding method adopts the trigonometric function encoding method mentioned above to encode, and the weights are assigned to the neighboring points of each center point. Indicates the multiplication operation of each element; After the features of each local area are extended, the features are aggregated to obtain the local feature information represented by each center point. : in Respectively represent the aggregation of neighbor feature information from the perspective of maximum pooling and average pooling; Finally, the first stage will get the response characteristics , where the characteristics are , represents the features of the first stage, where N represents the number of points in the sparse point cloud; Through four stages of different extraction, each stage focuses on different feature spaces and local information representation, and the local information of the upsampled point cloud can be obtained. , where the local features of the point cloud are .

9. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a program for a method for upsampling a point cloud at any scale based on key semantic correction. When the program for upsampling a point cloud at any scale based on key semantic correction is executed by a processor, the steps of a method for upsampling a point cloud at any scale based on key semantic correction as described in any one of claims 1 to 5 are implemented.

Images