Decompression point cloud quality enhancement method based on attention fusion multi-scale features

Abstract

The invention discloses a decompression point cloud quality enhancement method based on attention fusion multi-scale features. The method comprises the following steps: generating a reference true value of original point cloud data by using the original point cloud data and compressed point cloud data processed by a compression algorithm; according to the reference true value, segmenting the original point cloud and the compressed point cloud into a plurality of blocks through a block generation method to obtain a compressed point cloud block and a reference true value block; constructing a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block; generating a compressed point cloud block after position optimization according to the point cloud quality enhancement model; and carrying out block fusion processing on each compressed point cloud block after position optimization to obtain target point cloud data after position optimization. The method can improve the precision and accuracy of point cloud quality enhancement, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features. Background Art

[0002] Point cloud, as the main representation of three-dimensional scenes, is regarded as one of the most critical contents in the next generation of immersive media. Point cloud is a collection of sparsely and irregularly distributed points in three-dimensional space. Each valid point in the point cloud has its geometric coordinates, such as (X, Y, Z) in the Cartesian coordinate system, and related attribute information, such as RGB color, positive and negative, and reflectivity. Due to its flexibility and powerful representation capabilities, point clouds are increasingly used in immersive communications, robotics, geographic information systems, and autonomous driving.

[0003] Point clouds can be divided into point cloud objects and point cloud scenes according to the content they represent. Point cloud scenes are usually acquired dynamically through 3D scanning or LiDAR, covering the entire environment or scene. Point cloud objects represent specific objects or targets, and are further divided into static and dynamic. Static point clouds describe the geometry and attribute information of a single object, while dynamic point clouds, in addition to describing the geometry and attribute information of the object, also include its changes over time. This invention mainly focuses on static point cloud objects.

[0004] The data size of a point cloud may contain up to tens of millions of points, which is not conducive to storage and transportation. For this reason, the Moving Picture Experts Group (MPEG) of the International Organization for Standardization has been conducting in-depth research and promoting potential technologies for efficient point cloud compression since 2017, and finally formed two specifications, namely Geometry-Based Point Cloud Compression (G-PCC) and Video-Based Point Cloud Compression (V-PCC).

[0005] When bandwidth is limited, the compression bit rate must be reduced by increasing the quantization step size, which may cause geometric artifacts and attribute artifacts in the reconstructed point cloud. The compressed point cloud will have local blur or missing and unsmooth edges. These compression-induced artifacts will seriously affect the application of point clouds. Therefore, researchers have gradually developed algorithms designed to remove point cloud artifacts and enhance their quality.

[0006] Current research on point cloud compression artifacts mainly focuses on removing attribute artifacts, that is, improving the representation of color and texture in point clouds. However, geometric artifacts, that is, distortions or errors in geometric shapes that appear in point clouds, will also have a negative impact on the overall quality of point clouds. Since geometric information is closely related to attribute information, geometric artifacts will directly lead to errors in attribute information. Therefore, simply removing attribute artifacts is not enough to significantly improve the reconstruction quality of point clouds, and the removal of geometric artifacts is also crucial. In addition, in the research on point clouds, there is a common problem of insufficient local contextual information, which cannot effectively capture the correlation between global information and local information in point cloud data, limiting the precision and accuracy of point cloud quality enhancement. Summary of the invention

[0007] The main purpose of the embodiment of the present invention is to propose a decompressed point cloud quality enhancement method based on attention fusion multi-scale features, which can improve the precision and accuracy of point cloud quality enhancement.

[0008] To achieve the above object, an embodiment of the present invention proposes a method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features, comprising the following steps:

[0009] Generating a reference truth value of the original point cloud data using the original point cloud data and the compressed point cloud data processed by the compression algorithm;

[0010] According to the reference truth value, the original point cloud and the compressed point cloud are divided into a plurality of blocks by a block generation method to obtain compressed point cloud blocks and reference truth value blocks;

[0011] Constructing a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block;

[0012] Generate a compressed point cloud block with optimized position according to the point cloud quality enhancement model;

[0013] The compressed point cloud blocks after optimization in each position are subjected to block fusion processing to obtain the target point cloud data after optimization in position.

[0014] In some embodiments, the method of dividing the original point cloud and the compressed point cloud into a plurality of blocks by a block generation method according to the reference truth value to obtain compressed point cloud blocks and reference truth value blocks includes the following steps:

[0015] According to the reference true value, the original point cloud data is divided into a plurality of cube blocks of equal size according to the spatial position, and the points in each cube block are grouped as one block to obtain a reference true value block;

[0016] The compressed point cloud data is divided into a plurality of cube blocks of equal size according to spatial positions, and the points in each cube block are grouped as one block to obtain a compressed point cloud block.

[0017] In some embodiments, constructing a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block comprises the following steps:

[0018] In the encoder part, the input data generated by the multi-scale input module is received and multi-scale features are extracted; wherein the first layer of the encoder is a 5×5×5 discrete convolution layer; the expression of this process is: E0=CONV(MFIM(I)), wherein MFIM() represents a multi-scale feature extraction unit, CONV() represents discrete convolution; E0 is the initial feature extracted from the input data;

[0019] Then use a 3×3×3 discrete convolution layer with a step size of 2 for downsampling. During the downsampling process, the feature dimension doubles, while the number of points decreases. The expression of this process is: E n =D_CONV(E n-1 ), where D_CONV() represents the sparse convolution operation on the output of the previous layer with a step size of 2; E n is the intermediate output of each downsampling layer;

[0020] In the decoder part, a 3×3×3 transposed sparse convolution is used with a step size of 2 for upsampling. The expression of this process is: n =Tr_CONV(D n-1 ), where Tr_CONV() represents the transposed sparse convolution operation of the previous layer output, and D n is the intermediate output of each up-sampled layer;

[0021] In the last layer of the decoder, 1×1×1 sparse convolution is used to refine the features, generate a final output with the same shape as the input, and combine the information of the encoder path and the decoder path to obtain context information;

[0022] A cross-scale feature fusion module is used to fuse attribute features of all scales.

[0023] In some embodiments, the method of fusing attribute features of all scales using a cross-scale feature fusion module comprises the following steps:

[0024] In the pre-fusion transformation stage: the weight of the geometric distance is used to determine the attribute characteristics of the upsampled points to preserve the spatial consistency between the upsampled and downsampled point clouds;

[0025] In the cross-scale attention feature learning stage: the cross-scale attention mechanism is used to effectively fuse the detail information in the original resolution point cloud and the context information in the twice-downsampled point cloud;

[0026] In the feature fusion stage: enhanced features are obtained through cross-scale attention feature learning, and these features are stacked. The stacked features are smoothed using a multi-layer perceptron to obtain fused multi-scale features.

[0027] In some embodiments, the method of using the weight of the geometric distance to determine the attribute characteristics of the up-sampled point to preserve the spatial consistency between the up-sampled and down-sampled point clouds includes the following steps:

[0028] According to the number of points in the downsampled point cloud and the number of points in the upsampled point cloud, the attribute characteristics of each upsampled point are calculated. The calculation expression is: Among them, ψ j (x i ) is a weight function based on geometric distance, indicating that point x i and its jth neighbor x j The inverse square of the distance between bi Represents the i th The attribute features of the upsampled points; K represents the attribute features of each upsampled point in the low-resolution point cloud n d The number of adjacent points in a point; n d Represents the number of points in the downsampled point cloud;

[0029] The cross-scale attention mechanism is used to effectively fuse the detail information in the original resolution point cloud and the context information in the twice downsampled point cloud, including the following steps:

[0030] Perform element-wise multiplication of the point cloud downsampled once and the point cloud downsampled twice to calculate the semantic attention mask;

[0031] Perform element-by-element addition of the downsampled point cloud and the original resolution point cloud to calculate the detail attention mask;

[0032] Applying the semantic attention mask to the twice downsampled point cloud and applying the detail attention mask to the original resolution point cloud to obtain enhanced features;

[0033] The enhanced features are stacked to aggregate information of different scales, and the stacked features are smoothed using a multi-layer perceptron to obtain fused multi-scale features.

[0034] In some embodiments, the method further comprises a step of training a point cloud quality enhancement model, the step comprising the following steps:

[0035] The mean square error is used as the loss function of MPNet, and the model is trained using the point cloud data of the training set;

[0036] The expression of the loss function is:

[0037] Among them, l i represents the loss of the component at the i-th position in the block; C ij represents the geometric position in the ground truth; C i ' j Represents the geometric location after quality enhancement in the reconstructed patch.

[0038] In some embodiments, the method further includes: after completing the training of the point cloud quality enhancement model, using the test set data to perform an effect test on the model, specifically including:

[0039] The peak signal-to-noise ratio is calculated by the mean square error and is used to measure the geometric fidelity.

[0040] Another aspect of the embodiment of the present invention further provides a decompressed point cloud quality enhancement system based on attention fusion multi-scale features, including:

[0041] The first module is used to generate a reference truth value of the original point cloud data by using the original point cloud data and the compressed point cloud data processed by the compression algorithm;

[0042] The second module is used to divide the original point cloud and the compressed point cloud into multiple blocks according to the reference true value by a block generation method to obtain compressed point cloud blocks and reference true value blocks;

[0043] A third module is used to construct a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block;

[0044] A fourth module is used to generate a compressed point cloud block with optimized position according to the point cloud quality enhancement model;

[0045] The fifth module is used to perform block fusion processing on the compressed point cloud blocks after optimization of each position to obtain the target point cloud data after optimization of the position.

[0046] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0047] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0048] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0049] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features, which generates a reference truth value of the original point cloud data using original point cloud data and compressed point cloud data processed by a compression algorithm; according to the reference truth value, the original point cloud and the compressed point cloud are divided into multiple blocks by a block generation method to obtain compressed point cloud blocks and reference truth value blocks; according to the compressed point cloud blocks and the reference truth value blocks, a point cloud quality enhancement model is constructed; according to the point cloud quality enhancement model, a compressed point cloud block with optimized position is generated; and each compressed point cloud block with optimized position is subjected to block fusion processing to obtain the target point cloud data with optimized position. The present invention can improve the precision and accuracy of point cloud quality enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0051] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0052] Figure 3 It is an implementation process diagram provided by an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of the structure of a point cloud quality enhancement network provided by an embodiment of the present invention;

[0054] Figure 5 is a structural diagram of a multi-scale feature fusion block based on cross-scale attention provided by an embodiment of the present invention;

[0055] Figure 6 is a schematic diagram of a point-to-point distortion metric D1 provided in an embodiment of the present invention;

[0056] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0058] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0061] The method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features provided in an embodiment of the present invention relates to the field of computer technology. The method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features provided in an embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a vehicle-mounted terminal, etc., but is not limited thereto; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features, etc., but is not limited to the above forms.

[0062] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0064] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0065] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0066] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0067] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features. The following is an example of applying the method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features to the server 101. It can be understood that the method can also be applied to the terminal 102.

[0068] Reference Figure 2 , Figure 2 The flowchart of the method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features applied to the server provided in the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0069] Generating a reference truth value of the original point cloud data using the original point cloud data and the compressed point cloud data processed by the compression algorithm;

[0070] According to the reference truth value, the original point cloud and the compressed point cloud are divided into a plurality of blocks by a block generation method to obtain compressed point cloud blocks and reference truth value blocks;

[0071] Constructing a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block;

[0072] Generate a compressed point cloud block with optimized position according to the point cloud quality enhancement model;

[0073] The compressed point cloud blocks after optimization in each position are subjected to block fusion processing to obtain the target point cloud data after optimization in position.

[0074] In some embodiments, the method of dividing the original point cloud and the compressed point cloud into a plurality of blocks by a block generation method according to the reference truth value to obtain compressed point cloud blocks and reference truth value blocks includes the following steps:

[0075] According to the reference true value, the original point cloud data is divided into a plurality of cube blocks of equal size according to the spatial position, and the points in each cube block are grouped as one block to obtain a reference true value block;

[0076] The compressed point cloud data is divided into a plurality of cube blocks of equal size according to spatial positions, and the points in each cube block are grouped as one block to obtain a compressed point cloud block.

[0077] In some embodiments, constructing a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block comprises the following steps:

[0078] In the encoder part, the input data generated by the multi-scale input module is received and multi-scale features are extracted; wherein the first layer of the encoder is a 5×5×5 discrete convolution layer; the expression of this process is: E0=CONV(MFIM(I)), wherein MFIM() represents a multi-scale feature extraction unit, CONV() represents discrete convolution; E0 is the initial feature extracted from the input data;

[0079] Then use a 3×3×3 discrete convolution layer with a step size of 2 for downsampling. During the downsampling process, the feature dimension doubles, while the number of points decreases. The expression of this process is: E n =D_CONV(E n-1 ), where D_CONV() represents the sparse convolution operation on the output of the previous layer with a step size of 2; E n is the intermediate output of each downsampling layer;

[0080] In the decoder part, a 3×3×3 transposed sparse convolution is used with a step size of 2 for upsampling. The expression of this process is: n =Tr_CONV(D n-1 ), where Tr_CONV() represents the transposed sparse convolution operation of the previous layer output, and D n is the intermediate output of each up-sampled layer;

[0081] In the last layer of the decoder, 1×1×1 sparse convolution is used to refine the features, generate a final output with the same shape as the input, and combine the information of the encoder path and the decoder path to obtain context information;

[0082] A cross-scale feature fusion module is used to fuse attribute features of all scales.

[0083] In some embodiments, the method of fusing attribute features of all scales using a cross-scale feature fusion module comprises the following steps:

[0084] In the pre-fusion transformation stage: the weight of the geometric distance is used to determine the attribute characteristics of the upsampled points to preserve the spatial consistency between the upsampled and downsampled point clouds;

[0085] In the cross-scale attention feature learning stage: the cross-scale attention mechanism is used to effectively fuse the detail information in the original resolution point cloud and the context information in the twice-downsampled point cloud;

[0086] In the feature fusion stage: enhanced features are obtained through cross-scale attention feature learning, and these features are stacked. The stacked features are smoothed using a multi-layer perceptron to obtain fused multi-scale features.

[0087] In some embodiments, the method of using the weight of the geometric distance to determine the attribute characteristics of the up-sampled point to preserve the spatial consistency between the up-sampled and down-sampled point clouds includes the following steps:

[0088] According to the number of points in the downsampled point cloud and the number of points in the upsampled point cloud, the attribute characteristics of each upsampled point are calculated. The calculation expression is: Among them, ψ j (x i ) is a weight function based on geometric distance, indicating that point x i and its jth neighbor x j The inverse square of the distance between bi Represents the i th The attribute features of the upsampled points; K represents the attribute features of each upsampled point in the low-resolution point cloud n d The number of adjacent points in a point; n d Represents the number of points in the downsampled point cloud;

[0089] The cross-scale attention mechanism is used to effectively fuse the detail information in the original resolution point cloud and the context information in the twice downsampled point cloud, including the following steps:

[0090] Perform element-wise multiplication of the point cloud downsampled once and the point cloud downsampled twice to calculate the semantic attention mask;

[0091] Perform element-by-element addition of the downsampled point cloud and the original resolution point cloud to calculate the detail attention mask;

[0092] Applying the semantic attention mask to the twice downsampled point cloud and applying the detail attention mask to the original resolution point cloud to obtain enhanced features;

[0093] The enhanced features are stacked to aggregate information of different scales, and the stacked features are smoothed using a multi-layer perceptron to obtain fused multi-scale features.

[0094] In some embodiments, the method further comprises a step of training a point cloud quality enhancement model, the step comprising the following steps:

[0095] The mean square error is used as the loss function of MPNet, and the model is trained using the point cloud data of the training set;

[0096] The expression of the loss function is:

[0097] Among them, l i represents the loss of the component at the i-th position in the block; C ij represents the geometric position in the ground truth; C i ' j Represents the geometric location after quality enhancement in the reconstructed patch.

[0098] In some embodiments, the method further includes: after completing the training of the point cloud quality enhancement model, using the test set data to perform an effect test on the model, specifically including:

[0099] The peak signal-to-noise ratio is calculated by the mean square error and is used to measure the geometric fidelity.

[0100] The specific implementation process of the present invention is described in detail below in conjunction with the accompanying drawings:

[0101] like Figure 3 As shown, the present invention provides a method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features, comprising the following steps:

[0102] Step 1: Generate benchmark truth values ​​using the original point cloud data and the compressed point cloud data processed by the compression algorithm. The purpose of this step is to generate a set of aligned benchmark truth values ​​based on the original point cloud to provide a reference for the subsequent calculation of the error between the compressed point cloud and the original point cloud.

[0103] Step 2: Divide the point cloud into smaller blocks by block generation method to reduce the amount of calculation and adapt to computing resources. The fast division method adopted by the present invention is a block generation method based on the physical position of the point cloud. The specific operation is to divide the point cloud into multiple equal-sized cubic blocks according to the spatial position, and group the points in each cubic block into one block. This not only avoids the sampling problem of duplicate and missed points, but also reduces the computational complexity.

[0104] This method is used to divide the original point cloud and the compressed point cloud into blocks, so that the blocks in the physical position of the two are the same, that is, the points in each block have similar geometric features. This consistency helps the model learn the error between the compressed point cloud and the original point cloud more effectively, thereby improving the reconstruction quality.

[0105] Step 3: Use the Pytorch deep learning framework to build a point cloud quality enhancement network (MPNet) based on the Minkowski Engine. The structure of the network model is as follows: Figure 4 It mainly includes a multi-scale input module, a discrete convolution module and a cross-scale feature fusion module based on point attention.

[0106] Specifically, the proposed network structure is based on the U-Net architecture and implements discrete convolution through the Minkowski Engine library. The network consists of two parts: an encoder network and a decoder network. The encoder gradually extracts the features of the input data through a series of downsampling convolutional layers and normalization layers, while the decoder gradually restores the spatial resolution through deconvolutional layers and normalization layers, and reconstructs the output by combining the features extracted by the encoder.

[0107] In the encoder part, the data generated by the multi-scale input module is first received and multi-scale features are extracted. The first layer of the encoder is a 5×5×5 discrete convolution layer:

[0108] E0=CONV(MFIM(I)) (1)

[0109] Here, MFIM() represents a multi-scale feature extraction unit and CONV() represents a discrete convolution. E0 is the initial feature extracted from the input data. Next, a 3×3×3 discrete convolution layer is used with a stride of 2 for downsampling. During downsampling, the feature dimension doubles while the number of points decreases:

[0110] E n =D_CONV(E n-1 ) (2)

[0111] Where D_CONV() represents the sparse convolution operation on the output of the previous layer, with a step size of 2. nis the intermediate output of each downsampling layer. In the decoder part, a 3×3×3(3 3 ) Transposed sparse convolution, upsampling with a stride of 2:

[0112] D n =Tr_CONV(D n-1 ) (3)

[0113] Among them, Tr_CONV() represents the transposed sparse convolution operation of the previous layer output, D n is the intermediate output of each up-sampled layer. In the last layer of the decoder, 1×1×1(1 3 ) sparse convolution to refine the features and generate a final output with the same shape as the input. The intermediate outputs of the intermediate encoder (E1, E2, …, E5) and decoder (D1, D2, …, D5) are stored during the network forwarding process. In order to obtain contextual information, the information of the encoder path and the decoder path is combined.

[0114] Finally, the cross-scale feature fusion module is used to fuse the attribute features of all scales:

[0115] F n =CAFFM(D n ) (4)

[0116] Step 4: The present invention designs a cross-scale feature fusion module (CAFFM) based on point attention to extract point cloud features. The structure of the CAFFM module is as follows: Figure 5 As shown in Figure 1. The multi-scale input module downsamples the input point cloud to m different scales and extracts the geometric features of each scale in the discrete convolution module. These geometric features of different scales contain information at different levels, such as local details and global context. Therefore, the present invention designs a multi-scale feature fusion module based on point attention to integrate the geometric features of each scale. The module structure is shown in Figure 1. Figure 5 As shown in the figure, it mainly includes the following steps: pre-fusion transformation, cross-scale attention feature learning, and feature fusion.

[0117] In the pre-fusion transformation stage, in order to effectively upsample the low-resolution point cloud to the original resolution, the present invention adopts a method based on geometric distance weighted interpolation. This method uses the weight of the geometric distance to determine the attribute characteristics of the upsampled points, so that the spatial consistency between the upsampled and downsampled point clouds can be retained. Specifically, geometric distance weighted interpolation is a method of attribute interpolation of points commonly used in point cloud processing. Its basic principle is to calculate the attribute value of the unknown point based on the geometric distance between the known points in the point cloud. This method can make full use of the geometric relationship between points and maintain the consistency of the geometric features of the point cloud during the upsampling process.

[0118] Assume that the number of points in the downsampled point cloud is n d points, and the upsampled point cloud contains n points n(n d < n). The attribute feature h′ of the i th -th upsampled point is calculated as follows: bi

[0119]

[0120] where i ∈ {1, …, n} is a weight function based on geometric distance, representing the reciprocal square of the distance between point x i and its j-th neighboring point x j . d(x i , x j ) is the Euclidean distance between two points. K represents the number of adjacent points of each upsampled point in the low-resolution point cloud n d points.

[0121] The purpose of using this method for upsampling is to ensure that during the upsampling process, the new point cloud can maintain the same geometric features as the original point cloud. This weighted interpolation method can effectively smooth the transition and capture more accurate geometric information by weighted averaging the features of neighboring points, thus providing better input for the subsequent feature fusion module.

[0122] In the cross-scale attention feature learning stage, the present invention defines the point cloud at the original resolution as A, the point cloud downsampled once as B, and the point cloud downsampled twice as C. Among them, A comes from a lower level (i.e., high resolution, representing details), and C comes from a higher level (i.e., low resolution, representing context). Compared with B, A has more detailed information, while C contains richer context information. Therefore, the present invention designs B as the basic scale, and on this basis, uses the cross-scale attention mechanism to effectively fuse the detailed information in A and the context information in C using the cross-scale attention mechanism.

[0123] Specifically, first, perform an element-wise multiplication operation (“MUL”) on B and C to calculate the semantic attention mask M sem , and similarly, perform an element-wise addition operation (“ADD”) on B and A to calculate the detail attention mask M res . This operation can avoid the problem of boundary blurring that may occur during simple element-wise addition and the problem of suppressing important signals that may occur during element-wise multiplication.

[0124] M res = σ(z(A)+z(B)) (6)

[0125] M sem = σ(z(C)·z(B)) (7) ​

[0126] Among them, σ represents the sigmoid activation function, and z represents the feature transformation operation before fusion. According to the above steps, the corresponding mask M is applied to A and C respectively. res and M sem :

[0127] A′=A⊙M res (8)

[0128] C′=C⊙M sem (9)

[0129] This feature fusion strategy not only improves the network's ability to comprehensively utilize features of different scales, but also enhances the accuracy of point cloud feature representation.

[0130] In the feature fusion stage, the present invention obtains enhanced features through cross-scale attention feature learning, and stacks these features A′, B and C′ (corresponding to Figure 5 F in , in order to aggregate information from different scales. Then, a multi-layer perceptron (MLP, see Figure 5 The T in (T) is used to smooth the stacked features to obtain the fused multi-scale features.

[0131] Step 5: Use the point cloud data of the training set to train the model. In order to measure the difference between the quality enhancement block of the geometric component and the benchmark true value, the present invention uses the mean square error (MSE) as the loss function of MPNet:

[0132]

[0133] Among them, l i represents the loss of the i-th position component in the block, C and C' represent the geometric position in the ground truth and the geometric position after quality enhancement in the reconstructed patch, respectively. Since the model aims to minimize the difference between the noise input and the ground truth, MSE as a loss function can effectively measure this difference and intuitively reflect the performance of the model.

[0134] Step 6: After completing the training of the point cloud quality enhancement model, the test set data is used to perform the final effect test on the model. The present invention uses the peak signal-to-noise ratio (PSNR) to measure the geometric fidelity. The higher the PSNR, the better the reconstruction quality under the same number of coding bits. Specifically, the PSNR is calculated by the mean square error (MSE), where the MSE is the square average of the symmetrical point-to-point distances between the reference point cloud A and its reconstructed point cloud B. The definitions of D1 and D1-PSNR are as follows:

[0135]

[0136]

[0137] Where D1 represents the point-to-point distortion; D1-PSNR represents the peak signal-to-noise ratio calculated based on the point-to-point distortion; |A| and |B| represent the number of points in point clouds A and B, respectively. a Represents the normal vector between ab and point b The angle between b Represents the normal vector between ba and point a The angle between. Figure 6 Point-to-point error and point-to-plane error are shown.

[0138] In summary, current research on point cloud compression artifacts mainly focuses on removing attribute artifacts, that is, improving the representation of color and texture in point clouds. However, geometric artifacts, that is, distortions or errors in geometric shapes that appear in point clouds, will also have a negative impact on the overall quality of point clouds. Since geometric information is closely related to attribute information, geometric artifacts will directly lead to errors in attribute information. Therefore, simply removing attribute artifacts is not enough to significantly improve the reconstruction quality of point clouds, and the removal of geometric artifacts is also crucial.

[0139] In addition, in the research on point clouds, there is a common problem of insufficient local context information, which cannot effectively capture the correlation between global information and local information in point cloud data, limiting the precision and accuracy of point cloud quality enhancement. Point cloud features of different scales can often provide complementary information. Focusing on high-scale detail information and low-scale context information can more comprehensively describe point cloud data, thereby improving the precision and accuracy of reconstructed point clouds. Therefore, in response to the above problems, the present invention designs a point cloud quality enhancement method based on attention fusion of multi-scale features, with the following main advantages:

[0140] 1) The present invention designs a network for enhancing the geometric information quality of compressed point clouds based on multi-scale. The network adopts a multi-branch feature extraction structure combined with farthest point sampling to extract geometric artifact features in parallel and obtain a reconstructed point cloud with enhanced geometric quality.

[0141] 2) The present invention designs a multi-scale feature fusion module based on point attention, which effectively fuses the point cloud detail information and context information in the multi-scale features to improve the point cloud quality.

[0142] 3) Experimental results show that the average peak signal-to-noise ratio of the method designed by the present invention on the public data set is improved compared with the standard compression algorithm.

[0143] Another aspect of the embodiment of the present invention further provides a decompressed point cloud quality enhancement system based on attention fusion multi-scale features, including:

[0144] The first module is used to generate a reference truth value of the original point cloud data by using the original point cloud data and the compressed point cloud data processed by the compression algorithm;

[0145] The second module is used to divide the original point cloud and the compressed point cloud into multiple blocks according to the reference true value by a block generation method to obtain compressed point cloud blocks and reference true value blocks;

[0146] A third module is used to construct a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block;

[0147] A fourth module is used to generate a compressed point cloud block with optimized position according to the point cloud quality enhancement model;

[0148] The fifth module is used to perform block fusion processing on the compressed point cloud blocks after optimization of each position to obtain the target point cloud data after optimization of the position.

[0149] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0150] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned decompressed point cloud quality enhancement method based on attention fusion multi-scale features when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0151] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0153] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0154] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the decompressed point cloud quality enhancement method based on attention fusion multi-scale features of the embodiment of the present invention;

[0155] Input / output interface 703, used to implement information input and output;

[0156] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0157] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0158] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0159] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned decompressed point cloud quality enhancement method based on attention fusion multi-scale features.

[0160] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0163] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0164] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0165] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0167] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0168] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0169] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0173] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features, characterized in that: The following steps are involved: Generating a reference truth value of the original point cloud data using the original point cloud data and the compressed point cloud data processed by the compression algorithm; According to the reference truth value, the original point cloud and the compressed point cloud are divided into a plurality of blocks by a block generation method to obtain compressed point cloud blocks and reference truth value blocks; Constructing a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block; Generate a compressed point cloud block with optimized position according to the point cloud quality enhancement model; The compressed point cloud blocks after optimization in each position are subjected to block fusion processing to obtain the target point cloud data after optimization in position.

2. According to claim 1, a method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features is characterized in that: The method of dividing the original point cloud and the compressed point cloud into a plurality of blocks according to the reference true value by a block generation method to obtain compressed point cloud blocks and reference true value blocks includes the following steps: According to the reference true value, the original point cloud data is divided into a plurality of cube blocks of equal size according to the spatial position, and the points in each cube block are grouped as one block to obtain a reference true value block; The compressed point cloud data is divided into a plurality of cube blocks of equal size according to spatial positions, and the points in each cube block are grouped as one block to obtain a compressed point cloud block.

3. According to claim 1, a method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features is characterized in that: The step of constructing a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block comprises the following steps: In the encoder part, the input data generated by the multi-scale input module is received and multi-scale features are extracted; wherein the first layer of the encoder is a 5×5×5 discrete convolution layer; the expression of this process is: E0=CONV(FMIM(I)), wherein FMIM() represents a multi-scale feature extraction unit, CONV() represents discrete convolution; E0 is the initial feature extracted from the input data; Then use a 3×3×3 discrete convolution layer with a step size of 2 for downsampling. During the downsampling process, the feature dimension doubles, while the number of points decreases. The expression of this process is: E n =D_CONV(E n-1 ), where D_CONV() represents the sparse convolution operation on the output of the previous layer with a step size of 2; E n is the intermediate output of each downsampling layer; In the decoder part, a 3×3×3 transposed sparse convolution is used with a step size of 2 for upsampling. The expression of this process is: n =Tr_CONV(D n-1 ), where Tr_CONV() represents the transposed sparse convolution operation of the previous layer output, and D n is the intermediate output of each up-sampled layer; In the last layer of the decoder, 1×1×1 sparse convolution is used to refine the features, generate a final output with the same shape as the input, and combine the information of the encoder path and the decoder path to obtain context information; A cross-scale feature fusion module is used to fuse attribute features of all scales.

4. According to claim 3, a method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features is characterized in that: The cross-scale feature fusion module is used to fuse attribute features of all scales, including the following steps: In the pre-fusion transformation stage: the weight of the geometric distance is used to determine the attribute characteristics of the upsampled points to preserve the spatial consistency between the upsampled and downsampled point clouds; In the cross-scale attention feature learning stage: the cross-scale attention mechanism is used to effectively fuse the detail information in the original resolution point cloud and the context information in the twice-downsampled point cloud; In the feature fusion stage: enhanced features are obtained through cross-scale attention feature learning, and these features are stacked. The stacked features are smoothed using a multi-layer perceptron to obtain fused multi-scale features.

5. The method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features according to claim 4, characterized in that: The method of using the weight of the geometric distance to determine the attribute characteristics of the up-sampled point to retain the spatial consistency between the up-sampled and down-sampled point clouds includes the following steps: According to the number of points in the downsampled point cloud and the number of points in the upsampled point cloud, the attribute characteristics of each upsampled point are calculated. The calculation expression is: Among them, ψ j (x i ) is a weight function based on geometric distance, indicating that point x i and its jth neighbor x j The inverse square of the distance between bi Represents the i th The attribute features of the upsampled points; K represents the attribute features of each upsampled point in the low-resolution point cloud n d The number of adjacent points in a point; n d Represents the number of points in the downsampled point cloud; The cross-scale attention mechanism is used to effectively fuse the detail information in the original resolution point cloud and the context information in the twice downsampled point cloud, including the following steps: Perform element-wise multiplication of the point cloud downsampled once and the point cloud downsampled twice to calculate the semantic attention mask; Perform element-by-element addition of the downsampled point cloud and the original resolution point cloud to calculate the detail attention mask; Applying the semantic attention mask to the twice downsampled point cloud and applying the detail attention mask to the original resolution point cloud to obtain enhanced features; The enhanced features are stacked to aggregate information of different scales, and the stacked features are smoothed using a multi-layer perceptron to obtain fused multi-scale features.

6. The method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features according to claim 1, characterized in that: The method also includes a step of training a point cloud quality enhancement model, which includes the following steps: The mean square error is used as the loss function of MPNet, and the model is trained using the point cloud data of the training set; The expression of the loss function is: Among them, l i represents the loss of the i-th position component in the block; C ij represents the geometric position in the ground truth; C i ′ j Represents the geometric location after quality enhancement in the reconstructed patch.

7. The method for enhancing the quality of decompressed point cloud based on attention fusion multi-scale features according to claim 6, characterized in that: The method further includes: after completing the training of the point cloud quality enhancement model, using the test set data to test the model effect, specifically including: The peak signal-to-noise ratio is calculated by the mean square error and is used to measure the geometric fidelity.

8. A decompressed point cloud quality enhancement system based on attention fusion multi-scale features, characterized in that: include: The first module is used to generate a reference truth value of the original point cloud data by using the original point cloud data and the compressed point cloud data processed by the compression algorithm; The second module is used to divide the original point cloud and the compressed point cloud into multiple blocks according to the reference true value by a block generation method to obtain compressed point cloud blocks and reference true value blocks; A third module is used to construct a point cloud quality enhancement model according to the compressed point cloud block and the reference true value block; A fourth module is used to generate a compressed point cloud block with optimized position according to the point cloud quality enhancement model; The fifth module is used to perform block fusion processing on the compressed point cloud blocks after optimization of each position to obtain the target point cloud data after optimization of the position.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images