A method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features

Abstract

This invention discloses a method for enhancing the quality of decompressed point clouds based on attention-based multi-scale feature fusion. The method generates a baseline ground truth value for the original point cloud data using original point cloud data and compressed point cloud data processed by a compression algorithm. Based on the baseline ground truth value, the original and compressed point clouds are divided into multiple blocks using a block generation method, resulting in compressed point cloud blocks and baseline ground truth value blocks. A point cloud quality enhancement model is constructed based on the compressed point cloud blocks and the baseline ground truth value blocks. Position-optimized compressed point cloud blocks are generated based on the point cloud quality enhancement model. Block fusion processing is performed on each position-optimized compressed point cloud block to obtain position-optimized target point cloud data. This invention can improve the accuracy and precision of point cloud quality enhancement and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features. Background Art

[0002] Point clouds, as a primary representation of 3D scenes, are considered one of the most crucial elements in next-generation immersive media. A point cloud is a collection of sparsely and irregularly distributed points in 3D space. Each valid point in a point cloud has its geometric coordinates, such as (X, Y, Z) in a Cartesian coordinate system, and related attribute information, such as RGB color, positive / negative sign, and reflectivity. Due to its flexibility and powerful representational capabilities, point clouds are increasingly being used in fields such as immersive communication, robotics, geographic information systems, and autonomous driving.

[0003] Point clouds can be categorized into point cloud objects and point cloud scenes based on their representation. Point cloud scenes are typically acquired dynamically through 3D scanning or LiDAR, encompassing the entire environment or scene. Point cloud objects, on the other hand, represent specific objects or targets and are further divided into static and dynamic types. Static point clouds describe the geometry and attribute information of a single object, while dynamic point clouds, in addition to describing the object's geometry and attribute information, also include its temporal changes. This invention primarily focuses on static point cloud objects.

[0004] Point cloud data can contain tens of millions of points, which is not conducive to storage and transportation. To address this, the Moving Picture Experts Group (MPEG) of the international standardization organization has been conducting in-depth research and promoting potential technologies for efficient point cloud compression since 2017, and has finally formed two specifications: geometry-based point cloud compression (G-PCC) and video-based point cloud compression (V-PCC).

[0005] When bandwidth is limited, the compression bit rate must be reduced by increasing the quantization step size, which may lead to geometric and attribute artifacts in the reconstructed point cloud. The compressed point cloud may exhibit local blurring or missing parts and unsmooth edges. These compression-induced artifacts can severely affect the application of point clouds. Therefore, researchers have gradually developed algorithms aimed at removing point cloud artifacts and enhancing their quality.

[0006] Current research on point cloud compression artifacts mainly focuses on removing attribute artifacts, i.e., improving the representation of color and texture in point clouds. However, geometric artifacts, i.e., distortions or errors in the geometric shapes appearing in point clouds, also negatively impact the overall quality of the point cloud. Since geometric information is closely related to attribute information, geometric artifacts directly lead to errors in attribute information. Therefore, simply removing attribute artifacts is insufficient to significantly improve the reconstruction quality of point clouds; the removal of geometric artifacts is equally crucial. Furthermore, research on point clouds generally suffers from insufficient local contextual information, failing to effectively capture the correlation between global and local information in point cloud data, thus limiting the precision and accuracy of point cloud quality enhancement. Summary of the Invention

[0007] The main objective of this invention is to propose a method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features, which can improve the accuracy and precision of point cloud quality enhancement.

[0008] To achieve the above objectives, one aspect of this invention proposes a method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features, comprising the following steps:

[0009] A baseline true value for the original point cloud data is generated using the original point cloud data and the compressed point cloud data processed by the compression algorithm.

[0010] Based on the baseline true value, the original point cloud and the compressed point cloud are divided into multiple blocks by the block generation method to obtain the compressed point cloud block and the baseline true value block.

[0011] Based on the compressed point cloud block and the baseline true value block, a point cloud quality enhancement model is constructed;

[0012] Based on the point cloud quality enhancement model, a position-optimized compressed point cloud block is generated;

[0013] Block fusion processing is performed on the compressed point cloud blocks optimized at each location to obtain the target point cloud data with optimized location.

[0014] In some embodiments, the step of dividing the original point cloud and compressed point cloud into multiple blocks using a block generation method based on the reference ground truth to obtain compressed point cloud blocks and reference ground truth blocks includes the following steps:

[0015] Based on the baseline true value, the original point cloud data is divided into multiple cubes of equal size according to spatial location, and the points in each cube are grouped into one block to obtain the baseline true value block.

[0016] The compressed point cloud data is divided into multiple cubes of equal size according to spatial location, and the points in each cube are grouped into one block to obtain a compressed point cloud block.

[0017] In some embodiments, constructing a point cloud quality enhancement model based on the compressed point cloud block and the baseline ground truth block includes the following steps:

[0018] In the encoder section, the input data generated by the multi-scale input module is received, and multi-scale features are extracted. The first layer of the encoder is a 5×5×5 discrete convolutional layer. The expression for this process is: E0 = CONV(MFIM(I)), where MFIM() represents the multi-scale feature extraction unit, CONV() represents discrete convolution, and E0 is the initial feature extracted from the input data.

[0019] Next, a 3×3×3 discrete convolutional layer is used for downsampling with a stride of 2. During downsampling, the feature dimension doubles while the number of points decreases. The expression for this process is: E n =D_CONV(E n-1 ), where D_CONV() represents a sparse convolution operation on the output of the previous layer with a stride of 2; E n It is the intermediate output of each downsampling layer;

[0020] In the decoder section, a 3×3×3 transposed sparse convolution is used, with upsampling at a stride of 2. The expression for this process is: D n =Tr_CONV(D n-1 ), where Tr_CONV() represents the transpose sparse convolution operation of the previous layer's output, D n It is the intermediate output of each layer's upsampling;

[0021] In the last layer of the decoder, 1×1×1 sparse convolutions are used to refine the features, generate a final output with the same shape as the input, and combine the information of the encoder path and the decoder path to obtain contextual information.

[0022] The cross-scale feature fusion module is used to fuse attribute features from all scales.

[0023] In some embodiments, the method of using a cross-scale feature fusion module to fuse attribute features across all scales includes the following steps:

[0024] In the pre-fusion transformation stage: the weights of geometric distance are used to determine the attribute features of the upsampled points in order to preserve the spatial consistency between the upsampled and downsampled point clouds;

[0025] In the cross-scale attention feature learning stage: the cross-scale attention mechanism is used to effectively fuse the detailed information in the point cloud at the original resolution and the contextual information in the point cloud downsampled twice;

[0026] In the feature fusion stage: enhanced features are obtained through cross-scale attention feature learning, and these features are stacked. The stacked features are then smoothed using a multilayer perceptron to obtain fused multi-scale features.

[0027] In some embodiments, the step of using the weights of geometric distance to determine the attribute features of upsampled points in order to preserve spatial consistency between upsampled and downsampled point clouds includes the following steps:

[0028] Based on the number of points in the downsampled point cloud and the number of points in the upsampled point cloud, the attribute features of each upsampled point are calculated using the following expression: Where, ψ j (x i () is a weighting function based on geometric distance, representing the point x i Its j-th neighboring point x j The reciprocal square of the distance between them; h′ bi Representing the i-th th The attribute features of each upsampled point; K represents the attribute features of each upsampled point in the low-resolution point cloud n. d The number of adjacent points in a given point; n d Represents the number of points in the downsampled point cloud;

[0029] The method of effectively fusing detailed information from the original resolution point cloud and contextual information from the point cloud after two downsampling steps using a cross-scale attention mechanism includes the following steps:

[0030] Perform element-wise multiplication on the point cloud downsampled once and the point cloud downsampled twice to calculate the semantic attention mask;

[0031] The detail attention mask is calculated by performing element-wise addition between the downsampled point cloud and the point cloud at the original resolution.

[0032] The semantic attention mask is applied to the point cloud that has been downsampled twice, and the detail attention mask is applied to the point cloud at the original resolution to obtain the enhanced features.

[0033] The enhanced features are stacked to aggregate information at different scales. A multilayer perceptron is then used to smooth the stacked features, resulting in fused multi-scale features.

[0034] In some embodiments, the method further includes a training step for the point cloud quality enhancement model, which includes the following steps:

[0035] Mean squared error is used as the loss function for MPNet, and the model is trained using point cloud data from the training set.

[0036] The expression for the loss function is:

[0037] Among them, l i C represents the loss of the component at position i in the block; ij Represents the geometric location in the ground truth; C i ′ j This represents the geometric position after quality enhancement in the rebuild patch.

[0038] In some embodiments, the method further includes: after training the point cloud quality enhancement model, testing the model's performance using test set data, specifically including:

[0039] Peak signal-to-noise ratio (PSNR) is calculated using mean square error, and PSNR is used to measure geometric fidelity.

[0040] Another aspect of this invention provides a decompressed point cloud quality enhancement system based on attention fusion of multi-scale features, comprising:

[0041] The first module is used to generate the baseline truth value of the original point cloud data using the original point cloud data and the compressed point cloud data processed by the compression algorithm.

[0042] The second module is used to divide the original point cloud and the compressed point cloud into multiple blocks according to the reference true value using a block generation method, so as to obtain a compressed point cloud block and a reference true value block.

[0043] The third module is used to construct a point cloud quality enhancement model based on the compressed point cloud block and the reference true value block.

[0044] The fourth module is used to generate position-optimized compressed point cloud blocks based on the point cloud quality enhancement model.

[0045] The fifth module is used to perform block fusion processing on the compressed point cloud blocks optimized at each location to obtain the target point cloud data with optimized location.

[0046] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0047] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0048] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0049] The embodiments of this invention include at least the following beneficial effects: This invention provides a method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features. This method utilizes original point cloud data and compressed point cloud data processed by a compression algorithm to generate a baseline ground truth value for the original point cloud data. Based on the baseline ground truth value, the original point cloud and compressed point cloud are divided into multiple blocks using a block generation method, resulting in compressed point cloud blocks and baseline ground truth value blocks. A point cloud quality enhancement model is constructed based on the compressed point cloud blocks and the baseline ground truth value blocks. Position-optimized compressed point cloud blocks are generated based on the point cloud quality enhancement model. Block fusion processing is performed on each position-optimized compressed point cloud block to obtain position-optimized target point cloud data. This invention can improve the accuracy and precision of point cloud quality enhancement. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0051] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0052] Figure 3 This is a flowchart illustrating the implementation process provided in this embodiment of the invention;

[0053] Figure 4 This is a schematic diagram of the point cloud quality enhancement network provided in an embodiment of the present invention;

[0054] Figure 5 This is a structural diagram of a multi-scale feature fusion block based on cross-scale attention provided in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the point-to-point distortion measurement D1 provided in an embodiment of the present invention;

[0056] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0058] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0059] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0061] The method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features provided in this invention relates to the field of computer technology. This method can be applied to terminals, servers, or software running on either terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features, but is not limited to the above forms.

[0062] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0063] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0064] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0065] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0066] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0067] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides a method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features. The following description uses the application of this method for enhancing the quality of decompressed point clouds based on attention fusion multi-scale features in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0068] Reference Figure 2 , Figure 2 This is a flowchart illustrating a method for enhancing the quality of decompressed point clouds based on attention fusion and multi-scale features, applied to a server, as provided in an embodiment of the present invention. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 2 The method may include the following steps:

[0069] A baseline true value for the original point cloud data is generated using the original point cloud data and the compressed point cloud data processed by the compression algorithm.

[0070] Based on the baseline true value, the original point cloud and the compressed point cloud are divided into multiple blocks by the block generation method to obtain the compressed point cloud block and the baseline true value block.

[0071] Based on the compressed point cloud block and the baseline true value block, a point cloud quality enhancement model is constructed;

[0072] Based on the point cloud quality enhancement model, a position-optimized compressed point cloud block is generated;

[0073] Block fusion processing is performed on the compressed point cloud blocks optimized at each location to obtain the target point cloud data with optimized location.

[0074] In some embodiments, the step of dividing the original point cloud and compressed point cloud into multiple blocks using a block generation method based on the reference ground truth to obtain compressed point cloud blocks and reference ground truth blocks includes the following steps:

[0075] Based on the baseline true value, the original point cloud data is divided into multiple cubes of equal size according to spatial location, and the points in each cube are grouped into one block to obtain the baseline true value block.

[0076] The compressed point cloud data is divided into multiple cubes of equal size according to spatial location, and the points in each cube are grouped into one block to obtain a compressed point cloud block.

[0077] In some embodiments, constructing a point cloud quality enhancement model based on the compressed point cloud block and the baseline ground truth block includes the following steps:

[0078] In the encoder section, the input data generated by the multi-scale input module is received, and multi-scale features are extracted. The first layer of the encoder is a 5×5×5 discrete convolutional layer. The expression for this process is: E0 = CONV(MFIM(I)), where MFIM() represents the multi-scale feature extraction unit, CONV() represents discrete convolution, and E0 is the initial feature extracted from the input data.

[0079] Next, a 3×3×3 discrete convolutional layer is used for downsampling with a stride of 2. During downsampling, the feature dimension doubles while the number of points decreases. The expression for this process is: E n =D_CONV(E n-1 ), where D_CONV() represents a sparse convolution operation on the output of the previous layer with a stride of 2; E n It is the intermediate output of each downsampling layer;

[0080] In the decoder section, a 3×3×3 transposed sparse convolution is used, with upsampling at a stride of 2. The expression for this process is: D n =Tr_CONV(D n-1 ), where Tr_CONV() represents the transpose sparse convolution operation of the previous layer's output, D n It is the intermediate output of each layer's upsampling;

[0081] In the last layer of the decoder, 1×1×1 sparse convolutions are used to refine the features, generate a final output with the same shape as the input, and combine the information of the encoder path and the decoder path to obtain contextual information.

[0082] The cross-scale feature fusion module is used to fuse attribute features from all scales.

[0083] In some embodiments, the method of using a cross-scale feature fusion module to fuse attribute features across all scales includes the following steps:

[0084] In the pre-fusion transformation stage: the weights of geometric distance are used to determine the attribute features of the upsampled points in order to preserve the spatial consistency between the upsampled and downsampled point clouds;

[0085] In the cross-scale attention feature learning stage: the cross-scale attention mechanism is used to effectively fuse the detailed information in the point cloud at the original resolution and the contextual information in the point cloud downsampled twice;

[0086] In the feature fusion stage: enhanced features are obtained through cross-scale attention feature learning, and these features are stacked. The stacked features are then smoothed using a multilayer perceptron to obtain fused multi-scale features.

[0087] In some embodiments, the step of using the weights of geometric distance to determine the attribute features of upsampled points in order to preserve spatial consistency between upsampled and downsampled point clouds includes the following steps:

[0088] Based on the number of points in the downsampled point cloud and the number of points in the upsampled point cloud, the attribute features of each upsampled point are calculated using the following expression: Where, ψ j (x i () is a weighting function based on geometric distance, representing the point x i Its j-th neighboring point x j The reciprocal square of the distance between them; h′ bi Representing the i-th th The attribute features of each upsampled point; K represents the attribute features of each upsampled point in the low-resolution point cloud n. d The number of adjacent points in a given point; n d Represents the number of points in the downsampled point cloud;

[0089] The method of effectively fusing detailed information from the original resolution point cloud and contextual information from the point cloud after two downsampling steps using a cross-scale attention mechanism includes the following steps:

[0090] Perform element-wise multiplication on the point cloud downsampled once and the point cloud downsampled twice to calculate the semantic attention mask;

[0091] The detail attention mask is calculated by performing element-wise addition between the downsampled point cloud and the point cloud at the original resolution.

[0092] The semantic attention mask is applied to the point cloud that has been downsampled twice, and the detail attention mask is applied to the point cloud at the original resolution to obtain the enhanced features.

[0093] The enhanced features are stacked to aggregate information at different scales. A multilayer perceptron is then used to smooth the stacked features, resulting in fused multi-scale features.

[0094] In some embodiments, the method further includes a training step for the point cloud quality enhancement model, which includes the following steps:

[0095] Mean squared error is used as the loss function for MPNet, and the model is trained using point cloud data from the training set.

[0096] The expression for the loss function is:

[0097] Among them, l i C represents the loss of the component at position i in the block; ij Represents the geometric location in the ground truth; C i ′ j This represents the geometric position after quality enhancement in the rebuild patch.

[0098] In some embodiments, the method further includes: after training the point cloud quality enhancement model, testing the model's performance using test set data, specifically including:

[0099] Peak signal-to-noise ratio (PSNR) is calculated using mean square error, and PSNR is used to measure geometric fidelity.

[0100] The specific implementation process of the present invention will now be described in detail with reference to the accompanying drawings:

[0101] like Figure 3 As shown, this invention provides a method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features, comprising the following steps:

[0102] Step 1: Generate baseline ground truth using the original point cloud data and the compressed point cloud data processed by the compression algorithm. The purpose of this step is to generate a set of aligned baseline ground truths based on the original point cloud, so as to provide a reference for subsequent calculation of the error between the compressed point cloud and the original point cloud.

[0103] Step 2: The point cloud is divided into smaller blocks using a block generation method to reduce computational load and adapt to computational resources. The block generation method used in this invention is based on the physical location of the point cloud. Specifically, the point cloud is divided into multiple equal-sized cubes according to their spatial location, and the points within each cube are grouped into one block. This not only avoids the sampling problems of duplicate and missed points but also reduces computational complexity.

[0104] This method divides both the original and compressed point clouds into blocks, ensuring that the blocks are physically identical, meaning that points within each block have similar geometric features. This consistency helps the model learn the errors between the compressed and original point clouds more effectively, thereby improving reconstruction quality.

[0105] Step 3: Use the PyTorch deep learning framework to build a point cloud quality enhancement network (MPNet) based on the Minkowski Engine. The structure of this network model is as follows: Figure 4 As shown, it mainly includes a multi-scale input module, a discrete convolution module, and a cross-scale feature fusion module based on point attention.

[0106] Specifically, the proposed network structure is based on the U-Net architecture and implements discrete convolutions using the Minkowski Engine library. The network consists of two parts: an encoder network and a decoder network. The encoder progressively extracts features from the input data through a series of downsampling convolutional layers and normalization layers, while the decoder progressively restores the spatial resolution through deconvolutional layers and normalization layers, and reconstructs the output by combining the features extracted by the encoder.

[0107] In the encoder section, it first receives data generated by the multi-scale input module and extracts multi-scale features. The first layer of the encoder is a 5×5×5 discrete convolutional layer:

[0108] E0 = CONV(MFIM(I)) (1)

[0109] Here, MFIM() represents a multi-scale feature extraction unit, and CONV() represents a discrete convolution. E0 is the initial feature extracted from the input data. Next, a 3×3×3 discrete convolutional layer is used for downsampling with a stride of 2. During downsampling, the feature dimension doubles, while the number of points decreases.

[0110] E n =D_CONV(E n-1 (2)

[0111] Where D_CONV() represents a sparse convolution operation on the output of the previous layer, with a stride of 2. E nThis is the intermediate output of each downsampling layer. In the decoder section, a 3×3×3(3 3 Transpose the sparse convolution and upsample with a stride of 2:

[0112] D n =Tr_CONV(D n-1 (3)

[0113] Where Tr_CONV() represents the transpose sparse convolution operation of the previous layer's output, D n This is the intermediate output of each upsampled layer. In the last layer of the decoder, 1×1×1(1 3 Sparse convolutions are used to refine features and generate a final output with the same shape as the input. During network forwarding, intermediate encoder (E1, E2, ..., E5) and decoder (D1, D2, ..., D5) intermediate outputs are stored. To obtain contextual information, information from the encoder and decoder paths is combined.

[0114] Finally, a cross-scale feature fusion module is used to fuse attribute features from all scales:

[0115] F n =CAFFM(D n (4)

[0116] Step 4: This invention designs a cross-scale feature fusion module (CAFFM) based on point attention to extract point cloud features. The structure of the CAFFM module is as follows: Figure 5 As shown, the multi-scale input module downsamples the input point cloud to m different scales and extracts geometric features at each scale in the discrete convolution module. These geometric features at different scales contain information at different levels, such as local details and global context. Therefore, this invention designs a multi-scale feature fusion module based on point attention to integrate geometric features at various scales. The module structure is as follows. Figure 5 As shown, the main steps include: transformation before fusion, cross-scale attention feature learning, and feature fusion.

[0117] In the pre-fusion transformation stage, to effectively upsample the low-resolution point cloud to the original resolution, this invention employs a geometric distance-weighted interpolation method. This method uses the weights of geometric distances to determine the attribute features of the upsampled points, thus preserving spatial consistency between the upsampled and downsampled point clouds. Specifically, geometric distance-weighted interpolation is a commonly used attribute interpolation method for points in point cloud processing. Its basic principle is to calculate the attribute values ​​of unknown points based on the geometric distances between known points in the point cloud. This method fully utilizes the geometric relationships between points, maintaining the consistency of the geometric features of the point cloud during the upsampling process.

[0118] Assume that the number of points in the downsampled point cloud is n d points, and the upsampled point cloud contains n points n(n d < n). The attribute feature h′ th of the i bi -th upsampled point is calculated as follows:

[0119]

[0120] where, i ∈ {1, …, n} is a weight function based on geometric distance, representing the reciprocal square of the distance between point x i and its j j -th neighboring point x i , and d(x j , x d ) is the Euclidean distance between two points. K represents the number of adjacent points of each upsampled point in the low-resolution point cloud n

[0121] The purpose of using this method for upsampling is to ensure that during the upsampling process, the new point cloud can maintain the same geometric features as the original point cloud. This weighted interpolation method can effectively smooth the transition and capture more accurate geometric information by weighted averaging the features of neighboring points, thereby providing better input for the subsequent feature fusion module.

[0122] In the cross-scale attention feature learning stage, the present invention defines the point cloud at the original resolution as A, the point cloud downsampled once as B, and the point cloud downsampled twice as C. Among them, A comes from a lower level (i.e., high resolution, representing details), and C comes from a higher level (i.e., low resolution, representing context). Compared with B, A has more detailed information, while C contains richer context information. Therefore, the present invention designs B as the basic scale, and on this basis, uses the cross-scale attention mechanism to effectively fuse the detailed information in A and the context information in C using the cross-scale attention mechanism.

[0123] Specifically, first perform an element-wise multiplication operation ("MUL") on B and C to calculate the semantic attention mask M sem , and similarly, perform an element-wise addition operation ("ADD") on B and A to calculate the detail attention mask M res . This operation can avoid the problem of boundary blurring that may occur during simple element-wise addition and the problem of suppressing important signals during element-wise multiplication.

[0124] M res = σ(z(A)+z(B)) (6)

[0125] M sem = σ(z(C)·z(B)) (7)

[0126] Where σ represents the sigmoid activation function, and z represents the feature transformation operation before fusion. Following the steps described above, the corresponding masks M are applied to A and C respectively. res and M sem :

[0127] A′=A⊙M res (8)

[0128] C′=C⊙M sem (9)

[0129] This feature fusion strategy not only improves the network's ability to comprehensively utilize features at different scales, but also enhances the accuracy of point cloud feature representation.

[0130] In the feature fusion stage, this invention obtains enhanced features through cross-scale attention feature learning, and stacks these features A′, B, and C′ (corresponding to...). Figure 5 The F in the model is used to aggregate information from different scales. Then, a multilayer perceptron (MLP) is used (see F) to aggregate information from different scales. Figure 5 The T in the model is used to smooth the stacked features, thus obtaining the fused multi-scale features.

[0131] Step 5: Train the model using the point cloud data from the training set. To measure the difference between the quality enhancement blocks of geometric components and the baseline true values, this invention uses mean squared error (MSE) as the loss function for MPNet.

[0132]

[0133] Among them, l i Let MSE represent the loss of the i-th location component in the block, and C and C' represent the geometric location in the ground truth and the quality-enhanced geometric location in the reconstructed patch, respectively. Since the model aims to minimize the difference between the noisy input and the ground truth, MSE, as the loss function, can effectively measure this difference and intuitively reflect the quality of the model's performance.

[0134] Step 6: After training the point cloud quality enhancement model, the model is tested using the test set data. This invention uses Peak Signal-to-Noise Ratio (PSNR) to measure geometric fidelity. A higher PSNR indicates better reconstruction quality with the same number of encoded bits. Specifically, PSNR is calculated using Mean Squared Error (MSE), where MSE is the squared average of the distances between symmetrical points in the reference point cloud A and its reconstructed point cloud B. D1 and D1-PSNR are defined as follows:

[0135]

[0136]

[0137] Where D1 represents the point-to-point distortion; D1-PSNR represents the peak signal-to-noise ratio calculated based on the point-to-point distortion; |A| and |B| represent the number of points in point clouds A and B, respectively. θ a Represents the normal vector between point ab and point b. The angle between them, and θ b Let ba be the normal vector to point a. The angle between them. Figure 6 It shows point-to-point error and point-to-plane error.

[0138] In summary, current research on point cloud compression artifacts mainly focuses on removing attribute artifacts, i.e., improving the representation of color and texture in point clouds. However, geometric artifacts, namely distortions or errors in the geometric shapes of point clouds, also negatively impact the overall quality of the point cloud. Since geometric information is closely related to attribute information, geometric artifacts directly lead to errors in attribute information. Therefore, simply removing attribute artifacts is insufficient to significantly improve the reconstruction quality of point clouds; the removal of geometric artifacts is equally crucial.

[0139] Furthermore, in point cloud research, there is a common problem of insufficient local contextual information, which fails to effectively capture the correlation between global and local information in point cloud data, limiting the accuracy and precision of point cloud quality enhancement. Point cloud features at different scales often provide complementary information; focusing on high-scale detail and low-scale contextual information can more comprehensively describe point cloud data, thereby improving the accuracy and precision of reconstructed point clouds. Therefore, to address the above problems, this invention designs a point cloud quality enhancement method based on attention fusion of multi-scale features, with the following main advantages:

[0140] 1) This invention designs a multi-scale compressed point cloud geometric information quality enhancement network. This network adopts a multi-branch feature extraction structure combined with farthest point sampling to extract geometric artifact features in parallel, thereby obtaining a geometrically enhanced reconstructed point cloud.

[0141] 2) This invention designs a multi-scale feature fusion module based on point attention, which effectively fuses point cloud detail information and context information in multi-scale features to improve point cloud quality.

[0142] 3) Experimental results show that the method designed in this invention improves the average peak signal-to-noise ratio on public datasets compared to standard compression algorithms.

[0143] Another aspect of this invention provides a decompressed point cloud quality enhancement system based on attention fusion of multi-scale features, comprising:

[0144] The first module is used to generate the baseline truth value of the original point cloud data using the original point cloud data and the compressed point cloud data processed by the compression algorithm.

[0145] The second module is used to divide the original point cloud and the compressed point cloud into multiple blocks according to the reference true value using a block generation method, so as to obtain a compressed point cloud block and a reference true value block.

[0146] The third module is used to construct a point cloud quality enhancement model based on the compressed point cloud block and the reference true value block.

[0147] The fourth module is used to generate position-optimized compressed point cloud blocks based on the point cloud quality enhancement model.

[0148] The fifth module is used to perform block fusion processing on the compressed point cloud blocks optimized at each location to obtain the target point cloud data with optimized location.

[0149] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0150] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0151] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0152] Please see Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0153] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0154] The memory 702 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and is called and executed by the processor 701 to execute the decompressed point cloud quality enhancement method based on attention fusion multi-scale features according to the embodiments of the present invention.

[0155] The input / output interface 703 is used to implement information input and output;

[0156] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0157] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0158] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0159] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features.

[0160] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0161] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0162] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0163] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0164] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0165] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0167] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0168] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0169] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0170] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0171] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0172] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for enhancing the quality of decompressed point clouds based on attention-based fusion of multi-scale features, characterized in that, Includes the following steps: A baseline true value for the original point cloud data is generated using the original point cloud data and the compressed point cloud data processed by the compression algorithm. Based on the baseline true value, the original point cloud and the compressed point cloud are divided into multiple blocks by the block generation method to obtain the compressed point cloud block and the baseline true value block. Based on the compressed point cloud block and the baseline true value block, a point cloud quality enhancement model is constructed; Based on the point cloud quality enhancement model, a position-optimized compressed point cloud block is generated; Block fusion processing is performed on the compressed point cloud blocks optimized at each location to obtain the target point cloud data with optimized location; The step of constructing a point cloud quality enhancement model based on the compressed point cloud block and the baseline ground truth block includes the following steps: In the encoder section, input data generated by the multi-scale input module is received, and multi-scale features are extracted. The first layer of the encoder is a 5×5×5 discrete convolutional layer. The expression for this process is: ,in, () represents a multi-scale feature extraction unit, and CONV() represents discrete convolution; These are the initial features extracted from the input data; Next, a 3×3×3 discrete convolutional layer is used for downsampling with a stride of 2. During downsampling, the feature dimension doubles while the number of points decreases. The expression for this process is: ,in, This indicates a sparse convolution operation on the output of the previous layer with a stride of 2; It is the intermediate output of each downsampling layer; In the decoder section, 3 is used. 3 3. Transpose the sparse convolution and upsample with a stride of 2. The expression for this process is: ,in, This represents the transpose of the sparse convolution operation of the previous layer's output. It is the intermediate output of each layer's upsampling; In the last layer of the decoder, using 1 1 1. Sparse convolution is used to refine features, generate a final output with the same shape as the input, and combine information from the encoder path and decoder path to obtain contextual information; A cross-scale feature fusion module is used to fuse attribute features from all scales; The method of using a cross-scale feature fusion module to fuse attribute features at all scales includes the following steps: In the pre-fusion transformation stage: the weights of geometric distance are used to determine the attribute features of the upsampled points in order to preserve the spatial consistency between the upsampled and downsampled point clouds; In the cross-scale attention feature learning stage: the cross-scale attention mechanism is used to effectively fuse the detailed information in the point cloud at the original resolution and the contextual information in the point cloud downsampled twice; In the feature fusion stage: enhanced features are obtained through cross-scale attention feature learning, and these features are stacked. The stacked features are then smoothed using a multilayer perceptron to obtain fused multi-scale features.

2. The method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features according to claim 1, characterized in that, The step of dividing the original point cloud and compressed point cloud into multiple blocks using a block generation method based on the reference true value to obtain compressed point cloud blocks and reference true value blocks includes the following steps: Based on the baseline true value, the original point cloud data is divided into multiple cubes of equal size according to spatial location, and the points in each cube are grouped into one block to obtain the baseline true value block. The compressed point cloud data is divided into multiple equal-sized cubes according to spatial location, and the points in each cube are grouped into one block to obtain a compressed point cloud block.

3. The method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features according to claim 1, characterized in that, The method of using the weights of geometric distance to determine the attribute features of upsampled points in order to preserve spatial consistency between upsampled and downsampled point clouds includes the following steps: Based on the number of points in the downsampled point cloud and the number of points in the upsampled point cloud, the attribute features of each upsampled point are calculated using the following expression: ,in, It is a weighting function based on geometric distance, representing a point. Its first Neighboring points The reciprocal square of the distance between them; Representing the The attribute features of each upsampled point; This indicates that each upsampled point is in the low-resolution point cloud. The number of adjacent points in a given point; Represents the number of points in the downsampled point cloud; The method of effectively fusing detailed information from the original resolution point cloud and contextual information from the point cloud after two downsampling steps using a cross-scale attention mechanism includes the following steps: Perform element-wise multiplication on the point cloud downsampled once and the point cloud downsampled twice to calculate the semantic attention mask; The detail attention mask is calculated by performing element-wise addition between the downsampled point cloud and the point cloud at the original resolution. The semantic attention mask is applied to the point cloud that has been downsampled twice, and the detail attention mask is applied to the point cloud at the original resolution to obtain the enhanced features. The enhanced features are stacked to aggregate information at different scales. A multilayer perceptron is then used to smooth the stacked features, resulting in fused multi-scale features.

4. The method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features according to claim 1, characterized in that, The method also includes a training step for the point cloud quality enhancement model, which includes the following steps: Mean squared error is used as the loss function for MPNet, and the model is trained using point cloud data from the training set. The expression for the loss function is: , in, Represents the first in the block Loss of each location component; Represents the geometric location in the ground truth; This represents the geometric position after quality enhancement in the rebuild patch.

5. The method for enhancing the quality of decompressed point clouds based on attention fusion of multi-scale features according to claim 4, characterized in that, The method further includes: after training the point cloud quality enhancement model, testing the model's performance using test set data, specifically including: Peak signal-to-noise ratio (PSNR) is calculated using mean square error, and PSNR is used to measure geometric fidelity.

6. A decompressed point cloud quality enhancement system based on attention fusion of multi-scale features for implementing the method as described in any one of claims 1-5, characterized in that, include: The first module is used to generate a baseline true value of the original point cloud data using the original point cloud data and the compressed point cloud data processed by the compression algorithm; The second module is used to divide the original point cloud and the compressed point cloud into multiple blocks according to the reference true value using a block generation method, so as to obtain a compressed point cloud block and a reference true value block. The third module is used to construct a point cloud quality enhancement model based on the compressed point cloud block and the reference true value block. The fourth module is used to generate position-optimized compressed point cloud blocks based on the point cloud quality enhancement model. The fifth module is used to perform block fusion processing on the compressed point cloud blocks optimized at each location to obtain the target point cloud data with optimized location.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images