A point cloud encoding processing method, device and electronic equipment

Abstract

The application discloses a point cloud encoding processing method and device and electronic equipment, and relates to the technical field of communication, to solve the problem that in the existing dynamic point cloud encoding, a user needs to obtain suitable setting parameters through multiple repeated experiments, so that the encoding operation is not quick enough. The method comprises the following steps: obtaining two-dimensional mapping images of each frame of point cloud in a dynamic point cloud, to obtain multiple two-dimensional mapping images; determining the similarity between each two adjacent two-dimensional mapping images in the multiple two-dimensional mapping images, to obtain N similarity values; determining the similarity parameter of the dynamic point cloud based on the N similarity values; and adjusting the encoding parameter of the dynamic point cloud according to the similarity parameter, wherein the encoding parameter comprises at least one of the number of key frames and the number of predicted frames. Through the similarity parameter of the dynamic point cloud, the application can help the user to quickly find the suitable encoding parameter without the need for the user to perform multiple repeated experiments.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a point cloud encoding processing method, apparatus and electronic device. Background Technology

[0002] Compression methods for dynamic point clouds can be broadly classified into two categories: compression methods based on three-dimensional structures and compression methods based on two-dimensional mappings. The two-dimensional mapping-based compression method involves projecting the dynamic point cloud data onto a two-dimensional plane according to certain rules to obtain a two-dimensional mapping sequence of the dynamic point cloud. This sequence is then compressed using video encoding and decoding techniques. Compared to three-dimensional structure-based compression methods, this approach improves compression efficiency in both time and space.

[0003] Currently, relevant organizations have released Test Model Category 2 (TMC2) for dynamic point cloud compression algorithms, which uses a two-dimensional mapping-based compression method to compress point clouds. However, due to the limited amount of dynamic point cloud data and the insufficient number of use cases, the parameter settings in this model lack unified usage indicators. This means that in different application scenarios, users can only obtain suitable settings through repeated experiments to achieve a good compression effect. Summary of the Invention

[0004] This application provides a point cloud encoding processing method, apparatus, and electronic device to solve the problem that existing dynamic point cloud encoding requires users to conduct repeated experiments to obtain suitable setting parameters, resulting in slow encoding operations.

[0005] In a first aspect, embodiments of this application provide a point cloud encoding processing method, including:

[0006] Obtain the two-dimensional mapping images of each frame of the point cloud in the dynamic point cloud to obtain multiple frames of two-dimensional mapping images;

[0007] Determine the similarity between each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images to obtain N similarity values, where N is an integer greater than 1;

[0008] Based on the N similarity values, the similarity parameters of the dynamic point cloud are determined;

[0009] The encoding parameters of the dynamic point cloud are adjusted according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames.

[0010] Optionally, determining the similarity parameters of the dynamic point cloud based on the N similarity values ​​includes:

[0011] The average of the N similarity values ​​is calculated to determine the overall average similarity of the dynamic point cloud.

[0012] Optionally, before adjusting the encoding parameters of the dynamic point cloud based on the similarity parameters, the method further includes:

[0013] The dynamic point cloud is segmented to obtain M segment point clouds, where M is an integer greater than 1;

[0014] Based on the N1 similarity values ​​corresponding to the first segment of the point cloud among the N similarity values, the average segment similarity of the first segment of the point cloud is determined, wherein the first segment of the point cloud is any segment of the M segment of the point cloud, N1 is a value related to the number of frames of the first segment of the point cloud, and N1 is an integer greater than 1 and less than N.

[0015] The step of adjusting the encoding parameters of the dynamic point cloud based on the similarity parameters includes:

[0016] The encoding parameters of the first segment point cloud are adjusted based on the average similarity of the segment and the overall average similarity.

[0017] Optionally, adjusting the encoding parameters of the first segment point cloud based on the segment average similarity and the overall average similarity includes:

[0018] If the average similarity of the segment is higher than the overall average similarity, reduce the number of prediction frames used for encoding the first segment point cloud.

[0019] If the average similarity of the segment is lower than the overall average similarity, the number of prediction frames used for encoding the first segment point cloud is increased.

[0020] Optionally, determining the similarity between any two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping image includes:

[0021] Based on the target parameters of each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, the structural similarity (SSIM) value between each two adjacent two-dimensional mapping images is determined, wherein the target parameters include at least one of brightness, contrast and structural parameters.

[0022] Optionally, the dynamic point cloud is a holographic digital human dynamic point cloud.

[0023] Secondly, embodiments of this application also provide a point cloud encoding processing apparatus, comprising:

[0024] The acquisition module is used to acquire two-dimensional mapping images of each frame of the point cloud in the dynamic point cloud, and obtain multiple frames of two-dimensional mapping images;

[0025] The first determining module is used to determine the similarity between each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, and obtain N similarity values, where N is an integer greater than 1;

[0026] The second determining module is used to determine the similarity parameters of the dynamic point cloud based on the N similarity values;

[0027] An adjustment module is used to adjust the encoding parameters of the dynamic point cloud according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames.

[0028] Optionally, the second determining module is used to average the N similarity values ​​to determine the overall average similarity of the dynamic point cloud.

[0029] Optionally, the point cloud encoding processing device further includes:

[0030] The segmentation module is used to segment the dynamic point cloud to obtain M segment point clouds, where M is an integer greater than 1;

[0031] The third determining module is used to determine the average segment similarity of the first segment point cloud based on N1 similarities corresponding to the first segment point cloud among the N similarity values, wherein the first segment point cloud is any segment of the M segment point cloud, N1 is a value related to the number of frames of the first segment point cloud, and N1 is an integer greater than 1 and less than N.

[0032] The adjustment module is used to adjust the encoding parameters of the first segment point cloud based on the segment average similarity and the overall average similarity.

[0033] Optionally, the adjustment module includes:

[0034] The first adjustment unit is used to reduce the number of prediction frames used for encoding the first segment point cloud when the average similarity of the segment is higher than the overall average similarity.

[0035] The second adjustment unit is used to increase the number of prediction frames used for encoding the first segment point cloud when the average similarity of the segment is lower than the overall average similarity.

[0036] Optionally, the first determining module is used to determine the SSIM value between each pair of adjacent two-dimensional mapping images based on the target parameters of each pair of adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, wherein the target parameters include at least one of brightness, contrast and structural parameters.

[0037] Thirdly, embodiments of this application also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the point cloud encoding processing method described above.

[0038] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the point cloud encoding processing method described above.

[0039] In this embodiment, two-dimensional mapping images of each frame of a dynamic point cloud are acquired to obtain multiple frames of two-dimensional mapping images. The similarity between any two adjacent frames of the multi-frame two-dimensional mapping images is determined to obtain N similarity values, where N is an integer greater than 1. Based on the N similarity values, similarity parameters of the dynamic point cloud are determined. According to the similarity parameters, the encoding parameters of the dynamic point cloud are adjusted, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames. Thus, by using the similarity parameters of the dynamic point cloud to guide the adjustment of its encoding parameters, users can quickly find suitable encoding parameters without having to conduct multiple repeated experiments. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of the point cloud encoding processing method provided in the embodiments of this application;

[0042] Figure 2 This is a schematic diagram of the visualization results of four dynamic point cloud sequences provided in the embodiments of this application;

[0043] Figure 3 This is a schematic diagram of the bitstream structure generated during the compression encoding process of the point cloud provided in the embodiments of this application;

[0044] Figure 4 The comparison histograms of relevant index parameters obtained by encoding and decoding point clouds with high SSIM values ​​using three different models are provided for embodiments of this application.

[0045] Figure 5 The comparison histograms of relevant index parameters obtained by encoding and decoding point clouds with low SSIM values ​​using three different models are provided for embodiments of this application.

[0046] Figure 6 This is a structural diagram of the point cloud encoding processing device provided in the embodiments of this application;

[0047] Figure 7 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] See Figure 1 , Figure 1 This is a flowchart of the point cloud encoding processing method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps:

[0050] Step 101: Obtain the two-dimensional mapping image of each frame of the point cloud in the dynamic point cloud to obtain multiple frames of two-dimensional mapping images.

[0051] Point clouds can be divided into two types: static point clouds and dynamic point clouds. A static point cloud consists of a set of points with three-dimensional positional information. Each point has one or more attributes to store additional information, such as color information. A dynamic point cloud, on the other hand, consists of multiple consecutive frames of static point clouds; each frame is called a frame of the dynamic point cloud. The positional information of the points in a point cloud is generally three-dimensional coordinates in the world coordinate system (x, y, z), used to represent the spatial topology and reconstruct the shape of three-dimensional objects.

[0052] In this embodiment of the application, the aforementioned dynamic point cloud can refer to a three-dimensional dynamic point cloud composed of multiple frames of continuous static point clouds. Furthermore, the dynamic point cloud can be a dynamic point cloud collected under various practical application scenarios; for example, it can be a dynamic point cloud collected continuously for a certain period of time from a specific object.

[0053] Optionally, the dynamic point cloud is a holographic digital human dynamic point cloud.

[0054] In one embodiment, the present application can be applied to the scenario of encoding dynamic point clouds of holographic digital humans. By analyzing the similarity parameters of the dynamic point clouds of holographic digital humans, the selection of the number of keyframes and prediction frames used in the encoding can be guided.

[0055] This application considers the characteristics of dynamic point clouds in holographic digital humans: the main body remains largely unchanged, with only minor, regular changes in some detailed organs. This results in significant redundancy between frames in the dynamic point cloud. Therefore, the compression coding rate can be improved by changing the number of keyframes and prediction frames in the compression coding of the holographic digital human's dynamic point cloud. The specific direction for adjusting the number of keyframes and prediction frames can be determined by analyzing the overall similarity parameters of the holographic digital human's dynamic point cloud.

[0056] The two-dimensional mapping images of each frame of the dynamic point cloud mentioned above can refer to the two-dimensional mapping images corresponding to each frame of the three-dimensional point cloud obtained by mapping each frame of the three-dimensional point cloud in the dynamic point cloud to a two-dimensional space. In one embodiment, each frame of the three-dimensional point cloud in the dynamic point cloud can be pre-mapped to a two-dimensional space to obtain the two-dimensional mapping images of each frame of the three-dimensional point cloud. Therefore, obtaining the two-dimensional mapping images of each frame of the dynamic point cloud can be done by directly obtaining the pre-mapped two-dimensional mapping images of each frame of the three-dimensional point cloud.

[0057] For example, for dynamic point clouds of holographic digital humans, considering the structure and movement trends of the human body, from a frontal view... Figure 1 Generally, holographic point clouds can reflect human movements and are suitable as two-dimensional mapping images. Therefore, the dynamic point cloud of a holographic digital human can be mapped from the front of the person, and the resulting two-dimensional mapping image is the frontal view of the person. Specifically, it can be done as follows: Figure 2 As shown.

[0058] By mapping each frame of the point cloud in a dynamic point cloud, a corresponding number of two-dimensional mapping images can be obtained. That is, the aforementioned multiple two-dimensional mapping images can be the acquired two-dimensional mapping images of each frame of the point cloud in the dynamic point cloud, and the number of frames in the two-dimensional mapping images is equal to the number of frames in the dynamic point cloud. Alternatively, for the dynamic point cloud, two-dimensional mapping images of only a portion of the frame points can be acquired. This can remove two-dimensional mapping images of point clouds with high repetition, thereby reducing the amount of subsequent computation.

[0059] Step 102: Determine the similarity between each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, and obtain N similarity values, where N is an integer greater than 1.

[0060] In this step, similarity calculation can be performed on adjacent two-dimensional mapping images in the obtained multi-frame two-dimensional mapping images to obtain the similarity between each adjacent two-dimensional mapping image, and a total of N similarity values ​​can be obtained. When performing similarity calculation on each adjacent two-dimensional mapping image in the multi-frame two-dimensional mapping images, N can be equal to the number of frames of the dynamic point cloud minus.

[0061] The N similarity values ​​can be used to characterize the similarity between the multiple frames of two-dimensional mapping images. Since the multiple frames of two-dimensional mapping images are projections of the dynamic point cloud onto a two-dimensional plane, the N similarity values ​​can also characterize the changes in the dynamic point cloud, or in other words, reflect the motion trend of the dynamic point cloud.

[0062] Optionally, step 102 includes:

[0063] Based on the target parameters of each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, the structural similarity (SSIM) value between each two adjacent two-dimensional mapping images is determined, wherein the target parameters include at least one of brightness, contrast and structural parameters.

[0064] In one embodiment, the SSIM value between two adjacent two-dimensional mapping images can be calculated based on parameters such as brightness, contrast, and structure of each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, and the similarity between the images can be judged by the SSIM value.

[0065] In practice, brightness, contrast, and structure can be used as measurement indicators, and the following formula can be used to calculate the similarity of two adjacent frames of two-dimensional mapped images in the three channels (brightness, contrast, and structure):

[0066]

[0067] Where l represents brightness, c represents contrast, s represents structure, and μ x Let μ be the mean of the image x. y The mean of the image y, σ x Let σ be the variance of the image x. y Let σ be the variance of the image y. xy Let c1 be the covariance of image x and image y, where c1 = (k1L). 2 Let c2 = (k2L) 2 c3 = c2 / 2.

[0068] Then, the similarity of the three channels can be multiplied by different indices to obtain the final result, i.e., the SSIM value:

[0069] SSIM(x,y)=[l(x,y) α ·c(x,y) β ·s(x,y) γ ]

[0070] Among them, α, β and γ are used to adjust the importance of the three channels. They can all be set to 1 by default, or they can be flexibly set according to the actual situation.

[0071] In this way, by calculating the structural similarity (SSIM) of images based on the characteristics of adjacent two-dimensional mapped images in terms of brightness, contrast, and structural parameters, the accuracy of the evaluated similarity between images can be ensured, making the similarity values ​​valuable for reference.

[0072] In one implementation, the SSIM value can be calculated and motion estimation performed using the computer programming language Python 3.8.

[0073] In another implementation, the SSIM value can be calculated using the OpenCV-Python and scikit-image libraries. OpenCV-Python is an open-source Python library that provides many real-time optimization algorithms for computer vision, used to read two-dimensional mapped images and convert them into RGB three-channel Python arrays. Scikit-image (also known as skimage) is an external library of SciPy that uses computer vision algorithms to process images and calculate the SSIM value between two images.

[0074] Step 103: Based on the N similarity values, determine the similarity parameters of the dynamic point cloud.

[0075] In this step, the N similarity values ​​can be used to determine the similarity parameters of the dynamic point cloud. For example, the average of the N similarity values ​​can be taken, or a portion of the N similarity values ​​can be taken, such as averaging the higher similarity values, or the highest similarity among the N similarity values ​​can be taken as the similarity parameter for evaluating the dynamic point cloud.

[0076] Optionally, step 103 includes:

[0077] The average of the N similarity values ​​is calculated to determine the overall average similarity of the dynamic point cloud.

[0078] In one implementation, to comprehensively measure the overall average similarity of the dynamic point cloud and ensure the reliability of the similarity assessment, the average similarity of the N similarity values ​​can be obtained and used as the overall average similarity of the dynamic point cloud. For example, the N similarity values ​​can be summed and then divided by N, or the geometric mean, weighted average, etc., can be calculated from the N similarity values.

[0079] Step 104: Adjust the encoding parameters of the dynamic point cloud according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames.

[0080] The similarity parameters of the dynamic point cloud can characterize the changes in the dynamic point cloud. For dynamic point clouds of moving objects, the similarity parameters of the dynamic point cloud can be used to evaluate the motion trend of the object, that is, it can be used as motion estimation for the dynamic point cloud.

[0081] Therefore, in this step, the changes / motion estimation of the dynamic point cloud can be determined based on the similarity parameters. Then, the encoding parameters of the dynamic point cloud can be adjusted based on the changes / motion estimation of the dynamic point cloud, such as adjusting the number of keyframes (Intra-coded Frames, I-frames) and / or the number of predictive frames (Predictive-coded Frames, P-frames) used when encoding the dynamic points. Keyframes can also be called I-frames, and predictive frames can also be called P-frames.

[0082] See Figure 3 , Figure 3 The structure of the bitstream generated during the compression of point cloud video is shown. Each bitstream consists of a header file and many different Group of Frames (GOFs). For each GOF, it can be encoded as a keyframe (I-frame) or a predicted frame (P-frame) for internal coding or motion compensation.

[0083] An I-frame is a keyframe containing all the information and can be decoded independently without referencing other images; it can be simply considered a still image. The first frame of a video sequence is usually an I-frame because it is a crucial keyframe, moderately compressed, and used as a reference point for random access. If an I-frame is lost during transmission, the video will appear choppy because subsequent frames cannot be decoded correctly.

[0084] During the Moving Picture Experts Group (MPEG) encoding process, some video frame sequences are compressed into I-frames, and some are compressed into P-frames.

[0085] P-frames require reference to previous I-frames for encoding and decoding, displaying the differences between the current and previous frames. Because P-frames incorporate motion estimation and compensation, encoding requires calculating the differences between the two frames, unlike I-frames which simply compress the current image. Decoding necessitates overlaying the stored differences from the current frame onto the previously cached image. P-frames typically occupy less space than I-frames because they store only the difference values, not the entire image. However, due to their complex correlation with the motion trends of the previous frame, they are highly sensitive to transmission errors; even minor errors can easily damage the image. Loss or corruption of P-frames results in a mosaic effect in the video.

[0086] Since P-frames utilize motion estimation and compensation, storing only the difference between two frames, it's clear that the motion trend and speed of the video being compressed directly affect compression performance. If the video depicts a nearly static scene, encoding with fewer I-frames and more P-frames can achieve good compression results; however, if the video is fast-paced and action-packed like an action movie, more I-frames should be included in the encoding process to ensure smooth, stutter-free visuals.

[0087] Therefore, in this step, specifically, a larger similarity parameter indicates a greater degree of change in the dynamic point cloud, or a larger range of motion and faster movement speed. This allows for the use of more I-frames and fewer P-frames during encoding, ensuring higher compression performance and avoiding information loss or large errors. Conversely, a smaller similarity parameter indicates a smaller degree of change in the dynamic point cloud, or a smaller range of motion and more static images. This allows for the use of more P-frames and fewer P-frames during encoding, achieving good compression results while improving encoding efficiency.

[0088] This application considers the dynamic point cloud characteristics of holographic digital humans, where the main body remains largely unchanged, with only minor, regular changes in some detailed organs. This results in significant redundancy between dynamic point cloud frames. Therefore, the compression coding efficiency can be improved by changing the number of I-frames and P-frames in the point cloud compression coding. Furthermore, since the video to be encoded is a sequence composed of two-dimensional mappings of dynamic point clouds, further decoding is required at the receiving end to reconstruct the point cloud data. Therefore, the error rate of video encoding should be as low as possible.

[0089] In compression methods based on 2D mapping, motion estimation is required based on the original 3D object. However, the number of points per frame in a dynamic point cloud is not constant; it depends on many different factors, such as the object's state, preprocessing steps, or the point cloud acquisition device. Due to the influence of noise, motion estimation of the geometry constructed from these points is very challenging. Furthermore, even in adjacent frames, it is difficult to find the correspondence between two sets of points, which is the main reason for the low compression efficiency of 3D-based methods. Although many algorithms applying motion estimation have been proposed, directly processing 3D point clouds requires a very large amount of computation. Therefore, in this embodiment, to reduce the cost of processing point cloud geometry, attribute 2D mapping can be used to significantly improve motion estimation.

[0090] Optionally, before step 104, the method further includes:

[0091] The dynamic point cloud is segmented to obtain M segment point clouds, where M is an integer greater than 1;

[0092] Based on the N1 similarity values ​​corresponding to the first segment of the point cloud among the N similarity values, the average segment similarity of the first segment of the point cloud is determined, wherein the first segment of the point cloud is any segment of the M segment of the point cloud, N1 is a value related to the number of frames of the first segment of the point cloud, and N1 is an integer greater than 1 and less than N.

[0093] Step 104 includes:

[0094] The encoding parameters of the first segment point cloud are adjusted based on the average similarity of the segment and the overall average similarity.

[0095] In one implementation, the dynamic point cloud can be segmented and the encoding parameters can be adjusted segment by segment.

[0096] In practice, the dynamic point cloud can be segmented. For example, the dynamic point cloud can be segmented into segments of several frames to obtain multiple point cloud segments. Let's say a total of M point cloud segments are obtained.

[0097] Then, based on the calculated N similarity values, and according to the correspondence between each frame of the two-dimensional mapped image and the corresponding frame of the dynamic point cloud, several similarity values ​​corresponding to each segment of the point cloud can be determined. For example, for any segment of the M segment of the point cloud, designated as the first segment, N1 similarity values ​​corresponding to that segment can be found from the N similarity values. The average similarity of the N1 similarity values ​​can then be calculated to determine the average segment similarity of the first segment of the point cloud, which characterizes the similarity of the first segment.

[0098] Finally, the direction of adjusting the encoding parameters of the corresponding segment point cloud can be determined by combining the average similarity of each segment and the overall average similarity. For example, the average similarity of each segment can be compared with the overall average similarity to determine whether the similarity of each segment is high or low, and then determine whether to adjust the encoding parameters of the segment to use more I frames or more P frames.

[0099] In this way, by segmenting the dynamic point cloud and referring to the average similarity of each segment and the overall average similarity, the coding parameters of each segment can be adjusted separately. This ensures that the coding parameter adjustment is more targeted, better takes into account the coding performance of each segment, and thus improves the overall coding rate.

[0100] Further, adjusting the encoding parameters of the first segment point cloud based on the segment average similarity and the overall average similarity includes:

[0101] If the average similarity of the segment is higher than the overall average similarity, reduce the number of prediction frames used for encoding the first segment point cloud.

[0102] If the average similarity of the segment is lower than the overall average similarity, the number of prediction frames used for encoding the first segment point cloud is increased.

[0103] In one specific implementation, the overall average similarity of the dynamic point cloud can be used as a reference benchmark. The average similarity of each segment can be compared with the overall average similarity. If the average similarity of a segment is higher than the overall average similarity, it indicates that the segment has a large degree of change or a large range of motion. Therefore, when encoding the point cloud segment, the number of key frames (I-frames) can be increased while the number of prediction frames (P-frames) can be decreased. For example, one I-frame can be added and one P-frame can be decreased each time an adjustment is made. If the average similarity of a segment is lower than the overall average similarity, it indicates that the segment has a small degree of change or a small range of motion. Therefore, when encoding the point cloud segment, the number of key frames (I-frames) can be reduced while the number of prediction frames (P-frames) can be increased. For example, one I-frame can be reduced and one P-frame can be added each time an adjustment is made.

[0104] In this way, the implementation method can ensure the accuracy of the direction of the encoding parameter adjustment for each segment, as well as ensure that each segment has a high encoding compression ratio and achieves a good compression effect.

[0105] The point cloud encoding processing method of this application embodiment acquires two-dimensional mapping images of each frame of a dynamic point cloud to obtain multiple frames of two-dimensional mapping images; determines the similarity between each adjacent pair of two-dimensional mapping images in the multiple frames of two-dimensional mapping images to obtain N similarity values, where N is an integer greater than 1; determines similarity parameters of the dynamic point cloud based on the N similarity values; and adjusts the encoding parameters of the dynamic point cloud according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames. In this way, by using the similarity parameters of the dynamic point cloud to guide the adjustment of its encoding parameters, users can quickly find suitable encoding parameters without having to conduct multiple repeated experiments.

[0106] The purpose of this application is to propose a motion estimation method for detecting the three-dimensional redundancy of dynamic point cloud data of holographic digital humans. In view of the high redundancy of dynamic point cloud data transmitted by digital twins, the method improves the compression coding rate by changing the number of I-frames and P-frames in the point cloud compression coding.

[0107] Dynamic point clouds are typically used to represent a scene or an object. Since 3D scenes are always static, motion estimation of point clouds primarily focuses on the object. Many algorithms exist for implementing dynamic point cloud mapping, such as the simplest direct projection. For simplicity, a frontal view of the 3D object can be used.

[0108] To address motion estimation in two-dimensional mapped images, this application proposes using SSIM to calculate the similarity between adjacent frames. SSIM is commonly used to measure image similarity or assess the quality of compressed images, using brightness, contrast, and structure as measurement metrics.

[0109] After calculating the SSIM values ​​of every two adjacent frames, the average value can be taken to find the similarity of the entire sequence. Since two-dimensional images are mapped from three-dimensional objects, the geometric structure and color attributes under motion can be deduced from the two-dimensional images. If the SSIM value is high, it indicates that the dynamic point cloud has a large range of motion and a fast motion speed. Therefore, the motion estimation should be more accurate during compression, i.e., more I-frames should be used.

[0110] This application can be implemented on different regions of a holographic digital human. Considering the data characteristics of a holographic digital human, to examine the versatility of the algorithm, the effectiveness of the embodiments of this application can be verified through relevant experiments. Specifically, four dynamic human point clouds can be used for evaluation in the experimental verification. The main evaluation indicators are compression time and bitstream size. In the High Efficiency Video Coding (HEVC) algorithm, the most important step in removing temporal redundancy is determining the number of I-frames. Therefore, changing the number of I-frames is a suitable method for testing the compression effect and motion estimation effect.

[0111] In the experiment, each dynamic human point cloud was compressed using the same test model under three conditions: all frames were encoded as I-frames, only some frames were encoded as I-frames, and auxiliary information was used for 3D motion compensation. In the experiment, different portions of the same number of frames from the same dynamic point cloud were selected to control different SSIM values ​​for each portion. The independent variables in the experiment were the SSIM value and different point cloud datasets; other factors such as the number of frames, test model, and hardware environment were kept the same.

[0112] The experiment has four dependent variables that can indicate compression efficiency: encoding time, decoding time, bitstream size, and mean square error (MSE) between the original sequence and the reconstructed sequence. These variables can have different weights in different application scenarios.

[0113] In this experiment, we chose to use them with the same weight to obtain more general results. Considering their different numerical scales and units, we used a relative change value to evaluate the effect; that is, we used the compression method with all I-frames as the benchmark and evaluated the improvement effect by comparing it to other methods. The expression for the improvement value is as follows:

[0114]

[0115] Where v is the value of any one of the four dependent variables mentioned above, v all-intra The value of the dependent variable is given when a compression method using only I-frames is employed.

[0116] After obtaining the improvement value for each dependent variable, summing them together yields the overall improvement. Since the compression method using only I-frames is used as the baseline, the base value is 4. If the value is greater than 4, it can be inferred that this method is generally more efficient than the compression method using only I-frames; if the value is less than 4, it can be inferred that this method is generally less efficient.

[0117] The SSIM calculated in this application can be implemented and used for motion estimation using Python 3.8. The dataset used in the implementation example is the 8iVSLTF (8i Voxelized Surface Light Field) dataset, which contains dynamic point clouds of human bodies in four different poses, all characterized by minimal changes in the main subject and minor changes in details, serving as a substitute for digital holographic human point cloud data. To simplify preprocessing, the .png images provided in the dataset are used as 2D mapping images, such as... Figure 2 The image shown is a frontal view of a human body. Considering the structure and movement of the human body, a frontal view is taken... Figure 1 Generally, it can reflect human movements and is suitable as a two-dimensional mapping image.

[0118] SSIM is used to calculate the similarity between two images. However, the number of frames in a dynamic point cloud is much greater than two frames. Therefore, to represent the overall motion of the dynamic point cloud, the SSIM values ​​of all adjacent frames are summed and then averaged. The motion estimation implementation method is as follows:

[0119] S1. Read the PNG image mapping of two adjacent frames;

[0120] S2. Calculate the SSIM values ​​of the two images;

[0121] S3. Repeat the above two steps to obtain all SSIM values ​​of adjacent frames;

[0122] S4. Calculate the average SSIM value to obtain the SSIM of the dynamic point cloud.

[0123] After implementing motion estimation computation, the performance of motion estimation was tested experimentally. The 8iVSLF dataset contains four dynamic point cloud sequences: loot, redandblack, longdress, and soldier. Figure 2 The visualization results are shown below. Each human dynamic point cloud was acquired at a rate of 30 frames per second for a total duration of approximately 10 seconds. Due to time and hardware limitations, only 3 frames were used for compression in this experiment.

[0124] For each sequence, two sets of dynamic point clouds with consecutive different SSIM values ​​were selected and compressed using different methods. One set had high SSIM, and the other had low SSIM. Three compression models were used for each set: 3D-based compression, 2D compression based on all I-frames, and 2D compression based on one I-frame and two P-frames. The improvement was evaluated using encoding time, decoding time, bitstream size, and MSE value.

[0125] Experiments were conducted on four dynamic point clouds, with two sequences of large and small SSIM values ​​selected for each point cloud. The compression method based on 3D compensation and the compression method based on two P-frames (one I-frame) were compared as a benchmark. A total of 24 experimental groups were conducted, comprising 4 datasets × 2 sets of SSIM sequences × 3 compression methods. All evaluation metrics related to the experimental groups were recorded, such as encoding time (seconds), decoding time (seconds), MSE, and bitstream size (bytes). Table 1 below shows the performance using a soldier point cloud as an example.

[0126] To more directly analyze the improvement of each indicator, Figure 4 and Figure 5 The visualization results of the improved calculations for the soldier are shown. The first column represents encoding all 3 frames as I-frames, used as a baseline; the second column represents adding 3D compensation during encoding and decoding; the third column represents encoding 1 frame as an I-frame and the other 2 frames as P-frames, thus eliminating a significant amount of temporal redundancy. Experiments calculated improvements in encoding time, decoding time, bitstream size, and MSE (Mean Sequence Size). The overall improvement is a simple sum of all other improvements, used to represent a comprehensive evaluation. According to the definition of improvement introduced earlier, the smaller the value, the better the performance compared to all I-frame encoding methods. Therefore, if a column height is lower than the first column, it indicates an improvement in the corresponding performance.

[0127] Table 1

[0128]

[0129] Experimental results demonstrate that SSIM is an excellent method for representing the motion of 3D objects. Point clouds with high SSIM values ​​show better overall improvement than those with low SSIM values, indicating that the algorithm removes some temporal redundancy caused by 3D continuity, provides an accurate estimate of point cloud motion, and offers a good reference for determining whether there is significant redundancy that needs to be removed. For example, for point clouds of dynamic human figures, an SSIM value above 85 is considered relatively high and can be interpreted as indicating substantial redundancy.

[0130] For each single point cloud, 3D compensation primarily reduces encoding time but increases decoding time, and using more P-frames further increases both encoding and decoding time. This is because encoding and decoding P-frames requires calculating the difference between the current and previous frames, which consumes additional time compared to encoding I-frames. However, the bitstream size and MSE are reduced, providing better performance for reconstructing the point cloud. If the 3D object moves too fast to calculate the difference, the encoding time for generating P-frames will be very long, potentially even twice that of all I-frame conditions. In this case, adding P-frames in GOF is unwise.

[0131] contrast Figure 4 and Figure 5 The high and low SSIM histograms shown clearly demonstrate that high SSIM point clouds perform better in removing temporal redundancy. These point clouds exhibit a slow motion trend because their 2D mappings are highly similar. Therefore, the differences stored in P-frames can be calculated quickly with minimal loss of information. With higher SSIM values, the algorithm effectively removes temporal redundancy. Since the temporal redundancy of dynamic point clouds stems from the 3D continuity of motion, it can be inferred that SSIM can evaluate moving 3D objects.

[0132] In summary, the experimental results show that SSIM is a suitable metric for evaluating 3D human motion and can also be applied to point cloud data of digital holographic humans. High SSIM values ​​indicate significant temporal redundancy between different frames, which can be eliminated by adding more P-frames.

[0133] The method of this application can quantitatively estimate the motion trend of the dynamic point cloud of a holographic digital human and guide the modification of parameter settings in the compression encoding, thereby improving the compression ratio and further improving the transmission efficiency.

[0134] The motion estimation method for detecting 3D redundancy in dynamic point cloud data of holographic digital humans provided in this application can quickly calculate the differences stored in P frames by calculating the SSIM value, with minimal loss of information. When the SSIM value is high, the algorithm can effectively remove temporal redundancy. Since the temporal redundancy of dynamic point clouds originates from the 3D continuity of motion, it can be inferred that SSIM can evaluate moving 3D objects. Experimental results show that SSIM is a suitable metric for evaluating the motion of 3D objects. High SSIM values ​​indicate a large amount of temporal redundancy between different frames, which can be eliminated by adding more P frames. Therefore, the method of this invention can quantitatively estimate the motion trend of dynamic point clouds and guide the modification of parameter settings in compression coding, improving the compression ratio and further enhancing transmission efficiency.

[0135] This application also provides a point cloud encoding processing apparatus. See also... Figure 6 , Figure 6 This is a structural diagram of the point cloud encoding processing apparatus provided in the embodiments of this application. Since the principle of the point cloud encoding processing apparatus in solving the problem is similar to the point cloud encoding processing method in the embodiments of this application, the implementation of this point cloud encoding processing apparatus can refer to the implementation of the method, and the repeated parts will not be described again.

[0136] like Figure 6 As shown, the point cloud encoding processing device 600 includes:

[0137] The acquisition module 601 is used to acquire two-dimensional mapping images of each frame of the point cloud in the dynamic point cloud, and obtain multiple frames of two-dimensional mapping images;

[0138] The first determining module 602 is used to determine the similarity between each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, and obtain N similarity values, where N is an integer greater than 1;

[0139] The second determining module 603 is used to determine the similarity parameters of the dynamic point cloud based on the N similarity values;

[0140] The adjustment module 604 is used to adjust the encoding parameters of the dynamic point cloud according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames.

[0141] Optionally, the second determining module is used to calculate the average of the N similarity values ​​to determine the overall average similarity of the dynamic point cloud.

[0142] Optionally, the point cloud encoding processing device 600 further includes:

[0143] The segmentation module is used to segment the dynamic point cloud to obtain M segment point clouds, where M is an integer greater than 1;

[0144] The third determining module is used to determine the average segment similarity of the first segment point cloud based on N1 similarities corresponding to the first segment point cloud among the N similarity values, wherein the first segment point cloud is any segment of the M segment point cloud, N1 is a value related to the number of frames of the first segment point cloud, and N1 is an integer greater than 1 and less than N.

[0145] The adjustment module 604 is used to adjust the encoding parameters of the first segment point cloud based on the segment average similarity and the overall average similarity.

[0146] Optionally, the adjustment module 604 includes:

[0147] The first adjustment unit is used to reduce the number of prediction frames used for encoding the first segment point cloud when the average similarity of the segment is higher than the overall average similarity.

[0148] The second adjustment unit is used to increase the number of prediction frames used for encoding the first segment point cloud when the average similarity of the segment is lower than the overall average similarity.

[0149] Optionally, the first determining module 602 is used to determine the SSIM value between each pair of adjacent two-dimensional mapping images based on the target parameters of each pair of adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, wherein the target parameters include at least one of brightness, contrast and structural parameters.

[0150] The point cloud encoding processing device 600 provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0151] The point cloud encoding processing apparatus 600 of this application embodiment acquires two-dimensional mapping images of each frame of a dynamic point cloud to obtain multiple frames of two-dimensional mapping images; determines the similarity between each adjacent pair of two-dimensional mapping images in the multiple frames of two-dimensional mapping images to obtain N similarity values, where N is an integer greater than 1; determines similarity parameters of the dynamic point cloud based on the N similarity values; and adjusts the encoding parameters of the dynamic point cloud according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames. In this way, by using the similarity parameters of the dynamic point cloud to guide the adjustment of its encoding parameters, users can quickly find suitable encoding parameters without having to conduct multiple repeated experiments.

[0152] This application also provides an electronic device. Since the principle by which the electronic device solves the problem is similar to the point cloud encoding processing method in this application, the implementation of this electronic device can be found in the implementation of the method, and repeated details will not be described again. Figure 7 As shown, the electronic device according to an embodiment of this application includes:

[0153] Processor 700 is used to read the program from memory 720 and execute the following procedures:

[0154] Obtain the two-dimensional mapping images of each frame of the point cloud in the dynamic point cloud to obtain multiple frames of two-dimensional mapping images;

[0155] Determine the similarity between each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images to obtain N similarity values, where N is an integer greater than 1;

[0156] Based on the N similarity values, the similarity parameters of the dynamic point cloud are determined;

[0157] The encoding parameters of the dynamic point cloud are adjusted according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames.

[0158] Among them, Figure 7 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 700) and memory (memory 720). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. Processor 700 is responsible for managing the bus architecture and general processing, and memory 720 can store data used by processor 700 during operation.

[0159] Optionally, the processor 700 is also used to read the program from the memory 720 and perform the following steps:

[0160] The average of the N similarity values ​​is calculated to determine the overall average similarity of the dynamic point cloud.

[0161] Optionally, the processor 700 is also used to read the program from the memory 720 and perform the following steps:

[0162] The dynamic point cloud is segmented to obtain M segment point clouds, where M is an integer greater than 1;

[0163] Based on the N1 similarity values ​​corresponding to the first segment of the point cloud among the N similarity values, the average segment similarity of the first segment of the point cloud is determined, wherein the first segment of the point cloud is any segment of the M segment of the point cloud, N1 is a value related to the number of frames of the first segment of the point cloud, and N1 is an integer greater than 1 and less than N.

[0164] The encoding parameters of the first segment point cloud are adjusted based on the average similarity of the segment and the overall average similarity.

[0165] Optionally, the processor 700 is also used to read the program from the memory 720 and perform the following steps:

[0166] If the average similarity of the segment is higher than the overall average similarity, reduce the number of prediction frames used for encoding the first segment point cloud.

[0167] If the average similarity of the segment is lower than the overall average similarity, the number of prediction frames used for encoding the first segment point cloud is increased.

[0168] Optionally, the processor 700 is also used to read the program from the memory 720 and perform the following steps:

[0169] Based on the target parameters of each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, the structural similarity (SSIM) value between each two adjacent two-dimensional mapping images is determined, wherein the target parameters include at least one of brightness, contrast and structural parameters.

[0170] Optionally, the dynamic point cloud is a holographic digital human dynamic point cloud.

[0171] The electronic device provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0172] Furthermore, the computer-readable storage medium of this application embodiment is used to store a computer program, which can be executed by a processor. Figure 1 The steps in the method embodiment shown.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0175] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0176] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A point cloud encoding processing method, characterized in that, include: Obtain the two-dimensional mapping images of each frame of the point cloud in the dynamic point cloud to obtain multiple frames of two-dimensional mapping images; The dynamic point cloud is a three-dimensional dynamic point cloud composed of multiple consecutive static point clouds. Determine the similarity between each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images to obtain N similarity values, where N is an integer greater than 1; Based on the N similarity values, the similarity parameters of the dynamic point cloud are determined; Based on the similarity parameters, the encoding parameters of the dynamic point cloud are adjusted, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames; The step of determining the similarity parameters of the dynamic point cloud based on the N similarity values ​​includes: The average of the N similarity values ​​is calculated to determine the overall average similarity of the dynamic point cloud; Before adjusting the encoding parameters of the dynamic point cloud based on the similarity parameters, the method further includes: The dynamic point cloud is segmented to obtain M segment point clouds, where M is an integer greater than 1; Based on the N1 similarity values ​​corresponding to the first segment of the point cloud among the N similarity values, the average segment similarity of the first segment of the point cloud is determined, wherein the first segment of the point cloud is any segment of the M segment of the point cloud, N1 is a value related to the number of frames of the first segment of the point cloud, and N1 is an integer greater than 1 and less than N. The step of adjusting the encoding parameters of the dynamic point cloud based on the similarity parameters includes: The encoding parameters of the first segment point cloud are adjusted based on the average similarity of the segment and the overall average similarity.

2. The method according to claim 1, characterized in that, The step of adjusting the encoding parameters of the first segment point cloud based on the segment average similarity and the overall average similarity includes: If the average similarity of the segment is higher than the overall average similarity, reduce the number of prediction frames used for encoding the first segment point cloud. If the average similarity of the segment is lower than the overall average similarity, the number of prediction frames used for encoding the first segment point cloud is increased.

3. The method according to any one of claims 1 to 2, characterized in that, Determining the similarity between any two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping image includes: Based on the target parameters of each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, the structural similarity (SSIM) value between each two adjacent two-dimensional mapping images is determined, wherein the target parameters include at least one of brightness, contrast and structural parameters.

4. The method according to claim 1, characterized in that, The dynamic point cloud is a holographic digital human dynamic point cloud.

5. A point cloud encoding processing device, characterized in that, include: The acquisition module is used to acquire two-dimensional mapping images of each frame of the point cloud in the dynamic point cloud, and obtain multiple frames of two-dimensional mapping images; The dynamic point cloud is a three-dimensional dynamic point cloud composed of multiple consecutive static point clouds. The first determining module is used to determine the similarity between each two adjacent two-dimensional mapping images in the multi-frame two-dimensional mapping images, and obtain N similarity values, where N is an integer greater than 1; The second determining module is used to determine the similarity parameters of the dynamic point cloud based on the N similarity values; An adjustment module is used to adjust the encoding parameters of the dynamic point cloud according to the similarity parameters, wherein the encoding parameters include at least one of the number of keyframes and the number of prediction frames; The second determining module is used to determine the overall average similarity of the dynamic point cloud; The segmentation module is used to segment the dynamic point cloud to obtain M segment point clouds, where M is an integer greater than 1; The third determining module is used to determine the average segment similarity of the first segment point cloud based on N1 similarities corresponding to the first segment point cloud among the N similarity values, wherein the first segment point cloud is any segment of the M segment point cloud, N1 is a value related to the number of frames of the first segment point cloud, and N1 is an integer greater than 1 and less than N. The adjustment module is used to adjust the encoding parameters of the first segment point cloud based on the segment average similarity and the overall average similarity.

6. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program in the memory to implement the steps of the point cloud encoding processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the point cloud encoding processing method as described in any one of claims 1 to 4.

Images