A dynamic human-computer interaction point cloud compression method

Abstract

The application relates to a dynamic human-computer interaction point cloud compression method, comprising the following steps: calculating a maximum change coordinate system of adjacent point cloud frames in a human-computer interaction three-dimensional point cloud frame sequence based on a multi-layer projection method; segmenting the three-dimensional point cloud frame sequence; for each segment of the three-dimensional point cloud frame, calculating a fusion coordinate system according to the maximum change coordinate system of the adjacent point cloud frames; projecting each point cloud frame of the segment of the three-dimensional point cloud frame on a projection direction of the fusion coordinate system to obtain a complete projection image of each point cloud frame on each projection direction, and compressing the segment of the three-dimensional point cloud frame according to the similarity of the complete projection images of the adjacent two point cloud frames on each projection direction.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction point cloud technology, and in particular to a dynamic human-computer interaction point cloud compression method. Background Technology

[0002] Human-computer interaction point clouds can be divided into two types: static point clouds and dynamic human-computer interaction point clouds. A static point cloud consists of a set of points with three-dimensional positional information, each point having one or more attributes to store additional information. A dynamic human-computer interaction point cloud consists of multiple consecutive static point clouds, where each static point cloud is called a frame of the dynamic human-computer interaction point cloud. The positional information of points in a point cloud is generally represented by (x, y, z) coordinates in a Cartesian coordinate system, used to characterize the spatial topology and reconstruct the shape of three-dimensional objects. A common attribute of point clouds is color information, typically stored using RGB values ​​to store the red, green, and blue channels, or using intensity values ​​to record the amplitude of pulse returns to occupy the color field.

[0003] Point clouds play a crucial role in future real-time holographic human-computer interaction scenarios. Devices based on LiDAR and photogrammetry can rapidly acquire large-scale, high-frequency changing real-time point cloud information for human-computer interaction. However, due to the uneven distribution of computing and storage resources across networks, the primary requirements are to transmit data to the user end for display, to the computing center for processing, and to the storage center for storage. But point cloud data is massive, unstructured, and lacks metadata; direct transmission would consume enormous bandwidth, becoming a major constraint on point cloud transmission. Therefore, point cloud compression is necessary to reduce data volume.

[0004] Currently, compression methods for dynamic human-computer interaction point clouds can be broadly categorized into two types: compression methods based on 3D structures and compression methods based on 2D mappings. Compression methods based on 3D structures directly study the compression algorithm for the point cloud, using structured algorithms for traversal and matching. Most of these methods are based on octree data structures, and a common problem is that the computational cost increases exponentially with the depth of the octree. This problem is particularly severe for compressing high-precision, high-detail point cloud sequences. Compression methods based on 2D mappings basically project the point cloud data onto a 2D plane according to certain rules, and then use existing mature video encoding and decoding technologies to compress and encode the 2D mapping sequence of the dynamic human-computer interaction point cloud. Compared with 3D structure-based compression algorithms, these algorithms significantly improve compression efficiency in both time and space, achieving remarkable results. In 2017, the MPEG (Moving Picture Experts Group) under ISO / IEC JTC1 established relevant international standards and released TMC2 (Test Model Category 2) for dynamic human-computer interaction point cloud compression algorithms. They adopted a compression method based on two-dimensional mapping to compress point clouds, which has high compression efficiency. However, because the three-dimensional point cloud is mapped to two-dimensional space, the temporal and spatial continuity in the three-dimensional space is completely destroyed, resulting in the loss of some information and low compression efficiency. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide a dynamic human-computer interaction point cloud compression method to solve the problems of computational complexity and low compression efficiency of existing dynamic human-computer interaction point cloud compression methods.

[0006] On one hand, embodiments of the present invention provide a dynamic human-computer interaction point cloud compression method, including the following steps:

[0007] Calculate the coordinate system of the maximum change between adjacent point cloud frames in a human-computer interaction 3D point cloud frame sequence based on the multi-layer projection method;

[0008] The three-dimensional point cloud frame sequence is segmented; for each segment of the three-dimensional point cloud frame, a fusion coordinate system is calculated based on the coordinate system of the maximum change of adjacent point cloud frames; each point cloud frame of the segment is projected onto the projection direction of the fusion coordinate system to obtain a complete projection image of each point cloud frame in each projection direction; the segment of the three-dimensional point cloud frame is compressed based on the similarity of the complete projection images of two adjacent point cloud frames in each projection direction.

[0009] Based on further improvements to the above technical solution, a multi-layer projection algorithm is used to calculate the coordinate system of maximum change between adjacent point cloud frames in a 3D point cloud frame sequence, including:

[0010] S11. For each point cloud frame in the three-dimensional point cloud frame sequence, establish multiple projection coordinate systems with its centroid as the origin; based on the layered projection method, project each point cloud frame onto the projection directions of the corresponding multiple projection coordinate systems.

[0011] S12. Take the second point cloud frame in the point cloud frame sequence as the current point cloud frame;

[0012] S13. For each projection direction of each projection coordinate system corresponding to the current point cloud frame, calculate the chamfer distance between the current point cloud frame and the previous frame of the current point cloud frame.

[0013] S14. The projection direction corresponding to the maximum chamfer distance is the first projection direction. Among all projection coordinate systems other than the first projection direction, the projection direction with the largest chamfer distance is the second projection direction. The coordinate system containing the first and second projection directions is the coordinate system with the largest change from the previous frame to the current point cloud frame. Take the next point cloud frame as the current frame and return to step S13 until all point cloud frames have been traversed.

[0014] Furthermore, based on the layered projection method, each point cloud frame is projected onto the projection directions of multiple corresponding projection coordinate systems, including:

[0015] S1121. For the j-th projection coordinate system of the current point cloud frame, construct a set of points to be projected from all points in the current point cloud frame; let k = 1, L = 1;

[0016] S1122, take the k-th coordinate axis of the j-th projected coordinate system as the reference axis;

[0017] S1123. Sort each point in the set of points to be projected in descending order of their values ​​on the reference coordinate axis. For each sorted point, if a corresponding point already exists in the Lth layer of the projection direction corresponding to the reference axis, then do not project that point. Otherwise, project the point to the corresponding position in the Lth layer of the projection direction corresponding to the reference axis and delete that point from the set of points to be projected. If the set of points to be projected is empty, then the projection ends. Otherwise, proceed to step S1124.

[0018] S1124, k = k + 1. If k = 4, then L = L + 1. Set k to 1 and return to step S1122; otherwise, return directly to step S1122.

[0019] Furthermore, for each projection direction in each projection coordinate system, the chamfer distance between the current point cloud frame and the previous frame is calculated in the following manner:

[0020] For the current projection direction O i X jFind the projection of the current point cloud frame onto the current projection direction O. i X j The points on the O form the first set of points; where O i X j O represents the coordinate system of the j-th projection coordinate system of the i-th point cloud frame. i X j The projection direction of the axis;

[0021] Find the previous frame of the current point cloud frame projected in the projection direction O. i-1 X j The points on the O form the second set of points; i-1 X j O represents the coordinate system of the j-th projection coordinate system of the (i-1)-th point cloud frame. i-1 X j The projection direction of the axis;

[0022] Calculate the chamfer distance between the first point set and the second point set to obtain the chamfer distance between the current point cloud frame and the previous frame in the current projection direction.

[0023] Furthermore, the chamfer distance d(S1,S2) is calculated using the following formula:

[0024]

[0025] Where S1 represents the first set of points, Let S1 represent the number of points, and S2 represent the second set of points. This represents the number of points in S2. This represents the distance from point x to point y. This represents the distance from point y to point x.

[0026] Furthermore, for each point cloud frame in the 3D point cloud frame sequence, multiple projected coordinate systems are established with its centroid as the origin, including:

[0027] Establish a coordinate system with the centroid of the current point cloud frame as the origin, parallel to the coordinate axes of the reference coordinate system and in the same direction, as the first coordinate system;

[0028] Fix each coordinate axis in the first coordinate system, and rotate the two coordinate axes other than the fixed coordinate axis around the fixed coordinate axis according to the rotation step size to obtain multiple coordinate systems that are the same as the X-axis, Y-axis or Z-axis of the first projected coordinate system;

[0029] The first coordinate system and multiple coordinate systems that are identical to its X-axis, Y-axis or Z-axis constitute multiple projected coordinate systems for the current point cloud frame.

[0030] Furthermore, by projecting each point cloud frame of the 3D point cloud segment onto the projection direction of the fused coordinate system, a complete projected image of each point cloud frame in each projection direction is obtained, including:

[0031] The layered projection method is used to project each point of each point cloud frame in the segment of the three-dimensional point cloud frame onto the projection direction corresponding to the fusion coordinate system, so as to obtain a multi-layer block projection map of each point cloud frame in each projection direction corresponding to the fusion coordinate system.

[0032] By stitching together the multi-layered block projection images of each point cloud frame in each projection direction, a complete projection image of each point cloud frame in each projection direction is obtained.

[0033] Furthermore, the 3D point cloud frame is compressed based on the similarity of the complete projected images of two adjacent point cloud frames in each projection direction, including:

[0034] The first and last point cloud frames in the current segment of the 3D point cloud frame are both keyframe images in each projection direction;

[0035] For other point cloud frames in the current segment of the 3D point cloud frame, calculate the structural similarity between the point cloud frame and the complete projected image of the previous point cloud frame in each projection direction; if the structural similarity does not exceed the second threshold, then the complete projected image is used as the key frame image; all key frame images of the segment of the 3D point cloud frame constitute the compressed image of the segment of the 3D point cloud frame.

[0036] Furthermore, the structural similarity between the point cloud frame and the complete projected image of its predecessor point cloud frame in each projection direction is calculated using the following method:

[0037] For the current projection direction, according to the formula Calculate the SSIM value of the complete projected image of the point cloud frame and its predecessor point cloud frame on the kth image channel;

[0038] Where, μ k,a μ represents the element mean of the k-th channel of the complete projected image of the point cloud frame. k,b σ represents the mean of the elements in the k-th channel of the complete projected image of the previous point cloud frame; k,a σ represents the element variance of the k-th channel of the complete projected image of the point cloud frame. k,b σ represents the element variance of the k-th channel of the complete projected image of the previous point cloud frame. k,ab c1 and c2 represent the elemental covariance of the k-th channel of the complete projected image of the point cloud frame and the complete projected image of the previous point cloud frame, where c1 and c2 are constants.

[0039] The SSIM values ​​of all image channels are summed to obtain the structural similarity value of the point cloud frame and the complete projected image of the previous point cloud frame in the current projection direction.

[0040] Furthermore, for each 3D point cloud frame, a fused coordinate system is calculated based on the coordinate system of the maximum change between adjacent point cloud frames, including:

[0041] For the current segment point cloud frame, shift all the maximum change coordinate system translation values ​​corresponding to the current segment point cloud frame to the origin so that they coincide with the origin of the reference coordinate system;

[0042] For each coordinate axis of each coordinate system with the largest change, assign it to the type corresponding to the reference coordinate with which it has the smallest angle;

[0043] The unit vectors of the coordinate axes corresponding to each type are summed to obtain the coordinate axes of the fused coordinate system corresponding to that type; thus, the fused coordinate system corresponding to the current segment of the 3D point cloud frame is obtained.

[0044] Compared with existing technologies, this invention calculates the maximum change coordinate system of adjacent point cloud frames in the human-computer interaction 3D point cloud frame sequence to obtain the change direction of connected point cloud frames; for each 3D point cloud frame, a fusion coordinate system is calculated based on the maximum change coordinate system of adjacent point cloud frames, and the point cloud frame is projected onto the projection direction corresponding to the fusion coordinate system, so that the temporal and spatial correlation of the point cloud frames is still preserved after projection, so that the spatiotemporal consistency of the point cloud frames is preserved during compression, thereby improving the compression efficiency.

[0045] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from the description and drawings, which are particularly pointed out. Attached Figure Description

[0046] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0047] Figure 1 This is a flowchart of a dynamic human-computer interaction point cloud compression method according to an embodiment of the present invention. Detailed Implementation

[0048] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0049] A specific embodiment of the present invention discloses a dynamic human-computer interaction point cloud compression method, such as... Figure 1 As shown, it includes the following steps:

[0050] S1. Calculate the coordinate system of the maximum change between adjacent point cloud frames in the human-computer interaction 3D point cloud frame sequence based on the multi-layer projection method;

[0051] S2. Segment the 3D point cloud frame sequence; for each segment of the 3D point cloud frame, calculate the fusion coordinate system based on the coordinate system of the maximum change of adjacent point cloud frames; project each point cloud frame of the segment of the 3D point cloud frame onto the projection direction of the fusion coordinate system to obtain the projection image of each point cloud frame in each projection direction, and compress the segment of the 3D point cloud frame based on the similarity of the projection images of two adjacent point cloud frames in each projection direction.

[0052] Compared with existing technologies, this invention calculates the maximum change coordinate system of adjacent point cloud frames in the human-computer interaction 3D point cloud frame sequence to obtain the change direction of connected point cloud frames; for each 3D point cloud frame, a fusion coordinate system is calculated based on the maximum change coordinate system of adjacent point cloud frames, and the point cloud frame is projected onto the projection direction corresponding to the fusion coordinate system, so that the temporal and spatial correlation of the point cloud frames is still preserved after projection, so that the spatiotemporal consistency of the point cloud frames is preserved during compression, thereby improving the compression efficiency.

[0053] Specifically, step S1 calculates the coordinate system of the maximum change between adjacent point cloud frames in the 3D point cloud frame sequence based on a multi-layer projection algorithm, including:

[0054] S11. For each point cloud frame in the 3D point cloud frame sequence, establish multiple projection coordinate systems with its centroid as the origin; based on the layered projection method, project each point cloud frame onto the projection directions of the corresponding multiple projection coordinate systems.

[0055] S12. Take the second point cloud frame in the point cloud frame sequence as the current point cloud frame;

[0056] S13. For each projection direction of each projection coordinate system corresponding to the current point cloud frame, calculate the chamfer distance between the current point cloud frame and the previous frame of the current point cloud frame.

[0057] S14. The projection direction corresponding to the maximum chamfer distance is the first projection direction. Among all projection coordinate systems other than the first projection direction, the projection direction with the largest chamfer distance is the second projection direction. The coordinate system containing the first and second projection directions is the coordinate system with the largest change from the previous frame to the current point cloud frame. Take the next point cloud frame as the current frame and return to step S13 until all point cloud frames have been traversed.

[0058] Specifically, in step S11, for each point cloud frame in the 3D point cloud frame sequence, multiple projected coordinate systems are established with its centroid as the origin, including:

[0059] S1111. Establish a coordinate system with the centroid of the current point cloud frame as the origin, parallel to the coordinate axes of the reference coordinate system and in the same direction, as the first coordinate system.

[0060] S1112. Fix one of the coordinate axes in the first coordinate system in turn, and rotate the two coordinate axes other than the fixed coordinate axis multiple times around the fixed coordinate axis according to the rotation step size. Each rotation results in a coordinate system, thus obtaining multiple coordinate systems that are the same as the X-axis, Y-axis or Z-axis of the first projected coordinate system.

[0061] S1113, the first coordinate system and multiple coordinate systems that are the same as its X-axis, Y-axis or Z-axis constitute multiple projected coordinate systems of the current point cloud frame.

[0062] It should be noted that the reference coordinate system is a reference coordinate system; for example, the world coordinate system can be used as the reference coordinate system.

[0063] During implementation, for the i-th point cloud frame, the centroid of the point cloud is first calculated, and a coordinate system parallel to and aligned with the coordinate axes of the reference coordinate system is established as the first coordinate O. i X1Y1Z1.

[0064] O i With the X-axis fixed in the X1Y1Z1 coordinate system, the Y-axis and Z-axis are rotated around the X-axis according to the rotation step size to obtain multiple coordinate systems with the same X-axis; O i With the Y-axis of the X1Y1Z1 coordinate system fixed, the X and Z axes are rotated around the Y-axis according to the rotation step size to obtain multiple coordinate systems with the same Y-axis; O i The Z-axis in the X1Y1Z1 coordinate system is fixed. The X-axis and Y-axis are rotated around the Z-axis according to the rotation step size to obtain multiple coordinate systems with the same Z-axis. The first projected coordinate system and the multiple coordinate systems obtained by rotation that are the same as its X-axis, Y-axis or Z-axis constitute the multiple projected coordinate systems of the i-th point cloud frame.

[0065] During implementation, the rotation step size can be determined based on the calculation speed and accuracy of the changing direction. If high calculation accuracy is required, the rotation step size can be set to a smaller number, such as 10 degrees. If high calculation speed is required, the rotation step size can be set to a larger number, such as 30 degrees.

[0066] During implementation, the projected coordinate systems of each point cloud frame are sorted in the same way, ensuring that the three coordinate axes of the j-th coordinate system of any two point cloud frames are parallel and have the same direction. That is, after translating the j-th coordinate system of the i-th point cloud frame to the centroid of the k-th point cloud frame, it completely coincides with the j-th coordinate system of the k-th point cloud frame.

[0067] Each projection coordinate system of each point cloud frame corresponds to multiple projection directions. In one embodiment of the present invention, one projection coordinate system corresponds to three projection directions. For example, for the j-th coordinate system O of the i-th point cloud frame... i X j Y j Z j The three projection directions are O i X j The direction of the axis, O i Z j The direction of the axis and O i Y j The direction of the axis.

[0068] Specifically, in step S11, each point cloud frame is projected onto the projection directions of the corresponding multiple projection coordinate systems based on the layered projection method, including:

[0069] S1121. For the j-th projection coordinate system of the current point cloud frame, construct a set of points to be projected from all points in the current point cloud frame; let k = 1, L = 1;

[0070] It should be noted that k represents the k-th coordinate axis of the j-th projected coordinate system, and L represents the first projection layer of the j-th projected coordinate system. Initially, k = 1 and L = 1.

[0071] S1122, take the k-th coordinate axis of the j-th projected coordinate system as the reference axis;

[0072] S1123. Sort each point in the set of points to be projected in descending order of their values ​​on the reference coordinate axis. For each sorted point, if a corresponding point already exists in the Lth layer of the projection direction corresponding to the reference axis, then do not project that point. Otherwise, project the point to the corresponding position in the Lth layer of the projection direction corresponding to the reference axis and delete that point from the set of points to be projected. If the set of points to be projected is empty, then the projection ends. Otherwise, proceed to step S1124.

[0073] S1124, k = k + 1. If k = 4, then L = L + 1. Set k to 1 and return to step S1122; otherwise, return directly to step S1122.

[0074] During implementation, for the j-th coordinate system O of the current point cloud frame (e.g., the i-th point cloud frame) i X j Y j Z j The set of points to be projected is formed by all the points in the current point cloud frame.

[0075] Set the j-th coordinate system O i X j Yj Z j The first coordinate axis is used as the reference axis, for example, the X-axis is used as the reference axis, and the projection direction is O. i X j The direction of the axis is determined by sorting each point in the set of points to be projected in descending order of their coordinate values ​​on the X-axis, prioritizing points with larger coordinate values ​​for projection. Then, following the sorted order, each point is checked sequentially to determine if it can be projected onto the O-axis. i X j The first layer in the projection direction of the axis.

[0076] The first point after sorting can necessarily be projected onto the first layer in the axial direction. Projecting this point onto O i X j The first layer in the axial direction, mark the point at O. i X j The projection position of the first layer along the axis is occupied, so this point is removed from the set of points to be projected. Starting from the second point, it is necessary to determine whether the point is in position O. i X j If the projection position of the first layer in the axial direction is occupied, and if it is occupied (meaning a point is already projected onto that position), then the point will not be projected and will remain in the set of points to be projected. Otherwise, the point will be projected onto O. i X j The first layer in the axial direction, mark the point at O. i X j The projection position of the first layer in the axial direction is occupied, so this point is removed from the set of points to be projected.

[0077] After traversing the set of points to be projected once, if the set is empty, the projection ends; if the set is not empty, meaning there are points that have not been projected, then the j-th coordinate system O is... i X j Y j Z j The second coordinate axis is used as the reference axis, for example, the Y-axis is used as the reference axis, and the projection direction is O. i Y j The direction of the axis is determined by sorting each point in the set of points to be projected in descending order of their Y-axis coordinates, prioritizing points with larger coordinate values. Then, following the sorted order, each point is checked sequentially to determine if it can be projected onto point O. i Y j For the first layer along the axis, please refer to the previous content for the specific judgment process. After traversing all the sets of points to be projected, if the set of points to be projected is empty, the projection ends; if the set of points to be projected is still not empty, then the j-th coordinate system O is... i X j Y j Z jUsing the third coordinate axis, the Z-axis, as the reference axis, the points in the set to be projected are projected onto the third projection direction O following the same process. i Z j On the first layer in the axial direction.

[0078] O i X j Y j Z j After iterating through all three coordinate axes as reference axes, some points may still not be projected due to occlusion. That is, the set of points to be projected is still not empty. So, the layer number is incremented by one, i.e., L is incremented by 1. Following the above process, the points in the set of points to be projected are projected onto the second layer of the three projection directions. If there are still unprojected points, the points in the set of points to be projected are projected onto the third layer of the three projection directions, and so on, until the set of points to be projected is empty.

[0079] Compared with existing technologies, this invention uses a layered projection method to project point clouds onto multiple layers in the projection direction, thereby taking into account the occlusion relationship of points and providing a basis for accurately calculating the change direction of point cloud frames.

[0080] Specifically, in step S13, for each projection direction of each projection coordinate system, the chamfer distance between the current point cloud frame and the previous frame is calculated in the following way:

[0081] For the current projection direction O i X j Find the projection of the current point cloud frame onto the current projection direction O. i X j The points on the O form the first set of points; where O i X j O represents the coordinate system of the j-th projection coordinate system of the i-th point cloud frame. i X j The projection direction of the axis.

[0082] Find the previous frame of the current point cloud frame projected in the projection direction O. i-1 X j The points on the O form the second set of points; i-1 X j O represents the coordinate system of the j-th projection coordinate system of the (i-1)-th point cloud frame. i-1 X j The projection direction of the axis;

[0083] Calculate the chamfer distance between the first point set and the second point set to obtain the chamfer distance between the current point cloud frame and the previous frame in the current projection direction.

[0084] For example, the current point cloud frame in the j-th projection coordinate system O i X j Yj Z j If 8 layers are projected downwards, then the projection onto O in the current point cloud frame is taken. i X j All points on all 8 layers of the direction constitute the first point set. The point cloud frame in the previous point cloud frame is projected onto its j-th projection coordinate system O. i-1 X j Y j Z j O i-1 X j Points on all layers (e.g., 10 layers) in a direction constitute the second set of points.

[0085] Specifically, the chamfer distance d(S1,S2) between the first set of points and the second set of points is calculated using the following formula:

[0086]

[0087] Where S1 represents the first set of points, Let S1 represent the number of points, and S2 represent the second set of points. This represents the number of points in S2. This represents the distance from point x to point y. This represents the distance from point y to point x.

[0088] The calculated chamfer distance between the first and second point sets represents the distance between the current point cloud frame (the i-th point cloud frame) and its predecessor point cloud frame in the j-th coordinate system. i X j The difference along the axis increases with the chamfer distance. The same process is used to calculate the difference between the current point cloud frame (the i-th point cloud frame) and its previous point cloud frame at O. i Y j Shaft and O i Z j Differences in the axial direction.

[0089] For the current point cloud frame, after calculating the chamfer distance for each projection direction in all projection coordinate systems, find the projection direction with the maximum chamfer distance (e.g., O). i X j The direction in which the coordinates are located is taken as the first projection direction. Since the first projection direction may correspond to multiple projection coordinate systems (and O), i X j Y j Z j O of the coordinate system i X j Since the projection coordinate systems are coaxial, the projection coordinate systems containing the first projection direction constitute a set of candidate coordinate systems. The candidate coordinate system set includes all systems except the one containing the first projection direction O. i X jAmong the other projection directions, the projection direction with the largest chamfer distance is the second projection direction. The coordinate system containing the first projection direction and the second projection direction is the coordinate system with the largest change from the previous frame to the current point cloud frame.

[0090] The first projection direction is the direction of the greatest change from the previous frame to the current point cloud frame.

[0091] After calculating the coordinate system of the maximum change between any two adjacent point cloud frames in the 3D point cloud frame sequence, the 3D point cloud frame sequence is segmented. In practice, the point cloud sequence can be segmented according to a fixed number of frames, for example, every 10 frames per segment.

[0092] Segmentation can also be based on the direction of maximum change between adjacent point cloud frames. Specifically, this includes:

[0093] S211. Take the first point cloud frame of the three-dimensional point cloud frame sequence as the starting point cloud frame of the first segment; take the first segment as the current segment; take the second point cloud frame as the current point cloud frame;

[0094] S212. If the angle between the maximum change direction of the current point cloud frame and the maximum change direction of the starting point cloud frame of the current segment is less than the first threshold, then add the current point cloud frame to the current segment; otherwise, take the current point cloud frame as the starting point cloud frame of a new segment, take the new segment as the current segment, take the next point cloud frame of the current point cloud frame as the current point cloud frame, and return to step S212; if there is no next point cloud frame, then the segmentation ends.

[0095] In practice, the first threshold can be determined based on the required accuracy of the calculation, for example, it can be set to 15 degrees.

[0096] That is, frames with basically the same direction of change are grouped into one segment, thereby improving the compression ratio and the accuracy of compressed data. For each segment of 3D point cloud frame, the coordinate systems of the most changed points cloud frames are fused together, which facilitates the compression of the point cloud.

[0097] Specifically, in step S2, for each 3D point cloud frame, the fused coordinate system is calculated based on the coordinate system of the maximum change between adjacent point cloud frames, including:

[0098] S221. For the current segment point cloud frame, translate all the coordinate systems with the largest changes corresponding to the current segment point cloud frame to coincide with the origin of the reference coordinate system.

[0099] During implementation, the coordinate system of the largest change in all adjacent point cloud frames in the current segment of the three-dimensional point cloud frame is translated to coincide with the origin O0 of the reference coordinate system O0X0Y0Z0.

[0100] S222. For each coordinate axis of each coordinate system with the largest change, classify it into the type corresponding to the reference coordinate axis with the smallest angle.

[0101] For each coordinate axis of the coordinate system with the largest change, determine the angle between it and the three coordinate axes of the reference coordinate system. If the current coordinate axis has the smallest angle with the O0X0 axis of the reference coordinate system, then the current coordinate axis is classified into the first category; if the current coordinate axis has the smallest angle with the O0Y0 axis of the reference coordinate system, then the current coordinate axis is classified into the second category; if the current coordinate axis has the smallest angle with the O0Z0 axis of the reference coordinate system, then the current coordinate axis is classified into the third category.

[0102] It should be noted that there may be a coordinate axis of the coordinate system with the maximum change that has the same and smallest angle with the two coordinate axes of the reference coordinate system. In this case, it is sufficient to ensure that the three coordinate axes of the current coordinate system with the maximum change belong to different categories.

[0103] S223. Sum the unit vectors of the coordinate axes corresponding to each type to obtain the coordinate axes of the fused coordinate system corresponding to that type; thus obtaining the fused coordinate system corresponding to the current segment of the 3D point cloud frame.

[0104] The unit vectors corresponding to the coordinate axes of the same type are summed to obtain the coordinate axes of the fused coordinate system. The fused coordinate system corresponding to the s-th segment of the 3D point cloud frame sequence is... express.

[0105] Specifically, in step S2, each point cloud frame of the 3D point cloud segment is projected onto the projection direction of the fused coordinate system to obtain a complete projection image of each point cloud frame in each projection direction, including:

[0106] S231. Using the layered projection method, each point of each point cloud frame in the segment of the three-dimensional point cloud frame is projected onto the projection direction corresponding to the fusion coordinate system, so as to obtain a multi-layer block projection map of each point cloud frame in each projection direction corresponding to the fusion coordinate system.

[0107] During implementation, for the i-th point cloud frame in the s-th segment of the 3D point cloud frame sequence, the coordinate system is first fused. Translate to the centroid of the point cloud frame to obtain the fused projection coordinate system corresponding to the i-th point cloud frame. The corresponding projection direction is The direction of the axis The direction of the axis and The direction in which it is located.

[0108] Since the fusion coordinate system of each point cloud frame is obtained by translating the fusion coordinate system corresponding to the current segment, the projection direction of each point cloud frame is the same as the projection direction of the fusion coordinate system corresponding to the current segment.

[0109] During implementation, the i-th point cloud frame in the s-th segment of the 3D point cloud frame sequence is projected onto the fused projection coordinate system using the same method as in steps S1121-S1124. In the corresponding projection direction, the multi-layer block projection map of the i-th point cloud frame in the s-th segment of the 3D point cloud frame sequence is obtained in each projection direction. For example, if the projection of the i-th point cloud frame is divided into 7 layers, then in Projection direction Projection direction and The projection directions correspond to the seven block projection diagrams.

[0110] It should be noted that each pixel in each projection map records the 3D coordinates and attribute index of the point in the point cloud frame projected to that location in the fused projection coordinate system, represented as (x, y, z, a). If no point is projected to that location, the corresponding pixel is (0, 0, 0, 0). In practice, the attribute can be color.

[0111] S232. Stitch together the multi-layer block projection images of each point cloud frame in each projection direction to obtain the complete projection image of each point cloud frame in each projection direction.

[0112] After obtaining the multi-layer block projection map of each point cloud frame in the current segment of the 3D point cloud frame sequence in each projection direction, for each point cloud frame, such as the i-th point cloud frame in the s-th segment of the 3D point cloud frame sequence, the multi-layer block projection map in each projection direction is stitched together to obtain the complete projection image of the i-th point cloud frame in the s-th segment of the 3D point cloud frame sequence on each projection plane.

[0113] In implementation, assume that the s-th segment of the 3D point cloud frame sequence contains 10 point cloud frames. For each block projection map of these 10 point cloud frames, a two-dimensional coordinate system is established with the point corresponding to the block projection map as the origin. The maximum absolute value of the coordinate values ​​in the two-dimensional coordinate system is the side length of the block projection map. The maximum side length of all block projection maps is L. max The dimension of the baseline block diagram is 2L. max ×2L max For example, if the maximum side length is 100, then the dimension of the baseline block map is 200x200. Each block projection map is expanded to the size of the baseline block map; that is, a two-dimensional coordinate system is established with the point in the block projection map corresponding to the origin of the projection coordinate system of the block projection map as the origin. The x-axis of the two-dimensional coordinate system is -2L. max Up to 2L max -2L on the Y-axis max Up to 2L max The positions within the range that have no projection points are filled with 0, resulting in a filled block projection map.

[0114] For the i-th point cloud frame, The i-th point cloud frame is obtained by stitching together the block projection images corresponding to the projection direction in order from the first layer to the last layer. A complete projected image along the projection direction; The i-th point cloud frame is obtained by stitching together the block projection images corresponding to the projection direction in order from the first layer to the last layer. A complete projected image along the projection direction; The i-th point cloud frame is obtained by stitching together the block projection images corresponding to the projection direction in order from the first layer to the last layer. A complete projected image along the projection direction.

[0115] Specifically, in step S2, the 3D point cloud frame is compressed based on the similarity of the complete projected images of two adjacent point cloud frames in each projection direction, including:

[0116] S241. The complete projection images of the first and last point cloud frames in the current segment of the 3D point cloud frame in each projection direction are keyframe images.

[0117] S242. For other point cloud frames in the current segment of the 3D point cloud frame, calculate the structural similarity of the point cloud frame with the complete projected image of the previous point cloud frame in each projection direction; if the structural similarity does not exceed the second threshold, then the complete projected image is used as the key frame image; all key frame images of the segment of the 3D point cloud frame constitute the compressed image of the segment of the 3D point cloud frame.

[0118] For example, for the i-th point cloud frame, we obtain its position in... Projection direction Projection direction and The complete projected image along the projection direction is used to calculate the i-th point cloud frame. The complete projected image along the projection direction and the (i-1)th point cloud frame are in The structural similarity of the complete projected image in the projection direction, and the i-th point cloud frame in The complete projected image along the projection direction and the (i-1)th point cloud frame are in The structural similarity of the complete projected image in the projection direction, and the i-th point cloud frame in The complete projected image along the projection direction and the (i-1)th point cloud frame are in Structural similarity of complete projected images in the projection direction.

[0119] It needs to be explained that, The fused projection coordinate system is the fused coordinate system. The coordinate system obtained by translating to the centroid of the i-th point cloud frame in the s-th segment The fused projection coordinate system is the fused coordinate system. The coordinate system is obtained by translating to the centroid of the (i-1)th point cloud frame in the s-th segment. Therefore... Projection direction and If the projection directions are the same, it is called a fused coordinate system. of The direction of the axis; Projection direction and If the projection directions are the same, it is called a fused coordinate system. of axis; Projection direction and If the projection directions are the same, it is called a fused coordinate system. of axis.

[0120] If the i-th point cloud frame is in The complete projected image along the projection direction and the (i-1)th point cloud frame are in The structural similarity of the complete projected image in the projection direction is less than the second threshold, that is, the i-th point cloud frame and the (i-1)-th point cloud frame are similar. If the difference in the projection direction is large, then the i-th point cloud frame will be projected onto the i-th point cloud frame. The complete projected image in the projection direction is used as the keyframe image; otherwise, the i-th point cloud frame is discarded. A complete projected image in the projection direction.

[0121] Specifically, step S242 calculates the structural similarity between the point cloud frame and the complete projected image of the previous point cloud frame in each projection direction in the following manner:

[0122] S2421. For the current projection direction, according to the formula...

[0123] Calculate the SSIM value of the complete projected image of the point cloud frame and its predecessor point cloud frame on the kth image channel;

[0124] Where, μ k,a μ represents the element mean of the k-th channel of the complete projected image of the point cloud frame. k,b σ represents the mean of the elements in the k-th channel of the complete projected image of the previous point cloud frame; k,a σ represents the element variance of the k-th channel of the complete projected image of the point cloud frame. k,b σ represents the element variance of the k-th channel of the complete projected image of the previous point cloud frame. k,ab c1 and c2 represent the elemental covariance of the k-th channel of the complete projected image of the point cloud frame and the complete projected image of the previous point cloud frame, where c1 and c2 are constants.

[0125] It should be noted that each image channel, i.e., each coordinate and color index, corresponds to four channels.

[0126] S2422. Add the SSIM values ​​of all image channels to obtain the structural similarity value of the point cloud frame and the complete projected image of the previous point cloud frame in the current projection direction.

[0127] The SSIM values ​​calculated from the four channels are added together to obtain the structural similarity value between the point cloud frame and the previous point cloud frame in the current projection direction.

[0128] After obtaining the keyframe image of the s-th segment of the 3D point cloud, the fused coordinates, keyframe image, and point cloud frame number, centroid, and projection method corresponding to each keyframe image are used as the compressed data for the current point cloud segment. Since highly similar complete projection images are not used as keyframe images, fewer complete projection images are retained for directions with minimal change, significantly reducing the amount of compressed data and greatly improving the compression ratio. Furthermore, more complete projection images are retained as keyframes for directions with significant change, reducing compression distortion. When reconstructing the point cloud from the compressed data, the reconstruction accuracy is high, retaining information from the original point cloud frames, resulting in higher compression accuracy.

[0129] During implementation, the compressed data can be decompressed using the following methods:

[0130] S31. Use the linear interpolation method to calculate the non-key frame images and corresponding point cloud frame numbers between key frame images with the same segment number and the same projection direction.

[0131] S32. For each point cloud frame number, according to the corresponding fusion coordinate system, perform 3D point reconstruction on the corresponding key frame image and / or non-key frame image to obtain the point cloud corresponding to the point cloud frame number; the point clouds corresponding to all point cloud frame numbers constitute a 3D point cloud frame sequence.

[0132] Specifically, in step S31, the linear interpolation method is used to calculate the non-key frame images and corresponding point cloud frame numbers between key frame images with the same segment number and the same projection direction, including:

[0133] S311. Keyframe images with the same segment number and the same projection direction are sorted in ascending order of point cloud frame number as a group of keyframe images.

[0134] S312. For each set of keyframe images, if the corresponding point cloud frame numbers are not consecutive between two adjacent keyframe images, the non-keyframe images between the two adjacent keyframe images are calculated according to the following formula:

[0135]

[0136] Where si represents the point cloud frame number corresponding to the first keyframe image in two adjacent keyframe images, sj represents the point cloud frame number corresponding to the second keyframe image in two adjacent keyframe images, and I si I represents the image data of the first keyframe image. sj This represents the image data of the second keyframe image, |I sj -I si | Represents the computational image I si and|I sj The difference between corresponding elements, where sk represents the point cloud frame number between si and sj, si < sk < sj, P sk This represents the image data of the non-keyframe image corresponding to the point cloud frame number sk.

[0137] Since the fusion coordinate systems corresponding to point cloud frames in different segments are different, the calculation of non-keyframe images (i.e., the images corresponding to the complete projection images discarded in step S332) needs to be obtained based on the keyframe images of the same segment. In practice, the keyframe images of each projection direction of the s-th segment (segment number s) are first sorted in ascending order of point cloud frame number. For example, the fusion coordinate system... of If the point cloud frame numbers are 12, 15, and 19 (i.e., the 12th, 15th, and 19th point cloud frames) along the axis projection direction, then the point cloud frames with frame numbers 13, 14, 16, 17, and 18 (i.e., the 13th, 14th, 16th, 17th, and 18th point cloud frames) need to be calculated. Non-keyframe images along the axis projection direction. For the 13th point cloud frame in The non-keyframe images along the axis projection direction need to be calculated based on the projected images of the 12th and 15th point cloud frames.

[0138] According to the formula Calculate the 13th point cloud frame in Non-keyframe image P in the axis projection direction sk Where si = 12, sj = 15, sk = 13, I si For the 12th point cloud frame in Keyframe images along the axis projection direction, I sj For the 15th point cloud frame in Keyframe images along the axis projection direction, I sj -I si This indicates that the 12th point cloud frame is in Keyframe images and the 15th point cloud frame in the axial projection direction The difference between corresponding elements of the keyframe image along the axis projection direction (calculated separately for each channel).

[0139] After calculating the non-keyframe images, for each point cloud frame number, based on the corresponding fusion coordinate system, the corresponding keyframe image and / or non-keyframe image are used to perform 3D point reconstruction to obtain the point cloud corresponding to that point cloud frame number, including:

[0140] Based on the relationship between the fused projection coordinate system and the reference coordinate system, the points corresponding to each pixel in each keyframe image and / or non-keyframe image corresponding to the current point cloud frame number are transformed to the reference coordinate system to obtain the point cloud corresponding to the current point cloud frame number.

[0141] For each point cloud frame, the image projected in each projection direction may be a keyframe image or a non-keyframe image calculated according to the above steps. The corresponding projection coordinate system is the coordinate system obtained by translating the corresponding fusion coordinate system to the centroid of the point cloud frame (i.e., the fusion projection coordinate system). The coordinate information recorded in the keyframe and non-keyframe images is their coordinate information under the corresponding fusion projection coordinate system. Therefore, after extracting the pixel information of the keyframe images and / or non-keyframes corresponding to all projection directions of the point cloud frame, the point cloud of the point cloud frame in the reference coordinate system can be obtained by reconstructing it according to the relationship between the fusion projection coordinate system and the reference coordinate system. The point clouds corresponding to all point cloud frame numbers constitute a three-dimensional point cloud frame sequence, which yields the decompressed point cloud sequence.

[0142] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0143] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A dynamic human-robot interaction point cloud compression method, characterized in that, The method comprises the following steps: calculating a maximum change coordinate system of adjacent point cloud frames in a three-dimensional point cloud frame sequence based on a multi-layer projection method; segmenting the three-dimensional point cloud frame sequence according to a first projection direction of the adjacent point cloud frames; for each segment of the three-dimensional point cloud frames, calculating a fusion coordinate system according to the maximum change coordinate system of the adjacent point cloud frames; projecting each point cloud frame of the segment of the three-dimensional point cloud frames on a projection direction of the fusion coordinate system to obtain a complete projection image of each point cloud frame on each projection direction, and compressing the segment of the three-dimensional point cloud frames according to the similarity of the complete projection images of two adjacent point cloud frames on each projection direction; calculating a maximum change coordinate system of adjacent point cloud frames in a three-dimensional point cloud frame sequence based on a multi-layer projection algorithm, comprising: S11, for each point cloud frame in the three-dimensional point cloud frame sequence, establishing a plurality of projection coordinate systems with the centroid of the point cloud frame as the origin; projecting each point cloud frame on the projection directions of the corresponding plurality of projection coordinate systems based on the layered projection method; S12, taking the second point cloud frame in the point cloud frame sequence as the current point cloud frame; S13, for each projection direction of each projection coordinate system corresponding to the current point cloud frame, calculating the chamfer distance between the current point cloud frame and the previous frame of the current point cloud frame; S14, the projection direction corresponding to the maximum chamfer distance is the first projection direction, the projection direction corresponding to the maximum chamfer distance among the projection directions other than the first projection direction of all the projection coordinate systems where the first projection direction is located is the second projection direction, and the coordinate system where the first projection direction and the second projection direction are located is the maximum change coordinate system from the previous frame of the current point cloud frame to the current point cloud frame; taking the next point cloud frame as the current frame, returning to step S13 until all point cloud frames are traversed.

2. The dynamic human-robot interaction point cloud compression method of claim 1, wherein, projecting each point cloud frame on the projection directions of the corresponding plurality of projection coordinate systems based on the layered projection method, comprising: S1121, for the jth projection coordinate system of the current point cloud frame, all points of the current point cloud frame form a set of points to be projected; let k = 1 and L = 1; S1122, taking the kth coordinate axis of the jth projection coordinate system as the reference axis; S1123, sorting each point of the set of points to be projected in descending order of the value on the reference axis, for each sorted point, if there is a projected point at the corresponding position of the Lth layer of the projection direction corresponding to the reference axis, the point is not projected, otherwise the point is projected on the corresponding position of the Lth layer of the projection direction corresponding to the reference axis, and the point is deleted from the set of points to be projected; if the set of points to be projected is empty, the projection is ended; otherwise, step S1124 is executed; S1124, k = k + 1, if k = 4, then L = L + 1, k is set to 1, and step S1122 is returned; otherwise, step S1122 is returned directly.

3. The dynamic human-robot interaction point cloud compression method of claim 1, wherein, For each projection direction of each projection coordinate system, the chamfer distance between the current point cloud frame and the previous frame of the current point cloud frame is calculated in the following manner: For the current projection direction O i X j , find the points of the current point cloud frame projected on the current projection direction O i X j , to form a first point set; wherein O i X j represents the projection direction of the jth projection coordinate system of the ith point cloud frame; i X j axis; Finding the points of the previous frame of the current point cloud frame projected on the projection direction O i-1 X j , constituting a second point set; O i-1 X j O i-1 X j axis of the jth projection coordinate system of the i-1th point cloud frame; calculate the chamfer distance of the first point set and the second point set to obtain the chamfer distance of the current point cloud frame and the previous frame of the current point cloud frame on the current projection direction.

4. The dynamic human-robot interaction point cloud compression method of claim 3, wherein, The chamfer distance d(S1, S2) is calculated using the following formula: wherein S1 represents a first set of points, n S1 represents the number of points in S1, S2 represents a second set of points, n S2 represents the number of points in S2, represents the distance from point x to point y, represents the distance from point y to point x.

5. The dynamic human-robot interaction point cloud compression method of claim 1, wherein, For each point cloud frame in the sequence of three-dimensional point cloud frames, a plurality of projection coordinate systems are established with its centroid as the origin, comprising: A coordinate system parallel to and consistent with the coordinate axes of the reference coordinate system is established with the centroid of the current point cloud frame as the coordinate origin as the first coordinate; Each coordinate axis in the first coordinate system is fixed respectively, and the two coordinate axes other than the fixed coordinate axis are rotated around the fixed coordinate axis according to the rotation step, to obtain a plurality of coordinate systems identical to the X-axis, Y-axis or Z-axis of the first projection coordinate system; The first coordinate system and the plurality of coordinate systems identical to the X-axis, Y-axis or Z-axis of the first projection coordinate system constitute the plurality of projection coordinate systems of the current point cloud frame.

6. The dynamic human-robot interaction point cloud compression method of claim 1, wherein, Projecting each point cloud frame of the segment of three-dimensional point cloud frames on the projection direction of the fusion coordinate system to obtain a complete projection image of each point cloud frame on each projection direction, comprising: Using a hierarchical projection method to project each point of each point cloud frame of the segment of three-dimensional point cloud frames on the projection direction corresponding to the fusion coordinate system to obtain a multi-layer partition projection map of each point cloud frame on each projection direction corresponding to the fusion coordinate system; Splicing the multi-layer partition projection map of each point cloud frame on each projection direction to obtain a complete projection image of each point cloud frame on each projection direction.

7. The dynamic human-robot interaction point cloud compression method of claim 1, wherein, According to the similarity of the complete projection images of adjacent two point cloud frames on each projection direction, the segment of three-dimensional point cloud frames is compressed, comprising: The complete projection images of the first point cloud frame and the last point cloud frame in the current segment of three-dimensional point cloud frames on each projection direction are key frame images; For other point cloud frames in the current segment of three-dimensional point cloud frames, the structural similarity of the complete projection images of the point cloud frame and its previous point cloud frame on each projection direction is calculated; if the structural similarity does not exceed a second threshold, the complete projection image is taken as a key frame image; all key frame images of the segment of three-dimensional point cloud frames constitute the compressed images of the segment of three-dimensional point cloud frames.

8. The dynamic human-robot interaction point cloud compression method of claim 7, wherein, The structural similarity of the complete projection images of the point cloud frame and its previous point cloud frame on each projection direction is calculated in the following way: For the current projection direction, the SSIM value of the complete projection image of the point cloud frame and its previous point cloud frame on the kth image channel is calculated according to the formula respectively. wherein μ k,a denotes the element mean of the k-th channel of the full projected image of the point cloud frame; μ k,b denotes the element mean of the k-th channel of the full projected image of the previous point cloud frame of the point cloud frame; σ k,a denotes the element variance of the k-th channel of the full projected image of the point cloud frame; σ k,b denotes the element variance of the k-th channel of the full projected image of the previous point cloud frame of the point cloud frame, σ k,ab denotes the element covariance of the k-th channel of the full projected image of the point cloud frame and the full projected image of the previous point cloud frame of the point cloud frame, and c1 and c2 are constants. The SSIM values of all image channels are added to obtain the structural similarity value of the complete projection images of the point cloud frame and its previous point cloud frame on the current projection direction.

9. The dynamic human-robot interaction point cloud compression method of claim 1, wherein, For each segment of three-dimensional point cloud frames, a fusion coordinate system is calculated according to the maximum change coordinate system of adjacent point cloud frames, comprising: For the current segment of point cloud frames, the translation values of all maximum change coordinate systems corresponding to the current segment of point cloud frames are translated to the origin to coincide with the origin of the reference coordinate system; For each coordinate axis of each maximum change coordinate system, it is attributed to the type corresponding to the coordinate axis of the reference coordinate with the smallest included angle; The unit vectors of the coordinate axes corresponding to each type are added to obtain the coordinate axes of the fusion coordinate system corresponding to the type; the fusion coordinate system corresponding to the current segment of three-dimensional point cloud frames is obtained.

Images