Point cloud projection processing method, system and receiving processing method based on user perspective

Abstract

The present invention provides a point cloud projection processing method, system and receiving processing method based on user perspective, including: step 1: distinguishing the main segment and multiple edge segments of the point cloud object according to the user perspective; step 2: maintaining the projection vector of the edge segment or changing it based on the user perspective according to the spatial connection between the edge segment and the main segment; step 3: projecting according to the adjusted projection vector to obtain a compressible two-dimensional image and perform point cloud processing. The present invention optimizes the projection of the point cloud segment by taking into account the user perspective and the spatial correlation with the main vector, combining the characteristics of the user observing the point cloud from a specific perspective, optimizing the projection vector of the point cloud, and focusing more on the direction of higher user attention, thereby improving the visual effect when presented after decompression.

Description

Technical Field

[0001] The present invention relates to the field of point cloud technology, and in particular to a point cloud projection processing method based on a user perspective, and a system receiving and processing method. Background Art

[0002] In recent years, media transmission technology has developed and improved rapidly. The transmission cost of image and video information has been continuously reduced while the accuracy has been improved, which has further promoted the maturity of related application fields. 3D point cloud data is a new type of image media data that can provide users with more accurate and vivid image information, bringing a more realistic and interactive communication experience.

[0003] Point cloud data is the surface information of an object obtained by scanning, including three-dimensional coordinate data, depth information, color information, etc. The maturity of 3D scanning technology and the improvement of system architecture have improved the accuracy of point cloud data, and also greatly increased the amount of point cloud data. There are different point cloud data compression algorithms for static and dynamic point cloud data, as well as different types of point cloud data such as machine perception and human eye perception. For example, for dynamic human eye perception point cloud data, a typical point cloud compression algorithm is to convert 3D point cloud data into 2D image data, and then perform data processing. One of them is the video-based point cloud compression (VPCC) algorithm. This compression method first projects the 3D point cloud onto a 2D plane to obtain occupancy map information, geometric information, attribute information, and auxiliary information. The attribute information usually includes texture information and color information. Therefore, the compressed information is usually divided into four types of data for transmission. This method achieves the overall compression transmission of the point cloud sequence.

[0004] However, in actual application scenarios, users cannot observe the entire content of a point cloud object. For example, when a user observes a point cloud object from the front, the user cannot see the media content on the back of the point cloud. In addition, users have different levels of attention to each side. For example, in a scenario where a user views point cloud media from the front, based on the direction of the user's perspective, the user's attention to the front content of the point cloud object will be higher than the attention to the left and right sides and the upper and lower bottom surfaces of the point cloud object. The visual quality of the side with high attention has a greater impact on the user's perception. In the current point cloud projection processing process, the projection design and point cloud compression scheme are not optimized and adjusted according to the characteristics of the user's attention when viewing the point cloud object. How to optimize the visual quality of the point cloud without affecting the compression rate and the objective quality of the point cloud has become a problem that needs to be studied and solved.

[0005] In the existing projection processing method, since the user's perspective is not taken into account, and of course the spatial relationship between the edge fragment and the main fragment is not taken into account, if the point cloud object is only divided into multiple point cloud fragments according to multiple standard vectors, the following situation will occur:

[0006] The spatial distance between the points contained in the point cloud fragments corresponding to the same standard vector may be very large. Such a simple rough division operation has nothing to do with the user's perspective, and the point cloud fragments corresponding to the same standard vector appear fragmented due to the lack of geometric spatial correlation. In other words, the related points with very close spatial distances in the point cloud object are divided and projected into different standard vectors due to different standard vectors.

[0007] Patent document CN109565579A (application number: 201780048234.5) discloses a device for combining image projection and surface scanning. Generally, the device and method of the present invention utilize at least one laser source to generate a laser beam, and a scanning mirror that reflects the laser beam into a scan line pattern. The light source is controlled to selectively generate projection image pixels during the first part of the scan line pattern, and to selectively generate depth mapping pulses during the second part of the scan line pattern. The projection image pixels generate a projection image, and the depth mapping pulses are reflected, received and used to generate a three-dimensional point cloud describing the surface depth measured at each point. Summary of the invention

[0008] In view of the defects in the prior art, the object of the present invention is to provide a point cloud projection processing method, system and receiving processing method based on user perspective.

[0009] The point cloud projection processing method based on user perspective provided by the present invention includes:

[0010] Step 1: distinguish the main segment and multiple edge segments of the point cloud object according to the user's perspective;

[0011] Step 2: According to the spatial relationship between the edge fragment and the main fragment, the projection vector of the edge fragment is maintained or changed based on the user's perspective;

[0012] Step 3: Project according to the adjusted projection vector to obtain a compressible two-dimensional image and perform point cloud processing.

[0013] Preferably, the step 1 comprises:

[0014] Dividing the point cloud object into a plurality of first-level point cloud fragments based on a plurality of standard normal vectors;

[0015] In the same first-level point cloud segment, points with the same normal vector and a distance less than a threshold are further divided into second-level point cloud segments as cloud segments to be processed;

[0016] Determine the principal vector in the standard normal vector according to the user's viewing angle;

[0017] Based on the main vector, the cloud fragment to be processed is divided into a main fragment and a plurality of edge fragments.

[0018] Preferably, the step 2 comprises:

[0019] Determine the principal vector in the standard normal vector according to the user's perspective;

[0020] Project the 3D edge box of the main segment along the direction of the main vector to obtain a main 2D projection image; Project the 3D edge box of the edge segment along the direction of the main vector to obtain an edge 2D projection image;

[0021] Based on each edge two-dimensional projection map, the intersection and union ratio is calculated with each main two-dimensional projection map respectively;

[0022] Multiple intersection-and-union ratios are compared with a preset threshold value as a basis for determining spatial connections.

[0023] Preferably, the preset threshold value depends on the surface characteristics of the point cloud object, distinguishes the surface characteristics of the object point cloud object from the surface characteristics of the person point cloud object, and the threshold value is adjusted and determined according to the experimental results;

[0024] If there is a case where the multiple intersection-and-union ratios are greater than a preset threshold value, the projection vector of the edge segment is changed to a principal vector; if there is no case where the multiple intersection-and-union ratios are greater than the preset threshold value, the projection vector of the edge segment is maintained.

[0025] Preferably, according to the user's position when viewing the point cloud object, a vector pointing from the point cloud to the user's observation point is used to represent the user's observation perspective, which is called a perspective vector; the distance between the perspective vector and the six standard normal vectors is calculated, and the standard normal vector closest to the perspective vector is called the main vector under the observation perspective.

[0026] Preferably, irrelevant segments of the point cloud object are also distinguished according to the user's viewing angle;

[0027] The standard normal vector closest to the user's viewing angle is determined as the principal vector;

[0028] Determine the to-be-processed point cloud fragment having the same main vector as the main vector among the plurality of standard normal vectors as the main fragment;

[0029] Determine the to-be-processed point cloud fragments that are opposite to the main vector among the plurality of standard normal vectors as irrelevant fragments;

[0030] Determine the to-be-processed point cloud fragment associated with the main vector in the plurality of standard normal vectors as an edge fragment;

[0031] The multiple standard normal vectors are in one-to-one correspondence with the multiple faces of the three-dimensional edge box, and the standard normal vectors are perpendicular to the corresponding faces;

[0032] The projection vector of the irrelevant fragment is the normal vector of the irrelevant fragment;

[0033] The three-dimensional edge box enclosing the point cloud object includes: a main fragment projection aggregation surface of the main fragment aggregation, five edge fragment projection aggregation surfaces of the edge fragment aggregation, and an irrelevant fragment projection aggregation surface of the irrelevant fragment aggregation.

[0034] Preferably, step 3 comprises:

[0035] The main segment, the edge segment after projection processing and the irrelevant segment are projected according to the adjusted projection vectors to generate compressible two-dimensional images, which are further processed to obtain a two-dimensional video sequence, and the two-dimensional video sequence is compressed using a video codec.

[0036] Preferably, in the subsequent point cloud processing after projection, the projection relationship identification field is used to identify the projection vector of the point in the point cloud object after processing and adjustment, including the projection vector value and the corresponding index value.

[0037] The point cloud projection processing system based on user perspective provided by the present invention includes:

[0038] Fragment segmentation module: distinguishes the main fragment and multiple edge fragments of the point cloud object according to the user's perspective;

[0039] Projection vector adjustment module: maintains the projection vector of the edge segment or changes it based on the user's perspective according to the spatial connection between the edge segment and the main segment;

[0040] Projection module: Projecting the points in the three-dimensional point cloud object according to the projection vector adjusted by projection to obtain a compressible two-dimensional image;

[0041] Code stream processing module: receives point cloud code stream and processes it to obtain a two-dimensional image;

[0042] Reverse projection module: Use the projection relationship identification field to perform reverse projection processing to restore the two-dimensional image to a three-dimensional point cloud object;

[0043] The projection relationship identification field indicates: the projection vector after processing and adjustment at the sending end, including the projection vector value and the corresponding index value.

[0044] The point cloud receiving and processing method based on user perspective provided by the present invention includes:

[0045] Receive the point cloud code stream and process it to obtain a two-dimensional image, use the projection relationship identification field to perform inverse projection processing, and restore the two-dimensional point cloud fragment to a three-dimensional point cloud object;

[0046] The projection relationship identification field indicates: the projection vector after processing and adjustment at the sending end, including the projection coordinate value and the corresponding index value.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention improves on the existing point cloud sequence coding transmission scheme, introduces the user's observation perspective in the process of compressing the three-dimensional point cloud into a two-dimensional image to optimize and adjust the projection processing process of the point cloud, thereby improving the compression performance;

[0049] 2. The present invention optimizes and adjusts the projection direction to avoid scattered and fragmented conditions, and also overcomes the defect that the related points with close spatial distances in the point cloud object are divided and projected into different standard vectors due to different standard vectors;

[0050] 3. The present invention optimizes and adjusts the projection of point cloud fragments by taking into account the user's perspective and the spatial correlation with the main vector, combining the characteristics of the user observing the point cloud from a specific perspective, optimizing the projection vector of the point cloud, and concentrating it more on the direction where the user pays more attention, thereby improving the visual effect when presented after decompression. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0052] Attached Figure 1 is a stereoscopic schematic diagram of a three-dimensional edge box of a point cloud object in an embodiment of the present invention;

[0053] Attached Figure 2 A corresponding diagram of the relationship between the normal vector and the principal vector of different types of point cloud fragments in an embodiment of the present invention;

[0054] Attached Figure 3 A schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment used for projection adjustment according to the main vector direction in an embodiment of the present invention;

[0055] Attached Figure 4 For attachment Figure 3 A schematic diagram of a two-dimensional projection obtained by projecting the three-dimensional edge boxes of the main segment and the edge segment onto a two-dimensional plane;

[0056] Attached Figure 5 Schematic diagram of the process of a point cloud projection processing method based on a user perspective in an embodiment of the present invention;

[0057] Attached Figure 6 Schematic diagram of the process of a point cloud receiving and processing method based on a user perspective in an embodiment of the present invention;

[0058] Attached Figure 7 is a schematic block diagram of a point cloud projection processing system based on a user perspective in an embodiment of the present invention;

[0059] Attached Figure 8 It is a schematic block diagram of a point cloud receiving and processing system based on user perspective in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0061] Example:

[0062] In the prior art solution, the processing steps of the point cloud sequence encoding method include: dividing the point cloud object into multiple point cloud fragments according to the normal vector, subdividing the fragments according to the connection relationship of the point cloud fragments, projecting the subdivided point cloud fragments according to the normal vector, filling the projection plane with blanks, and compressing and encoding the video sequence obtained by the projection using a two-dimensional encoding tool such as HEVC.

[0063] In an embodiment of the present invention, an improvement is made on the existing point cloud sequence coding and transmission scheme, and the user's observation perspective is introduced in the process of compressing the three-dimensional point cloud into a two-dimensional image to optimize and adjust the projection processing process of the point cloud, thereby improving the compression performance.

[0064] According to the point cloud projection processing method, receiving processing method and system based on user perspective of the present invention, when the point cloud object is divided into multiple point cloud segments according to multiple standard vectors, the user perspective factor is introduced to distinguish the multiple point cloud segments into a main segment and multiple edge segments. Among them, the user pays more attention to the front content of the point cloud object, then the main segment is determined based on the user perspective, and in addition, the edge segments associated with the main segment, such as the point cloud segments of the left and right sides and the upper and lower bottom surfaces, are taken as edge segments. The projection direction of these edge segments further considers the distance relationship between these edge segments and the main segment in terms of spatial connection, and the projection vector of the point in the edge segment is adjusted and optimized as necessary. If the spatial connection is large, the projection vector of the point in the edge segment is changed based on the user perspective so that it is projected in the direction of the main vector consistent with the main segment. In addition, if the spatial connection is small, the projection vector of the point in the edge segment is maintained. After the optimization adjustment of the above-mentioned projection vector is completed, the projection operation is performed on the multiple point cloud segments of the point cloud object, namely, the main segment and multiple edge segments, respectively, according to the optimized and adjusted projection vector, and subsequent point cloud compression, encoding and transmission processing, etc.

[0065] The above-mentioned optimization and adjustment of the projection direction of the edge segment takes into account the user's perspective and the spatial correlation with the main vector, so that the points in the edge segment that have a large spatial correlation with the main segment are re-projected according to the main vector of the main segment, and are more concentrated in the direction where the user pays more attention. Moreover, these adjusted and concentrated points are highly correlated and close to the main segment with higher attention in space, avoiding scattered and fragmented conditions, and overcoming the defect that the related points with close spatial distances in the point cloud object are divided and projected into different standard vectors due to different standard vectors.

[0066] The projection vector of the point cloud fragment is selected by comprehensively considering the two factors of the user's observation perspective and the normal vector of the fragment. This embodiment provides a point cloud projection processing method based on the user's perspective, including: distinguishing the main fragment and multiple edge fragments of the point cloud object according to the user's perspective; considering the spatial connection between the edge fragment and the main fragment, the projection vector of the edge fragment is maintained or changed based on the user's perspective; and performing subsequent point cloud processing after projection.

[0067] Attached Figure 1 Schematic diagram of a three-dimensional edge box of a point cloud object in an embodiment of the present invention.

[0068] When observing a point cloud object from a specific direction such as a user's viewing angle, Figure 1As shown, the point cloud object is projected using a three-dimensional bounding box of the point cloud object to include the point cloud object. Usually, the three-dimensional bounding box of the point cloud object is a rectangular cube set according to the specific point cloud object situation, and the specific size and shape are not limited by this embodiment.

[0069] Not shown in the figure, with the point cloud object as the origin (0, 0, 0), the standard normal vectors (x, y, z) of the six projection surfaces corresponding to the point cloud object are first defined: (1, 0, 0), (-1, 0, 0), (0, 1, 0), (0, -1, 0), (0, 0, 1), (0, 0, -1).

[0070] According to the user's position when viewing the point cloud object, a vector pointing from the point cloud to the user's observation point is used to represent the user's viewing angle, which is called the viewing angle vector. Then the distance between the viewing angle vector and the six standard normal vectors is calculated, and the standard normal vector closest to the viewing angle vector is called the main vector under the viewing angle.

[0071] Usually, six different Index values ​​are used to indicate six different normal vectors. Table 1 is a corresponding table of index parameter values ​​and projection vector values.

[0072] In this embodiment, 0 is used to indicate the direction of the first standard normal vector, which is perpendicular to the (1, 0, 0) plane, 1 is used to indicate the direction of the sixth standard normal vector, which is perpendicular to the (-1, 0, 0) plane, 2 is used to indicate the direction of the second standard normal vector, which is perpendicular to the (0, 1, 0) plane, 3 is used to indicate the direction of the third standard normal vector, which is perpendicular to the (0, -1, 0) plane, 4 is used to indicate the direction of the fourth standard normal vector, which is perpendicular to the (0, 0, 1) plane, and 5 is used to indicate the direction of the fifth standard normal vector, which is perpendicular to the (0, 0, -1) plane.

[0073] In this embodiment, the main vector is the first standard normal vector (1, 0, 0).

[0074] It can be seen that the six standard normal vectors are in one-to-one correspondence with the six faces of the three-dimensional edge box, and the standard normal vectors are perpendicular to their corresponding faces.

[0075] Index parameter value (Index) Projected vector surfaceNormal 0 (1，0，0) 1 (-1，0，0) 2 (0，1，0) 3 (0，-1，0) 4 (0，0，1) 5 (0，0，-1)

[0076] Table 1

[0077] The normal vector of each point is calculated by combining the neighbor point information of the point cloud object, and then the distance between the normal vector of each point and the first to sixth standard normal vectors is calculated. The standard normal vector with the closest distance is the standard normal vector corresponding to the point. Through this step, all points in the point cloud object are divided into 6 first-level point cloud fragments. Through the first-level division, points with the same standard normal vector belong to the same first-level point cloud fragment. The 6 first-level point cloud fragments are subdivided, and points that belong to the same first-level point cloud fragment and are close to each other are divided into a second-level point cloud fragment. The method for calculating the distance between points can be selected according to the specific situation, and the Euclidean distance is usually used. The 6 first-level point cloud fragments are subdivided according to this method, and the normal vectors of the second-level point cloud fragments obtained after the subdivision correspond to one of the above 6 standard normal vectors.

[0078] In this embodiment, the secondary point cloud fragments are further processed as the to-be-processed cloud fragments, and the to-be-processed cloud fragments located on the six standard normal vector planes are further distinguished according to the direction of the viewing vector. The normal vectors of the secondary point cloud fragments are calculated. Then, the main direction of the projection is determined according to the user's viewing angle, and the secondary point cloud fragments are divided into main fragments, edge fragments, and irrelevant fragments according to the relationship between the normal vectors of the secondary point cloud fragments and the main direction.

[0079] Attached Figure 2 is a corresponding table of the relationship between normal vectors and principal vectors of different point cloud fragment types in an embodiment of the present invention; Figure 2 The criteria for distinguishing different fragment types are given.

[0080] Among them Figure 2 As shown, the secondary point cloud fragments with the same normal vector as the main vector are defined as main fragments. These main fragments are also called the first standard fragments. These main fragments form a main fragment set. The first standard fragment set (i.e., the main fragment set) is gathered on the main fragment projection gathering surface of the point cloud three-dimensional edge box.

[0081] The secondary point cloud fragments whose normal vectors are opposite to the main vectors are defined as irrelevant fragments. These irrelevant fragments are also called second standard fragments. These irrelevant fragments form an irrelevant fragment set. The second standard fragment set (i.e., the irrelevant point cloud fragment set) is gathered on the irrelevant fragment projection aggregation surface of the point cloud three-dimensional edge box.

[0082] The secondary point cloud segments whose normal vectors are perpendicular to the main vectors are defined as edge segments. These edge segments are distributed on four edge segment projection aggregation surfaces perpendicular to the main vectors. Figure 1 Only one of the four edge fragment projection aggregation surfaces is explicitly shown, and the other three surfaces are omitted.

[0083] Among them, the secondary point cloud segments located on the projection gathering surface of the first edge segment, that is, these edge segments form the set of the first edge segment, and the first edge segment comes from the third standard segment. Similarly, the secondary point cloud segments located on the other three edge segment gathering surfaces are respectively called the set of the second edge segment, the set of the third edge segment, and the set of the fourth edge segment, and they form sets on their gathering surfaces respectively, and the second edge segment to the fourth edge segment come from the fourth to the sixth standard segments respectively.

[0084] Next, the projection vector of the main segment is projected with its canonical normal vector, i.e., the main vector. The projection vector of the irrelevant segment is projected with its canonical normal vector, i.e., opposite to the main vector.

[0085] The key difference between this embodiment and the prior art is that the spatial connection between the edge segment and the main segment is taken into consideration to determine whether the projection vector of the edge segment needs to be further adjusted and optimized.

[0086] In short, the size of the spatial connection is determined by calculating the intersection-over-union (IOU) of each edge segment with each main segment in turn. If there is an IOU greater than the threshold value, it means that the spatial connection between the edge point cloud segment and the main point cloud segment is large, and the main vector is selected as the projection vector of the edge point cloud segment; if there is no IOU greater than the threshold value, it means that the spatial connection between the edge point cloud segment and the main point cloud segment is small, and the normal vector of the edge point cloud segment is selected as its projection vector.

[0087] Combination Figure 3 and Figure 4 , the process of judging the intersection-over-union (IOU) between one edge segment and one of the main segments is specifically described.

[0088] Attached Figure 3 Schematic diagram of the relative positions of the three-dimensional edge box of the main segment and the three-dimensional edge box of the edge segment for projection adjustment according to the main vector direction in an embodiment of the present invention.

[0089] The three-dimensional edge box shown by the solid line (referred to as edge box 1) corresponds to the main segment, and the three-dimensional edge box shown by the dotted line (referred to as edge box 2) corresponds to the edge segment.

[0090] Figure 3 and Figure 4 It is just a brief enumeration of the 3D edge boxes of a certain main segment and a certain edge segment, indicating the projection orientation and the overlap of the 2D projection images between the 3D boxes. The spatial association between the overlap calculation of a certain edge segment and other main segments, or between other edge segments and other main segments, will not be listed one by one. Figure 3 and Figure 4 Just for example.

[0091] Attached Figure 4 The 2D projection of the rectangle obtained by projecting the 3D edge boxes of the main fragment and the edge fragment onto the 2D plane.

[0092] The three-dimensional edge box of the main segment is projected along the direction of the main vector to obtain a main two-dimensional projection image, and the three-dimensional edge box of the edge segment is projected along the direction of the main vector to obtain an edge two-dimensional projection image. Figure 4 In the figure, the grid filling part is the projection of the edge box 1 of the main fragment along the main vector direction (called rectangular area 1), the point filling part is the projection of the edge box 2 of the edge fragment along the main vector direction (called rectangular area 2), and the grid point filling part is the common part where rectangular area 1 and rectangular area 2 overlap (called rectangular area 3).

[0093] For each edge segment, the Intersection over Union (IOU) ratio between it and each main segment is calculated in turn. Figure 1 and attached Figure 2 The calculation formula is as follows:

[0094]

[0095] S 1 =(y i1 -y i0 )×(z i1 -z i0 ) (2)

[0096] S 2 =(y j1 -y j0 )×(z j1 -z j0 ) (3)

[0097] S 3 =(y i1 -y j0 )×(z i1 -z j0 ) (4)

[0098] S 3 Corresponding attachment Figure 4 The area of ​​the middle rectangular region 3, i.e., the common part where the main two-dimensional projection image and the edge two-dimensional projection image overlap;

[0099] S 1 Corresponding attachment Figure 4 The main two-dimensional projection excludes the area of ​​the common parts;

[0100] S 2 Corresponding attachment Figure 4 The two-dimensional projection of the middle edge excludes the area of ​​the common part.

[0101] As attached Figure 3 As shown in the solid three-dimensional edge box (edge ​​box 1), A i and B i They are the vertices closest to and farthest from the origin O on the three-dimensional edge box of the main fragment, i.e., the edge box 1; as shown in the attached figure Figure 3 As shown in the dashed three-dimensional edge box (edge ​​box 2), A j and B j They are respectively the vertices closest to and farthest from the origin O on the three-dimensional edge box of the edge fragment, i.e., the edge box 2.

[0102] As attached Figure 4 As shown, A' i , A' j , B' i , B' j A i , A j , B i , B j The points projected along the direction of the principal vector. Their coordinates in the XYZ coordinate system with O as the origin are: A i (x i0 ,y i0 ,z i0 ),B i (x i1 ,y i1 ,z i1 ),A j (x j0 ,y j0 ,z j0 ),B j (x j1 ,y j1 ,z j1 ),A' i (y i0 ,z i0 ),B' i (y i1 ,z i1 ),A' j (y j0 ,z j0 ),B' j (y j1 ,z j1 ).

[0103] Based on the two-dimensional projection graph, for each edge segment, multiple IOUs are calculated with multiple main segments, and the number of IOUs is equal to the number of main segments.

[0104] For an edge fragment, its various IOUs are compared with an IOU threshold value. If there is an IOU greater than the IOU threshold value, it means that the edge fragment has a large spatial connection with the main fragment. In order to optimize the point cloud visual quality in the direction of the main vector, the projection direction of the edge fragment is changed, and the main vector is selected as the projection vector of the edge fragment; if there is no IOU greater than the IOU threshold value, it means that the edge fragment has a small spatial connection with the main fragment, and the normal vector of the edge fragment is kept as its projection vector.

[0105] It can be seen that the final projection vector of the edge fragment is determined by the normal vector of the edge fragment and the spatial position relationship between the edge fragment and the main fragment.

[0106] The selection of the intersection-and-union ratio threshold value depends on the surface characteristics of the point cloud object to be processed. For example, the surface characteristics of the object point cloud object and the person point cloud object are different, so the intersection-and-union ratio threshold value selected for comparison is different. For common person point cloud objects, the threshold value is generally selected to be 0.2-0.3. Further, the threshold value is determined mainly through adjustment and selection based on experimental results.

[0107] Through the above process, the adjustment, optimization or maintenance of the projection vector of the first edge segment is completed, and the same processing as the above process is used to distinguish and adjust each edge segment one by one until all edge segments in the set of the second edge segment, the set of the third edge segment, and the set of the fourth edge segment have been processed.

[0108] For the main fragment, its projection vector is the main vector; for an irrelevant fragment, its projection vector is the normal vector of the irrelevant fragment.

[0109] After the above process, the projection vectors of all secondary point cloud fragments, i.e., the cloud fragments to be processed, have been optimized and adjusted. In the subsequent point cloud processing after projection, the projection relationship identification field is used to identify: the projection vectors of the points in the point cloud object after processing and adjustment, including the optimized and adjusted projection vector coordinate values ​​and the corresponding index parameter values.

[0110] The change process of Table 2 is explained based on Table 1. Table 2 is an example table of corresponding relationships before and after the projection vectors of multiple edge segments are adjusted and optimized.

[0111] For example, in Table 2, for the first edge segment, before the projection adjustment and optimization, its original standard normal vector is index parameter value 2, (0, 1, 0). After the above optimization and adjustment, the projection direction is changed to the main vector, that is, index parameter value 0, (1, 0, 0). Whether other edge segments are changed depends on the spatial correlation between each edge segment and the main segment. In short, for the first to k edge segments, they are either adjusted and optimized to 0 or kept.

[0112] The main segment, i.e., the index parameter value is 0 and the standard normal vector coordinate value is (1, 0, 0), is not optimized and adjusted, and the main vector is kept for projection. The irrelevant edge segment, i.e., the index parameter value is 1 and the standard normal vector coordinate value is (-1, 0, 0), is not optimized and adjusted, and the original standard normal vector direction is kept for projection.

[0113]

[0114]

[0115] Table 2

[0116] Continue to complete the subsequent projection processing, including:

[0117] The main segment, the edge segment after projection processing, and the irrelevant segment are projected according to the adjusted projection vectors to generate compressible two-dimensional images, which are further processed to obtain a two-dimensional video sequence, and the two-dimensional video sequence is compressed using a video codec.

[0118] Attached Figure 5 Schematic diagram of the process of a point cloud projection processing method based on a user perspective in an embodiment of the present invention;

[0119] The steps of the point cloud projection processing method based on the user's perspective include:

[0120] Input point cloud object;

[0121] Generate point cloud fragments and calculate the normal vector of each point in turn;

[0122] Each point is preliminarily divided, and the point cloud object is divided to generate a first-level point cloud fragment. At this time, the first-level point cloud fragment is preliminarily divided according to the differences of six standard vectors;

[0123] Generate secondary point cloud fragments, wherein the primary point cloud fragments are further subdivided. Since the vectors between the primary point cloud fragments of the same standard vector are the same, the points in the primary point cloud fragments are further clustered and divided according to the distance between the points. It is worth noting that the processing process of generating secondary point cloud fragments from primary point cloud fragments can be completed by using the usual technical means in the existing point cloud projection step, and the secondary point cloud fragments are used as the cloud fragments to be processed to perform the following steps. In the present invention, obtaining the cloud fragments to be processed is not limited to the above-mentioned steps of the primary point cloud fragments and the secondary point cloud fragments. Any preprocessing steps of point cloud fragments in the prior art are within the scope of the present invention.

[0124] Calculate the main vector according to the user's viewing angle, and take the standard vector closest to the viewing angle vector as the main vector;

[0125] The cloud fragments to be processed are distinguished. According to the relationship between the normal vector and the main vector of the secondary point cloud fragments, the secondary point cloud fragments are divided into main fragments, edge fragments, and irrelevant fragments. Among them, the main fragments are those with the same normal vector as the main vector, the irrelevant fragments are those with the opposite normal vector to the main vector, and the edge fragments are those with the normal vector perpendicular to the main vector;

[0126] The projection direction of the secondary point cloud fragment is calculated, and the projection direction of the edge fragment in the secondary point cloud fragment is optimized and adjusted. The specific optimization and adjustment process is as described above;

[0127] Projecting the secondary point cloud fragments, projecting the optimized and adjusted secondary point cloud fragments, namely, the main fragments, the edge fragments, and the irrelevant fragments according to the adjusted projection direction;

[0128] After projection, a compressible two-dimensional image is generated;

[0129] generating a two-dimensional video sequence;

[0130] compressing a two-dimensional video sequence; and

[0131] Finally, the compressed two-dimensional video is output as a code stream.

[0132] Attached Figure 6 Schematic diagram of the process of a point cloud receiving and processing method based on a user perspective in an embodiment of the present invention;

[0133] The point cloud receiving and processing method based on the user perspective in this embodiment is the inverse process of the above-mentioned point cloud projection processing method based on the user perspective, and the decompression on the receiving end side is the inverse process of the compression on the sending end side.

[0134] Attached Figure 6 As shown, first input the compressed code stream;

[0135] After decompression, a two-dimensional video sequence is generated;

[0136] Then obtain a two-dimensional image (usually including three categories: geometric map, attribute map, occupancy map);

[0137] Generate secondary point cloud segments using 2D image information;

[0138] Then, the secondary point cloud fragments are used to generate the primary point cloud fragments;

[0139] The point cloud object is generated by combining the primary point cloud fragment with the auxiliary information, and the projection relationship identification field is used to perform reverse projection processing to restore the two-dimensional point cloud fragment to a three-dimensional point cloud object;

[0140] Finally, after processing, a complete point cloud object is obtained for output.

[0141] In addition, in this embodiment, a point cloud projection processing system based on the user's perspective is also provided on the transmitting end side. Figure 7 4 is a schematic block diagram of a point cloud projection processing system based on user perspective in an embodiment of the present invention.

[0142] like Figure 7 As shown, the point cloud projection processing system based on user perspective includes a point cloud input module, a fragment generation module, a fragment division module, a projection vector adjustment module, a projection module, a compression module, and a code stream output module.

[0143] Point cloud input module, used to input point cloud objects;

[0144] The fragment generation module is used to generate the cloud fragments to be processed according to the predetermined rules from the point cloud objects, and perform subsequent projection optimization and adjustment;

[0145] A fragment division module, used for dividing the cloud fragments to be processed into main fragments, edge fragments and irrelevant fragments;

[0146] A projection vector adjustment module, for maintaining the projection vector of the edge segment or changing it based on the user's perspective according to the projection vector adjustment rule, taking into account the spatial connection between the edge segment and the main segment;

[0147] A projection module, used for projecting the main segment, edge segment and irrelevant segment after the projection adjustment is completed;

[0148] The compression module and the code stream output module are used to compress the projected two-dimensional image; and subsequently output the code stream.

[0149] On the transmitting end side, the projection module uses the optimized and adjusted projection vector for projection, and identifies it through a projection relationship identification field, which includes an index parameter value of the optimized and adjusted projection vector and a corresponding projection vector value.

[0150] In addition, in this embodiment, a point cloud receiving and processing system based on the user perspective is also provided on the receiving end side. Figure 8 It is a schematic block diagram of a point cloud receiving and processing system based on user perspective in an embodiment of the present invention.

[0151] like Figure 8 As shown, the point cloud receiving and processing system based on the user perspective includes: a code stream processing module and a reverse projection module, the code stream processing module is used for receiving the input of the code stream;

[0152] The reverse projection module is used to reversely project the two-dimensional point cloud fragments and restore the two-dimensional point cloud fragments to point cloud objects according to the instructions of the projection relationship identification field. The projection relationship identification field contains the index parameter value of the optimized and adjusted projection vector and the corresponding projection vector value.

[0153] In the present invention, the functions of each module in the point cloud projection processing system based on user perspective and the point cloud receiving processing system based on user perspective correspond to the optimization and adjustment process in the above-mentioned point cloud projection processing method based on user perspective and the point cloud receiving processing method based on user perspective, and can be inferred. Then, the structure and technical elements possessed by the device can be formed by the corresponding conversion of the generating method and the receiving method, and the repeated description is omitted here.

[0154] Functions and effects of this embodiment

[0155] The above-mentioned optimized adjustment projection of the point cloud fragment takes into account the user's perspective and the spatial correlation with the main vector, combines the characteristics of the user observing the point cloud from a specific perspective, optimizes the projection vector of the point cloud, and focuses more on the direction where the user pays more attention, thereby improving the visual effect when presented after decompression.

[0156] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0157] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0158] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A point cloud projection processing method based on user perspective, characterized in that: include: Step 1: distinguish the main segment and multiple edge segments of the point cloud object according to the user's perspective; Step 2: According to the spatial relationship between the edge fragment and the main fragment, the projection vector of the edge fragment is maintained or changed based on the user's perspective; Step 3: Project according to the adjusted projection vector to obtain a compressible two-dimensional image and perform point cloud processing.

2. The point cloud projection processing method based on user perspective according to claim 1, characterized in that: The step 1 comprises: Dividing the point cloud object into a plurality of first-level point cloud fragments based on a plurality of standard normal vectors; In the same first-level point cloud segment, points with the same normal vector and a distance less than a threshold are further divided into second-level point cloud segments as cloud segments to be processed; Determine the principal vector in the standard normal vector according to the user's viewing angle; Based on the main vector, the cloud fragment to be processed is divided into a main fragment and a plurality of edge fragments.

3. The point cloud projection processing method based on user perspective according to claim 1, characterized in that: The step 2 comprises: Determine the principal vector in the standard normal vector according to the user's perspective; Project the 3D edge box of the main segment along the direction of the main vector to obtain a main 2D projection image; Project the 3D edge box of the edge segment along the direction of the main vector to obtain an edge 2D projection image; Based on each edge two-dimensional projection map, the intersection and union ratio is calculated with each main two-dimensional projection map respectively; Multiple intersection-and-union ratios are compared with a preset threshold value as a basis for determining spatial connections.

4. The point cloud projection processing method based on user perspective according to claim 3 is characterized in that: The preset threshold value depends on the surface characteristics of the point cloud object, distinguishing the surface characteristics of the object point cloud object from the surface characteristics of the person point cloud object. The threshold value is adjusted and determined based on the experimental results; If there is a case where the multiple intersection-and-union ratios are greater than a preset threshold value, the projection vector of the edge segment is changed to a principal vector; if there is no case where the multiple intersection-and-union ratios are greater than the preset threshold value, the projection vector of the edge segment is maintained.

5. The point cloud projection processing method based on user perspective according to claim 1, characterized in that: According to the user's position when viewing the point cloud object, a vector pointing from the point cloud to the user's observation point is used to represent the user's viewing angle, which is called the viewing angle vector. The distance between the viewing angle vector and the six standard normal vectors is calculated, and the standard normal vector closest to the viewing angle vector is called the main vector under the viewing angle.

6. The point cloud projection processing method based on user perspective according to claim 1, characterized in that: The irrelevant fragments of the point cloud object are also distinguished according to the user's perspective; The standard normal vector closest to the user's viewing angle is determined as the principal vector; Determine the to-be-processed point cloud fragment having the same main vector as the main vector among the plurality of standard normal vectors as the main fragment; Determine the to-be-processed point cloud fragments that are opposite to the main vector among the plurality of standard normal vectors as irrelevant fragments; Determine the to-be-processed point cloud fragment associated with the main vector in the plurality of standard normal vectors as an edge fragment; The multiple standard normal vectors are in one-to-one correspondence with the multiple faces of the three-dimensional edge box, and the standard normal vectors are perpendicular to the corresponding faces; The projection vector of the irrelevant fragment is the normal vector of the irrelevant fragment; The three-dimensional edge box enclosing the point cloud object includes: a main fragment projection aggregation surface of the main fragment aggregation, five edge fragment projection aggregation surfaces of the edge fragment aggregation, and an irrelevant fragment projection aggregation surface of the irrelevant fragment aggregation.

7. The point cloud projection processing method based on user perspective according to claim 6, characterized in that: The step 3 comprises: The main segment, the edge segment after projection processing and the irrelevant segment are projected according to the adjusted projection vectors to generate compressible two-dimensional images, which are further processed to obtain a two-dimensional video sequence, and the two-dimensional video sequence is compressed using a video codec.

8. The point cloud projection processing method based on user perspective according to claim 7, characterized in that: In the subsequent point cloud processing after projection, the projection relationship identification field is used to identify the projection vector of the point in the point cloud object after processing and adjustment, including the projection vector value and the corresponding index value.

9. A point cloud projection processing system based on user perspective, characterized in that: The point cloud projection processing method based on user perspective described in any one or more of claims 1 to 8 comprises: Fragment segmentation module: distinguishes the main fragment and multiple edge fragments of the point cloud object according to the user's perspective; Projection vector adjustment module: maintains the projection vector of the edge segment or changes it based on the user's perspective according to the spatial connection between the edge segment and the main segment; Projection module: Projecting the points in the three-dimensional point cloud object according to the projection vector adjusted by projection to obtain a compressible two-dimensional image; Code stream processing module: receives point cloud code stream and processes it to obtain a two-dimensional image; Reverse projection module: Use the projection relationship identification field to perform reverse projection processing to restore the two-dimensional image to a three-dimensional point cloud object; The projection relationship identification field indicates: the projection vector after processing and adjustment at the sending end, including the projection vector value and the corresponding index value.

10. A point cloud receiving and processing method based on user perspective, characterized in that: The point cloud projection processing method based on user perspective described in any one or more of claims 1 to 8 comprises: Receive the point cloud code stream and process it to obtain a two-dimensional image, use the projection relationship identification field to perform inverse projection processing, and restore the two-dimensional point cloud fragment to a three-dimensional point cloud object; The projection relationship identification field indicates: the projection vector after processing and adjustment at the sending end, including the projection coordinate value and the corresponding index value.

Images