Point cloud encoding and decoding methods and codecs

Abstract

The present application discloses a point cloud encoding and decoding method and a codec, which relates to the field of encoding and decoding technology and is used to provide a technical solution for effectively encoding and decoding point cloud data. The point cloud encoding method includes: obtaining auxiliary information of the union occupancy map of a point cloud group; wherein the point cloud group includes N frames of point cloud, N≥2, N is an integer, and the union occupancy map is the union of occupancy maps of N patches with matching relationships in the N frames of point cloud; the auxiliary information of the union occupancy map is used to determine the auxiliary information of the N patches; and the syntax element is encoded into the bitstream, and the syntax element includes the auxiliary information of the union occupancy map. The technical solution provided by the present application helps to save bitstream transmission overhead without affecting the encoding and decoding performance.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on September 30, 2018, with application number 201811166410.8 and application name “Point Cloud Encoding and Decoding Method and Codec”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of coding and decoding technology, and in particular to a point cloud coding and decoding method and a codec. Background Art

[0003] With the continuous development of 3D sensor (such as 3D scanner) technology, it is becoming more and more convenient to collect point cloud data, and the scale of the collected point cloud data is also getting larger and larger. Therefore, how to effectively encode point cloud data has become an urgent problem that needs to be solved. Summary of the invention

[0004] The embodiments of the present application provide a point cloud encoding and decoding method and a codec, which are used to provide a technical solution for effectively encoding and decoding point cloud data. Specifically, it helps to save code stream transmission overhead without affecting the encoding and decoding performance.

[0005] In a first aspect, a point cloud encoding method is provided, comprising: obtaining auxiliary information of a union occupancy map of a point cloud group; wherein the point cloud group comprises N frames of point cloud, N≥2, N is an integer, and the union occupancy map is a union of occupancy maps of N point cloud patches having a matching relationship in the N frames of point cloud; the auxiliary information of the union occupancy map is used to determine (specifically, for a decoder to determine) auxiliary information of the N patches; and encoding a syntax element into a bitstream, the syntax element comprising the auxiliary information of the union occupancy map. The executor of the technical solution may be an encoder.

[0006] Each of the N patches comes from one of the N frames of point clouds, and different patches come from different point clouds. For example, assuming N=4, the four patches with matching relationships in the four frames of point clouds come from or are included in the four frames of point clouds. A patch that has a matching relationship with a patch is a matching patch of a target patch, and the target patch is the patch or a matching patch of the patch. Each of the N patches can be called a global matched patch.

[0007] Due to the correlation between the union occupancy map and the occupancy maps of the N patches corresponding to the union occupancy map, such as the position information of the union occupancy map in the global occupancy map, the position information of the occupancy maps of the N patches in the occupancy map of the point cloud to which they belong, and / or the union occupancy map is the union of the occupancy maps of the N patches, etc., the auxiliary information of the union occupancy map can be represented or approximately represented. In this way, the bitstream transmission overhead can be saved without affecting the encoding and decoding performance.

[0008] In one possible design, the auxiliary information of the union occupancy map includes at least one of the following information: a coordinate value of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and a coordinate value of the union occupancy map corresponding to the three-dimensional space. The global occupancy map is an occupancy map obtained by packaging multiple union occupancy maps in the point cloud group.

[0009] The coordinate values ​​of the union occupancy map in the global occupancy map include: u0 and / or v0 of the union occupancy map.

[0010] The size information of the union occupancy map in the global occupancy map includes: sizeU0 and / or sizeV0 of the union occupancy map.

[0011] The coordinate value of the union occupancy map in the three-dimensional space refers to the minimum (or maximum) coordinate value of the N patches corresponding to the union occupancy map in the three-dimensional space. Taking the minimum coordinate value as an example, the coordinate value of the union occupancy map in the three-dimensional space may include: the minimum u1 value, the minimum v1 value and / or the minimum d1 value of the N patches in the three-dimensional space.

[0012] In one possible design, the difference between the corresponding auxiliary information of one or more patches in the above-mentioned N patches and the auxiliary information of the union occupancy map. The corresponding auxiliary information of the one or more patches refers to the auxiliary information of the same type as the auxiliary information of the union occupancy map, for example, all are u0, etc. This possible design uses a differential method to transmit the auxiliary information of one or more patches corresponding to the union occupancy map, which helps to improve the accuracy and / or setting flexibility of the auxiliary information of the one or more patches in the N patches. In the specific implementation process, optionally, if the auxiliary information of a patch in the N patches is the same as the auxiliary information of the union occupancy map, the difference between the auxiliary information of the patch and the auxiliary information of the union occupancy map is 0, and the difference may not be encoded into the bitstream.

[0013] In one possible design, obtaining auxiliary information of the union occupancy map of the point cloud group includes: obtaining auxiliary information of the union occupancy map based on the auxiliary information of the N patches.

[0014] In a possible design, based on the auxiliary information of the N patches, the auxiliary information of the union occupancy map is obtained, including:

[0015] When the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map in the global occupancy map, the maximum coordinate value or the minimum coordinate value of the occupancy map of the N patches in the occupancy map of the corresponding point cloud is used as the coordinate value of the union occupancy map in the global occupancy map. For example, the minimum or maximum value of u0 of the occupancy maps of the N patches is used as u0 of the union occupancy map; and / or the minimum or maximum value of v0 of the occupancy maps of the N patches is used as v0 of the union occupancy map.

[0016] And / or, when the auxiliary information of the union occupancy map includes the size information of the union occupancy map in the global occupancy map, the maximum size information of the occupancy maps of the N patches in the occupancy map of the corresponding point cloud is used as the size information of the union occupancy map in the global occupancy map. For example, the maximum value of sizeU0 of the occupancy maps of the N patches is used as the sizeV0 of the union occupancy map, and / or the maximum value of sizeU0 of the occupancy maps of the N patches is used as the sizeV0 of the union occupancy map.

[0017] And / or, when the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map in the three-dimensional space, the maximum coordinate value or the minimum coordinate value of the N patches in the corresponding point cloud is used as the coordinate value of the union occupancy map in the three-dimensional space. For example, taking the minimum coordinate value as an example, the minimum value of u1 of the N patches is used as the value of u1 of the union occupancy map, the minimum value of v1 of the N patches is used as the value of v1 of the union occupancy map, and the minimum value of d1 of the N patches is used as the value of d1 of the union occupancy map.

[0018] Among them, the coordinate axis of the occupancy map of a patch in the occupancy map of the corresponding point cloud is the coordinate value of the occupancy map of the patch in the occupancy map of the point cloud to which the patch belongs. The coordinate value of a patch in the corresponding point cloud is the coordinate value of the patch in the point cloud to which the patch belongs.

[0019] In a possible design, the method further includes: reconstructing the N frames of point cloud according to auxiliary information of multiple patches (such as all patches) in the N frames of point cloud, wherein the multiple patches include the N patches.

[0020] In a possible design, the point cloud group is a group of frame (GOF) GOF or a sub-GOF, and the sub-GOF includes continuous multi-frame point clouds in a GOF.

[0021] In one possible design, if the point cloud group is a sub-GOF, the above-mentioned syntax element also includes information indicating the position of the point cloud contained in the sub-GOF (such as the point cloud of each frame) in the GOF to which the sub-GOF belongs, such as the number of the starting frame of the sub-GOF and / or the number of frames contained in the sub-GOF.

[0022] In a second aspect, a point cloud decoding method is provided, comprising: parsing a bitstream to obtain a syntax element, the syntax element comprising auxiliary information of a union occupancy map of a point cloud group, the point cloud group comprising N frames of point clouds, N≥2, N is an integer, the union occupancy map is the union of occupancy maps of N patches having a matching relationship in the N frames of point clouds; determining auxiliary information of the N patches based on the auxiliary information of the union occupancy map obtained by parsing; reconstructing the N frames of point clouds based on the auxiliary information of multiple patches in the N frames of point clouds; wherein the multiple patches include the N patches. The executor of the technical solution may be a decoder. Optionally, the method may further include displaying the reconstructed point cloud.

[0023] In a possible design, the auxiliary information of the union occupancy map includes at least one of the following information: a coordinate value of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and a coordinate value of the union occupancy map corresponding to the three-dimensional space. The global occupancy map is an occupancy map obtained by packaging multiple union occupancy maps in the point cloud group.

[0024] In one possible design, the syntax element also includes: the difference between the corresponding auxiliary information of one or more patches in the N patches and the auxiliary information of the union occupancy map. In this case, the auxiliary information of the N patches is determined according to the auxiliary information of the union occupancy map obtained by parsing, including: determining the auxiliary information of the N patches according to the auxiliary information of the union occupancy map obtained by parsing, and the difference between the corresponding auxiliary information of one or more patches in the N patches and the auxiliary information of the union occupancy map. For example, the auxiliary information of the union occupancy map and the difference between the corresponding auxiliary information of one patch of the N patches and the auxiliary information of the union occupancy map are summed to obtain the corresponding auxiliary information of the patch.

[0025] In a possible design, the point cloud group is a GOF or a sub-GOF. A sub-GOF includes multiple consecutive frames of point clouds in a GOF.

[0026] In a possible design, if the point cloud group is a sub-GOF, the syntax element further includes: information indicating the position of each frame of point cloud contained in the sub-GOF in the GOF to which the sub-GOF belongs, such as the number of the starting frame of the sub-GOF and / or the number of frames contained in the sub-GOF, etc. In this case, the method further includes: determining the position of each frame of point cloud contained in the sub-GOF in the GOF to which the sub-GOF belongs according to the information obtained by parsing indicating the position of each frame of point cloud contained in the sub-GOF in the GOF to which the sub-GOF belongs.

[0027] The second aspect or any possible design provided by the second aspect corresponds to the first aspect or the corresponding possible design of the first aspect. Therefore, the explanation of the relevant content, the specific implementation method, and the description of the beneficial effects in the second aspect or any possible design provided by the second aspect can be referred to the description in the above-mentioned first aspect or the possible design of the first aspect, or inferred based on the description in the above-mentioned first aspect or the possible design of the first aspect, and will not be repeated here.

[0028] In a third aspect, a point cloud encoding method is provided, comprising: determining indication information, and encoding the indication information into a bitstream. The indication information indicates whether to use a target point cloud encoding method to encode the auxiliary information of N patches in a point cloud group, and the target point cloud encoding method includes the point cloud encoding method described in the first aspect or any possible design provided in the first aspect; the point cloud group includes N frames of point cloud, N≥2, N is an integer; the N patches are N patches with a matching relationship in the N frames of point cloud. In other words, the technical solution provided in the embodiment of the present application supports the technical solution of selecting a target point cloud encoding method from at least two candidate point cloud encoding methods.

[0029] In a fourth aspect, a point cloud decoding method is provided, comprising: parsing a bitstream to obtain indication information, the indication information indicating whether to use a target point cloud decoding method to decode the auxiliary information of N patches in a point cloud group, the target point cloud decoding method including the point cloud decoding method described in the second aspect or any possible design provided in the second aspect; the point cloud group includes N frames of point cloud, N≥2, N is an integer; the N patches are N patches with a matching relationship in the N frames of point cloud; when the indication information indicates that the target point cloud decoding method is used for decoding, the target point cloud decoding method is used to process the auxiliary information of the N patches. In other words, the technical solution provided in the embodiment of the present application supports the technical solution of selecting a target point cloud decoding method from at least two candidate point cloud decoding methods.

[0030] In a fifth aspect, an encoder is provided, comprising: various modules for executing the method provided by the first aspect or any possible design of the first aspect, or various modules for executing the method provided by the third aspect or any possible design of the third aspect.

[0031] In one possible implementation, the encoder can be divided into functional modules according to the method provided by the first / third aspect or any possible implementation in the first / third aspect. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module.

[0032] In a possible implementation, the encoder includes a packing module and an auxiliary information encoding module.

[0033] For example, when applied to the technical solution of the first aspect, the packing module is used to obtain the auxiliary information of the union occupancy map of the point cloud group; the point cloud group includes N frames of point clouds, N≥2, N is an integer, and the union occupancy map is the union of occupancy maps of N patches with matching relationships in the N frames of point clouds; the auxiliary information of the union occupancy map is used to determine the auxiliary information of the N patches. The auxiliary information encoding module is used to encode the syntax elements containing the auxiliary information of the union occupancy map into the bitstream.

[0034] For another example, when applied to the technical solution of the third aspect, the packaging module is used to determine indication information, which indicates whether to use a target point cloud encoding method to encode the auxiliary information of N patches in the point cloud group, and the target point cloud encoding method includes the point cloud encoding method described in the first aspect or any possible design provided in the first aspect; the point cloud group includes N frames of point cloud, N≥2, N is an integer; the N patches are N patches with matching relationships in the N frames of point cloud. The auxiliary information encoding module is used to encode the indication information into the bitstream.

[0035] In a sixth aspect, a decoder is provided, comprising: various modules for executing the method provided by the second aspect or any possible design of the second aspect, or various modules for executing the method provided by the fourth aspect or any possible design of the fourth aspect.

[0036] In one possible implementation, the decoder can be divided into functional modules according to the method provided by the second / fourth aspect or any possible implementation in the second / fourth aspect. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module.

[0037] In a possible implementation, the decoder includes an auxiliary information decoding module and a point cloud reconstruction module.

[0038] For example, when applied to the technical solution of the second aspect, the auxiliary information decoding module is used to parse the bitstream to obtain a syntax element, which includes auxiliary information of the union occupancy map of the point cloud group, and the union occupancy map is the union of occupancy maps of N patches with matching relationships in the N-frame point cloud; according to the auxiliary information of the union occupancy map obtained by parsing, the auxiliary information of the N patches is determined. The point cloud reconstruction module is used to reconstruct the N-frame point cloud based on the auxiliary information of multiple patches in the N-frame point cloud; wherein the multiple patches include the N patches.

[0039] For another example, when applied to the technical solution of the fourth aspect, the auxiliary information decoding module is used to parse the code stream to obtain indication information, which indicates whether to use the target point cloud decoding method to decode the auxiliary information of N patches in the point cloud group, and the target point cloud decoding method includes the point cloud decoding method described in the second aspect or any possible design provided in the second aspect; the point cloud group includes N frames of point cloud, N≥2, N is an integer; the N patches are N patches with matching relationships in the N frames of point cloud; and, when the indication information indicates to use the target point cloud decoding method for decoding, the target point cloud decoding method is used to decode the auxiliary information of the N patches. The point cloud reconstruction module is used to reconstruct the N frames of point cloud based on the auxiliary information of multiple patches in the N frames of point cloud obtained by decoding; the multiple patches include the N patches.

[0040] In a seventh aspect, a device for encoding point cloud data is provided, and the device may include:

[0041] A memory is used to store point cloud data.

[0042] An encoder is used to execute the point cloud encoding method described in the first aspect or any possible design of the first aspect, or the third aspect or any possible design of the third aspect.

[0043] In an eighth aspect, a coding device is provided, the device comprising a memory and a processor. The memory is used to store program code; the processor is used to call the program code to execute the point cloud coding method described in the first aspect or any possible design of the first aspect, or the third aspect or any possible design of the third aspect.

[0044] In a ninth aspect, a device for decoding point cloud data is provided, and the device may include:

[0045] A memory is used to store point cloud data.

[0046] A decoder is used to execute the point cloud decoding method described in the above-mentioned second aspect or any possible design of the second aspect, or the fourth aspect or any possible design of the fourth aspect.

[0047] In a tenth aspect, a decoding device is provided, the device comprising a memory and a processor. The memory is used to store program code; the processor is used to call the program code to execute the point cloud decoding method described in the second aspect or any possible design of the second aspect, or the fourth aspect or any possible design of the fourth aspect.

[0048] In the eleventh aspect, a computer-readable storage medium is provided, which stores a program code. When the program code runs on a computer, the computer executes part or all of the steps of the point cloud encoding method described in the first aspect or any possible design of the first aspect, or the third aspect or any possible design of the third aspect.

[0049] In the twelfth aspect, a computer program product is provided. When the computer program product runs on a computer, the computer executes part or all of the steps of the point cloud encoding method described in the first aspect or any possible design of the first aspect, or the third aspect or any possible design of the third aspect.

[0050] In the thirteenth aspect, a computer-readable storage medium is provided, which stores a program code. When the program code runs on a computer, the computer executes part or all of the steps of the point cloud decoding method described in the second aspect or any possible design of the second aspect, or the fourth aspect or any possible design of the fourth aspect.

[0051] In the fourteenth aspect, a computer program product is provided. When the computer program product runs on a computer, the computer executes part or all of the steps of the point cloud decoding method described in the second aspect or any possible design of the second aspect, or the fourth aspect or any possible design of the fourth aspect.

[0052] It should be understood that the beneficial effects of the above-mentioned related devices / equipment / computer-readable storage media / computer program products can all refer to the beneficial effects of the method embodiments provided in the corresponding aspects, and will not be repeated here.

[0053] It should be understood that the term "at least one (kind / frame)" in the embodiments of the present application includes one (kind frame) or multiple (kind frames). "Multiple (kind frames)" refers to two (kind frames) or more than two (kind frames). For example, at least one of A, B and C includes: A exists alone, B exists alone, A and B exist at the same time, A and C exist at the same time, B and C exist at the same time, and A, B and C exist at the same time. The term "and / or" in the embodiments of the present application is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The term character " / " in the embodiments of the present application generally indicates that the previously associated objects are in an "or" relationship. In a formula, the character " / " represents a division operation, such as A / B represents A divided by B. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of a packaging method applicable to an example of an embodiment of the present application;

[0055] Figure 2 and Figure 3 Based on Figure 1 An example process diagram of the packaging method shown;

[0056] Figure 4 is a schematic block diagram of a point cloud decoding system that can be applied to an example of an embodiment of the present application;

[0057] Figure 5 is a schematic block diagram of an encoder that can be used in an example of an embodiment of the present application;

[0058] Figure 6 is a schematic diagram of a point cloud, a patch, and an occupancy map of a point cloud that can be applied to an example of an embodiment of the present application;

[0059] Figure 7 is a schematic block diagram of a decoder that can be used in an example of an embodiment of the present application;

[0060] Figure 8 A schematic diagram of a point cloud encoding method provided in an embodiment of the present application;

[0061] Fig. 9 A schematic diagram of a flow chart of a point cloud decoding method provided in an embodiment of the present application;

[0062] Fig.10 A schematic diagram of a flow chart of another point cloud encoding method provided in an embodiment of the present application;

[0063] Fig.11 A schematic diagram of a flow chart of another point cloud decoding method provided in an embodiment of the present application;

[0064] Fig.12 A schematic block diagram of an encoder provided in an embodiment of the present application;

[0065] Fig.13 A schematic block diagram of a decoder provided in an embodiment of the present application;

[0066] Fig.14 A schematic block diagram of a decoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, the technologies and terms involved in the embodiments of the present application are explained below.

[0068] In the moving picture expert group (MPEG) point cloud coding method, the encoder first divides the point cloud to be encoded (i.e., the current frame or the current frame point cloud) into several patches according to certain criteria, and these patches have no overlapping areas. Then, each patch is projected from the three-dimensional space to the two-dimensional plane to obtain a two-dimensional image (i.e., the occupancy map of the patch). Next, the occupancy maps of all patches (or the occupancy maps of the patches after reducing the resolution) are tightly arranged on a two-dimensional image according to a certain rule to obtain the occupancy map of the current frame. This method of arranging the occupancy map of the patch is called packing. Specifically, the patches of the point cloud are arranged in a specific order, for example, in descending order of the width / height of the occupancy map of each patch; then, according to the order of the arranged patches, the occupancy maps of the patches are inserted into the available area of ​​the point cloud occupancy map in turn to obtain the occupancy map of the point cloud. Subsequently, the current frame depth map and the current frame texture map are generated in the packing order. Among them, the current frame depth map is a two-dimensional image generated by the depth obtained by the projection of each patch in the packing order. The current frame texture map is a two-dimensional image generated by projecting the texture maps of each patch in the order of packing. The current frame occupancy map is a two-value two-dimensional image, which is used to indicate whether each pixel position of the two-dimensional image is occupied by a point in the point cloud. Usually, in order to save coding bits, the resolution of the current frame occupancy map is lower than that of the current frame depth map and the current frame texture map. For the convenience of description, the following description will take the example of not considering whether the resolution is reduced.

[0069] To improve encoding or compression efficiency, a packaging method such as Figure 1 As shown, Figure 1 The method shown includes the following steps:

[0070] Step 1: Get the global matching patch of the point cloud group. The point cloud group includes N frames of point clouds. N ≥ 2, N is an integer.

[0071] Among them, the global matching patch of a frame point cloud refers to a patch in the frame point cloud and a patch with a matching relationship can be found in each frame point cloud in the point cloud group except the frame point cloud. At the same time, the patch that has a matching relationship with the patch is a global matching patch. Among them, a patch that has a matching relationship with a patch is a matching patch of a target patch, and the target patch is the patch or a matching patch of the patch. Optionally, a matching patch of a patch in a frame point cloud in another frame point cloud can be a patch in the other frame point cloud that has the largest intersection over union (IoU) with the patch and the IoU is greater than or equal to a preset threshold.

[0072] For any global matching patch, the number of patches that have a matching relationship with the global matching patch is N-1.

[0073] Step 2: Determine M union occupancy maps corresponding to M sets, each set includes N global matching patches, and the N global matching patches are patches with matching relationships in N frame point clouds (or patches with matching relationships across N frame point clouds). Optionally, the union occupancy map corresponding to the mth set is the union of the occupancy maps of each global matching patch in the mth set; 1≤m≤M, m and M are both integers.

[0074] Among them, the union occupancy map can also be called the union occupancy map.

[0075] The N global matching patches included in a set are patches with matching relationships across N frames of point clouds. It can be understood that: each of the N global matching patches comes from (or belongs to) a frame of point cloud, different global matching patches come from different point clouds, and the N global matching patches have a matching relationship.

[0076] The union occupancy map corresponding to the set to which a global matching patch belongs overwrites the occupancy map of the global matching patch.

[0077] Optionally, after the encoder executes step 2, the method may further include:

[0078] Step 2a: Obtain auxiliary information of the union occupancy map. The specific description of the auxiliary information of the union occupancy map and the specific implementation method of obtaining the auxiliary information of the union occupancy map can be referred to below and will not be repeated here. Of course, the order of step 2a and steps 3 to 4 is not limited in the embodiment of the present application.

[0079] Step 3: Pack the above M union occupancy maps to obtain a global occupancy map (i.e., the final occupancy map). The global occupancy map is used to determine the positions of the M union occupancy maps in the global occupancy map. The packing process can be considered as a process of updating the global occupancy map, such as arranging the M union occupancy maps in the order from top to bottom and from left to right in the available space of the global occupancy map to obtain the final occupancy map. The specific method can refer to the prior art and will not be repeated here.

[0080] Step 4: Pack each point cloud in the N-frame point cloud to obtain the occupancy map of the N-frame point cloud. The position (first position) of the occupancy map of the m-th global matching patch in the n-frame point cloud in the occupancy map of the n-frame point cloud corresponds to the position (second position) of the m-th union occupancy map of the M union occupancy maps in the global occupancy map; 1≤n≤N, n is an integer.

[0081] Among them, the first position corresponding to the second position may include: the coordinate value of the first position is the same as the coordinate value of the second position; or the coordinate value of the first position in its coordinate system is substantially the same as the coordinate value of the second position in its coordinate system; or the coordinate value of the first position is different from the coordinate value of the second position, but the position range of the second position covers the position range of the first position. Among them, the coordinate value of the first position can be represented by the position coordinates of the occupancy map of the mth global matching patch in the occupancy map of the nth frame point cloud in the occupancy map of the nth frame point cloud. The coordinate value of the second position can be represented by the position coordinates of the mth union occupancy map in the global occupancy map. The position range where the first position is located is the area occupied by the occupancy map of the mth global matching patch, and the position range where the second position is located is the area occupied by the mth union occupancy map.

[0082] The following is a simple example. Figure 1 The packaging method in is described below.

[0083] Figure 2 and Figure 3 Based on Figure 1 A process diagram of an example of a packaging method is provided. Figure 2 and Figure 3In , it is assumed that the point cloud group includes 4 frames of point clouds, each frame of point cloud includes 10 patches, and the global matching patches are: patch11, patch12 and patch13 in point cloud 1; patch21, patch22 and patch23 in point cloud 2; patch31, patch32 and patch33 in point cloud 3; patch41, patch42 and patch43 in point cloud 4. In addition, patch11, patch21, patch31 and patch41 have a matching relationship, and these patches form set 1. Patch12, patch22, patch32 and patch42 have a matching relationship, and these patches form set 2. Patch13, patch23, patch33 and patch43 have a matching relationship, and these patches form set 3. In other words, the above M sets are specifically sets 1 to 3, and each set contains 4 patches with matching relationships.

[0084] The global occupancy map obtained based on this example can be Figure 2 shown. Figure 2 The larger rectangle in represents the global occupancy map, Figure 2 The ellipse, triangle and smaller rectangle in the figure represent the union patch occupancy maps corresponding to sets 1 to 3, respectively.

[0085] Based on this example, after executing step 4 above, the occupancy map of point cloud 1 can be as follows Figure 3 As shown in (a) in the figure, the occupancy map of point cloud 2 can be Figure 3 As shown in (b) in . Figure 3 Only the occupancy maps of point cloud 1 and some patches in the occupancy map of point cloud 2 are illustrated, and the occupancy maps of point cloud 3 and point cloud 4 are not illustrated.

[0086] contrast Figure 2 and Figure 3 It can be seen that the position of the occupancy map of patch11 in the occupancy map of point cloud 1 and the position of the occupancy map of patch21 in the occupancy map of point cloud 2 both correspond to the union patch occupancy Figure 1 The position of the occupancy map of patch12 in the occupancy map of point cloud 1 and the position of the occupancy map of patch22 in the occupancy map of point cloud 2 correspond to the union patch occupancy map. Figure 2 The position of the occupancy map of patch13 in the occupancy map of point cloud 1 and the position of the occupancy map of patch23 in the occupancy map of point cloud 2 correspond to the union patch occupancy map. Figure 3 Position in the global occupancy map.

[0087] As an example, regarding Figure 1 The explanation of the relevant contents in the technical solution shown and the description of the specific implementation method can, for example but not limited to, refer to other patents previously applied for by the applicant.

[0088] According to the traditional point cloud encoding method, the encoder needs to encode the information of each patch in a frame of point cloud into the bitstream. This will result in a large bitstream transmission overhead. To this end, the embodiment of the present application provides a point cloud encoding and decoding method and a codec to save bitstream transmission overhead without affecting the encoding and decoding performance.

[0089] In order to explain the patch information, we first explain the coordinate system defined in the point cloud decoding system. For the frame-level occupancy map (two-dimensional image), the horizontal axis is usually defined as the U0 axis and the vertical axis is the V0 axis. For the point cloud (three-dimensional space), the horizontal axis is usually defined as the U1 axis, the vertical axis is the V1 axis, and the coordinate axis perpendicular to the U1 axis and the V1 axis is the D1 axis. In other words, the U1 axis is the X axis in the world coordinate system, the V1 axis is the Y axis in the world coordinate system, and the D axis is the Z axis in the world coordinate system.

[0090] For an example and description of patch information, please refer to the following Table 1:

[0091] Table 1

[0092]

[0093] Based on Table 1, when the patch-level occupancy map is the occupancy map of the patch in the point cloud, the frame-level occupancy map can be the occupancy map of the point cloud. When the patch-level occupancy map is the union occupancy map, the frame-level occupancy map can be the global occupancy map. The explanation of the patch information involved below can refer to Table 1, which is explained here uniformly and will not be repeated below.

[0094] The following describes a schematic block diagram of a system architecture and a schematic block diagram of a device that can be used in an embodiment of the present application.

[0095] Figure 4 1 is a schematic block diagram of a point cloud decoding system 1 that can be used in an example of an embodiment of the present application. The term "point cloud decoding" or "decoding" can generally refer to point cloud encoding or point cloud decoding. The encoder 100 of the point cloud decoding system 1 can encode the point cloud to be encoded according to any point cloud encoding method proposed in the present application. The decoder 200 of the point cloud decoding system 1 can decode the point cloud to be decoded according to the point cloud decoding method corresponding to the point cloud encoding method used by the encoder proposed in the present application.

[0096] like Figure 4As shown, the point cloud decoding system 1 includes a source device 10 and a destination device 20. The source device 10 generates encoded point cloud data. Therefore, the source device 10 can be referred to as a point cloud encoding device. The destination device 20 can decode the encoded point cloud data generated by the source device 10. Therefore, the destination device 20 can be referred to as a point cloud decoding device. Various implementations of the source device 10, the destination device 20, or both may include one or more processors and a memory coupled to the one or more processors. The memory may include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures that can be accessed by a computer, as described herein.

[0097] Source device 10 and destination device 20 may comprise a variety of devices, including desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called “smart” phones, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, or the like.

[0098] The destination device 20 may receive the encoded point cloud data from the source device 10 via the link 30. The link 30 may include one or more media or devices capable of moving the encoded point cloud data from the source device 10 to the destination device 20. In one example, the link 30 may include one or more communication media that enable the source device 10 to send the encoded point cloud data directly to the destination device 20 in real time. In this example, the source device 10 may modulate the encoded point cloud data according to a communication standard (e.g., a wireless communication protocol), and may send the modulated point cloud data to the destination device 20. The one or more communication media may include wireless and / or wired communication media, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). The one or more communication media may include routers, switches, base stations, or other equipment that facilitates communication from the source device 10 to the destination device 20.

[0099] In another example, the encoded data may be output from the output interface 140 to the storage device 40. Similarly, the encoded point cloud data may be accessed from the storage device 40 via the input interface 240. The storage device 40 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded point cloud data.

[0100] In another example, storage device 40 may correspond to a file server or another intermediate storage device that may hold the encoded point cloud data generated by source device 10. Destination device 20 may access the stored point cloud data from storage device 40 via streaming or downloading. The file server may be any type of server capable of storing the encoded point cloud data and sending the encoded point cloud data to destination device 20. Example file servers include a network server (e.g., for a website), a file transfer protocol (FTP) server, a network attached storage (NAS) device, or a local disk drive. Destination device 20 may access the encoded point cloud data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of both that is suitable for accessing the encoded point cloud data stored on a file server. The transmission of the encoded point cloud data from storage device 40 may be a streaming transmission, a download transmission, or a combination of both.

[0101] Figure 4 The point cloud decoding system 1 described in is merely an example, and the technology of the present application may be applicable to point cloud decoding (e.g., point cloud encoding or point cloud decoding) devices that do not necessarily include any data communication between the point cloud encoding device and the point cloud decoding device. In other examples, data is retrieved from a local memory, streamed over a network, and so on. The point cloud encoding device may encode the data and store the data to the memory, and / or the point cloud decoding device may retrieve the data from the memory and decode the data. In many examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to the memory and / or retrieve data from the memory and decode the data.

[0102] exist Figure 4In an example of , source device 10 includes data source 120, encoder 100, and output interface 140. In some examples, output interface 140 may include a modem and / or a transmitter. Data source 120 may include a point cloud capture device (e.g., a camera), a point cloud archive containing previously captured point cloud data, a point cloud feed interface for receiving point cloud data from a point cloud content provider, and / or a computer graphics system for generating point cloud data, or a combination of these sources of point cloud data.

[0103] Encoder 100 may encode point cloud data from data source 120. In some examples, source device 10 sends the encoded point cloud data directly to destination device 20 via output interface 140. In other examples, the encoded point cloud data may also be stored on storage device 40 for later access by destination device 20 for decoding and / or playback.

[0104] exist Figure 4 In an example of , destination device 20 includes input interface 240, decoder 200, and display device 220. In some examples, input interface 240 includes a receiver and / or a modem. Input interface 240 may receive encoded point cloud data via link 30 and / or from storage device 40. Display device 220 may be integrated with destination device 20 or may be external to destination device 20. In general, display device 220 displays decoded point cloud data. Display device 220 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0105] although Figure 4 1, but in some aspects, encoder 100 and decoder 200 may be integrated with an audio encoder and decoder, respectively, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software to handle the encoding of both audio and video in a common data stream or in separate data streams. In some examples, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP), if applicable.

[0106] The encoder 100 and the decoder 200 can each be implemented as any of a variety of circuits such as one or more microprocessors, digital signal processors (digital signal processing, DSP), application specific integrated circuits (application specific integrated circuits, ASIC), field programmable gate arrays (field-programmable gate arrays, FPGA), discrete logic, hardware, or any combination thereof. If the present application is partially implemented in software, the device can store instructions for the software in a suitable non-volatile computer-readable storage medium, and can use one or more processors to execute the instructions in hardware to implement the technology of the present application. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) can be considered as one or more processors. Each of the encoder 100 and the decoder 200 can be included in one or more encoders or decoders, any of which can be integrated as part of a combined encoder / decoder (encoder-decoder) in a corresponding device.

[0107] The application may generally refer to encoder 100 as "signaling" or "sending" certain information to another device, such as decoder 200. The term "signaling" or "sending" may generally refer to the transmission of syntax elements and / or other data used to decode the compressed point cloud data. This transmission may occur in real time or near real time. Alternatively, this communication may occur over a period of time, such as when the syntax elements are stored in the encoded bitstream to a computer-readable storage medium at the time of encoding, and the decoding device may then retrieve the syntax elements at any time after the syntax elements are stored to such medium.

[0108] like Figure 5 , which is a schematic block diagram of an encoder 100 that can be used in an example of an embodiment of the present application. Figure 5 The MPEG (Moving Picture Expert Group) point cloud compression (PCC) coding framework is used as an example to illustrate. Figure 5 In the example, the encoder 100 may include a patch information generation module 101, a packing module 102, a depth map generation module 103, a texture map generation module 104, a first filling module 105, an image or video based encoding module 106, an occupancy map encoding module 107, an auxiliary information encoding module 108 and a multiplexing module 109. In addition, the encoder 100 may also include a point cloud filtering module 110, a second filling module 111 and a point cloud reconstruction module 112. Among them:

[0109] The patch information generation module 101 is used to use a certain method to segment a frame of point cloud to generate multiple patches, and obtain relevant information of the generated patches, etc. Among them, patch refers to a set of some points in a frame of point cloud, and usually one connected area corresponds to one patch. The relevant information of the patch may include but is not limited to at least one of the following information: the number of patches divided into the point cloud, the position information of the patch in three-dimensional space, the index of the normal coordinate axis of the patch, the depth map generated by projecting the patch from three-dimensional space to two-dimensional space, the size of the depth map of the patch (such as the width and height of the depth map), the occupancy map generated by projecting the patch from three-dimensional space to two-dimensional space, etc. Part of the relevant information, such as the number of patches divided into the point cloud, the index of the normal coordinate axis of the patch, the size of the depth map of the patch, the position information of the patch in the point cloud, the size information of the occupancy map of the patch, etc., can be sent to the auxiliary information encoding module 108 as auxiliary information for encoding (i.e., compression encoding). The occupancy map of each patch can be sent to the packaging module 102 for packaging. On the other hand, the specific position information of each patch in the point cloud occupancy map and the depth map of each patch can be sent to the depth map generation module 103.

[0110] After the packing module 102 obtains the occupancy map of the point cloud, on the one hand, the occupancy map of the point cloud can be filled by the second filling module 111 and then sent to the occupancy map encoding module 107 for encoding. On the other hand, the occupancy map of the point cloud can be used to guide the depth map generation module 103 to generate the depth map of the point cloud and guide the texture map generation module 104 to generate the texture map of the point cloud.

[0111] When applied to the technical solution provided in the embodiment of the present application, as an example, the packing module 102 can also send the auxiliary information of the union occupancy map generated during the packing process to the auxiliary information encoding module 108 to encode the auxiliary information of the union occupancy map into the bitstream. In this way, there is no need to transmit the relevant information of the global matching patch corresponding to the union occupancy map through the bitstream, so that the bitstream overhead can be saved. For the terms and specific implementation methods in the technical solution, please refer to the following. In addition, the module for encoding the auxiliary information of the union occupancy map into the bitstream can also be a module independent of the auxiliary information encoding module 108, and the embodiment of the present application does not limit this.

[0112] The depth map generation module 103 is used to generate a depth map of the point cloud according to the occupancy map of the point cloud, the occupancy map of each patch of the point cloud and the depth information, and send the generated depth map to the first filling module 105 to fill the blank pixels in the depth map to obtain a filled depth map.

[0113] The texture map generation module 104 is used to generate a texture map of the point cloud according to the occupancy map of the point cloud, the occupancy maps of each patch of the point cloud and the texture information, and send the generated texture map to the first filling module 105 to fill the blank pixels in the texture map to obtain a filled texture map.

[0114] The filled depth map and the filled texture map are sent by the first filling module 105 to the image or video based encoding module 106 for image or video based encoding.

[0115] On the one hand, the image or video based encoding module 106, the occupancy map encoding module 107, and the auxiliary information encoding module 108 send the obtained encoding result (ie, code stream) to the multiplexing module 109 to be merged into a code stream, which can be sent to the output interface 140.

[0116] On the other hand, the encoding result (i.e., code stream) obtained by the encoding module 106 based on the image or video is sent to the point cloud reconstruction module 112 for point cloud reconstruction to obtain a reconstructed point cloud (i.e., obtain the reconstructed point cloud geometric information). Specifically, the encoded depth map obtained by the encoding module 106 based on the image or video is subjected to video decoding to obtain a decoded depth map of the point cloud, and the reconstructed point cloud geometric information is obtained by using the decoded depth map, the occupancy map of the point cloud, and the auxiliary information of each patch. Among them, the geometric information of the point cloud refers to the coordinate values ​​of the points in the point cloud (e.g., each point in the point cloud) in the three-dimensional space.

[0117] Optionally, the point cloud reconstruction module 112 may also send the texture information of the point cloud and the reconstructed point cloud geometry information to the coloring module, and the coloring module is used to color the reconstructed point cloud to obtain the texture information of the reconstructed point cloud.

[0118] Optionally, the texture map generation module 104 may also generate a texture map of the point cloud based on information obtained by filtering the reconstructed point cloud geometric information by the point cloud filtering module 110 .

[0119] like Figure 6 , which is a schematic diagram of a point cloud, a patch, and an occupancy map of a point cloud that can be applied to an example of an embodiment of the present application. Figure 6 (a) is a schematic diagram of a frame of point cloud. Figure 6 (b) is based on Figure 6 (a) is a schematic diagram of the patch of the point cloud obtained. Figure 6 (c) in Figure 6 Schematic diagram of the occupancy map of the point cloud obtained by mapping each patch shown in (b) to a two-dimensional plane and then packing the occupancy map of each patch. Figure 6The occupancy map of the point cloud shown in (c) can be based on the above Figure 1 obtained by the method shown.

[0120] like Figure 7 FIG. 2 is a schematic block diagram of a decoder 200 that can be used in an embodiment of the present application. Figure 7 The MPEG PCC decoding framework is used as an example. Figure 7 In the example of , the decoder 200 may include a demultiplexing module 201, an image or video based decoding module 202, an occupancy map decoding module 203, an auxiliary information decoding module 204, a point cloud reconstruction module 205, a point cloud filtering module 206 and a point cloud texture information reconstruction module 207. Among them:

[0121] The demultiplexing module 201 is used to send the input code stream (i.e., the merged code stream) to the corresponding decoding module. Specifically, the code stream containing the encoded texture map and the code stream containing the encoded depth map are sent to the image or video based decoding module 202; the code stream containing the encoded occupancy map is sent to the occupancy map decoding module 203, and the code stream containing the encoded auxiliary information is sent to the auxiliary information decoding module 204.

[0122] The image or video based decoding module 202 is used to decode the received encoded texture map and the encoded depth map; then, the texture map information obtained by decoding is sent to the texture information reconstruction module 207 of the point cloud, and the depth map information obtained by decoding is sent to the point cloud reconstruction module 205. The occupancy map decoding module 203 is used to decode the received bitstream containing the encoded occupancy map, and send the occupancy map information obtained by decoding to the point cloud reconstruction module 205. The auxiliary information decoding module 204 is used to decode the received encoded auxiliary information, and send the information indicating the auxiliary information obtained by decoding to the point cloud reconstruction module 205.

[0123] The point cloud reconstruction module 205 is used to reconstruct the geometric information of the point cloud according to the received occupancy map information and auxiliary information. The reconstructed geometric information of the point cloud is filtered by the point cloud filtering module 206 and then sent to the point cloud texture information reconstruction module 207. As an example, the reconstruction process performed by the point cloud reconstruction module 205 can refer to the reconstruction process performed by the point cloud reconstruction module 112 above.

[0124] The point cloud texture information reconstruction module 207 is used to reconstruct the texture information of the point cloud to obtain a reconstructed point cloud.

[0125] Understandably, Figure 7 The decoder 200 shown is only an example. In a specific implementation, the decoder 200 may include Figure 7More or fewer modules are shown in the embodiment of the present application.

[0126] The point cloud encoding and decoding method and codec provided in the embodiments of the present application are described below in conjunction with the accompanying drawings.

[0127] like Figure 8 As shown, it is a schematic diagram of a flow chart of a point cloud encoding method provided in an embodiment of the present application. As an example, in combination with Figure 4 , Figure 8 The execution subject of the method shown may be the encoder 100 . Figure 8 The method shown includes the following steps:

[0128] S101: Obtain auxiliary information of a union occupancy map of a point cloud group; wherein the point cloud group includes N frames of point clouds, N≥2, N is an integer, and the union occupancy map is the union of occupancy maps of N patches having a matching relationship in the N frames of point clouds; the auxiliary information of the union occupancy map is used to determine the auxiliary information of the N patches. For example, based on the auxiliary information of the N patches, the auxiliary information of the union occupancy map is obtained.

[0129] The point cloud group includes at least two frames of point clouds, for example, at least two frames of point clouds that are temporally continuous.

[0130] Optionally, the point cloud group may be one or more GOFs, usually one GOF. The number of frames of the point cloud in a GOF is configurable. The embodiment of the present application does not limit the number of frames of the point cloud contained in a GOF. For example, a GOF may contain 32 frames of point cloud. The method for determining a GOF may refer to the prior art.

[0131] Optionally, the point cloud group may be a sub-GOF, which may be composed of multiple frames of point clouds that are temporally continuous in a GOF. A GOF may include at least two sub-GOFs, and the number of frames of point clouds contained in any two of the at least two sub-GOFs may be equal or unequal.

[0132] In the specific implementation process, the encoder and the decoder may divide a GOF into at least two sub-GOFs based on the same predefined method (such as predefined by the protocol). Alternatively, the encoder may divide a GOF into at least two sub-GOFs (such as dividing a GOF into at least two sub-GOFs based on the continuity between the contents of the point clouds contained in the GOF), and then encode the information indicating the positions of the at least two sub-GOFs in the GOF (i.e., the information about which point clouds each sub-GOF includes) into the bitstream, so that the decoder obtains the positions of the at least two sub-GOFs in the GOF by parsing the information obtained from the bitstream.

[0133] Optionally, the information indicating the positions of the at least two sub-GOFs in the GOF may include: the number of the starting frame (i.e., the first frame point cloud) of each sub-GOF and / or the number of frames included in each sub-GOF, etc. It can be understood that the number of the starting frame of the first sub-GOF may not be encoded into the bitstream and / or the number of frames included in one of the GOFs may not be encoded into the bitstream. This optional embodiment can be applied to a scenario in which each of the at least two sub-GOFs includes the same number of frames, and can also be applied to a scenario in which some or all of the at least two sub-GOFs include different numbers of frames.

[0134] That is, in an optional implementation, when the point cloud group is a sub-GOF, the above-mentioned syntax elements may also include information indicating the position of each frame of the point cloud in the sub-GOF in the GOF to which the sub-GOF belongs, for example, the number of the starting frame contained in the sub-GOF and / or the number of frames of the point cloud contained in the sub-GOF.

[0135] Optionally, if each of the at least two sub-GOFs includes the same number of frames, the information indicating the positions of the at least two sub-GOFs in the GOF may include: the number of frames of the point cloud included in a sub-GOF, or the number of sub-GOFs contained in a GOF.

[0136] The number of occupancy maps of the union of point cloud groups can be one or more, for example, combined Figure 1 In the packing method shown, the number of union occupancy maps of the point cloud group is M. The union occupancy map in S101 can be any union occupancy map in the point cloud group. The description of the union occupancy map, the patch with matching relationship and other related terms can be referred to above, and will not be repeated here.

[0137] Each of the N patches in S101 can be considered as the global matching patch described above. Each of the N patches comes from a frame of point cloud, and different patches come from different point clouds. For the relevant explanation of the global matching patch, please refer to the above, which will not be repeated here.

[0138] The auxiliary information of the union occupancy map refers to information used to describe the union occupancy map, such as but not limited to at least one of the following information: the coordinate value of the union occupancy map in the global occupancy map, the size information of the union occupancy map in the global occupancy map, and the coordinate value of the union occupancy map in the three-dimensional space. The global occupancy map is an occupancy map obtained by packaging multiple union occupancy maps in the point cloud group. For the relevant description of the global occupancy map, please refer to the above, which will not be repeated here.

[0139] The coordinate values ​​of the union occupancy map in the global occupancy map include: the coordinate values ​​of the union occupancy map on the U0 axis in the global occupancy map (such as u0 of the union occupancy map), and / or the coordinate values ​​on the V0 axis (such as v0 of the union occupancy map). In specific implementation, the encoder may use the minimum or maximum value of u0 of the occupancy maps of the N patches corresponding to the union occupancy map as u0 of the union occupancy map; and / or use the minimum or maximum value of v0 of the occupancy maps of the N patches corresponding to the union occupancy map as v0 of the union occupancy map.

[0140] The size information of the union occupancy map in the global occupancy map includes: the size information of the union occupancy map on the U0 axis in the global occupancy map (such as the sizeU0 of the union occupancy map), and / or the size information on the V0 axis (such as the sizeV0 of the union occupancy map). In specific implementation, the encoder may use the maximum value of the sizeU0 of the occupancy maps of the N patches corresponding to the union occupancy map as the sizeU0 of the union occupancy map, and / or use the maximum value of the sizeU0 of the occupancy maps of the N patches corresponding to the union occupancy map as the sizeV0 of the union occupancy map.

[0141] The coordinate value corresponding to the union occupancy map in the three-dimensional space refers to the minimum (or maximum, etc.) coordinate value of the N patches corresponding to the union occupancy map in the three-dimensional space. Taking the minimum coordinate value as an example, the coordinate value corresponding to the union occupancy map in the three-dimensional space may include: the minimum coordinate value of the N patches on the U1 axis in the three-dimensional space (such as the minimum u1 value), the minimum coordinate value on the V1 axis (such as the minimum v1 value), and / or the minimum coordinate value on the D1 axis (such as the minimum d1 value).

[0142] In one implementation scenario, the step of obtaining auxiliary information of the union occupancy map in S101 can be obtained after the encoder obtains the union occupancy map, for example, in combination with Figure 1 The packaging method shown can be obtained during the execution of the packaging method, and of course can also be obtained after the execution of the packaging method. The embodiment of the present application does not limit this.

[0143] The auxiliary information of the union occupancy map is used to determine the auxiliary information of the N patches, which can be understood as: the auxiliary information of the union occupancy map is used by the decoder to determine the auxiliary information of the N patches.

[0144] For the convenience of description, in the embodiment of the present application, the N patches in S101 are referred to as the N patches corresponding to the union occupancy map in S101. They are uniformly described here and will not be repeated in the following.

[0145] S102: Encode a syntax element into a bitstream, where the syntax element includes auxiliary information of the union occupancy map.

[0146] The above S101~S102 illustrate the point cloud encoding method from the perspective of N patches corresponding to a union occupancy map. In specific implementation, the point cloud encoding method can be executed based on S101~S102 for any one or more patches corresponding to the union occupancy map in the point cloud group (i.e., global matching patches). For other patches in the point cloud group, the embodiment of the present application does not limit the point cloud encoding method applicable to them. For example, they can be encoded according to any point cloud encoding method provided by the prior art, such as the MPEG point cloud encoding method. The other patches may include all non-global matching patches, and may further include other global matching patches except for the global matching patches for executing the point cloud encoding method according to the above S101~S102.

[0147] In the specific implementation process, as an example, the encoder may execute S101-S102 for N patches corresponding to one union occupancy map, and then execute S101-102 for N patches corresponding to another union occupancy map, until all patches corresponding to the union occupancy maps that need to be encoded using the technical solution provided in the embodiment of the present application are executed. As another example, the encoder may execute the above S101 for the patches corresponding to the union occupancy maps that need to be encoded using the technical solution provided in the embodiment of the present application in sequence or simultaneously, and then execute the above S102 for the patches corresponding to the union occupancy maps in sequence or simultaneously, and the embodiment of the present application is not limited to this.

[0148] It can be understood that due to the correlation between the union occupancy map and the occupancy maps of the N patches corresponding to the union occupancy map, such as the position information of the union occupancy map in the global occupancy map, the position information of the occupancy maps corresponding to the above-mentioned N patches in the occupancy map of the point cloud to which they belong, and / or the union occupancy map is the union of the occupancy maps of the N patches, etc., therefore, the auxiliary information of the N patches can be represented or approximately represented by the auxiliary information of the union occupancy map.

[0149] In the point cloud encoding method provided in the embodiment of the present application, the encoder encodes the auxiliary information of the union occupancy map into the bitstream, so that the decoder determines the auxiliary information of the N patches corresponding to the union occupancy map based on the auxiliary information of the union occupancy map. In this way, compared with the technical solution of directly transmitting the auxiliary information of the N patches in the bitstream in the prior art, the technical solution provided in the embodiment of the present application helps to save the bitstream transmission overhead without affecting the encoding and decoding performance.

[0150] Optional, Figure 8 The method shown also includes the following steps S103:

[0151] S103: Reconstruct the N frames of point cloud according to the auxiliary information of multiple patches (such as all patches) in the N frames of point cloud; wherein the multiple patches include the N patches. The specific implementation of S103 can be referred to above and will not be repeated here.

[0152] The embodiment of the present application does not limit the execution order of S101-S102 and S103. For example, S101-S102 are executed first and then S103, or S103 is executed first and then S101-S102, or S103 is executed during the execution of S101-S102, etc.

[0153] Optional, Figure 8 The syntax element in the illustrated embodiment further includes a difference between the corresponding auxiliary information of one or more patches (eg, all patches) among the N patches and the auxiliary information of the union occupancy map.

[0154] The corresponding auxiliary information of the one or more patches refers to the auxiliary information of the same type as the auxiliary information of the union occupancy map. For example, the auxiliary information of the union occupancy map and the corresponding auxiliary information of the one or more patches are both u0 or both v0; or both are sizeU0 or both are sizeV0; or both are u1 or both are d1 or both are v1.

[0155] This optional implementation can be understood as using a differential method to transmit the auxiliary information of one or more patches corresponding to the union occupancy map. Although there is a correlation between the union occupancy map and the occupancy maps of the N patches corresponding to the union occupancy map, the auxiliary information of the union occupancy map and the auxiliary information of the N patches may not be exactly the same. Therefore, using a differential method to transmit the auxiliary information of one or more patches corresponding to the union occupancy map helps to improve the accuracy and / or setting flexibility of the auxiliary information of the one or more patches in the N patches. For example, the accuracy of u0, v0, u1, v1 and / or d1 of the one or more patches can be improved. For example, the setting flexibility of sizeU0 and / or sizeV0 of the one or more patches can be improved, that is, it is not necessary to limit the sizeU0 of the one or more patches to be equal to the sizeU0 of the union occupancy map, and / or it is not necessary to limit the sizeV0 of the one or more patches to be equal to the sizeV0 of the union occupancy map.

[0156] During the specific implementation process, whether the syntax element includes the difference between the corresponding auxiliary information of one or more patches (for example, all patches) in the above-mentioned N patches and the auxiliary information of the union occupancy map can be predefined, such as predefined by a protocol, or the encoder can encode the indication information indicating whether the difference is included into the bitstream and send it to the decoder.

[0157] like Fig. 9 As shown, it is a schematic diagram of a flow chart of a point cloud encoding method provided in an embodiment of the present application. As an example, in combination with Figure 4 , Fig. 9 The execution subject of the method shown may be the decoder 200 . Fig. 9 The method shown includes the following steps:

[0158] S201: Parse the bitstream to obtain a syntax element, where the syntax element includes auxiliary information of a union occupancy map of a point cloud group, where the point cloud group includes N frames of point clouds, where N≥2, and N is an integer, and the union occupancy map is a union of occupancy maps of N patches having a matching relationship in the N frames of point clouds.

[0159] Optionally, the syntax element further includes the difference between the corresponding auxiliary information of one or more patches (eg, all patches) in the above N patches and the auxiliary information of the union occupancy map. Figure 8 The illustrated embodiment corresponds to an embodiment in which the syntax element includes the difference value.

[0160] S202: Determine the auxiliary information of the N patches according to the auxiliary information of the union occupancy map obtained by parsing.

[0161] For example, for any union occupancy map, if the syntax element includes auxiliary information of the union occupancy map, but does not include the difference between the corresponding auxiliary information of any one of the N patches corresponding to the union occupancy map and the auxiliary information of the union occupancy map, the decoder may use the auxiliary information of the union occupancy map as the corresponding auxiliary information of each patch in the N patches.

[0162] For another example, for any union occupancy map, if the syntax element includes auxiliary information of the union occupancy map, and includes the difference between the corresponding auxiliary information of one or more patches in the N patches corresponding to the union occupancy map and the auxiliary information of the union occupancy map, then for each of the one or more patches, the decoder may determine the corresponding auxiliary information of the patch based on the difference between the auxiliary information of the union occupancy map and the corresponding auxiliary information of the patch, specifically, summing the auxiliary information of the union occupancy map and the difference to obtain the corresponding auxiliary information of the patch. Optionally, if the N patches also include other patches in addition to the one or more patches, then for each of the other patches, the decoder may use the auxiliary information of the union occupancy map as the corresponding auxiliary information of the N patches.

[0163] It can be understood that for each patch corresponding to the non-union occupancy map in the point cloud group (ie, the non-global matching patch in the point cloud group), the decoder can obtain the auxiliary information of the patch by parsing the bitstream, and the method can refer to the prior art.

[0164] S203: Reconstruct the N frames of point cloud based on the auxiliary information of multiple patches (eg, all patches) in the N frames of point cloud; wherein the multiple patches include the N patches. The specific implementation of S203 can be referred to above and will not be repeated here.

[0165] Any point cloud decoding method provided in this embodiment is Figure 8 The corresponding point cloud encoding method in the shown embodiment corresponds, therefore, the description of the relevant terms, specific implementation methods and beneficial effects in this embodiment can refer to the above-mentioned embodiment of the point cloud encoding method, which will not be repeated here. For example, the point cloud group includes GOF or sub-GOF, etc. Among them, how the decoder obtains the position of the sub-GOF in the GOF to which the sub-GOF belongs can be referred to above, which will not be repeated here. For another example, the relevant description of the N patches in this embodiment, etc. can all be referred to above.

[0166] like Fig.10 As shown, it is a schematic diagram of a flow chart of a point cloud encoding method provided in an embodiment of the present application. As an example, in combination with Figure 4 , Fig.10 The execution subject of the method shown may be the encoder 100 . Fig.10 The method shown includes the following steps:

[0167] S301: Determine indication information, the indication information indicating whether to use a target point cloud encoding method to encode the auxiliary information of N patches in the point cloud group. The target point cloud encoding method can be any point cloud encoding method provided in the embodiment of the present application, such as Figure 8 The point cloud encoding method shown in the figure, wherein the point cloud group includes N frames of point clouds, N≥2, and N is an integer; and the N patches are N patches having a matching relationship in the N frames of point clouds.

[0168] In the specific implementation process, the encoder can support at least two encoding methods, one of which can be any point cloud encoding method provided in the embodiment of the present application. Figure 8 The point cloud encoding method shown, other point cloud encoding methods may be any point cloud encoding method provided in the prior art or in the future.

[0169] Optionally, the indication information may be an index of a target point cloud encoding / decoding method. During the specific implementation process, the encoder and the decoder may pre-agree on the indexes of at least two point cloud encoding / decoding methods supported by the encoder / decoder, and then, after the encoder determines the target encoding method, the index of the target encoding method or the index of the decoding method corresponding to the target encoding method is encoded into the bitstream as indication information. The embodiment of the present application does not limit how the encoder determines which of the at least two encoding methods supported by the encoder the target encoding method is, for example, the target encoding method is determined based on the bitstream transmission overhead.

[0170] S302: Encode the indication information into a bitstream.

[0171] In the specific implementation process, the method further includes the following step S303:

[0172] S303: Encode the auxiliary information of the N patches according to the target point cloud encoding method. For example, according to Figure 8 The point cloud encoding method shown encodes the auxiliary information of the N patches.

[0173] This embodiment provides a technical solution for selecting a target encoding method, which can be applied to a scenario where the encoder supports at least two point cloud encoding methods. In other words, the embodiment of the present application supports the technical solution for the encoder to select a target point cloud encoding method from at least two candidate point cloud encoding methods.

[0174] like Fig.11 As shown, it is a flowchart of a point cloud decoding method provided by an embodiment of the present application. As an example, in combination with Figure 4 , Fig.11 The execution subject of the method shown may be the decoder 200 . Fig.11The method shown includes the following steps:

[0175] S401: Parse the bitstream to obtain indication information, which indicates whether to use the target point cloud decoding method to decode the auxiliary information of the N patches in the point cloud group. The target point cloud decoding method can be any point cloud decoding method provided in the embodiment of the present application, such as Fig. 9 The point cloud decoding method shown in the figure, wherein the point cloud group includes N frames of point clouds, N≥2, and N is an integer; and the N patches are N patches having a matching relationship in the N frames of point clouds.

[0176] S402: When the indication information indicates to use the target point cloud decoding method for decoding, the target point cloud decoding method is used to process the auxiliary information of the N patches. For details, please refer to the above S201 to S203.

[0177] The point cloud decoding method provided in this embodiment is Fig.10 Therefore, the explanation of the relevant contents, specific implementation methods and description of the beneficial effects in this embodiment can all refer to the above Fig.10 The embodiment shown. The embodiment of the present application supports the technical solution that an encoder selects a target point cloud decoding method from at least two candidate point cloud decoding methods.

[0178] The following takes the point cloud group as a subGOF as an example to illustrate an example of implementing the above-provided bitstream format. Specifically, the format of the bitstream representing the auxiliary information (group_of_frames_auxiliary_information) of the global matching patch in the GOF can be as shown in Table 2:

[0179] Table 2

[0180]

[0181] Among them, group_of_frames_size in Table 2 represents the number of frames of the point cloud included in the GOF, and subGOFsize represents the number of frames of the point cloud included in the sub-GOF.

[0182] Based on Table 2, the format of the code stream representing the auxiliary information (subGOF_global_matched_patch_auxiliary_information) of the global matching patch in the sub-GOF (ie, the global matching patch with a matching relationship) may be as shown in Table 3:

[0183] Table 3

[0184]

[0185] Among them, global_matched_patch_count represents the number of globally matched patches in a frame. Table 3 is illustrated by taking the case where u0 and d0 of each global patch in the set are transmitted in a differential manner, and u1, v1 and d1 all use the corresponding auxiliary information of the union occupancy map corresponding to the set as an example. In addition, unionpatch[i].normal_axis represents the normal coordinate axis index of the union occupancy map. It can be understood that the normal coordinate axes of all patches corresponding to a union occupancy map are the same. Therefore, the normal coordinate axis index of the union occupancy map can be used to represent the normal coordinate axes of all patches.

[0186] Based on Table 2, the format of the code stream representing the auxiliary information of all patches in a frame (specifically u1, v1, and d1) can be shown in Table 4:

[0187] Table 4

[0188]

[0189] Table 4 takes the differential transmission of u1, v1 and d1 of the global matching patch as an example for explanation.

[0190] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0191] The embodiment of the present application can divide the encoder and decoder into functional modules according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0192] Fig.12 A schematic block diagram of an encoder 120 provided in an embodiment of the present application. The encoder 120 may be used to perform Figure 8 or Fig.10 The point cloud encoding method shown in FIG. 1 is a diagram showing a point cloud encoding method. The encoder 120 includes a packing module 1201 and an auxiliary information encoding module 1202 .

[0193] In some embodiments: the packing module 1201 is used to obtain auxiliary information of the union occupancy map of the point cloud group; the point cloud group includes N frames of point cloud, N ≥ 2, N is an integer, and the union occupancy map is the union of occupancy maps of N patches with matching relationships in the N frames of point cloud; the auxiliary information of the union occupancy map is used to determine the auxiliary information of the N patches. The auxiliary information encoding module 1202 is used to encode a syntax element into a bitstream, and the syntax element includes the auxiliary information of the union occupancy map. For example, in combination Figure 8 The packing module 1201 can be used to execute S101, and the auxiliary information encoding module 1202 can be used to execute S102.

[0194] Optionally, the auxiliary information of the union occupancy map includes at least one of the following information: coordinate values ​​of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and coordinate values ​​of the union occupancy map corresponding to the three-dimensional space. The global occupancy map is an occupancy map obtained by packaging multiple union occupancy maps in the point cloud group.

[0195] Optionally, the above-mentioned syntax element also includes: the difference between the corresponding auxiliary information of one or more patches in the N patches and the auxiliary information of the union occupancy map.

[0196] Optionally, the packing module 1201 is specifically used to obtain auxiliary information of the union occupancy map based on the auxiliary information of the N patches.

[0197] Optionally, when the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map in the global occupancy map, the packing module 1201 is specifically used to: use the maximum coordinate value or the minimum coordinate value of the occupancy maps of the N patches in the occupancy map of the corresponding point cloud as the coordinate value of the union occupancy map in the global occupancy map;

[0198] Optionally, when the auxiliary information of the union occupancy map includes size information of the union occupancy map in the global occupancy map, the packing module 1201 is specifically used to: use information of the maximum size of the occupancy maps of the N patches in the occupancy map of the corresponding point cloud as size information of the union occupancy map in the global occupancy map;

[0199] Optionally, when the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map corresponding to the three-dimensional space, the packing module 1201 is specifically used to: use the maximum coordinate value or the minimum coordinate value of the N patches in the corresponding point cloud as the coordinate value of the union occupancy map corresponding to the three-dimensional space.

[0200] Optionally, the encoder 120 further includes a point cloud reconstruction module 1203, which is used to reconstruct the N frames of point cloud according to the auxiliary information of multiple patches in the N frames of point cloud; wherein the multiple patches include the N patches. Figure 8 , the point cloud reconstruction module 1203 can be used to execute S103.

[0201] Optionally, the point cloud group is a GOF or a sub-GOF, and the sub-GOF includes continuous multi-frame point clouds in a GOF.

[0202] Optionally, if the point cloud group is the sub-GOF, the above syntax elements also include: information indicating the position of the point cloud (such as each frame point cloud) contained in the sub-GOF in the GOF to which the sub-GOF belongs. For example, the number of the starting frame contained in the sub-GOF, and / or the number of frames of the point cloud contained in the sub-GOF. In other words, the position of each frame of the point cloud contained in the sub-GOF in the GOF to which the sub-GOF belongs can be directly or indirectly derived based on the number of the starting frame contained in the sub-GOF, and / or the number of frames of the point cloud contained in the sub-GOF.

[0203] In some other embodiments: the packing module 1201 is used to determine the indication information, which indicates whether to use the target point cloud encoding method to encode the auxiliary information of the N patches in the point cloud group, and the target point cloud encoding method includes the above Fig.10 The point cloud encoding method shown in the figure; the point cloud group includes N frames of point cloud, N ≥ 2, N is an integer; the N patches are N patches with matching relationship in the N frames of point cloud. The auxiliary information encoding module 1202 is used to encode the indication information into the bit stream. In the specific implementation process, the encoder 120 also includes a point cloud reconstruction module 1203, which is used to reconstruct the N frames of point cloud according to the auxiliary information of multiple patches in the N frames of point cloud; the multiple patches include the N patches. For example, combined with Fig.10 The packing module 1201 can be used to execute S301, the auxiliary information encoding module 1202 can be used to execute S302, and the point cloud reconstruction module 1203 can be used to execute S303.

[0204] As an example, the packaging module 1201 may correspond to Figure 5 The packing module 102 in the auxiliary information encoding module 1202 may correspond to Figure 5 The auxiliary information encoding module 108 in the point cloud reconstruction module 1203 may specifically correspond to Figure 5 It should be noted that the packing module 1201 and the reconstruction module 1203 can be directly connected or indirectly connected through other functional modules. For example, an example of indirect connection can be referred to. Figure 5The indirect connection relationship between the packing module 102 and the point cloud reconstruction module 112.

[0205] Fig.13 A schematic block diagram of a decoder 130 provided in an embodiment of the present application. The decoder 130 can be used to perform Fig. 9 or Fig.11 The point cloud decoding method shown in FIG. 1 includes an auxiliary information decoding module 1301 and a point cloud reconstruction module 1302 .

[0206] In some embodiments, the auxiliary information decoding module 1301 is used to parse the bitstream to obtain a syntax element, which includes auxiliary information of a union occupancy map of a point cloud group, the point cloud group includes N frames of point clouds, N ≥ 2, N is an integer, and the union occupancy map is the union of occupancy maps of N patches with matching relationships in the N frames of point clouds; according to the auxiliary information of the union occupancy map obtained by parsing, the auxiliary information of the N patches is determined. The point cloud reconstruction module 1302 is used to reconstruct the N frames of point clouds based on the auxiliary information of multiple patches in the N frames of point clouds; wherein the multiple patches include the N patches. For example, in combination Fig. 9 The auxiliary information decoding module 1301 can be used to execute S201 and S202, and the point cloud reconstruction module 1302 can be used to execute S203.

[0207] Optionally, the auxiliary information of the union occupancy map includes at least one of the following information: coordinate values ​​of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and coordinate values ​​of the union occupancy map corresponding to the three-dimensional space. The global occupancy map is an occupancy map obtained by packaging multiple union occupancy maps in the point cloud group.

[0208] Optionally, the syntax element further includes: a difference between the corresponding auxiliary information of one or more patches in the N patches and the auxiliary information of the union occupancy map. In this case, in terms of determining the auxiliary information of the N patches according to the auxiliary information of the union occupancy map obtained by parsing, the auxiliary information decoding module 1301 is specifically used to: determine the auxiliary information of the N patches according to the auxiliary information of the union occupancy map obtained by parsing, and a difference between the corresponding auxiliary information of one or more patches in the N patches and the auxiliary information of the union occupancy map.

[0209] Optionally, the point cloud group is a GOF or a sub-GOF, and the sub-GOF includes continuous multi-frame point clouds in a GOF.

[0210] Optionally, if the point cloud group is a sub-GOF, the above syntax elements also include: information indicating the position of the point cloud (such as the point cloud of each frame) contained in the sub-GOF in the GOF to which the sub-GOF belongs, for example, the number of the starting frame contained in the sub-GOF, and / or the number of frames of the point cloud contained in the sub-GOF. In this case, the auxiliary information decoding module 1301 is also used to determine the position of the point cloud (such as the point cloud of each frame) contained in the sub-GOF in the GOF to which the sub-GOF belongs according to the information obtained by parsing that indicates the position of the point cloud contained in the sub-GOF in the GOF to which the sub-GOF belongs.

[0211] In some other embodiments: the auxiliary information decoding module 1301 is used to parse the code stream to obtain indication information, which indicates whether to use the target point cloud decoding method to decode the auxiliary information of N patches in the point cloud group. The target point cloud decoding method includes the above Fig.11 The point cloud decoding method shown in FIG. 1 is a point cloud decoding method; the point cloud group includes N frames of point cloud, N ≥ 2, and N is an integer; the N patches are N patches with matching relationships in the N frames of point cloud. When the indication information indicates that the target point cloud decoding method is used for decoding, the auxiliary information decoding module 1301 and the point cloud reconstruction module 1302 use the target point cloud decoding method to process the auxiliary information of the N patches. For example, in combination with Fig.11 , the auxiliary information decoding module 1301 can be used to execute S401, and the auxiliary information decoding module 1301 and the point cloud reconstruction module 1302 can be used to execute S402. In this process, the steps executed by the auxiliary information decoding module 1301 and the point cloud reconstruction module 1302 can refer to the above embodiment. For example, in this process, combined with Fig. 9 The auxiliary information decoding module 1301 can be used to execute S201 and S202, and the point cloud reconstruction module 1302 can be used to execute S203.

[0212] As an example, the auxiliary information decoding module 1301 may correspond to Figure 7 The auxiliary information decoding module 204 in the point cloud reconstruction module 1302 may correspond to Figure 7 The point cloud reconstruction module 205 in.

[0213] It can be understood that the various modules in the codec of the embodiment of the present application are functional entities that implement the various execution steps included in the point cloud encoding and decoding method corresponding to the present application, that is, they have the functional entities that can fully implement the various steps in the point cloud encoding and decoding method corresponding to the present application and the extension and deformation of these steps. For details, please refer to the introduction of the image filtering method in this article. For the sake of brevity, this article will not go into details.

[0214] Fig.14A schematic block diagram of an implementation method of an encoding device or decoding device (referred to as a decoding device 160, or a point cloud decoding device 160) for an embodiment of the present application. The decoding device 160 may include a processor 1610, a memory 1630, and a bus system 1650. The processor 1610 and the memory 1630 are connected via the bus system 1650, the memory 1630 is used to store instructions, and the processor 1610 is used to execute the instructions stored in the memory 1630 to execute the various point cloud encoding or decoding methods described in the present application, especially the method of filtering the current image block based on the block size of the current image block. To avoid repetition, it will not be described in detail here.

[0215] In the embodiment of the present application, the processor 1610 may be a central processing unit (CPU), or may be other general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0216] The memory 1630 may include a ROM device or a RAM device. Any other suitable type of storage device may also be used as the memory 1630. The memory 1630 may include code and data 1631 accessed by the processor 1610 using the bus system 1650. The memory 1630 may further include an operating system 1633 and an application 1635, which includes at least one program that allows the processor 1610 to execute the point cloud encoding or decoding method described in the present application (especially the method of filtering the current image block based on the block size of the current image block described in the present application). For example, the application 1635 may include applications 1 to N, which further include a point cloud encoding or decoding application (referred to as a point cloud decoding application) that executes the point cloud encoding or decoding method described in the present application.

[0217] In addition to the data bus, the bus system 1650 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the bus system 1650 in the figure.

[0218] Optionally, the decoding device 160 may also include one or more output devices, such as a display 1670. In one example, the display 1670 may be a touch-sensitive display that combines a display with a touch-sensitive unit operable to sense touch input. The display 1670 may be connected to the processor 1610 via the bus system 1650.

[0219] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any communication media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include computer-readable media.

[0220] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are actually directed to non-temporary tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, DVDs, and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0221] Instructions may be executed by one or more processors such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. In addition, in some aspects, the functions described by the various illustrative logic blocks, modules, and steps described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Moreover, the technology may be fully implemented in one or more circuits or logic elements. In one example, the various illustrative logic blocks, units, and modules in the encoder 100 and the decoder 200 may be understood as corresponding circuit devices or logic elements.

[0222] The techniques of the present application may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this application to emphasize functional aspects of devices for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).

[0223] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A point cloud encoding method, characterized in that: include: Acquire auxiliary information of a union occupancy map of a point cloud group; wherein the point cloud group includes N frames of point clouds, N≥2, N is an integer, and the union occupancy map is the union of occupancy maps of N point cloud patches having a matching relationship in the N frames of point clouds; the auxiliary information of the union occupancy map is used to determine auxiliary information of the N patches, each of the N patches comes from a frame of point cloud in the N frames of point clouds, and each of the N patches comes from point clouds of different frames; Encoding a syntax element into a bitstream, the syntax element comprising auxiliary information of the union occupancy map; The syntax element further includes: a difference between corresponding auxiliary information of one or more patches in the N patches and auxiliary information of the union occupancy map.

2. The point cloud encoding method according to claim 1, characterized in that: The auxiliary information of the union occupancy map includes at least one of the following information: a coordinate value of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and a coordinate value of the union occupancy map corresponding to the three-dimensional space; The global occupancy map is an occupancy map obtained by packaging a plurality of the union occupancy maps in the point cloud group.

3. The point cloud coding method according to claim 1 or 2, characterized in that: The obtaining of auxiliary information of the union occupancy map of the point cloud group includes: Based on the auxiliary information of the N patches, auxiliary information of the union occupancy map is obtained.

4. The point cloud encoding method according to claim 3, characterized in that: The obtaining of auxiliary information of the union occupancy map based on the auxiliary information of the N patches includes: When the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map in the global occupancy map, the maximum coordinate value or the minimum coordinate value of the occupancy maps of the N patches in the occupancy map of the corresponding point cloud is used as the coordinate value of the union occupancy map in the global occupancy map; and / or, when the auxiliary information of the union occupancy map includes size information of the union occupancy map in the global occupancy map, using information of the maximum size of the occupancy maps of the N patches in the occupancy map of the corresponding point cloud as the size information of the union occupancy map in the global occupancy map; And / or, when the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map corresponding to the three-dimensional space, the maximum coordinate value or the minimum coordinate value of the N patches in the corresponding point cloud is used as the coordinate value of the union occupancy map corresponding to the three-dimensional space.

5. The point cloud encoding method according to claim 3, characterized in that: The method further comprises: Reconstruct the N frames of point cloud according to the auxiliary information of multiple patches in the N frames of point cloud; wherein the multiple patches include the N patches.

6. The point cloud coding method according to claim 1 or 2, characterized in that: The point cloud group is a frame group GOF or a sub-GOF, and the sub-GOF includes multiple consecutive frames of point clouds in one GOF.

7. The point cloud encoding method according to claim 6, characterized in that: If the point cloud group is the sub-GOF, the syntax element further includes: information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs.

8. The point cloud encoding method according to claim 7, characterized in that: The information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs includes: the number of the starting frame included in the sub-GOF, and / or the number of frames of the point cloud included in the sub-GOF.

9. A point cloud decoding method, characterized in that: include: Parse the bitstream to obtain syntax elements, wherein the syntax elements include auxiliary information of a union occupancy map of a point cloud group, wherein the point cloud group includes N frames of point clouds, N≥2, and N is an integer, and the union occupancy map is a union of occupancy maps of N point cloud patches having a matching relationship in the N frames of point clouds, wherein each of the N patches comes from one frame of point clouds in the N frames of point clouds, and each of the N patches comes from point clouds of different frames; Determine the auxiliary information of the N patches according to the auxiliary information of the union occupancy map obtained by parsing; Reconstruct the N frames of point cloud based on auxiliary information of multiple patches in the N frames of point cloud; wherein the multiple patches include the N patches.

10. The point cloud decoding method according to claim 9, characterized in that: The auxiliary information of the union occupancy map includes at least one of the following information: a coordinate value of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and a coordinate value of the union occupancy map corresponding to the three-dimensional space; The global occupancy map is an occupancy map obtained by packaging a plurality of the union occupancy maps in the point cloud group.

11. The point cloud decoding method according to claim 9 or 10, characterized in that: The syntax element further includes: a difference between corresponding auxiliary information of one or more patches in the N patches and auxiliary information of the union occupancy map; and determining the auxiliary information of the N patches according to the auxiliary information of the union occupancy map obtained by parsing, including: The auxiliary information of the N patches is determined according to the auxiliary information of the union occupancy map obtained by parsing and the difference between the corresponding auxiliary information of one or more patches in the N patches and the auxiliary information of the union occupancy map.

12. The point cloud decoding method according to claim 9 or 10, characterized in that: The point cloud group is a frame group GOF or a sub-GOF, and the sub-GOF includes multiple consecutive frames of point clouds in one GOF.

13. The point cloud decoding method according to claim 12, characterized in that: If the point cloud group is the sub-GOF, the syntax element further includes: information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs; the method further includes: The position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs is determined according to the information obtained by parsing that represents the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs.

14. The point cloud decoding method according to claim 13, characterized in that: The information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs includes: the number of the starting frame included in the sub-GOF and / or the number of frames of the point cloud included in the sub-GOF.

15. A point cloud encoding method, characterized in that: include: Determine indication information, the indication information indicating whether to use a target point cloud encoding method to encode auxiliary information of N point cloud block patches in a point cloud group, the target point cloud encoding method comprising the point cloud encoding method according to any one of claims 1 to 9; the point cloud group comprises N frames of point cloud, N≥2, N is an integer; the N patches are N patches having a matching relationship in the N frames of point cloud; The indication information is encoded into a bitstream.

16. A point cloud decoding method, characterized in that: include: Parse the bitstream to obtain indication information, wherein the indication information indicates whether to use a target point cloud decoding method to decode the auxiliary information of N point cloud block patches in the point cloud group, wherein the target point cloud decoding method includes the point cloud decoding method according to any one of claims 10 to 15; the point cloud group includes N frames of point cloud, N≥2, and N is an integer; and the N patches are N patches having a matching relationship in the N frames of point cloud; When the indication information indicates to adopt the target point cloud decoding method for decoding, the target point cloud decoding method is adopted to process the auxiliary information of the N patches.

17. An encoder, characterized in that: include: A packing module, used to obtain auxiliary information of a union occupancy map of a point cloud group; wherein the point cloud group includes N frames of point clouds, N≥2, N is an integer, and the union occupancy map is the union of occupancy maps of N point cloud patches having a matching relationship in the N frames of point clouds; the auxiliary information of the union occupancy map is used to determine auxiliary information of the N patches, each of the N patches comes from a frame of point cloud in the N frames of point clouds, and each of the N patches comes from point clouds of different frames; an auxiliary information encoding module, configured to encode a syntax element into a bitstream, wherein the syntax element includes auxiliary information of the union occupancy map; The syntax element further includes: a difference between corresponding auxiliary information of one or more patches in the N patches and auxiliary information of the union occupancy map.

18. The encoder according to claim 17, characterized in that The auxiliary information of the union occupancy map includes at least one of the following information: a coordinate value of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and a coordinate value of the union occupancy map corresponding to the three-dimensional space; The global occupancy map is an occupancy map obtained by packaging a plurality of the union occupancy maps in the point cloud group.

19. The encoder according to claim 17 or 18, characterized in that The packing module is specifically used to obtain auxiliary information of the union occupancy map based on the auxiliary information of the N patches.

20. The encoder according to claim 19, characterized in that The packaging module is specifically used for: When the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map in the global occupancy map, the maximum coordinate value or the minimum coordinate value of the occupancy maps of the N patches in the occupancy map of the corresponding point cloud is used as the coordinate value of the union occupancy map in the global occupancy map; and / or, when the auxiliary information of the union occupancy map includes size information of the union occupancy map in the global occupancy map, using information of the maximum size of the occupancy maps of the N patches in the occupancy map of the corresponding point cloud as the size information of the union occupancy map in the global occupancy map; And / or, when the auxiliary information of the union occupancy map includes the coordinate value of the union occupancy map corresponding to the three-dimensional space, the maximum coordinate value or the minimum coordinate value of the N patches in the corresponding point cloud is used as the coordinate value of the union occupancy map corresponding to the three-dimensional space.

21. The encoder according to claim 19, characterized in that The encoder further comprises: A point cloud reconstruction module is used to reconstruct the N frames of point cloud according to the auxiliary information of multiple patches in the N frames of point cloud; wherein the multiple patches include the N patches.

22. The encoder according to claim 17 or 18, characterized in that The point cloud group is a frame group GOF or a sub-GOF, and the sub-GOF includes multiple consecutive frames of point clouds in one GOF.

23. The encoder according to claim 22, characterized in that If the point cloud group is the sub-GOF, the syntax element further includes: information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs.

24. The encoder according to claim 23, characterized in that The information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs includes: the number of the starting frame included in the sub-GOF and / or the number of frames of the point cloud included in the sub-GOF.

25. A decoder, characterized in that: include: An auxiliary information decoding module is used to parse the bitstream to obtain syntax elements, wherein the syntax elements include auxiliary information of a union occupancy map of a point cloud group, wherein the point cloud group includes N frames of point clouds, N≥2, and N is an integer, and the union occupancy map is a union of occupancy maps of N point cloud block patches having a matching relationship in the N frames of point clouds; according to the auxiliary information of the union occupancy map obtained by parsing, determine the auxiliary information of the N patches, wherein each patch of the N patches comes from a frame of point clouds in the N frames of point clouds, and each patch of the N patches comes from point clouds of different frames; A point cloud reconstruction module is used to reconstruct the N frames of point cloud based on auxiliary information of multiple patches in the N frames of point cloud; wherein the multiple patches include the N patches.

26. The decoder according to claim 25, characterized in that The auxiliary information of the union occupancy map includes at least one of the following information: a coordinate value of the union occupancy map in the global occupancy map, size information of the union occupancy map in the global occupancy map, and a coordinate value of the union occupancy map corresponding to the three-dimensional space; The global occupancy map is an occupancy map obtained by packaging a plurality of the union occupancy maps in the point cloud group.

27. The decoder according to claim 25 or 26, characterized in that The syntax element further includes: a difference between corresponding auxiliary information of one or more patches in the N patches and auxiliary information of the union occupancy map; In the aspect of determining the auxiliary information of the N patches based on the auxiliary information of the union occupancy map obtained by parsing, the auxiliary information decoding module is specifically used to determine the auxiliary information of the N patches based on the auxiliary information of the union occupancy map obtained by parsing, and the difference between the corresponding auxiliary information of one or more patches in the N patches and the auxiliary information of the union occupancy map.

28. The decoder according to claim 25 or 26, characterized in that The point cloud group is a frame group GOF or a sub-GOF, and the sub-GOF includes multiple consecutive frames of point clouds in one GOF.

29. The decoder according to claim 28, characterized in that If the point cloud group is the sub-GOF, the syntax element further includes: information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs; The auxiliary information decoding module is also used to determine the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs based on the information obtained by parsing that represents the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs.

30. The decoder according to claim 29, characterized in that The information indicating the position of the point cloud included in the sub-GOF in the GOF to which the sub-GOF belongs includes: the number of the starting frame included in the sub-GOF and / or the number of frames of the point cloud included in the sub-GOF.

31. An encoder, characterized in that include: A packing module, used to determine indication information, wherein the indication information indicates whether to use a target point cloud encoding method to encode auxiliary information of N point cloud block patches in a point cloud group, wherein the target point cloud encoding method includes the point cloud encoding method according to any one of claims 1 to 8; the point cloud group includes N frames of point cloud, N≥2, and N is an integer; and the N patches are N patches having a matching relationship in the N frames of point cloud; The auxiliary information encoding module is used to encode the indication information into a bit stream.

32. A decoder, characterized in that: include: Auxiliary information decoding module and point cloud reconstruction module; The auxiliary information decoding module is used to parse the code stream to obtain indication information, wherein the indication information indicates whether to use a target point cloud decoding method to decode the auxiliary information of N point cloud block patches in the point cloud group, wherein the target point cloud decoding method includes the point cloud decoding method according to any one of claims 9 to 14; the point cloud group includes N frames of point cloud, N≥2, and N is an integer; the N patches are N patches having a matching relationship in the N frames of point cloud; When the indication information indicates to use the target point cloud decoding method for decoding, the auxiliary information decoding module and the point cloud reconstruction module use the target point cloud decoding method to process the auxiliary information of the N patches.

33. A coding device, characterized in that: It comprises a memory and a processor; wherein the memory is used to store program code; and the processor is used to call the program code to execute the point cloud encoding method as described in any one of claims 1 to 8 or claim 15.

34. A decoding device, characterized in that: It comprises a memory and a processor; wherein the memory is used to store program code; and the processor is used to call the program code to execute the point cloud decoding method as described in any one of claims 9 to 14 or claim 16.

35. A computer-readable storage medium, characterized in that: The method comprises a program code, and when the program code is run on a computer, the computer executes the point cloud encoding method according to any one of claims 1 to 8 or claim 15.

36. A computer-readable storage medium, characterized in that: The method comprises a program code, and when the program code is run on a computer, the computer is caused to execute the point cloud decoding method according to any one of claims 9 to 14 or claim 16.

Images