Resolution-based decoding method, encoding method, decoder, and encoder

Abstract

The embodiment of the present application provides a resolution-based decoding method, an encoding method, a decoder and an encoder, which comprises the following steps: obtaining a current subgraph block by parsing a code stream; determining whether the current subgraph block is a subgraph block which needs to change resolution; when the current subgraph block is a subgraph block which needs to change resolution, acquiring a scaling rate parameter of the current subgraph block by parsing the code stream; determining a scaling factor of the current subgraph block based on the scaling rate parameter of the current subgraph block; and obtaining coordinates of the current subgraph block in a 3D space coordinate system based on the scaling factor of the current subgraph block. The decoding method provided by the present application can improve the quality of a decoded image.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image coding, and more particularly, to a resolution-based decoding method, a resolution-based encoding method, a decoder and an encoder. BACKGROUND

[0002] In a Moving Picture Experts Group Immersive Video (MIV) decoding process, the same scaling factor can be used to convert a stitch from a stitch coordinate system to a 3D space coordinate system to obtain a 3D view at a user viewpoint position orientation. Each stitch can include one or more sub-tiles, and each sub-tile can be a sub-tile at a certain viewpoint. For all sub-tiles in the stitch whose viewpoint is a special viewpoint, a stitch-level scaling parameter read from a bitstream can be calculated to obtain a scaling factor for all sub-tiles in the stitch whose viewpoint is a special viewpoint. The scaling factor for each sub-tile can include a width scaling factor and a height scaling factor. In addition, for sub-tiles in the stitch whose viewpoint is a normal viewpoint, a scaling factor can not be read from the bitstream, and a default scaling factor, such as a default width scaling factor and a default height scaling factor, whose value is 1, can be used directly. That is, all sub-tiles in the special viewpoint use the same scaling factor, and all sub-tiles in the normal viewpoint use the default unchanged scaling factor.

[0003] However, if all sub-tiles in the special viewpoint use the same scaling factor, when the importance of different sub-tiles at the special viewpoint is different, the sub-tiles parsed at the decoding end can not be able to express the information completely, reducing the quality of the decoded image. Similarly, if the scaling factor of all sub-tiles in the normal viewpoint is defaulted to 1, when there is a relatively important sub-tile at the normal viewpoint, the sub-tile parsed at the decoding end can not be able to express the information completely, reducing the quality of the decoded image. SUMMARY

[0004] Embodiments of the present application provide a resolution-based decoding method, a resolution-based encoding method, a decoder and an encoder, which can improve the quality of the decoded image.

[0005] In a first aspect, the present application provides a resolution-based decoding method, comprising:

[0006] parsing a bitstream to obtain a current sub-tile;

[0007] determining whether the current sub-tile is a sub-tile whose resolution needs to be changed;

[0008] when the current sub-tile is a sub-tile whose resolution needs to be changed, parsing the bitstream to obtain a scaling rate parameter of the current sub-tile;

[0009] determining a scaling factor of the current sub-tile based on the scaling ratio parameter of the current sub-tile;

[0010] obtaining coordinates of the current sub-tile in a 3D space coordinate system based on the scaling factor of the current sub-tile.

[0011] In a second aspect, the present application provides a resolution-based encoding method, comprising:

[0012] obtaining a current sub-tile;

[0013] determining whether the current sub-tile is a sub-tile that needs to change resolution;

[0014] when the current sub-tile is the sub-tile that needs to change resolution, writing a scaling ratio parameter of the current sub-tile into a bitstream.

[0015] In a third aspect, the present application provides a decoder, comprising:

[0016] a first parsing unit configured to parse a bitstream to obtain a current sub-tile;

[0017] a first determining unit configured to determine whether the current sub-tile is a sub-tile that needs to change resolution;

[0018] a second parsing unit configured to, when the current sub-tile is the sub-tile that needs to change resolution, parse the bitstream to obtain a scaling ratio parameter of the current sub-tile;

[0019] a second determining unit configured to determine a scaling factor of the current sub-tile based on the scaling ratio parameter of the current sub-tile;

[0020] an obtaining unit configured to obtain coordinates of the current sub-tile in a 3D space coordinate system based on the scaling factor of the current sub-tile.

[0021] In a fourth aspect, the present application provides an encoder, comprising:

[0022] an obtaining unit configured to obtain a current sub-tile;

[0023] a determining unit configured to determine whether the current sub-tile is a sub-tile that needs to change resolution;

[0024] an encoding unit configured to, when the current sub-tile is the sub-tile that needs to change resolution, write a scaling ratio parameter of the current sub-tile into a bitstream.

[0025] In a fifth aspect, the present application provides an electronic device, comprising:

[0026] a processor adapted to implement computer instructions; and

[0027] The computer readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by the processor and performing the encoding method or the decoding method in any one of the first aspect to the second aspect or the implementation manner thereof.

[0028] In an implementation manner, the processor is one or more, and the memory is one or more.

[0029] In an implementation manner, the computer readable storage medium can be integrated with the processor, or the computer readable storage medium is arranged separately from the processor.

[0030] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device performs the encoding method or the decoding method in any one of the first aspect to the second aspect or the implementation manner thereof.

[0031] In a seventh aspect, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method in any one of the first aspect to the second aspect or the implementation manner thereof.

[0032] Based on the above technical solutions, when the current sub-block is a sub-block that needs to change the resolution, the scaling ratio parameter of the current sub-block is introduced in the code stream, that is, the scaling ratio parameter at the sub-block level is introduced in the code stream, which is beneficial to use different scaling parameters for different sub-blocks in the same stitching picture or use different scaling parameters for different sub-blocks under the same view, can improve the expression effect of some important sub-blocks, and further can improve the quality of the decoded image. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is an example of the coding system of the MIV provided by the embodiment of the present application.

[0034] Figure 2 is an example of the preprocessing process provided by the embodiment of the present application.

[0035] Figure 3 is an example of the generation process of the stitching picture provided by the embodiment of the present application.

[0036] Figure 4 is an example of the video encoding framework provided by the embodiment of the present application.

[0037] Figure 5 is an example of a video decoding framework provided by embodiments of the present application.

[0038] Figure 6 is a schematic flowchart of a resolution-based decoding method provided by embodiments of the present application.

[0039] Figure 7 is an example of obtaining coordinates of a current sub-tile in a 3D space coordinate system based on a scaling factor of the current sub-tile provided by embodiments of the present application.

[0040] Figure 8 is an example of projecting a current sub-tile from a 2D stitching map coordinate system to a 3D sub-tile coordinate system provided by embodiments of the present application.

[0041] Figure 9 is a schematic flowchart of a resolution-based encoding method provided by embodiments of the present application.

[0042] Figure 10 is a schematic block diagram of a decoder provided by embodiments of the present application.

[0043] Figure 11 is a schematic block diagram of an encoder provided by embodiments of the present application.

[0044] Figure 12 is a schematic block diagram of an electronic device provided by embodiments of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below with reference to the drawings.

[0046] Figure 1 is an example of a codec system 10 of MIV provided by embodiments of the present application.

[0047] As shown in Figure 1 , the codec system 10 of MIV can be roughly divided into the following modules according to the task line: a data acquisition module 11, a preprocessing module 12, a video encoding framework 13, a video decoding framework 14, and a post-processing module 15.

[0048] The data acquisition module 11 is configured to acquire data. The data acquired by the data acquisition module 11 can include not only texture information but also depth information corresponding to the texture information. The texture information can be a three-channel color image, and the depth information can be a depth map. Optionally, the pixel value of the depth map can be used to reflect the distance measure of the corresponding point to the acquisition device, i.e., the depth map can be geometric information.

[0049] The pre-processing module 12 is configured to organize and express the data collected by the data collection module 11 to obtain a video to be encoded. The input source of the pre-processing module 12 is a multi-view texture plus depth video, and the video can be a perspective projection planar video or a panoramic video. Specifically, after the data collection module 11 collects the corresponding video sequences through cameras at different positions, the pre-processing module 12 can perform, for the images in the video sequences, illumination equalization, color correction, camera calibration, and image calibration. If the video collected by the data collection module 11 is a panoramic video, the pre-processing module 12 can also perform image stitching on the images in the panoramic video by using an image stitching technology, and map the stitched image into a two-dimensional planar video.

[0050] Figure 2 is an example of a pre-processing process provided by an embodiment of the present application.

[0051] As shown in Figure 2 , the reference view is a view used by the data collection module 11 to collect data, and the reference view can be associated through a view parameter, texture data, and geometry data. The pre-processing module 12 can select a limited number of views as base views from the reference view, and the selected base views can express the visible range of the scene as much as possible. The images under the base views can be transmitted as complete images. Further, the redundant pixels between the remaining non-base views and the base views can be removed, that is, only the effective information that is not redundantly expressed is retained, and then the effective information is extracted as sub-tiles and the images under the base views are reorganized to form larger rectangular images, which can also be referred to as stitched images or stitched images. The stitched images can be used to generate video data, which can include texture video data and geometry video data. The video data can be used as input images of the video encoding framework 13, and the output of the video encoding framework 13 can be used as input of the video decoding framework 14.

[0052] Of course, for the stitched images, there can also be auxiliary data related to the stitching of the sub-tiles.

[0053] As shown in Figure 1 , the auxiliary data related to the stitching information of the sub-tiles can also be written into a bitstream, so that the pixel rate of transmission can be reduced while the scene information is retained as much as possible, and thus there is enough information for the post-processing module 15 to render the reconstructed images to obtain the final view.

[0054] Figure 3 is an example of a pre-processing process provided by an embodiment of the present application.

[0055] As shown in Figure 3As shown, the reference viewpoints can include viewpoint 0, viewpoint 1 and viewpoint 2, the reference viewpoint image captured under the viewpoint 0 can be used to extract sub-block 2 and sub-block 4, the reference viewpoint image captured under the viewpoint 1 can be used to extract sub-block 8, the reference viewpoint image captured under the viewpoint 2 can be used to extract sub-block 3 and sub-block 7, further, the stitching image 0 can be obtained based on the sub-block 2, the sub-block 4 and the sub-block 8. The stitching image 1 can be obtained based on the sub-block 3 and the sub-block 7. Figure 1 ; the stitching image 0 and the stitching Figure 1 may be input to the video encoding framework 13.

[0056] The video encoding framework 13 is used for encoding compression of the video, and the video decoding framework 14 is used for decoding reconstruction of the video.

[0057] It should be noted that the specific implementation of the video encoding framework 13 and the video decoding framework 14 is not limited in the present application. For example, a coding and decoding framework that applies three-dimensional video coding technologies such as MV-HEVC and 3D-HEVC to multi-viewpoint video can be used. The coding and decoding efficiency of this coding and decoding framework is higher than that of HEVC. However, as the number of input reference viewpoints increases, MV-HEVC and 3D-HEVC reduce the bandwidth load but introduce higher computational complexity. In addition, there are devices that do not support MV-HEVC and 3D-HEVC technologies. Therefore, a coding and decoding framework of traditional planar video hybrid coding technologies such as HEVC and VVC can also be used.

[0058] The post-processing module 15 is used for compositing and rendering the images decoded by the video decoding framework 14 to obtain the final view. For example, the target view at the user's viewpoint position and orientation can be compositely rendered according to the decoded and reconstructed images and the user's current pose information.

[0059] In order to facilitate understanding of the scheme of the present application, the video encoding framework 13 and the video decoding framework 14 are briefly introduced as follows.

[0060] Figure 4 is a schematic block diagram of the encoding framework 100 provided by an embodiment of the present application.

[0061] The encoding framework 100 calculates a residual block by calculating a residual between a prediction block and a to-be-encoded image block, and transmits the residual block to a decoding end via a process such as transformation and quantization. After receiving and analyzing the code stream, the decoding end obtains the residual block via a step such as inverse transformation and inverse quantization, and obtains a reconstructed block by superimposing a prediction block obtained by prediction of the decoding end on the residual block. As shown in FIG. 1, the encoding framework 100 includes a prediction module 101, a residual calculation module 102, a transform and quantization module 103, an entropy coding module 104 and a decoding end 105. Figure 4As shown, the encoding framework 100 can include an intra prediction unit 180, an inter prediction unit 170, a residual unit 110, a transform and quantization unit 120, an entropy encoding unit 130, an inverse transform and inverse quantization unit 140, and a loop filter unit 150. Optionally, the encoding framework 100 can further include a decoded picture buffer unit 160. The encoding framework 100 can also be referred to as a hybrid framework encoding mode.

[0062] The intra prediction unit 180 or the inter prediction unit 170 can perform prediction on a to-be-encoded image block to output a predicted block.

[0063] The intra prediction unit 180 can be used for intra prediction, which only refers to information of the same frame image to predict pixel information in a to-be-encoded image block for eliminating spatial redundancy. The frame used for intra prediction can be an I frame. For example, according to an encoding order from left to right and from top to bottom, a to-be-encoded image block can refer to a left upper image block, an upper image block, and a left image block as reference information to predict the to-be-encoded image block, and the to-be-encoded image block is used as reference information of a next image block. In this way, the entire image can be predicted. If the input digital video is in a color format, for example, a YUV 4:2:0 format, each 4 pixel points of each image frame of the digital video are composed of 4 Y components and 2 UV components, and the encoding framework 100 can encode the Y components (i.e., luminance blocks) and the UV components (i.e., chrominance blocks) respectively. Similarly, the decoding end can also perform corresponding decoding according to the format.

[0064] Exemplarily, the intra prediction can predict a to-be-encoded image block by means of an angular prediction mode and a non-angular prediction mode to obtain a predicted block. For example, an optimal prediction mode of the to-be-encoded image block can be selected according to rate-distortion information calculated from the predicted block and the to-be-encoded image block, and the prediction mode is transmitted to the decoding end through a bitstream. The decoding end analyzes the prediction mode, predicts a predicted block of a target decoding block, and superimposes a time-domain residual block obtained through the bitstream transmission to obtain a reconstructed block. The non-angular mode remains relatively stable, and has a mean value mode and a planar mode; the angular mode is continuously increased with the evolution of the digital video coding standard. Taking the H series of international digital video coding standards as an example, the H.264 / AVC standard has only 8 angular prediction modes and 1 non-angular prediction mode; the H.265 / HEVC is expanded to 33 angular prediction modes and 2 non-angular prediction modes. In the H.266 / VVC, the intra prediction mode is further expanded, and for a luminance block, there are 67 traditional prediction modes and a non-traditional matrix weighted intra-frame prediction (MIP) mode. The traditional prediction modes include: a planar mode with mode number 0, a DC mode with mode number 1, and angular prediction modes with mode numbers 2 to 66.

[0065] The inter prediction unit 170 can be configured to perform inter prediction, which can refer to image information of different frames, and search for motion vector information that best matches a to-be-encoded image block using motion estimation, for removing temporal redundancy; the frames used for inter prediction can be P frames and / or B frames, where P frames refer to forward prediction frames, and B frames refer to bidirectional prediction frames.

[0066] The residual unit 110 can calculate a residual block based on the prediction block and the to-be-encoded image block, i.e., the difference between the prediction block and the to-be-encoded image block. The residual block can be transformed and quantized by the transform and quantization unit 120, so as to remove information that is not sensitive to human eyes, for removing visual redundancy. Alternatively, the residual block before being transformed and quantized by the transform and quantization unit 120 can be referred to as a time-domain residual block, and the residual block after being transformed and quantized by the transform and quantization unit 120 can be referred to as a frequency residual block or a frequency-domain residual block.

[0067] The entropy encoding unit 130 can output a bitstream based on the transform and quantization coefficients output by the transform and quantization unit 120. For example, the entropy encoding unit 130 can remove character redundancy based on a target context model and probability information of a binary bitstream. For example, the entropy encoding unit 130 can be configured to perform context-based adaptive binary arithmetic coding (CABAC). The entropy encoding unit 130 can also be referred to as a header information encoding unit.

[0068] It should be understood that the to-be-encoded image block referred to in the present application can also be referred to as an original image block or a target image block, the prediction block can also be referred to as a predicted image block or an image prediction block, and can also be referred to as prediction signal or prediction information, and the reconstructed block can also be referred to as a reconstructed image block or an image reconstructed block, and can also be referred to as a reconstructed signal or reconstructed information. In addition, the to-be-encoded image block can also be referred to as an encoding block or an encoding image block for the encoding end, and can also be referred to as a decoding block or a decoding image block for the decoding end. The to-be-encoded image block can be a CTU or a CU. The present application does not make specific limitations on this.

[0069] It should be noted that the inverse transform and inverse quantization unit 140, the loop filter unit 150, and the decoded picture buffer unit 160 can be used to form a decoder. In other words, the intra prediction unit 180 or the inter prediction unit 170 can predict the to-be-encoded image block based on the already reconstructed block, so as to ensure that the understanding of the reference frame at the encoding end and the decoding end is consistent. In other words, the encoder can copy the processing loop of the decoder, so as to generate the same prediction as the decoding end. Specifically, the quantized transform coefficient is inverse transformed and inverse quantized by the inverse transform and inverse quantization unit 140, so as to copy the approximate residual block of the decoding end. The approximate residual block is added to the prediction block, and then the loop filter unit 150 is used to smooth the block effect and the like caused by the block-based processing and quantization. The image block output by the loop filter unit 150 can be stored in the decoded picture buffer unit 160, so as to be used for prediction of subsequent images.

[0070] It should be understood that, Figure 4 The foregoing merely illustrates some examples of the present application, and should not be understood to limit the present application.

[0071] For example, the loop filter unit 150 in the encoding framework 100 can include a deblocking filter (DBF), a sample adaptive offset filter (SAO), and an adaptive loop filter (ALF). The DBF is used to remove the blocking effect, and the SAO is used to remove the ringing effect. In other embodiments of the present application, the encoding framework 100 can use a neural network-based loop filtering algorithm to improve the compression efficiency of the video. In other words, the encoding framework 100 can be a hybrid framework of the neural network-based video coding based on deep learning. In one implementation, the neural network-based calculation can be used to filter the pixel result on the basis of the deblocking filter and the sample adaptive offset filter. The network structure of the loop filter unit 150 on the luminance component and the chrominance component can be the same or different. Considering that the luminance component contains more visual information, the luminance component can also be used to guide the filtering of the chrominance component, so as to improve the reconstruction quality of the chrominance component.

[0072] Figure 5 FIG. 2 is a schematic block diagram of a decoding framework 200 provided by an embodiment of the present application.

[0073] As Figure 5 shown, the decoding framework 200 can include an entropy decoding unit 210, an inverse transform and inverse quantization unit 220, a residual unit 230, an intra prediction unit 240, an inter prediction unit 250, a loop filter unit 260, and a decoded picture buffer unit 270.

[0074] The entropy decoding unit 210 receives and parses the bitstream to obtain a prediction block and a frequency domain residual block. For the frequency domain residual block, the inverse transform and inverse quantization are performed by the inverse transform and inverse quantization unit 220, and a time domain residual block is obtained. The residual unit 230 superimposes the prediction block obtained by the intra prediction unit 240 or the inter prediction unit 250 to the time domain residual block after the inverse transform and inverse quantization by the inverse transform and inverse quantization unit 220, and a reconstructed block is obtained. For example, the intra prediction unit 240 or the inter prediction unit 250 can obtain the prediction block by decoding the header information of the bitstream.

[0075] In the MIV decoding process, the same scaling factor can be used to convert the stitching atlas from the stitching atlas coordinate system to the 3D space coordinate system to obtain the 3D view at the user's viewpoint position. Each stitching atlas can include one or more sub-atlases, and each sub-atlas can be a sub-atlas at a certain viewpoint. For all sub-atlases in the stitching atlas whose viewpoints are special viewpoints, the scaling parameters of the stitching atlas level read from the bitstream can be calculated to obtain the scaling factors of all sub-atlases in the stitching atlas whose viewpoints are special viewpoints. The scaling factor of each sub-atlas can include a width scaling factor and a height scaling factor. In addition, for sub-atlases in the stitching atlas whose viewpoints are ordinary viewpoints, the scaling factors can not be read from the bitstream, and the default scaling factors, such as the default width scaling factor and the default height scaling factor, can be directly used. That is, the same scaling factor is used for all sub-atlases in the special viewpoint, and the default scaling factor is used for all sub-atlases in the ordinary viewpoint.

[0076] Exemplarily, the decoding end can decode the sub-atlases in the following manner:

[0077] Step 1:

[0078] The compensation flag asme_inpaint_enabled_flag of the stitching atlas level is obtained from the bitstream. The asme_inpaint_enabled_flag is used to indicate whether the up-sampling pixel compensation (inpaint) is used for the corresponding stitching atlas (atlas).

[0079] Step 2:

[0080] The X-axis coordinate scaling parameter afme_inpaint_lod_scale_x_minus1 and the Y-axis coordinate scaling parameter afme_inpaint_lod_scale_y_idc of the stitching atlas are obtained from the bitstream.

[0081] Step 3:

[0082] When the asme_inpaint_enabled_flag flag is true, a sub-tile level compensation flag pdu_inpaint_flag is obtained from the bitstream, and the pdu_inpaint_flag is used to indicate whether to use up-sampling pixel compensation (inpaint) for the corresponding sub-tile.

[0083] Step 4:

[0084] When the pdu_inpaint_flag flag is true, it indicates that the corresponding sub-tile corresponds to a special view point, and the following method 1 can be used to perform up-sampling pixel compensation on the corresponding sub-tile.

[0085] Method 1:

[0086] A width scaling factor (TilePatchLoDScaleX[tileID][p]) is obtained according to the afme_inpaint_lod_scale_x_minus1, and a height scaling factor (TilePatchLoDScaleY[tileID][p]) is obtained according to the afme_inpaint_lod_scale_y_idc.

[0087] Method 2:

[0088] The width scaling factor and the height scaling factor are 1 by default.

[0089] Step 5:

[0090] After obtaining the width scaling factor and the height scaling factor, the width and the height of each position (x, y) in the sub-tile in the special view point are expanded, so as to convert the 2D stitching map coordinates (X, Y) of the sub-tile into local 3D sub-tile coordinates (u, v, D).

[0091] Step 6:

[0092] After obtaining the local 3D sub-tile coordinates of the sub-tile, the coordinates are mapped to a 3D view point space coordinate system through different back projection schemes.

[0093] For example, the patch P1 with tile index tileID1 and patch index p1 in the tile picture, when the patch level flag is true, means that the corresponding view is a special view. After the decoder gets afme_inpaint_lod_scale_x_minus1 and afme_inpaint_lod_scale_y_idc, the width scaling factor TilePatchLoDScaleX[tileID1][p1] and the height scaling factor TilePatchLoDScaleY[tileID1][p1] can be obtained.

[0094] For example, the patch P2 with tile index tileID1 and patch index p2 in the tile picture, when the patch level flag is true, means that the corresponding view is a special view. After the decoder gets afme_inpaint_lod_scale_x_minus1 and afme_inpaint_lod_scale_y_idc, the width scaling factor TilePatchLoDScaleX[tileID1][p2] and the height scaling factor TilePatchLoDScaleY[tileID1][p2] can be obtained, which are the same as those of P1.

[0095] For example, the patch P3 with tile index tileID2 and patch index p1 in the tile picture, when the patch level flag is true, means that the corresponding view is a special view. After the decoder gets afme_inpaint_lod_scale_x_minus1 and afme_inpaint_lod_scale_y_idc, the width scaling factor TilePatchLoDScaleX[tileID2][p1] and the height scaling factor TilePatchLoDScaleY[tileID2][p1] can be obtained, which are the same as those of P1.

[0096] For example, the patch P4 with tile index tileID3 and patch index p3 in the tile picture, when the patch level flag is false, means that the corresponding view is a normal view. The width scaling factor TilePatchLoDScaleX[tileID3][p3] and the height scaling factor TilePatchLoDScaleY[tileID3][p3] can be obtained as the default value 1.

[0097] However, in the above scheme, if all sub-tiles in the special view use the same scaling factor, when the importance of different sub-tiles in the special view is different, the sub-tiles parsed at the decoding end can not be able to express the information completely, reducing the quality of the decoded image. Similarly, if the scaling factor of all sub-tiles in the normal view is defaulted to 1, when there is a relatively important sub-tile in the normal view, the sub-tile parsed at the decoding end can not be able to express the information completely, reducing the quality of the decoded image.

[0098] Therefore, embodiments of the present application provide a resolution-based decoding method, a resolution-based encoding method, a decoder and an encoder, which can improve the quality of the decoded image.

[0099] Figure 6 FIG. 3 is a schematic flowchart of a resolution-based decoding method 300 provided by embodiments of the present application.

[0100] As shown in FIG. 3, the decoding method 300 can include the following steps. Figure 6

[0101] S310, parsing, by a decoding end, a code stream to obtain a current sub-tile;

[0102] S320, determining, by the decoding end, whether the current sub-tile is a sub-tile that needs to change resolution;

[0103] S330, when the current sub-tile is a sub-tile that needs to change resolution, parsing, by the decoding end, the code stream to obtain a scaling rate parameter of the current sub-tile;

[0104] S340, determining, by the decoding end, a scaling factor of the current sub-tile based on the scaling rate parameter of the current sub-tile;

[0105] S350, obtaining, by the decoding end, coordinates of the current sub-tile in a 3D space coordinate system based on the scaling factor of the current sub-tile.

[0106] Exemplarily, the scaling rate parameter of the current sub-tile can include a width scaling rate parameter pdu_sampling_rate_width[tileID][p] and a height scaling rate parameter pdu_sampling_rate_height[tileID][p]. Wherein, pdu_sampling_rate_width[tileID][p] represents the width scaling rate parameter of a sub-tile with tile index tileID and patch index p. pdu_sampling_rate_height[tileID][p] represents the height scaling rate parameter of the sub-tile with tile index tileID and patch index p. ​

[0107] In the embodiment, when the current sub-tile is a sub-tile that needs to change the resolution, the scaling rate parameter of the current sub-tile is introduced into the code stream, that is, the scaling rate parameter at the sub-tile level is introduced into the code stream, which is beneficial to use different scaling parameters for different sub-tiles in the same stitching picture or use different scaling parameters for different sub-tiles under the same view point, can improve the expression effect of some important sub-tiles, and further can improve the quality of the decoded image.

[0108] From the perspective of code stream and image quality, the decoding method provided in the application can achieve the following technical effects:

[0109] 1. The code stream can be reduced without changing the image / video quality.

[0110] Specifically, if the scaling factor of all sub-tiles in the normal view point is defaulted to 1, when there is a relatively important sub-tile in the normal view point, the sub-tile parsed by the decoding end may not be able to express the information completely, which reduces the quality of the decoded image. The application can use different scaling rate parameters for each sub-tile in the normal view point, for example, use a higher scaling rate parameter for some sub-tiles and use a lower scaling rate parameter for some sub-tiles, which is equivalent to avoiding using the default unchanged scaling rate parameter for all sub-tiles in the normal view point. Based on this, the code stream of some sub-tiles with high scaling rate parameters will be small but will not affect the overall quality.

[0111] 2. The decoding performance can be improved under the condition of low code rate.

[0112] Specifically, under the condition of low code rate, when the default unchanged scaling rate parameter is used for all normal view points, for some important sub-tiles of some important view points, the decoding end may not be able to parse the expression information completely, thereby reducing the quality of the decoded image. The application can use a higher sampling rate to obtain a smaller code stream for some important sub-tiles in some normal view points with low importance, thereby being beneficial to provide more code streams for other sub-tiles, and further being beneficial to achieve higher decoding performance.

[0113] 3. When there is an occlusion area in some view points, especially small block occlusion, the performance can be improved.

[0114] Specifically, for small block occlusion, a higher sampling rate can be used to obtain a smaller code stream, thereby being beneficial to provide more code streams for other sub-tiles, and further being beneficial to achieve higher decoding performance.

[0115] In some embodiments, the S320 can include:

[0116] The code stream is parsed to obtain a value of the first identifier; wherein the value of the first identifier is a first numerical value, indicating that the current sub-tile is a sub-tile that needs to change resolution, and the value of the first identifier is a second numerical value, indicating that the current sub-tile is a sub-tile that does not need to change resolution.

[0117] Illustratively, the first numerical value is 0 and the second numerical value is 1, or the first numerical value is 1 and the second numerical value is 0.

[0118] Illustratively, the first identifier is a sub-tile level identifier, which can be represented as pdu_scaled_flag.

[0119] Illustratively, the decoding end can parse the code stream to obtain a value of the pdu_scaled_flag; wherein the value of the pdu_scaled_flag is 1, indicating that the current sub-tile is a sub-tile that needs to change resolution, and the value of the pdu_scaled_flag is 0, indicating that the current sub-tile is a sub-tile that does not need to change resolution.

[0120] Illustratively, the decoding end can parse the code stream to obtain the pdu_scaled_flag of the current sub-tile in the spliced tile, and when the value of the pdu_scaled_flag of the current sub-tile is 1, the decoding end can parse the code stream to obtain a width scaling rate parameter pdu_sampling_rate_width[tileID][p] and a height scaling rate parameter pdu_sampling_rate_height[tileID][p] of the current sub-tile.

[0121] Illustratively, the pdu_scaled_flag, the pdu_sampling_rate_width[tileID][p] and the pdu_sampling_rate_height[tileID][p] can be realized as fields or elements in a patch data unit MIV extension syntax. For example, the patch data unit MIV extension syntax can be realized as the syntax shown in Table 1:

[0122] Table 1

[0123]

[0124] As shown in Table 1, when the value of asme_scaled_enabled_flag is 1, the decoding end can obtain pdu_scaled_flag[tileID][p] by parsing the code stream, and when the value of pdu_scaled_flag[tileID][p] is 1, the decoding end can obtain the scaling rate parameters of the current subpatch, i.e., the width scaling rate parameter pdu_sampling_rate_width[tileID][p] and the height scaling rate parameter pdu_sampling_rate_height[tileID][p] of the current subpatch by parsing the code stream. When the value of pdu_scaled_flag[tileID][p] is equal to 1, it indicates that the current subpatch is a subpatch that needs to change the sampling rate. When the value of pdu_scaled_flag[tileID][p] is equal to 0, it indicates that the current subpatch is not a subpatch that needs to change the sampling rate. When pdu_scaled_flag[tileID][p] does not exist, the value of pdu_scaled_flag[tileID][p] should be 0. pdu_sampling_rate_width[tileID][p] represents the width scaling rate parameter of the subpatch with tile index tileID and patch index p. pdu_sampling_rate_height[tileID][p] represents the height scaling rate parameter of the subpatch with tile index tileID and patch index p.

[0125] In some embodiments, the S320 can include:

[0126] determining whether the current patch in which the current subpatch is located includes subpatches with different resolutions;

[0127] when the current patch includes subpatches with different resolutions, determining whether the current subpatch is a subpatch that needs to change the resolution.

[0128] In the embodiments of the present application, before determining whether the current subpatch is a subpatch that needs to change the resolution, it is determined whether the current patch in which the current subpatch is located includes subpatches with different resolutions. When the current patch does not include subpatches with different resolutions, i.e., when the current patch includes subpatches with the same resolution, the introduction of the subpatch-level identifier, i.e., the first identifier, can be avoided, and the coding performance can be improved.

[0129] In some embodiments, the code stream is parsed to obtain a value of a second identifier; when the value of the second identifier is a third numerical value, it indicates that the current tile includes sub-tiles of different resolutions; and when the value of the second identifier is a fourth numerical value, it indicates that the current tile includes sub-tiles of the same resolution.

[0130] For example, the current tile including sub-tiles of different resolutions can mean that the current tile includes at least two sub-tiles of different resolutions. The current tile including sub-tiles of the same resolution can mean that all sub-tiles included in the current tile have the same resolution.

[0131] For example, the third numerical value is 0 and the fourth numerical value is 1, or the third numerical value is 1 and the fourth numerical value is 0.

[0132] For example, the second identifier is a tile-level identifier, which can be represented as asme_scaled_enabled_flag.

[0133] For example, the code stream can be parsed to obtain a value of asme_scaled_enabled_flag; when the value of asme_scaled_enabled_flag is 1, it indicates that the current tile includes sub-tiles of different resolutions; and when the value of asme_scaled_enabled_flag is 0, it indicates that the current tile includes sub-tiles of the same resolution.

[0134] For example, the decoding end can parse the code stream to obtain asme_scaled_enabled_flag of the current tile; when the value of asme_scaled_enabled_flag of the current tile is 1, the decoding end can parse the code stream to obtain pdu_scaled_flag of a current sub-tile in the tile, and when the value of pdu_scaled_flag of the current sub-tile is 1, the decoding end can parse the code stream to obtain scaling rate parameters of the current sub-tile, such as width scaling rate parameter pdu_sampling_rate_width[tileID][p] and height scaling rate parameter pdu_sampling_rate_height[tileID][p].

[0135] For example, the value of asme_scaled_enabled_flag can be carried in a field or element in an Atlas sequence parameter set MIV extension syntax.

[0136] Exemplarily, the patch map sequence parameter set MIV extension syntax can be implemented as the syntax shown in Table 2:

[0137] Table 2

[0138]

[0139] As shown in Table 2, the patch map level identifier asme_scaled_enabled_flag can be carried in the asps_miv_extension() syntax structure. Wherein, the value of asme_scaled_enabled_flag equal to 1 indicates that the pdu_sampling_rate_width and pdu_sampling_rate_height exist in the patch_data_unit() syntax structure. The value of asme_scaled_enabled_flag equal to 0 indicates that the pdu_sampling_rate_width and pdu_sampling_rate_height do not exist in the patch_data_unit() syntax structure.

[0140] In some embodiments, the S320 can include:

[0141] determining whether the current view point in which the current sub-tile is located includes sub-tiles of different resolutions;

[0142] when the current view point includes sub-tiles of different resolutions, determining whether the current sub-tile is a sub-tile that needs to change resolution.

[0143] In the embodiments of the present application, before determining whether the current sub-tile is a sub-tile that needs to change resolution, it is determined whether the current view point in which the current sub-tile is located includes sub-tiles of different resolutions. When the current view point does not include sub-tiles of different resolutions, i.e., when the current view point includes sub-tiles of the same resolution, the introduction of the sub-tile level identifier, i.e., the first identifier, can be avoided, and the coding performance can be improved.

[0144] In some embodiments, the value of the third identifier is obtained by parsing the code stream; when the value of the third identifier is a fifth numerical value, it indicates that the current view point includes sub-tiles of different resolutions, and when the value of the third identifier is a sixth numerical value, it indicates that the current view point includes sub-tiles of the same resolution.

[0145] Exemplarily, the current view point including sub-tiles of different resolutions can be understood as that the current view point includes at least two sub-tiles of different resolutions. The current view point including sub-tiles of the same resolution can be understood as that all the sub-tiles included in the current view point are of the same resolution.

[0146] Exemplarily, the fifth value is 0 and the sixth value is 1, or the fifth value is 1 and the sixth value is 0.

[0147] Exemplarily, the third identifier is a view level identifier, which can be represented as mvp_view_wise_enabled_flag[viewID].

[0148] Exemplarily, the decoding end can obtain the value of mvp_view_wise_enabled_flag[viewID] by parsing the code stream; wherein, when the value of mvp_view_wise_enabled_flag[viewID] is 1, it indicates that the current view includes sub-blocks of different resolutions, and when the value of mvp_view_wise_enabled_flag[viewID] is 0, it indicates that the current view includes sub-blocks of the same resolution.

[0149] Exemplarily, the value of mvp_view_wise_enabled_flag[viewID] can be carried in a field or element in the MIV view parameter list syntax (Atlas MIV view parameters list syntax).

[0150] Exemplarily, the MIV view parameter list syntax can be implemented as the syntax shown in Table 3:

[0151] Table 3

[0152]

[0153]

[0154] As shown in Table 3, mvp_view_wise_enabled_flag[viewID] can be carried in the miv_view_params_list() syntax structure. Wherein, the value of mvp_view_wise_enabled_flag[viewID] equal to 1 indicates that the view with index viewID is a view with different resolutions for different sub-tiles. The value of mvp_view_wise_enabled_flag[viewID] equal to 0 indicates that the view with index viewID is a view with the same resolution for all sub-tiles. When not present, the value of mvp_view_wise_enabled_flag[viewID] shall be 0. Exemplarily, the identification of (mvp_num_views_minus1+1) views can be used to form a one-dimensional array, which can be defined by the ID of the view as follows: mvp_view_wise_enabled_flag[viewID].

[0155] In some embodiments, the S340 can include:

[0156] The scaling ratio parameter of the current sub-tile is determined as the scaling factor of the current sub-tile.

[0157] It should be understood that the present application does not specifically limit the form of the scaling ratio parameter of the current sub-tile in the code stream. For example, the decoding end can directly obtain the scaling ratio parameter of the current sub-tile by parsing the code stream, that is, the decoding end can determine the parameter obtained by parsing the code stream as the scaling factor of the current sub-tile. For another example, the decoding end can obtain an index by parsing the code stream, and determine the scaling ratio parameter corresponding to the index value as the scaling ratio parameter of the current sub-tile; that is, the decoding end can determine the scaling ratio parameter corresponding to the index obtained by parsing the code stream as the scaling factor of the current sub-tile.

[0158] In some embodiments, the method 300 can further include:

[0159] When the current sub-tile is a sub-tile that does not need to change the resolution, the scaling ratio parameter of the current sub-tile is determined as a default value.

[0160] In some embodiments, a first coordinate of the current sub-tile in a 2D splicing map coordinate system is obtained; a second coordinate of the current sub-tile in a 3D sub-tile coordinate system is obtained based on the first coordinate and the scaling factor; and the second coordinate is back-projected to the 3D space coordinate system to obtain the coordinate of the current sub-tile in the 3D space coordinate system.

[0161] Exemplarily, after obtaining the second coordinates, the decoding end can combine the texture information or the depth information of the current sub-tile to back-project the second coordinates to the 3D space coordinate system, and finally obtain the coordinates of the current sub-tile in the 3D space coordinate system. Optionally, the depth information can be a depth map. Optionally, the pixel value of the depth map can be used to reflect the distance measurement of the corresponding point to the acquisition device, that is, the depth map can be used to express the geometric information of the visible surface of the current sub-tile.

[0162] Exemplarily, the 3D sub-tile coordinate system can also be understood as a 2D coordinate system in which the current stitching picture is located. For example, a tangent (U) double tangent (V) coordinate system. For example, a tangent (U) double tangent (V) normal (D) coordinate system, that is, the second coordinates are (U, V, 1).

[0163] Exemplarily, the 3D space coordinate system can be a 3D coordinate system in which a target viewpoint is located. For example, a 3D Cartesian coordinate system (x, y, z).

[0164] It should be understood that the embodiments of the present application do not limit the specific implementation mode of back-projecting the second coordinates to the 3D space coordinate system. For example, the back-projection process defined by standard Annex H2.4 can be implemented, that is, the coordinates (U, V, D) in the projected local 3D space are back-projected to the real 3D Cartesian coordinate system (x, y, z). Exemplarily, the back-projection operation involved in the present application includes but is not limited to: ERP back-projection operation, perspective back-projection operation and orthogonal back-projection operation. Optionally, the ERP back-projection operation can refer to the content described in standard Annex H2.4.1, the perspective back-projection operation can refer to the content described in standard Annex H2.4.2, and the orthogonal back-projection operation can refer to the content described in standard Annex H2.4.3. To avoid repetition, details are not described here.

[0165] In some embodiments, the current sub-tile is projected to the 3D sub-tile coordinate system based on the first coordinates, the scaling factor and transformation parameters to obtain the second coordinates; wherein the transformation parameters include a transformation process for characterizing an exchange operation and / or a transformation process for a flip operation.

[0166] Exemplarily, the transformation operation can be a mirror transformation operation along the U axis or the V axis, the flip operation can be a flip operation along the U axis or the V axis, etc. For example, the flip operation can be a 90-degree flip operation along the U axis or the V axis, etc.

[0167] In some embodiments, when the scaling parameter of the current sub-tile includes a width scaling parameter, the scaling factor of the current sub-tile includes the width scaling factor corresponding to the width scaling parameter; and / or, when the scaling parameter of the current sub-tile includes a height scaling parameter, the scaling factor of the current sub-tile includes the height scaling factor corresponding to the height scaling parameter.

[0168] Figure 7 This is an example provided in this application of obtaining the coordinates of the current sub-tile in a 3D spatial coordinate system based on the scaling factor of the current sub-tile.

[0169] like Figure 7 As shown in (a), after the decoder obtains the width scaling factor and height scaling factor of the current sub-tile, it can obtain the 2D mosaic width coordinates and height coordinates of the current sub-tile in the 2D mosaic coordinate system. Figure 7 As shown in (b), the decoder can transform the width and height coordinates of the current sub-tile in the 2D mosaic coordinate system to obtain the upper-left coordinates (starting from the origin) of the current sub-tile in the 3D sub-tile coordinate system (also called the local sub-tile coordinate system). For example, the 3D sub-tile coordinate system can be the coordinate system where the projection plane is located. Figure 7 As shown in (c), after the decoder obtains the upper-left coordinates of the current sub-tile in the 3D sub-tile coordinate system, it multiplies the current sub-tile's coordinates in the 3D sub-tile coordinate system by their respective scaling factors to obtain the final coordinates of the current sub-tile in the 3D sub-tile coordinate system. If a swap operation exists, the obtained final coordinates need to be swapped; if a flip operation exists, the rightmost coordinate (the coordinate furthest from the origin) of the 3D sub-tile coordinate system needs to be subtracted from the obtained final coordinates.

[0170] For example, the coordinates (x, y) of the current sub-tile in the 2D mosaic coordinate system can be converted to coordinates (l, u, v) in the 3D sub-tile coordinate system using the following formula:

[0171]

[0172] Among them, R o (swap) and R s (invert) represents the transformation and flipping process, lodX and lodY represent the width scaling factor and height scaling factor, respectively, posX and poxY represent the width and height positions of the current sub-tile in the current mosaic, respectively, and sizeX and sizeY represent the width and height of the current sub-tile, respectively.

[0173] Figure 8is an example of projecting the current sub-tile from the 2D atlas coordinate system to the 3D sub-tile coordinate system provided by the embodiment of the present application.

[0174] As shown in (a) of Figure 8 The current sub-tile in the atlas can include a plurality of pixel points, Figure 8 (b) and (c) of are respectively for the case that the scaling factor is 1 and is not 1 Figure 7 The projection process in (b) of. If the resolution of the current sub-image and the resolution of the viewpoint are consistent, as shown in (b) of Figure 8 A pixel point on the 2D atlas coordinate system of the current sub-tile can correspond to or be projected to a pixel point in the 3D sub-tile coordinate system of, and the pixel points in the 3D sub-tile coordinate system have continuity. If the resolution of the current sub-image and the resolution of the viewpoint are inconsistent, as shown in (c) of Figure 8 A pixel point on the 2D atlas coordinate system of the current sub-tile can correspond to or be projected to a plurality of pixel points in the 3D sub-tile coordinate system of, which is equivalent to that the pixel point on the 2D atlas coordinate system of the current sub-tile can be up-sampled to ensure that the pixel points in the 3D sub-tile coordinate system have continuity.

[0175] The scheme of the present application will be described below in conjunction with specific embodiments.

[0176] Embodiment 1

[0177] In the present embodiment, the decoding end can include part or all of the following steps:

[0178] Step 1

[0179] The decoding end can obtain the atlas level identifier asme_scaled_enabled_flag from the code stream, and asme_scaled_enabled_flag is used to indicate whether the current atlas in which the current sub-tile is located includes sub-tiles of different resolutions.

[0180] Step 2

[0181] When asme_scaled_enabled_flag is yes, the decoding end can obtain pdu_scaled_flag of the current sub-tile from the code stream, and pdu_scaled_flag is used to indicate whether the current sub-tile is a sub-tile that needs to change the resolution.

[0182] Step 3

[0183] When the pdu_scaled_flag is determined to be yes, the decoder can obtain the scaling ratio parameters of the current sub-tile from the bitstream, i.e., the width scaling ratio parameter pdu_sampling_rate_width[tileID][p] and the height scaling ratio parameter pdu_sampling_rate_height[tileID][p].

[0184] Step 4:

[0185] When the asme_scaled_enabled_flag and the pdu_scaled_flag are both determined to be yes, the decoder can determine the scaling factor of the current sub-tile (indexed as tileID, p) by using the following method 1. When the asme_scaled_enabled_flag is determined to be no or the pdu_scaled_flag is determined to be no, the decoder can determine the scaling factor of the current sub-tile (indexed as tileID, p) by using the following method 2.

[0186] Method 1:

[0187] The decoder can obtain the width scaling factor TilePatchLoDScaleX[tileID][p] of the current sub-tile according to the width scaling ratio parameter pdu_sampling_rate_width[tileID][p], and obtain the height scaling factor TilePatchLoDScaleY[tileID][p] of the current sub-tile according to the height scaling ratio parameter pdu_sampling_rate_height[tileID][p].

[0188] Method 2:

[0189] The decoder can determine that the width scaling factor and the height scaling factor of the current sub-tile are both 1 by default.

[0190] Step 5:

[0191] After obtaining the width scaling factor and the height scaling factor of the current sub-tile by using the method 1 or the method 2, the 2D splicing picture width coordinate and the 2D splicing picture height coordinate of the current sub-tile in the 2D splicing picture coordinate system can be transformed to obtain the top-left coordinate (with the origin as the starting point) of the current sub-tile in the 3D sub-tile coordinate system (which can also be referred to as a local sub-tile coordinate system). Then, the coordinates of the current sub-tile in the 3D sub-tile coordinate system are multiplied by the respective scaling factors to obtain the final coordinates of the current sub-tile in the 3D sub-tile coordinate system. If there is an exchange operation, the obtained final coordinates need to be exchanged. If there is a flipping operation, the rightmost coordinate (the coordinate farthest from the origin) of the 3D sub-tile coordinate system needs to be subtracted from the obtained final coordinates.

[0192] Exemplarily, the coordinates (x, y) of the current subpatch in the 2D mosaic coordinate system can be converted into the coordinates (1, u, v) in the 3D subpatch coordinate system by using the following formula:

[0193]

[0194] wherein, R o (swap) and R s (invert) represent the transformation process and the flipping process, lodX and lodY represent the width scaling factor and the height scaling factor respectively, posX and poxY represent the width and height positions of the current subpatch in the current mosaic respectively, and sizeX and sizeY represent the width and height of the current subpatch respectively.

[0195] Step 6:

[0196] After obtaining the local 3D subpatch coordinates of the current subpatch, the coordinates are mapped to the 3D viewpoint space coordinate system by using different back projection schemes (for specific schemes, see the standard Annex H2.4). Exemplarily, the back projection operations involved in the present application include but are not limited to: ERP back projection operation, perspective back projection operation and orthogonal back projection operation. Optionally, the ERP back projection operation can refer to the content described in the standard Annex H2.4.1, the perspective back projection operation can refer to the content described in the standard Annex H2.4.2, and the orthogonal back projection operation can refer to the content described in the standard Annex H2.4.3. To avoid repetition, details are not described herein.

[0197] Example 1:

[0198] Suppose that the current subpatch is a subpatch P1 with a tile index tileID1 and a patch index p1 in the mosaic, a viewpoint index viewID1, and the pdu_scaled_flag is true. After the decoding end obtains the scaling rate parameters of the subpatch P1, i.e., the width scaling rate pdu_sampling_rate_width[tileID1][p1] and the height scaling rate pdu_sampling_rate_height[tileID1][p1] of the subpatch P1, the width scaling factor lodX = TilePatchLoDScaleX[tileID1][p1] and the height scaling factor lodY = TilePatchLoDScaleY[tileID1][p1] of the subpatch P1 can be obtained.

[0199] Further, suppose that the decoding end determines that the subpatch P1 does not need any transformation operation, then and

[0200] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0201]

[0202] Further, assuming that the decoder can determine that P1 needs perspective back projection operation, at this time, the final 3D space coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.2.

[0203] Example 2:

[0204] Assuming that the current subpatch is a subpatch P2 with tile index tileID1, patch index p2, and view index viewID1 in the stitching picture, and that pdu_scaled_flag is true, after the decoding end obtains the scaling rate parameters of the subpatch P2, i.e., the width scaling rate pdu_sampling_rate_width[tileID1][p2] and the height scaling rate pdu_sampling_rate_height[tileID1][p2] of the subpatch P2, the width scaling factor lodX = TilePatchLoDScaleX[tileID1][p2] of the subpatch P2 and the height scaling factor lodY = TilePatchLoDScaleY[tileID1][p2] of the subpatch P2 can be obtained.

[0205] Further, assuming that the decoding end determines that the subpatch P2 needs swap transform operation, then and

[0206] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0207]

[0208] Further, assuming that the decoder can determine that P2 needs perspective back projection operation, at this time, the final 3D space coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.2.

[0209] Example 3:

[0210] Assuming the current subpatch is the subpatch P3 with tile index tileID2, patch index p1 in the tile, view index viewID1, and pdu_scaled_flag is false, the width scaling factor lodX = TilePatchLoDScaleX[tileID2][p1] and the height scaling factor lodY = TilePatchLoDScaleY[tileID2][p1] are obtained as default value 1.

[0211] Further, assuming the decoder determines that the subpatch P3 needs a height flipping transformation operation, then and

[0212] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0213]

[0214] Further, assuming the decoder can determine that the P3 needs an orthogonal back projection operation, the final 3D space coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.3.

[0215] Example 4:

[0216] Assuming the current subpatch is the subpatch P4 with tile index tileID2, patch index p2 in the tile, view index viewID1, and pdu_scaled_flag is true, the decoder obtains the scaling rate parameters of the subpatch P4, i.e. the width scaling rate pdu_sampling_rate_width[tileID2][p2] and the height scaling rate pdu_sampling_rate_height[tileID2][p2] of the subpatch P4, and can obtain the width scaling factor lodX = TilePatchLoDScaleX[tileID2][p2] of the subpatch P4 and the height scaling factor lodY = TilePatchLoDScaleY[tileID2][p2] of the subpatch P4 based on this.

[0217] Further, assuming the decoder determines that the subpatch P4 needs a width flipping transformation operation, then and

[0218] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0219]

[0220] Further, assuming the decoder determines that P3 requires an orthogonal inverse projection operation, the final 3D spatial coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.3.

[0221] Example 5:

[0222] Assuming the current subpatch is subpatch P5 with tile index tileID2, patch index p3, and view index viewID2, and pdu_scaled_flag is true, the decoder obtains the scaling parameters of subpatch P5, i.e., the width scaling rate pdu_sampling_rate_width[tileID2][p3] and the height scaling rate pdu_sampling_rate_height[tileID2][p3] of subpatch P5, and can obtain the width scaling factor lodX = TilePatchLoDScaleX[tileID2][p3] and the height scaling factor lodY = TilePatchLoDScaleY[tileID2][p3] of subpatch P5 based on the obtained scaling parameters.

[0223] Further, assuming the decoder determines that subpatch P5 requires a width flipping and swapping transformation operation, then and

[0224]

[0225] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0226]

[0227] Further, assuming the decoder determines that subpatch P5 requires an ERP inverse projection operation, the final 3D spatial coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.1.

[0228] Example 6:

[0229] Assume the current subpatch is subpatch P6 with tile index tileID3, patch index p1 in the mosaic, view index viewID2, and pdu_scaled_flag is false, then the width scaling factor lodX = TilePatchLoDScaleX[tileID3][p1] and the height scaling factor lodY = TilePatchLoDScaleY[tileID3][p1] are default value 1.

[0230] Further, assume the decoding end judges that subpatch P6 needs height flipping and swap transform operation, then and

[0231]

[0232] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0233]

[0234] Further, assume the decoding end judges that subpatch P6 needs ERP back projection operation, at this time, the final 3D space coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.1.

[0235] Example 7:

[0236] Assume the current subpatch is subpatch P7 with tile index tileID3, patch index p2 in the mosaic, view index viewID3, and pdu_scaled_flag is false, then the width scaling factor lodX = TilePatchLoDScaleX[tileID3][p2] and the height scaling factor lodY = TilePatchLoDScaleY[tileID3][p2] are default value 1.

[0237] Further, assume the decoding end judges that subpatch P7 needs width flipping, height flipping and swap transform operation, then and

[0238]

[0239] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0240]

[0241] Further, assuming the decoding end judges that subpatch P7 needs orthogonal inverse projection operation, at this time the final 3D space coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.3.

[0242] Example 8:

[0243] Assuming that the current subpatch is subpatch P8 with tile index tileID4, patch index p3, and view index viewID3 in the spliced picture, and that pdu_scaled_flag is yes, after the decoding end obtains the scaling rate parameters of subpatch P8, i.e., the width scaling rate pdu_sampling_rate_width[tileID4][p3] and the height scaling rate pdu_sampling_rate_height[tileID4][p3] of subpatch P8, the width scaling factor lodX = TilePatchLoDScaleX[tileID4][p3] of subpatch P8 and the height scaling factor lodY = TilePatchLoDScaleY[tileID4][p3] of subpatch P8 can be obtained.

[0244] Further, assuming that the decoding end judges that subpatch P8 needs width and height flipping transformation operation, then and

[0245]

[0246] Based on this, the final local 3D subpatch coordinates (1, u, v) can be obtained by the following formula:

[0247]

[0248] Further, assuming that the decoding end judges that subpatch P8 needs ERP inverse projection operation, at this time the final 3D space coordinates (sampleX, sampleY, sampleZ) can be obtained according to the method in Annex H2.4.1.

[0249] It should be noted that the above examples are only examples of the present application and should not be understood as limiting the present application.

[0250] The preferred embodiments of the present application are described in detail above with reference to the drawings, but the present application is not limited to the specific details in the above-described embodiments. Within the technical concept of the present application, various simple modifications can be made to the technical solutions of the present application, and these simple modifications all belong to the protection scope of the present application. For example, in the above-described specific embodiments, various specific technical features are described, and in the case of no contradiction, any suitable combination can be made. In order to avoid unnecessary repetition, various possible combination manners are not described again in the present application. For another example, various different embodiments of the present application can also be combined arbitrarily, as long as it does not deviate from the idea of the present application, and it should also be considered as the disclosed content of the present application.

[0251] It should also be understood that, in various method embodiments of the present application, the magnitude of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0252] The decoding method according to the embodiments of the present application is described in detail from the decoding end above, and the encoding method according to the embodiments of the present application will be described below from the encoding end. Figure 6 to Figure 8 Figure 9 The encoding method according to the embodiments of the present application is described from the encoding end.

[0253] Figure 9 is a schematic flowchart of the resolution-based encoding method 400 provided by the embodiments of the present application.

[0254] As shown in Figure 9 , the encoding method 400 can include:

[0255] S410, acquiring a current sub-graphic block;

[0256] S420, determining whether the current sub-graphic block is a sub-graphic block that needs to change resolution;

[0257] S430, when the current sub-graphic block is a sub-graphic block that needs to change resolution, writing a scaling rate parameter of the current sub-graphic block into a code stream.

[0258] In some embodiments, the S430 can include:

[0259] writing the scaling rate parameter of the current sub-graphic block and a first identifier into the code stream;

[0260] wherein, when the value of the first identifier is a first numerical value, it indicates that the current sub-graphic block is a sub-graphic block that needs to change resolution, and when the value of the first identifier is a second numerical value, it indicates that the current sub-graphic block is a sub-graphic block that does not need to change resolution.

[0261] In some embodiments, the S420 can include:​

[0262] determining whether the current tile belongs to a current stitching picture including sub-tiles of different resolutions;

[0263] when the current stitching picture includes sub-tiles of different resolutions, determining whether the current tile is a tile requiring resolution change.

[0264] In some embodiments, the S430 can include:

[0265] writing the scaling parameter of the current tile and a second identifier into the bitstream;

[0266] when the second identifier takes a third value, it indicates that the current stitching picture includes sub-tiles of different resolutions, and when the second identifier takes a fourth value, it indicates that the current stitching picture includes sub-tiles of the same resolution.

[0267] In some embodiments, the S420 can include:

[0268] determining whether the current tile belongs to a current view including sub-tiles of different resolutions;

[0269] when the current view includes sub-tiles of different resolutions, determining whether the current tile is a tile requiring resolution change.

[0270] In some embodiments, the S430 can include:

[0271] writing the scaling parameter of the current tile and a third identifier into the bitstream;

[0272] when the third identifier takes a fifth value, it indicates that the current view includes sub-tiles of different resolutions, and when the third identifier takes a sixth value, it indicates that the current view includes sub-tiles of the same resolution.

[0273] In some embodiments, the method 400 can further include:

[0274] when the current tile is a tile not requiring resolution change, writing a default value of the scaling parameter of the current tile into the bitstream.

[0275] In some embodiments, the method 400 can further include:

[0276] determining the scaling parameter of the current tile based on at least one of the following information:

[0277] a size of the current tile, a motion vector of the current tile, a rate-distortion cost of the current tile, a richness of detail information of the current tile, a variance of the current tile.

[0278] In some embodiments, the scaling ratio parameter of the current subpatch is determined as a first parameter when the current subpatch satisfies at least one of the following conditions: a size of the current subpatch is greater than a first threshold, a motion vector of the current subpatch is less than or equal to a second threshold, a rate-distortion cost of the current subpatch is greater than a third threshold, a richness of detail information of the current subpatch is less than or equal to a fourth threshold, a variance of the current subpatch is less than or equal to a fifth threshold; otherwise, the scaling ratio parameter of the current subpatch is determined as a second parameter: wherein the first parameter is greater than the second parameter.

[0279] In some embodiments, when the scaling ratio parameter of the current subpatch comprises a width scaling ratio parameter, the scaling factor of the current subpatch comprises a width scaling factor corresponding to the width scaling ratio parameter; and / or, when the scaling ratio parameter of the current subpatch comprises a height scaling ratio parameter, the scaling factor of the current subpatch comprises a height scaling factor corresponding to the height scaling ratio parameter.

[0280] The scheme of the present application will be described below in conjunction with specific embodiments.

[0281] Embodiment 2:

[0282] In this embodiment, the encoding end can include part or all of the following steps:

[0283] Step 1:

[0284] For all views, mvp_view_wise_enabled_flag[viewID] is written in the bitstream for each view to indicate whether subpatches of different resolutions are included. When mvp_view_wise_enabled_flag[viewID] is true, it indicates that subpatches of different resolutions are included; when mvp_view_wise_enabled_flag[viewID] is false, it indicates that subpatches of different resolutions are not included.

[0285] Step 2:

[0286] After it is determined whether a certain view (view) includes subpatches of different resolutions, all subpatches (patches) of the certain view can be obtained. For each subpatch of the certain view, a corresponding pdu_scaled_flag can be written in the bitstream, which can be used to indicate whether the corresponding subpatch is a subpatch that needs to change resolution.

[0287] When the flag is true, method one is adopted; when the flag is false, method two is adopted.

[0288] Method 1:

[0289] After determining that a certain sub-tile in a certain view is a sub-tile that needs to change resolution, a width scaling parameter and a height scaling parameter are obtained by a certain means, and the width scaling parameter and the height scaling parameter of the certain sub-tile need to be written in the code stream.

[0290] Method 2:

[0291] After determining that a certain sub-tile in a certain view is a sub-tile that does not need to change resolution, the width scaling parameter of the certain sub-tile can be written in the code stream as the default value 1, and the height scaling parameter of the certain sub-tile can be written in the code stream as the default value 1.

[0292] Step 3:

[0293] After the above operations are performed on all sub-tiles in all views, the sub-tiles are spliced to generate atlases, and it is determined for each atlas whether the atlas includes sub-tiles of different resolutions, and asme_scaled_enabled_flag is written in the code stream to indicate whether the corresponding atlas includes sub-tiles of different resolutions. asme_scaled_enabled_flag is yes when the corresponding atlas includes sub-tiles of different resolutions, and asme_scaled_enabled_flag is no when the corresponding atlas does not include sub-tiles of different resolutions.

[0294] Step 4:

[0295] The encoder can determine the scaling parameter of each sub-tile by using at least one of the following methods:

[0296] Method 1:

[0297] The encoder can determine the scaling parameter of a sub-tile by comparing the size of the sub-tile. For example, if the size of the sub-tile is greater than a certain threshold, a larger scaling parameter is selected; if the size of the sub-tile is less than or equal to a certain threshold, a smaller scaling parameter is selected.

[0298] Method 2:

[0299] The encoder can determine the scaling parameter of a sub-tile by comparing the motion vector of the sub-tile. For example, if the motion vector of the sub-tile is less than or equal to a certain threshold, a larger scaling parameter is selected; if the motion vector of the sub-tile is greater than a certain threshold, a smaller scaling parameter is selected.

[0300] Method 3:

[0301] The encoder can determine the scaling ratio parameter of a subpatch by comparing the rate-distortion cost of the subpatch. For example, if the rate-distortion cost of the subpatch is greater than a certain threshold, a larger scaling ratio parameter is selected; if the rate-distortion cost of the subpatch is less than or equal to a certain threshold, a smaller scaling ratio parameter is selected.

[0302] Method 4:

[0303] The encoder can determine the scaling ratio parameter of a subpatch by comparing the detail richness of the subpatch. For example, if the detail richness of the subpatch is small, a larger scaling ratio parameter is selected; if the detail richness of the subpatch is large, a smaller scaling ratio parameter is selected.

[0304] Method 5:

[0305] The encoder can determine the scaling ratio parameter of a subpatch by comparing the variance of the subpatch. For example, if the variance of the subpatch is less than a certain threshold, a larger scaling ratio parameter is selected; if the variance of the subpatch is greater than a certain threshold, a smaller scaling ratio parameter is selected.

[0306] Example 1:

[0307] Suppose the view with view index viewID1 includes subpatches with different resolutions. The encoder obtains subpatch P1 with subpatch index patch1, and subpatch P1 is a subpatch that needs to change resolution; at this time, when the size of subpatch P1 is 64*64, a larger width scaling ratio parameter and a larger height scaling ratio parameter can be selected. The encoder obtains subpatch P2 with subpatch index patch2, and subpatch P2 is a subpatch that needs to change resolution; at this time, when the size of subpatch P2 is 32*32, a smaller width scaling ratio parameter and a smaller height scaling ratio parameter can be selected. The encoder obtains subpatch P3 with subpatch index patch3, and subpatch P3 is a subpatch that does not need to change resolution; at this time, the width scaling ratio parameter and the height scaling ratio parameter of subpatch P3 can both be the default value 1. The encoder obtains subpatch P4 with subpatch index patch4, and subpatch P4 is a subpatch that needs to change resolution; at this time, when the variance of subpatch P4 is large, a smaller width scaling ratio parameter and a smaller height scaling ratio parameter can be selected.

[0308] Example 2:

[0309] Suppose the view with view index viewID2 includes subpatches with different resolutions. The encoder obtains subpatch P1 with subpatch index patch1, and subpatch P1 is a subpatch that needs to change resolution; at this time, when the texture of subpatch P1 is rich, a smaller width scaling ratio parameter and a smaller height scaling ratio parameter are selected. The encoder obtains subpatch P2 with subpatch index patch2, and subpatch P2 is a subpatch that does not need to change resolution; at this time, the width scaling ratio parameter and the height scaling ratio parameter of subpatch P2 can both be the default value 1.

[0310] Example 3:

[0311] Suppose the view with viewID3 includes sub-patches with the same resolution. The encoder obtains the sub-patch P1 with patchID1, and the sub-patch P1 is a sub-patch that does not need to change resolution. At this time, the width scaling parameter and the height scaling parameter of the sub-patch P1 can both be the default value 1. The encoder obtains the sub-patch P2 with patchID2, and the sub-patch P2 is a sub-patch that does not need to change resolution. At this time, the width scaling parameter and the height scaling parameter of the sub-patch P2 can both be the default value 1.

[0312] The method embodiments of the present application are described in detail above, and the device embodiments of the present application are described in detail below. Figure 10 to Figure 12

[0313] Figure 10 is a schematic block diagram of a decoder 500 of an embodiment of the present application.

[0314] As shown in Figure 10 , the decoder 500 can include:

[0315] A first parsing unit 510 is configured to parse a code stream to obtain a current sub-patch.

[0316] A first determining unit 520 is configured to determine whether the current sub-patch is a sub-patch that needs to change resolution.

[0317] A second parsing unit 530 is configured to parse the code stream to obtain a scaling parameter of the current sub-patch when the current sub-patch is a sub-patch that needs to change resolution.

[0318] A second determining unit 540 is configured to determine a scaling factor of the current sub-patch based on the scaling parameter of the current sub-patch.

[0319] An obtaining unit 550 is configured to obtain coordinates of the current sub-patch in a 3D space coordinate system based on the scaling factor of the current sub-patch.

[0320] In some embodiments, the first determining unit 520 is specifically configured to:

[0321] parse the code stream to obtain a value of a first identifier;

[0322] When the value of the first identifier is a first numerical value, it indicates that the current sub-patch is a sub-patch that needs to change resolution, and when the value of the first identifier is a second numerical value, it indicates that the current sub-patch is a sub-patch that does not need to change resolution.

[0323] In some embodiments, the first determining unit 520 is specifically configured to:​

[0324] determining whether the current tile belongs to a current stitching picture including sub-tiles of different resolutions;

[0325] when the current tile belongs to the current stitching picture including sub-tiles of different resolutions, determining whether the current tile is a sub-tile requiring resolution change.

[0326] In some embodiments, the first determining unit 520 is specifically configured to:

[0327] obtaining a value of a second identifier from the bitstream;

[0328] when the value of the second identifier is a third numerical value, it indicates that the current stitching picture includes sub-tiles of different resolutions, and when the value of the second identifier is a fourth numerical value, it indicates that the current stitching picture includes sub-tiles of the same resolution.

[0329] In some embodiments, the first determining unit 520 is specifically configured to:

[0330] determining whether the current tile belongs to a current view point including sub-tiles of different resolutions;

[0331] when the current tile belongs to the current view point including sub-tiles of different resolutions, determining whether the current tile is a sub-tile requiring resolution change.

[0332] In some embodiments, the first determining unit 520 is specifically configured to:

[0333] obtaining a value of a third identifier from the bitstream;

[0334] when the value of the third identifier is a fifth numerical value, it indicates that the current view point includes sub-tiles of different resolutions, and when the value of the third identifier is a sixth numerical value, it indicates that the current view point includes sub-tiles of the same resolution.

[0335] In some embodiments, the second determining unit 540 is specifically configured to:

[0336] determining the scaling rate parameter of the current tile as a scaling factor of the current tile.

[0337] In some embodiments, the second determining unit 540 is further configured to:

[0338] when the current tile is a sub-tile not requiring resolution change, determining the scaling rate parameter of the current tile as a default value.

[0339] In some embodiments, the obtaining unit 550 is specifically configured to:

[0340] obtaining a first coordinate of the current tile in a 2D stitching picture coordinate system;

[0341] obtaining second coordinates of the current sub-tile in a 3D sub-tile coordinate system based on the first coordinates and the scaling factor;

[0342] back-projecting the second coordinates to the 3D space coordinate system to obtain coordinates of the current sub-tile in the 3D space coordinate system.

[0343] In some embodiments, the obtaining unit 550 is specifically configured to:

[0344] obtaining the second coordinates by projecting the current sub-tile to the 3D sub-tile coordinate system based on the first coordinates, the scaling factor and a transformation parameter;

[0345] The transformation parameter includes a transformation process for representing a swapping operation and / or a transformation process for representing a flipping operation.

[0346] In some embodiments, when the scaling ratio parameter of the current sub-tile includes a width scaling ratio parameter, the scaling factor of the current sub-tile includes a width scaling factor corresponding to the width scaling ratio parameter; and / or, when the scaling ratio parameter of the current sub-tile includes a height scaling ratio parameter, the scaling factor of the current sub-tile includes a height scaling factor corresponding to the height scaling ratio parameter.

[0347] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. Specifically, Figure 10 The decoder 500 shown can correspond to the subject performing the method 300 of the embodiments of the present application, i.e., the foregoing and other operations and / or functions of each unit in the decoder 500 are respectively for realizing the corresponding processes in each method of the method 300, and for the sake of avoiding repetition, they will not be described here.

[0348] Figure 11 is a schematic block diagram of an encoder 600 of the embodiments of the present application.

[0349] As shown in Figure 11 The encoder 600 can include:

[0350] An obtaining unit 610 is configured to obtain a current sub-tile.

[0351] A determining unit 620 is configured to determine whether the current sub-tile is a sub-tile that needs to change resolution.

[0352] An encoding unit 630 is configured to, when the current sub-tile is a sub-tile that needs to change resolution, write a scaling ratio parameter of the current sub-tile into a bitstream.

[0353] In some embodiments, the encoding unit 630 is specifically configured to:

[0354] write the scaling ratio parameter of the current sub-tile and a first identifier into the bitstream;

[0355] wherein the first identifier has a first value to indicate that the current sub-tile is a sub-tile that needs to change resolution, and the first identifier has a second value to indicate that the current sub-tile is a sub-tile that does not need to change resolution.

[0356] In some embodiments, the determining unit 620 is specifically configured to:

[0357] determine whether the current stitching picture in which the current sub-tile is located includes sub-tiles of different resolutions;

[0358] when the current stitching picture includes sub-tiles of different resolutions, determine whether the current sub-tile is a sub-tile that needs to change resolution.

[0359] In some embodiments, the encoding unit 630 is specifically configured to:

[0360] write the scaling ratio parameter of the current sub-tile and a second identifier into the bitstream;

[0361] the second identifier has a third value to indicate that the current stitching picture includes sub-tiles of different resolutions, and the second identifier has a fourth value to indicate that the current stitching picture includes sub-tiles of the same resolution.

[0362] In some embodiments, the determining unit 620 is specifically configured to:

[0363] determine whether the current view in which the current sub-tile is located includes sub-tiles of different resolutions;

[0364] when the current view includes sub-tiles of different resolutions, determine whether the current sub-tile is a sub-tile that needs to change resolution.

[0365] In some embodiments, the encoding unit 630 is specifically configured to:

[0366] write the scaling ratio parameter of the current sub-tile and a third identifier into the bitstream;

[0367] the third identifier has a fifth value to indicate that the current view includes sub-tiles of different resolutions, and the third identifier has a sixth value to indicate that the current view includes sub-tiles of the same resolution.

[0368] In some embodiments, the encoding unit 630 is further configured to:

[0369] When the current sub-block is a sub-block that does not need to change resolution, a default value of the scaling ratio parameter of the current sub-block is written into the bitstream.

[0370] In some embodiments, the determining unit 620 is further configured to:

[0371] determine the scaling ratio parameter of the current sub-block based on at least one of the following information:

[0372] a size of the current sub-block, a motion vector of the current sub-block, a rate-distortion cost of the current sub-block, a richness of detail information of the current sub-block, a variance of the current sub-block.

[0373] In some embodiments, the determining unit 620 is specifically configured to:

[0374] determine the scaling ratio parameter of the current sub-block as a first parameter when the current sub-block satisfies at least one of the following conditions:

[0375] the size of the current sub-block is greater than a first threshold, the motion vector of the current sub-block is less than or equal to a second threshold, the rate-distortion cost of the current sub-block is greater than a third threshold, the richness of detail information of the current sub-block is less than or equal to a fourth threshold, and the variance of the current sub-block is less than or equal to a fifth threshold.

[0376] Otherwise, determine the scaling ratio parameter of the current sub-block as a second parameter: wherein the first parameter is greater than the second parameter.

[0377] In some embodiments, when the scaling ratio parameter of the current sub-block includes a width scaling ratio parameter, the scaling factor of the current sub-block includes a width scaling factor corresponding to the width scaling ratio parameter; and / or, when the scaling ratio parameter of the current sub-block includes a height scaling ratio parameter, the scaling factor of the current sub-block includes a height scaling factor corresponding to the height scaling ratio parameter.

[0378] It should be understood that the apparatus embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. Specifically, Figure 11 The illustrated encoder 600 can correspond to the subject performing the method 400 of the embodiments of the present application, i.e., the foregoing and other operations and / or functions of the various units in the encoder 600 are respectively for implementing the corresponding processes in the various methods, such as the method 400, to avoid repetition, which will not be described here.

[0379] It should also be understood that each unit in the decoder 500 or the encoder 600 involved in the embodiments of the present application can be combined into one or several other units respectively or entirely, or some of the units can be further split into multiple units with smaller functions to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above units are divided based on logical functions, and in actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the decoder 500 or the encoder 600 can also include other units, and in actual applications, these functions can also be assisted by other units, and can be implemented by multiple units in cooperation. According to another embodiment of the present application, the decoder 500 or the encoder 600 involved in the embodiments of the present application and the encoding method or the decoding method of the embodiments of the present application can be constructed by running a computer program (including program codes) capable of performing each step involved in the corresponding method on a general computing device including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), etc. The computer program can be recorded on, for example, a computer readable storage medium, and loaded into an electronic device through the computer readable storage medium, and run therein to implement the corresponding method of the embodiments of the present application.

[0380] In other words, the units involved above can be implemented in the form of hardware, in the form of instructions of software, or in the form of combination of software and hardware. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits and / or software instructions in the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware decoding processors for execution, or executed by hardware and software combination in the decoding processor. Alternatively, the software can be located in mature storage media in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps in the above method embodiments.

[0381] Figure 12 is a schematic structural diagram of the electronic device 700 provided by the embodiments of the present application.

[0382] As Figure 12As shown, the electronic device 700 at least includes a processor 710, a computer readable storage medium 720, and a transceiver 730. Among them, the processor 710 and the computer readable storage medium 720 can be connected through a bus or other means. The computer readable storage medium 720 is used to store a computer program 721, the computer program 721 includes computer instructions, and the processor 710 is used to execute the computer instructions stored in the computer readable storage medium 720. The processor 710 is the computing core and control core of the electronic device 700, which is suitable for implementing one or more computer instructions, and is particularly suitable for loading and executing one or more computer instructions to implement a corresponding method flow or a corresponding function.

[0383] As an example, the processor 710 can also be referred to as a central processing unit (CPU). The processor 710 can include but is not limited to: a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0384] By way of example, computer-readable storage media 720 can be RAM. Additionally or alternatively, computer-readable storage media 720 can be non-volatile memory, such as at least one disk storage device; optionally, computer-readable storage media 720 can reside at a computing site that is remote from the aforementioned processor(s) 710. Specifically, computer-readable storage media 720 includes, but is not limited to, volatile and / or nonvolatile memory. Nonvolatile memory can be, for example, read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. Volatile memory can be, for example, random access memory (RAM), which acts as external cache. By way of example, and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DR RAM). By way of further example, computer-readable storage media 720 can also include, but is not limited to, optical storage such as CD-ROM, Blu-ray discs, DVDs, and the like, solid state memory such as solid state drives (SSDs) and the like, and / or a suitable combination of the foregoing. Note that the

[0385] By way of example, the electronic device 700 can be an encoding end, an encoder, or an encoding framework related to embodiments of the present application. The computer-readable storage medium 720 stores a first computer instruction. The processor 710 loads and executes the first computer instruction stored in the computer-readable storage medium 720 to implement corresponding steps in the encoding method or the decoding method provided by embodiments of the present application. In other words, the first computer instruction in the computer-readable storage medium 720 is loaded and executed by the processor 710 to implement corresponding steps. To avoid repetition, details are not described here.

[0386] Exemplarily, the electronic device 700 can be a decoding end, a decoder or a decoding framework related to the embodiments of the present application; the second computer instructions are stored in the computer readable storage medium 720; the second computer instructions stored in the computer readable storage medium 720 are loaded and executed by the processor 710 to implement the corresponding steps in the encoding method or the decoding method provided by the embodiments of the present application; in other words, the second computer instructions in the computer readable storage medium 720 are loaded and executed by the processor 710 to implement the corresponding steps, and to avoid repetition, details are not described herein.

[0387] According to another aspect of the present application, the embodiments of the present application further provide a computer readable storage medium (Memory), which is a memory device in the electronic device 700, and is used to store programs and data. For example, the computer readable storage medium 720. It can be understood that the computer readable storage medium 720 herein can include a built-in storage medium in the electronic device 700, and of course can include an expansion storage medium supported by the electronic device 700. The computer readable storage medium provides a storage space, and the storage space stores an operating system of the electronic device 700. In addition, one or more computer instructions suitable for being loaded and executed by the processor 710 are stored in the storage space, and the computer instructions can be one or more computer programs 721 (including program codes).

[0388] According to another aspect of the present application, a computer program product or a computer program is provided, and the computer program product or the computer program includes computer instructions stored in a computer readable storage medium. For example, the computer program 721. At this time, the electronic device 700 can be a computer, the processor 710 reads the computer instructions from the computer readable storage medium 720, and the processor 710 executes the computer instructions to enable the computer to perform the encoding method or the decoding method provided in the various optional manners.

[0389] In other words, when implemented using software, the functions can be implemented in whole or in part using one or more computer programs that include one or more sets of instructions executable by a computer. When the computer program is loaded into the computer and executed, it can run the flowcharts or functions according to the embodiments of the present application in whole or in part, or enable the computer to run the flowcharts or functions according to the embodiments of the present application in whole or in part. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatuses. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner.

[0390] Those skilled in the art can realize that the units and flow steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, or in combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0391] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A resolution-based decoding method, characterized in that, include: Parse the bitstream to obtain the current sub-plot; Determine if the current sub-tile is one that needs its resolution changed; When the current sub-tile is a sub-tile whose resolution needs to be changed, the scaling parameter of the current sub-tile is obtained by parsing the bitstream; Based on the scaling parameter of the current sub-tile, determine the scaling factor of the current sub-tile; Based on the scaling factor of the current sub-tile, the coordinates of the current sub-tile in the 3D spatial coordinate system are obtained; The step of determining whether the current sub-tile is a sub-tile whose resolution needs to be changed includes: Determine whether the current mosaic image containing the current sub-tile includes sub-tiles of different resolutions; When the current mosaic image includes sub-tiles with different resolutions, determine whether the current sub-tile is a sub-tile whose resolution needs to be changed.

2. The method according to claim 1, characterized in that, Determining whether the current sub-tile is one that needs its resolution changed includes: The value of the first identifier is obtained by parsing the bitstream; Wherein, when the value of the first identifier is a first value, it indicates that the current sub-plot is a sub-plot that needs to have its resolution changed, and when the value of the first identifier is a second value, it indicates that the current sub-plot is a sub-plot that does not need to have its resolution changed.

3. The method according to claim 1, characterized in that, Determining whether the current mosaic image containing the current sub-tile includes sub-tiles of different resolutions includes: The value of the second identifier is obtained by parsing the bitstream; When the value of the second identifier is the third value, it indicates that the current mosaic image includes sub-patterns of different resolutions; when the value of the second identifier is the fourth value, it indicates that the current mosaic image includes sub-patterns of the same resolution.

4. The method according to any one of claims 1 to 3, characterized in that, Determining whether the current sub-tile is one that needs its resolution changed includes: Determine whether the current viewpoint containing the current sub-tile includes sub-tiles of different resolutions; When the current viewpoint includes sub-tiles of different resolutions, determine whether the current sub-tile is a sub-tile whose resolution needs to be changed.

5. The method according to claim 4, characterized in that, Determining whether the current sub-tile is a sub-tile whose resolution needs to be changed includes: The value of the third identifier is obtained by parsing the bitstream; When the value of the third identifier is the fifth value, it indicates that the current viewpoint includes sub-tiles of different resolutions; when the value of the third identifier is the sixth value, it indicates that the current viewpoint includes sub-tiles of the same resolution.

6. The method according to any one of claims 1 to 3, characterized in that, Determining the scaling factor of the current sub-tile based on its scaling parameter includes: The scaling parameter of the current sub-tile is determined as the scaling factor of the current sub-tile.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: When the current sub-tile is a sub-tile whose resolution does not need to be changed, the scaling parameter of the current sub-tile is determined to be the default value.

8. The method according to any one of claims 1 to 3, characterized in that, The process of obtaining the coordinates of the current sub-tile in the 3D spatial coordinate system based on the scaling factor of the current sub-tile includes: Obtain the first coordinates of the current sub-plot in the 2D mosaic coordinate system; Based on the first coordinate and the scaling factor, the second coordinate of the current sub-tile in the 3D sub-tile coordinate system is obtained; The second coordinate is back-projected onto the 3D spatial coordinate system to obtain the coordinates of the current sub-block in the 3D spatial coordinate system.

9. The method according to claim 8, characterized in that, The step of obtaining the second coordinates of the current sub-tile in the 3D sub-tile coordinate system based on the first coordinates and the scaling factor includes: Based on the first coordinates, the scaling factor, and the transformation parameters, the current sub-tile is projected onto the 3D sub-tile coordinate system to obtain the second coordinates; The transformation parameters include transformation processes used to characterize swap operations and / or flip operations.

10. The method according to any one of claims 1 to 3, characterized in that, When the scaling parameter of the current sub-plot includes a width scaling parameter, the scaling factor of the current sub-plot includes the width scaling factor corresponding to the width scaling parameter. And / or, when the scaling parameter of the current sub-tile includes a height scaling parameter, the scaling factor of the current sub-tile includes the height scaling factor corresponding to the height scaling parameter.

11. A resolution-based encoding method, characterized in that, include: Get the current sub-tile; Determine if the current sub-tile is one that needs its resolution changed; When the current sub-tile is a sub-tile whose resolution needs to be changed, the scaling parameter of the current sub-tile is written into the bitstream; The step of determining whether the current sub-tile is a sub-tile whose resolution needs to be changed includes: Determine whether the current mosaic image containing the current sub-tile includes sub-tiles of different resolutions; When the current mosaic image includes sub-tiles with different resolutions, determine whether the current sub-tile is a sub-tile whose resolution needs to be changed.

12. The method according to claim 11, characterized in that, The step of writing the scaling parameter of the current sub-tile into the bitstream includes: Write the scaling parameter and the first identifier of the current sub-plot into the bitstream; Wherein, when the value of the first identifier is a first value, it indicates that the current sub-plot is a sub-plot that needs to have its resolution changed, and when the value of the first identifier is a second value, it indicates that the current sub-plot is a sub-plot that does not need to have its resolution changed.

13. The method according to claim 11, characterized in that, The step of writing the scaling parameter of the current sub-tile into the bitstream includes: Write the scaling parameter and second identifier of the current sub-plot into the bitstream; When the value of the second identifier is the third value, it indicates that the current mosaic image includes sub-patterns of different resolutions; when the value of the second identifier is the fourth value, it indicates that the current mosaic image includes sub-patterns of the same resolution.

14. The method according to any one of claims 11 to 13, characterized in that, Determining whether the current sub-tile is one that needs its resolution changed includes: Determine whether the current viewpoint containing the current sub-tile includes sub-tiles of different resolutions; When the current viewpoint includes sub-tiles of different resolutions, determine whether the current sub-tile is a sub-tile whose resolution needs to be changed.

15. The method according to claim 14, characterized in that, The step of writing the scaling parameter of the current sub-tile into the bitstream includes: Write the scaling parameter and third identifier of the current sub-plot into the bitstream; When the value of the third identifier is the fifth value, it indicates that the current viewpoint includes sub-tiles of different resolutions; when the value of the third identifier is the sixth value, it indicates that the current viewpoint includes sub-tiles of the same resolution.

16. The method according to any one of claims 11 to 13, characterized in that, The method further includes: When the current sub-tile is a sub-tile whose resolution does not need to be changed, the default value of the scaling parameter of the current sub-tile is written into the bitstream.

17. The method according to any one of claims 11 to 13, characterized in that, The method further includes: The scaling parameter of the current sub-tile is determined based on at least one of the following information: The size of the current sub-plot, the motion vector of the current sub-plot, the rate-distortion cost of the current sub-plot, the richness of the detail information of the current sub-plot, and the variance of the current sub-plot.

18. The method according to claim 17, characterized in that, The scaling parameter of the current sub-tile is determined based on at least one of the following information: The scaling parameter of the current sub-tile is determined to be the first parameter when the current sub-tile satisfies at least one of the following conditions: The current sub-plot size is greater than a first threshold, the motion vector of the current sub-plot is less than or equal to a second threshold, the rate-distortion cost of the current sub-plot is greater than a third threshold, the richness of the detail information of the current sub-plot is less than or equal to a fourth threshold, and the variance of the current sub-plot is less than or equal to a fifth threshold. Otherwise, the scaling parameter of the current sub-plot is determined to be the second parameter: wherein the first parameter is greater than the second parameter.

19. The method according to any one of claims 11 to 13, characterized in that, When the scaling parameter of the current sub-plot includes a width scaling parameter, the scaling factor of the current sub-plot includes the width scaling factor corresponding to the width scaling parameter. And / or, when the scaling parameter of the current sub-tile includes a height scaling parameter, the scaling factor of the current sub-tile includes the height scaling factor corresponding to the height scaling parameter.

20. A decoder, characterized in that, include: The first parsing unit is used to parse the bitstream to obtain the current sub-block; The first determining unit is used to determine whether the current sub-tile is a sub-tile whose resolution needs to be changed; The second parsing unit is used to parse the bitstream to obtain the scaling parameter of the current sub-plot when the current sub-plot is a sub-plot that needs to change its resolution. The second determining unit is used to determine the scaling factor of the current sub-plot based on the scaling rate parameter of the current sub-plot; The acquisition unit is used to obtain the coordinates of the current sub-tile in the 3D spatial coordinate system based on the scaling factor of the current sub-tile; Specifically, the first determining unit is used for: Determine whether the current mosaic image containing the current sub-tile includes sub-tiles of different resolutions; When the current mosaic image includes sub-tiles with different resolutions, determine whether the current sub-tile is a sub-tile whose resolution needs to be changed.

21. An encoder, characterized in that, include: The acquisition unit is used to acquire the current sub-plot. The determining unit is used to determine whether the current sub-tile is a sub-tile whose resolution needs to be changed; The encoding unit is used to write the scaling parameter of the current sub-tile into the bitstream when the current sub-tile is a sub-tile whose resolution needs to be changed; Specifically, the determining unit is used for: Determine whether the current mosaic image containing the current sub-tile includes sub-tiles of different resolutions; When the current mosaic image includes sub-tiles with different resolutions, determine whether the current sub-tile is a sub-tile whose resolution needs to be changed.

22. An electronic device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the method as claimed in any one of claims 1 to 10 or the method as claimed in any one of claims 11 to 19.

23. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as claimed in any one of claims 1 to 10 or the method as claimed in any one of claims 11 to 19.

24. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1 to 10 or the method as described in any one of claims 11 to 19.

Images