Free viewpoint image encoding and decoding method and device based on deep learning

Abstract

The application relates to the field of image coding technology, and provides a free-viewpoint image coding method and device based on deep learning, which are used for improving the quality of image coding. The key viewpoint image and at least one reference viewpoint image in each viewpoint group divided by the free-viewpoint image sequence are respectively coded at the encoding end, which is beneficial to better learning the spatial correlation of images among different viewpoints, and when residual coding is performed on the reference viewpoint image, the depth image of at least one of the key viewpoint image of the viewpoint group, the key viewpoint image of the next viewpoint group and the previous reference viewpoint image and the corresponding viewpoint image is used for image prediction, which can effectively reduce the spatial redundancy among different viewpoints, improve the image coding quality, reduce the data amount of image coding, and improve the end-to-end transmission speed; meanwhile, the decoding end performs image prediction and reconstruction on the decoding results of different image code streams, so that high-quality decoded images are obtained.

Description

Technical Field

[0001] This application relates to the field of image encoding and decoding technology, and provides a free-viewpoint image encoding and decoding method and device based on deep learning. Background Technology

[0002] Free-viewpoint images are captured by a 6-DOF (Degree of Freedom, DoF) camera system, showing images of the same scene from multiple angles, providing users with an immersive visual experience. However, limitations in device storage and network transmission speed restrict the application of free-viewpoint images. Therefore, compressing and transmitting free-viewpoint images through encoding and decoding is of significant research importance for promoting their applications.

[0003] Typically, 6-DoF camera acquisition systems include cameras with multiple viewpoints. Compared to single-viewpoint image compression, free-viewpoint image compression focuses more on eliminating spatial redundancy between viewpoints to save bitrate. However, since free-viewpoint images are acquired through a 6-DoF camera system, each viewpoint undergoes translation and rotation transformations in three dimensions. Therefore, the changes between viewpoints are very flexible, resulting in significant parallax in free-viewpoint images. This increases the difficulty of utilizing the spatial correlation between different viewpoint images, making free-viewpoint images difficult to compress. Summary of the Invention

[0004] This application provides a deep learning-based free-viewpoint image encoding and decoding method and device to reduce spatial redundancy during free-viewpoint image compression.

[0005] On one hand, embodiments of this application provide a deep learning-based free-viewpoint image encoding / decoding method, applied to encoding, including:

[0006] The free viewpoint image sequence is segmented into multiple viewpoint groups and input into a trained encoding and decoding compression network model. Each viewpoint group contains a key viewpoint image and at least one reference viewpoint image, and each viewpoint image corresponds to a depth image.

[0007] The key viewpoint encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the key viewpoint images in each viewpoint group to obtain the first image bitstream.

[0008] The depth encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the depth image corresponding to each viewpoint image to obtain the second image bitstream;

[0009] Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, for the reference viewpoint image in each viewpoint group, image prediction is performed based on at least one of the key viewpoint image in the same viewpoint group, the key viewpoint image in the next viewpoint group, and the previous reference viewpoint image, along with the corresponding depth image, to obtain a base image. Combined with the reference viewpoint image, a viewpoint residual image is obtained, and the viewpoint residual image is encoded to obtain a third image bitstream.

[0010] The first, second, and third image streams corresponding to each viewpoint group are transmitted to the decoding end for decoding.

[0011] On the other hand, embodiments of this application provide a deep learning-based free-viewpoint image encoding / decoding method, applied at the decoding end, including:

[0012] The system receives the first image stream, second image stream, and third image stream of each of the multiple viewpoint groups corresponding to the free viewpoint image sequence sent by the encoding end, and inputs them into the trained encoding and decoding compression network model. The first image stream is the encoding result of the key viewpoint image in each viewpoint group, the second image stream is the encoding result of the depth image corresponding to each viewpoint image in each viewpoint group, and the third image stream is the encoding result of the reference viewpoint image in each viewpoint group.

[0013] The key viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to independently decode the first image bitstream of each viewpoint group to obtain the key viewpoint image within the corresponding viewpoint group.

[0014] The depth codec module in the aforementioned codec compression network model is used to independently decode the second image stream of each viewpoint group to obtain the depth image of each viewpoint image within the corresponding viewpoint group.

[0015] The reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to decode the third image stream of each viewpoint group to obtain the residual image of each reference viewpoint image within the corresponding viewpoint group. Based on at least one of the key viewpoint image of the viewpoint group where the decoded residual image is located, the key viewpoint image of the next viewpoint group, and the previous reference viewpoint image, as well as the corresponding depth image, image prediction is performed to obtain a base image. The base image and the residual image are then fused to obtain the corresponding reference viewpoint image.

[0016] On the other hand, embodiments of this application provide a first terminal device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0017] The communication interface is used to communicate with the second terminal device;

[0018] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0019] The free viewpoint image sequence is segmented into multiple viewpoint groups and input into a trained encoding and decoding compression network model. Each viewpoint group contains a key viewpoint image and at least one reference viewpoint image, and each viewpoint image corresponds to a depth image.

[0020] The key viewpoint encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the key viewpoint images in each viewpoint group to obtain the first image bitstream.

[0021] The depth encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the depth image corresponding to each viewpoint image to obtain the second image bitstream;

[0022] Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, for the reference viewpoint image in each viewpoint group, image prediction is performed based on the key viewpoint images and corresponding depth images in the same viewpoint group and the next viewpoint group, as well as the previous viewpoint image and corresponding depth image of the reference viewpoint image, to obtain a base image. Combined with the reference viewpoint image, a viewpoint residual image is obtained, and the viewpoint residual image is encoded to obtain a third image bitstream.

[0023] The first image stream, the second image stream, and the third image stream corresponding to each viewpoint group are transmitted to the second terminal device for decoding.

[0024] On the other hand, embodiments of this application provide a second terminal device, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0025] The communication interface is used to communicate with the first terminal device;

[0026] The memory stores computer program instructions, and the processor performs the following operations according to the computer program instructions:

[0027] The system receives the first image stream, second image stream, and third image stream of each of the multiple viewpoint groups corresponding to the free viewpoint image sequence sent by the first terminal device, and inputs them into the trained encoding and decoding compression network model; wherein, the first image stream is the encoding result of the key viewpoint image in each viewpoint group, the second image stream is the encoding result of the depth image corresponding to each viewpoint image in each viewpoint group, and the third image stream is the encoding result of the reference viewpoint image in each viewpoint group;

[0028] The key viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to independently decode the first image bitstream of each viewpoint group to obtain the key viewpoint image within the corresponding viewpoint group.

[0029] The depth codec module in the aforementioned codec compression network model is used to independently decode the second image stream of each viewpoint group to obtain the depth image of each viewpoint image within the corresponding viewpoint group.

[0030] Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, the third image stream of each viewpoint group is decoded to obtain the residual image of each reference viewpoint image within the corresponding viewpoint group. Based on the key viewpoint image and corresponding depth image in the viewpoint group and next viewpoint group where the decoded residual image is located, as well as the previous viewpoint image and corresponding depth image of the reference viewpoint image corresponding to the residual image, image prediction is performed to obtain a base image. The base image and the residual image are then fused to obtain the corresponding reference viewpoint image.

[0031] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a deep learning-based free-viewpoint image compression method provided in embodiments of this application.

[0032] The beneficial effects of the deep learning-based free-viewpoint image encoding and decoding method and device provided in this application are as follows:

[0033] At the encoding end, since each viewpoint group after the free viewpoint image sequence is segmented contains one key viewpoint image and at least one reference viewpoint image, the key viewpoint image and reference viewpoint image in each viewpoint group are encoded and transmitted separately, thereby better exploring the spatial correlation between images of different viewpoints. Furthermore, through the division of viewpoint images, each reference viewpoint image is associated with at least one of the key viewpoint image of its own viewpoint group, the key viewpoint image of the next viewpoint group, and the previous reference viewpoint image, and each viewpoint image corresponds to a depth image. In this way, image prediction can be performed using its associated viewpoint image and the corresponding depth image, and a residual image can be obtained by combining it with the original reference viewpoint image. The residual image is then encoded and transmitted. Since prediction is performed using multiple viewpoint images, spatial redundancy between different viewpoints can be effectively reduced, image encoding quality can be improved, the amount of image encoding data can be reduced, and the end-to-end transmission speed can be increased.

[0034] At the decoding end, after receiving the first image stream, third image stream, and second image stream of the key viewpoint image, each reference viewpoint image, and their corresponding depth images from each viewpoint group, the first and third image streams are independently decoded to obtain the key viewpoint image and the depth image of each viewpoint image. Thus, for the current reference viewpoint image to be decoded, prediction can be performed by acquiring at least one associated viewpoint image and its corresponding depth image. Combined with the residual image decoded from the third image stream, the current reference viewpoint image is obtained. Since prediction is performed using multiple decoded viewpoint images, spatial redundancy between different viewpoints is effectively reduced, improving image prediction quality. Furthermore, after fusion with the residual image, the quality of reference viewpoint image decoding is effectively enhanced.

[0035] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1A An architecture diagram of an end-to-end encoding / decoding compression network model provided in an embodiment of this application;

[0038] Figure 1B This is a structural diagram of the encoder and decoder provided in the embodiments of this application;

[0039] Figure 2 A flowchart of a deep learning-based free-viewpoint image encoding and decoding method provided in an embodiment of this application;

[0040] Figure 3 A flowchart of the viewpoint image segmentation method provided in the embodiments of this application;

[0041] Figure 4 Flowchart of the encoding method for key viewpoint images provided in the embodiments of this application;

[0042] Figure 5 A flowchart of a depth image encoding method provided in an embodiment of this application;

[0043] Figure 6AA schematic diagram illustrating the prediction process of a reference viewpoint image provided in an embodiment of this application;

[0044] Figure 6B A schematic diagram illustrating the prediction process of another reference viewpoint image provided in an embodiment of this application;

[0045] Figure 7 Flowchart of the encoding method for reference viewpoint images provided in embodiments of this application;

[0046] Figure 8 This is a schematic diagram of 3D projection relationships provided for embodiments of this application;

[0047] Figure 9A A flowchart of the hole-filling method provided in the embodiments of this application;

[0048] Figure 9B The hole-filling network structure diagram provided in the embodiments of this application;

[0049] Figure 10 A flowchart of a residual coding method for a reference viewpoint image provided in an embodiment of this application;

[0050] Figure 11 A flowchart of a deep learning-based free-viewpoint image encoding / decoding method for the decoding end provided in this application embodiment;

[0051] Figure 12 A flowchart illustrating the decoding method for key viewpoint images provided in embodiments of this application;

[0052] Figure 13 A flowchart of a depth image decoding method provided in an embodiment of this application;

[0053] Figure 14 A flowchart of a residual decoding method for a reference viewpoint image provided in an embodiment of this application;

[0054] Figure 15 This is a structural diagram of the residual fusion network provided in the embodiments of this application;

[0055] Figure 16 A flowchart of a reference viewpoint image reconstruction method provided in an embodiment of this application;

[0056] Figure 17 A comparison chart of PSNR indices for various models provided in the embodiments of this application;

[0057] Figure 18 A comparison chart of the image reconstruction quality of each model is provided for the embodiments of this application;

[0058] Figure 19 A comparison chart of the prediction quality of various models provided in the embodiments of this application;

[0059] Figure 20 A structural diagram of a first terminal device provided in an embodiment of this application;

[0060] Figure 21 This is a structural diagram of a first terminal device provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0062] Compression of free-viewpoint images differs from compression of binocular and light field images. It is more universal because binocular images consist of left and right eye viewpoint images, and the parallax between the two viewpoint images is mainly caused by the horizontal displacement between the left and right eyes, resulting in strong horizontal spatial correlation. Light field images, on the other hand, are composed of two-dimensional multi-viewpoint image arrays acquired by light field cameras or multi-camera systems. The parallax between viewpoint images is mainly caused by the different positions of the camera lens array, resulting in parallax in both horizontal and vertical spatial directions. Free-viewpoint images, however, are acquired through a 6-DOF camera system, where each viewpoint undergoes translation and rotation transformations in three dimensions, placing fewer restrictions on the viewpoint camera position.

[0063] Compression of free-viewpoint images mainly focuses on how to utilize the spatial correlation between viewpoints. However, since each viewpoint in a free-viewpoint image undergoes translation and rotation transformations in three dimensions, the changes between viewpoints are very flexible, resulting in significant parallax. This increases the difficulty of utilizing the spatial correlation between images from different viewpoints, making free-viewpoint images difficult to compress.

[0064] In view of this, embodiments of this application provide a free-viewpoint image compression method based on deep learning. This method divides a free-viewpoint image sequence into multiple viewpoint groups, each containing a key viewpoint image and at least one reference viewpoint image, with each viewpoint image corresponding to a depth image. Thus, for each type of image, an end-to-end encoding / decoding compression network model is built based on deep learning to encode and decode the images, thereby better exploring the spatial correlation between images from different viewpoints and improving the compression quality of free-viewpoint images. Simultaneously, when performing residual encoding and decoding on the reference viewpoint image, at least one of the key viewpoint image in its viewpoint group, the key viewpoint image in the next viewpoint group, and the previous reference viewpoint image, along with the corresponding depth image, is used for image prediction. This effectively eliminates spatial redundancy between different viewpoints, reduces the amount of image data, and enhances the image holes generated during prediction, further improving the quality of free-viewpoint image compression.

[0065] In the embodiments of this application, an end-to-end encoding / decoding compression network model is constructed using a deep learning network for both the encoding and decoding ends. This model is then trained using a constructed free-viewpoint image sample pool. The free-viewpoint image sample pool contains multiple key viewpoint images and at least one reference viewpoint image corresponding to each key viewpoint image. Each viewpoint image (including the key viewpoint image and the reference viewpoint image) corresponds to a depth image. This free-viewpoint image sample pool allows for multiple iterations of the encoding / decoding compression network model.

[0066] like Figure 1A The diagram shown is an architecture diagram of the end-to-end encoding / decoding compression network model provided in this application embodiment, mainly including a key viewpoint encoding / decoding module, a depth encoding / decoding module, and a reference viewpoint encoding / decoding module. Wherein:

[0067] The key viewpoint encoding and decoding module is divided into a main branch and an entropy branch. In the main branch, the key viewpoint image is encoded by an encoder, a quantizer, and an entropy encoder to obtain an image bitstream. Then, the image bitstream is decoded and reconstructed by an entropy decoder and a decoder. In the entropy branch, the super-prior network and the context network work together to estimate the parameters in the entropy encoder and entropy decoder using the entropy model.

[0068] The network structure of the depth encoding / decoding module is similar to that of the key viewpoint encoding module. However, since the depth image is a single-channel image, its network channel number is set to be smaller and the network layer number is set to be shallower to save bitrate compared to the key viewpoint encoding module.

[0069] The reference viewpoint encoding / decoding module adds a residual image generator (represented by "-") to the main branches of the key viewpoint encoding / decoding module and the depth encoding / decoding module. Simultaneously, a multi-viewpoint prediction branch is added. In this branch, the depth prediction unit performs image prediction based on multiple viewpoint images (including at least one of the key viewpoint image and the reference viewpoint image) and their corresponding depth images, combined with pre-calibrated intrinsic and extrinsic parameters of the viewpoint camera. This effectively eliminates spatial redundancy between different viewpoints and improves coding quality. Considering that image prediction may produce holes due to occlusion between viewpoints, after obtaining multiple prediction images, a multi-viewpoint reference strategy and a filtering unit are combined to fill the pixel holes generated by the prediction, achieving image enhancement. A residual image is generated by subtracting the current reference viewpoint image from the enhanced image, and this residual image is then compressed and encoded, reducing the image bitrate.

[0070] In the reference viewpoint encoding and decoding module, a residual fusion network is used to fuse and reconstruct the residual image and the prediction result when decoding the residual image. Compared with the traditional reconstruction method of superimposing the residual image and the prediction result, this method can improve the decoding quality of the image.

[0071] Optionally, in one example, the speed at which the entropy model estimates parameters can be improved by removing the context network from the deep codec module.

[0072] exist Figure 1A In this architecture, the encoder in each module performs feature extraction and downsampling on the input image; the quantizer performs integer quantization on the extracted features; the entropy encoder further encodes the quantized features (e.g., arithmetic encoding); and the entropy decoder upsamples and reconstructs the image bitstream. Both the super-prior network and the context network are composed of convolutions, predicting super-prior and autoregressive prior information, respectively. The entropy model uses prior information to predict the probabilistic model parameters (e.g., mean, variance) of the entropy encoder and decoder. Optionally, the entropy model can be a Gaussian model.

[0073] In the embodiments of this application, in order to achieve fast image encoding and decoding, most of the encoding and decoding modules in the encoding and decoding compression network model are built based on simple residual blocks and convolutional layers.

[0074] like Figure 1B The diagram shown is a network structure diagram of the encoder and decoder provided in the embodiments of this application. The network structures of the encoder and decoder in each module are basically the same, the difference lies in the different network parameter settings. Specifically, the reference viewpoint encoding and decoding module needs to normalize the residual image so that its distribution pattern is close to that of the key viewpoint image. Therefore, the network parameters set are generally larger than those of the reference viewpoint encoding and decoding module. On the other hand, the depth encoding and decoding module encodes and decodes a single-channel image, so the network parameters set are smaller.

[0075] For example, the number of channels in the key viewpoint codec module is set to 128, the number of channels in the reference viewpoint codec module is set to 192, and the number of channels in the depth codec module is set to 64.

[0076] For example, the number of residual blocks and downsampling layers in the encoder and decoder of the depth codec module is reduced by half compared to the number of residual blocks and downsampling layers in the encoder and decoder of the key viewpoint codec module.

[0077] In the embodiments of this application, images acquired by different viewpoint cameras (such as RGBD cameras, which can acquire RGB images and depth images) are divided into key viewpoint images and reference viewpoint images. Combined with the depth images corresponding to each viewpoint image, end-to-end encoding and decoding modules are designed for these three types of images. This improves the rationality of the encoding and decoding network structure and facilitates better exploration of the spatial correlation between different viewpoints. At the same time, for the residual encoding process of the reference viewpoint image, image prediction is performed using the depth information of multiple viewpoint images, thereby effectively reducing the spatial redundancy between different viewpoint images and improving the compression performance and quality of the reference viewpoint image.

[0078] In the embodiments of this application, the trained encoding and decoding compression network model can be deployed to the encoding end and the decoding end respectively to perform image encoding and decoding operations.

[0079] like Figure 2 The diagram illustrates the encoding process at the encoding end of the deep learning-based free-viewpoint image encoding / decoding method provided in this application embodiment. The process mainly includes the following steps:

[0080] S201: Segment the free viewpoint image sequence into multiple viewpoint groups and input them into the trained encoding / decoding compression network model.

[0081] In one example, to better explore the spatial correlation between images from different viewpoints, free viewpoint images were divided into key viewpoint images and reference viewpoint images, and the free viewpoint image sequence was grouped. For details, see [link to documentation]. Figure 3 It mainly includes the following steps:

[0082] S2011: Divide the consecutive free viewpoint images in the free viewpoint image sequence into equal parts according to the preset number of viewpoints to obtain multiple point groups.

[0083] For example, if the preset number of viewpoints is 4, then every 4 free viewpoint images are divided into a viewpoint group.

[0084] S2012: For each viewpoint group, extract the feature vector of each free viewpoint image in the viewpoint group.

[0085] In one example, the Mvsnet network can be used to determine the category of each free viewpoint image in each viewpoint group.

[0086] S2013: Based on the feature vectors of every two free viewpoint images, obtain the similarity between the two free viewpoint images. The similarity score is used to characterize the size of the overlapping area between the two free viewpoint images.

[0087] In one example, the similarity between two free-viewpoint images can be represented by the square of the difference between the feature vectors of the two free-viewpoint images, where a greater similarity indicates a larger overlapping area between the two free-viewpoint images.

[0088] S2014: Select the free viewpoint image with the highest similarity to other viewpoint images from among the free viewpoint images as the key viewpoint image, and use the remaining free viewpoint images as reference viewpoint images.

[0089] In each viewpoint group, the free viewpoint image with the largest overlap area (i.e., the highest similarity) with other viewpoint images is used as the key viewpoint image, and all other viewpoint images are used as reference viewpoint images. Therefore, each viewpoint group contains one key viewpoint image and at least one reference viewpoint image.

[0090] For example, each viewpoint group contains 4 free viewpoint images, including 1 key viewpoint image and 3 reference viewpoint images.

[0091] In the embodiments of this application, each viewpoint image corresponds to a depth image, which can extract the depth information of each pixel in the viewpoint image.

[0092] For the three types of images—key viewpoint images, reference viewpoint images, and depth images—three encoding and decoding modules are designed in the encoding and decoding compression network model to perform encoding and compression respectively. For the specific encoding process, please refer to S202-S204.

[0093] S202: Using the key viewpoint encoding / decoding module in the encoding / decoding compression network model, the key viewpoint images in each viewpoint group are independently encoded to obtain the first image bitstream.

[0094] In one example, the key viewpoint images within each viewpoint group are independently encoded using the key viewpoint encoding / decoding module in the encoding / decoding compression network model to ensure image compression quality. For example, the encoding process for a single viewpoint group's key viewpoint images is described below. Figure 4 It mainly includes the following steps:

[0095] S202 1: Input the key viewpoint image into the encoder in the key viewpoint encoding / decoding module for encoding to obtain key viewpoint image features.

[0096] In practice, the key viewpoint image features are obtained by performing operations such as convolution, residual connection, normalization, and downsampling on the key viewpoint image through the encoder in the key viewpoint encoding and decoding module.

[0097] S2022: Input the key viewpoint image features into the quantizer in the key viewpoint encoding / decoding module for quantization to obtain the quantized features of the key viewpoint image.

[0098] In practice, the key viewpoint image features are rounded to the nearest integer by the quantizer in the key viewpoint encoding / decoding module to obtain the quantized features of the key viewpoint image.

[0099] S2023: Input the quantized features of the key viewpoint image into the entropy encoder in the key viewpoint encoding and decoding module for arithmetic encoding to obtain the first image bitstream.

[0100] When training the encoding and decoding compression network model, the entropy model in the key viewpoint encoding and decoding module determines the probability model parameters of the entropy encoder based on two kinds of prior information extracted from the sample images by the context network and the super-prior network. Thus, the entropy encoder is used to perform arithmetic encoding on the quantization features of the key viewpoint image to obtain the first image bitstream.

[0101] S203: The depth encoding and decoding module in the encoding and decoding compression network model is used to independently encode the depth image corresponding to each viewpoint image to obtain the second image bitstream.

[0102] In the embodiments of this application, the depth image can be used in the encoding and decoding of subsequent reference viewpoint images, and is therefore also encoded by the depth encoding and decoding module in the encoding and decoding compression network model.

[0103] Taking a depth image as an example, this depth image can be the depth image corresponding to a key viewpoint image or the depth image corresponding to a reference viewpoint image. The encoding process of this depth image is described in [link to documentation]. Figure 5 It mainly includes the following steps:

[0104] S203 1: Input the depth image into the encoder in the depth encoder-decoder module for encoding to obtain depth image features.

[0105] In practice, the depth image features are obtained by performing operations such as convolution, residual connection, normalization and downsampling on the depth image through the encoder in the depth encoding and decoding module.

[0106] S2032: Input the depth image features into the quantizer in the depth codec module for quantization to obtain the quantized features of the key viewpoint image.

[0107] In practice, the quantizer in the depth encoding / decoding module performs rounding operations on the depth image features to obtain the quantized features of the depth image.

[0108] S2033: Input the quantization features of the depth image into the entropy encoder in the depth encoder-decoder module for arithmetic encoding to obtain the second image bitstream.

[0109] When training the encoding and decoding compression network model, the entropy model in the deep encoding and decoding module determines the probability model parameters of the entropy encoder based on the prior information extracted from the sample images by the super-prior network. In this way, the entropy encoder is used to perform arithmetic encoding on the quantization features of the depth image to obtain the second image bitstream.

[0110] S204: Using the reference viewpoint encoding / decoding module in the encoding / decoding compression network model, for the reference viewpoint image in each viewpoint group, image prediction is performed based on at least one of the key viewpoint images in the same viewpoint group, the key viewpoint images in the next viewpoint group, and the previous reference viewpoint image, as well as the corresponding depth image, to obtain a base image.

[0111] In one example, for each reference viewpoint image in each viewpoint group, residual coding is performed using the reference viewpoint encoding / decoding module in the encoding / decoding compression network model to reduce the compression bitrate.

[0112] Since the prediction effect based on optical flow is limited, in order to effectively eliminate the spatial redundancy between images of different viewpoints when performing residual coding on the reference viewpoint image, the depth prediction unit in the reference viewpoint encoding and decoding module uses at least one of the key viewpoint images in the same viewpoint group, the key viewpoint images in the next viewpoint group, and the previous reference viewpoint image, along with the corresponding depth image, and combines the pre-calibrated intrinsic and extrinsic parameters of each viewpoint camera to predict the reference viewpoint image.

[0113] Taking a viewpoint group containing 4 viewpoint images as an example, such as Figure 6A As shown, when predicting the second reference viewpoint image in the viewpoint group, it is necessary to utilize the first key viewpoint image and its corresponding depth image in the viewpoint group, the second key viewpoint image and its corresponding depth image in the next viewpoint group, and the first reference viewpoint image and its corresponding depth image in the viewpoint group; as Figure 6B As shown, when predicting the first reference viewpoint image in the viewpoint group, since there is no previous reference viewpoint image in the viewpoint group, prediction can be made using only the first key viewpoint image and its corresponding depth image in the viewpoint group, and the second key viewpoint image and its corresponding depth image in the next viewpoint group.

[0114] Taking a viewpoint group as an example, the prediction process for each reference viewpoint image is described in [link to documentation]. Figure 7 It mainly includes the following steps:

[0115] S2041: For the candidate viewpoint image and the corresponding depth image, the prediction unit in the reference viewpoint encoding and decoding module is used to extract the depth information of each pixel in the candidate viewpoint image from the depth image.

[0116] The candidate viewpoint images are key viewpoint images from the same viewpoint group and the next viewpoint group, as well as any viewpoint image from the previous viewpoint image. The depth information of each pixel in each viewpoint image can be obtained from the corresponding depth image, and the process of predicting the reference viewpoint image using each viewpoint image and its depth information is consistent.

[0117] In practical applications, by transforming between the pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system, a relationship between pixel coordinates and world coordinates can be established. Therefore, the prediction process based on depth information can be viewed as projecting pixels on the candidate viewpoint image plane onto the plane of the current reference viewpoint image, such as... Figure 8 As shown.

[0118] S2042: Based on depth information and the intrinsic and extrinsic parameters of the camera corresponding to the pre-calibrated candidate viewpoint images, obtain the 3D points of each pixel in the candidate viewpoint image in three-dimensional space.

[0119] In one example, using Figure 8 Image prediction is performed based on the 3D projection relationship between the pixel coordinate system and the world coordinate system. Taking a pixel in the candidate viewpoint image as an example, let's assume that pixel A... k The pixel coordinates are (u a v a If the intrinsic parameters K and extrinsic parameters R and t of the camera corresponding to the candidate viewpoint image are given, then the three-dimensional coordinates of the 3D point A projected onto the world coordinate system are (X... A Y A Z A The projection formula is expressed as follows:

[0120]

[0121] Among them, D a For pixel A k The depth information in the depth image corresponding to the candidate viewpoint image, where C1 represents the candidate viewpoint camera coordinate system and W represents the world coordinate system.

[0122] S2043: Based on the intrinsic and extrinsic parameters of the camera corresponding to the pre-calibrated reference viewpoint image, project each acquired 3D point onto the reference viewpoint image to obtain the corresponding 2D projection points.

[0123] After obtaining the 3D points of each pixel in three-dimensional space, based on the intrinsic and extrinsic parameters of the camera corresponding to the reference viewpoint image and the depth image corresponding to the reference viewpoint image, each 3D point can be projected onto the reference viewpoint image using Equation 1 to obtain the 2D projection points of each 3D point.

[0124] S2044: Render each obtained 2D projection point based on the reference viewpoint image to obtain a prediction image.

[0125] Based on the pixel coordinates of each 2D projection point, the corresponding color is obtained from the reference viewpoint image to obtain a predicted image.

[0126] Within a viewpoint group, a corresponding number of predicted images can be obtained based on the number of viewpoint images associated with each reference viewpoint image.

[0127] For example, Figure 6A The second reference viewpoint image is associated with three viewpoint images, therefore corresponding to three prediction images. Figure 6B The first reference viewpoint image is associated with two viewpoint images, and therefore corresponds to two prediction images.

[0128] S2045: Obtain a base image based on each predicted image.

[0129] In one example, considering that depth-based image prediction may produce image holes due to occlusion between viewpoints, a multi-viewpoint hole-filling strategy is proposed after obtaining multiple predicted images corresponding to each reference viewpoint image. For details, please refer to [link to relevant documentation]. Figure 9A It mainly includes the following steps:

[0130] S2045_1: From the obtained prediction images, select the candidate prediction image with the smallest hole, and use the remaining prediction images in each prediction image to fill in the hole in the candidate prediction image to obtain the image to be processed.

[0131] Taking the second reference viewpoint image as an example, after image prediction is performed using the first key viewpoint image, the second key viewpoint image, and the first reference viewpoint image respectively, three prediction images are obtained. By counting the size of the region with a pixel of 0 in the three images, the hole sizes of the three prediction images are obtained as S1, S2, and S3, respectively, where S3 < S1 < S2. Then, the prediction image corresponding to the first reference viewpoint image is used as the candidate prediction image, and the other two prediction images are used to fill in the holes in the candidate prediction image, thereby obtaining a high-quality image to be processed.

[0132] In one example, the multi-viewpoint hole completion strategy cannot completely solve the hole problem and other distortion problems. In order to further improve the quality of image prediction, the image to be processed after multi-viewpoint hole completion can be enhanced by a filtering network. For the specific filtering process, see S2046-S2047.

[0133] S20452: Input the image to be processed and the remaining predicted images into the filtering unit, and extract the feature vectors of the image to be processed and the remaining predicted images through the residual blocks and convolutional layers in the filtering unit.

[0134] Taking the prediction process of the second reference viewpoint image as an example, the following is obtained: Figure 9B As shown, the image to be processed obtained by filling based on the first reference viewpoint image, and the predicted image 1 and predicted image 2 obtained based on the first key viewpoint image and the second key viewpoint image are respectively input into the filtering unit in the reference viewpoint encoding and decoding module. The feature vectors of each image are extracted through the convolutional layer in the filtering unit, and the channel redundancy of each image is removed through multiple residual blocks, effectively extracting the feature domain information on each channel.

[0135] In the embodiments of this application, a filtering unit is built using simple residual blocks and convolutional layers, which can effectively improve the effectiveness of prediction without increasing the number of network parameters.

[0136] S2045_3: An attention mechanism is used to learn the channel weights of the image to be processed and the other predicted images, and the learned channel weights are used to fuse the feature vectors to obtain a base image.

[0137] To better compensate for voids and distorted areas, an attention mechanism is introduced into the multilayer perceptron (MLP) of the filtering unit to learn the channel weights of each image for feature domain information fusion. This allows for better fusion and reconstruction of channel features, supplementing information loss in certain areas of certain channels, and obtaining a high-quality base image.

[0138] S205: Combine the reference viewpoint image to obtain the viewpoint residual image, and encode the viewpoint residual image to obtain the third image bitstream.

[0139] After obtaining the base image after prediction and completion for each reference viewpoint image, residual coding can be performed by combining it with the original reference viewpoint image. This reduces the entropy of the resulting bitstream, lowers the compression bitrate, and thus improves the compression performance of the encoding / decoding compression network model. For details on the residual coding process, please refer to [link to documentation]. Figure 10 It mainly includes the following steps:

[0140] S2051: Input the viewpoint residual image into the encoder in the reference viewpoint encoding / decoding module for encoding to obtain residual image features.

[0141] In practice, the residual image features are obtained by performing operations such as convolution, residual concatenation, normalization, and downsampling on the residual image through the encoder in the reference viewpoint encoding and decoding module.

[0142] S2052: Input the residual image features into the quantizer in the reference viewpoint encoding / decoding module for quantization to obtain the quantized features of the residual image.

[0143] In practice, the residual image features are rounded to the nearest integer by the quantizer in the reference viewpoint encoding / decoding module to obtain the quantized features of the residual image.

[0144] S2053: Input the quantization features of the residual image into the entropy encoder in the reference viewpoint encoding and decoding module for arithmetic encoding to obtain the third image bitstream.

[0145] When training the encoding and decoding compression network model, the entropy model in the reference view encoding and decoding module determines the probability model parameters of the entropy encoder based on two kinds of prior information extracted from the sample images by the context network and the super-prior network. Then, the entropy encoder is used to perform arithmetic encoding on the quantization features of the residual image to obtain the third image bitstream.

[0146] S206: Transmit the first image stream, second image stream, and third image stream corresponding to each viewpoint group to the decoding end for decoding.

[0147] The encoding end encodes the key viewpoint images, reference viewpoint images, and corresponding depth images in each viewpoint group and transmits them to the decoding end for display.

[0148] It should be noted that the embodiments of this application do not impose restrictive requirements on the transmission methods of the three image bitstreams. They can be merged and transmitted through a single link, or they can be transmitted independently through multiple links.

[0149] It should be noted that since the key viewpoint image and the depth image are encoded and transmitted independently, there is no mandatory requirement for the encoding order of the two, and they can be executed in parallel.

[0150] In the embodiments of this application, at the encoding end, one key viewpoint image and at least one reference viewpoint image in each viewpoint group after the free viewpoint image sequence is segmented are encoded and transmitted separately, thereby better exploring the spatial correlation between images from different viewpoints. Furthermore, through the division of viewpoint groups, each reference viewpoint image is associated with at least one of the key viewpoint image of its own viewpoint group, the key viewpoint image of the next viewpoint group, and the previous reference viewpoint image, and each viewpoint image corresponds to a depth image. Thus, image prediction can be performed using its associated viewpoint images and corresponding depth images, effectively reducing spatial redundancy between different viewpoints and improving prediction quality. Simultaneously, by performing hole-filling and enhancement processing on the predicted image, the prediction quality is further improved. When combined with the original reference viewpoint image for residual coding, the entropy of the residual image is effectively reduced, ensuring image coding quality, reducing the amount of data encoded, and improving end-to-end transmission speed.

[0151] In one example, deploying a trained encoding / decoding compression network model at the decoding end allows for corresponding decoding operations. For example... Figure 11 The diagram illustrates the decoding process at the decoding end of the deep learning-based free-viewpoint image encoding / decoding method provided in this application embodiment. The process mainly includes the following steps:

[0152] S1101: Receive the first image stream, second image stream, and third image stream of each of the multiple viewpoint groups corresponding to the free viewpoint image sequence sent by the encoding end, and input them into the trained encoding / decoding compression network model.

[0153] The first image stream is the encoding result of the key viewpoint image in each viewpoint group, the second image stream is the encoding result of the depth image corresponding to each viewpoint image in each viewpoint group, and the third image stream is the encoding result of the reference viewpoint image in each viewpoint group.

[0154] S1102: The key viewpoint encoding / decoding module in the encoding / decoding compression network model is used to independently decode the first image bitstream of each viewpoint group to obtain the key viewpoint image within the corresponding viewpoint group.

[0155] In one example, since the key viewpoint images in each viewpoint group are encoded and transmitted independently, the decoding end can directly decode the key viewpoint images from the first image bitstream. For example, the decoding process for a single viewpoint group's key viewpoint images is described below. Figure 12 It mainly includes the following steps:

[0156] S1102_1: Input the first image bitstream into the entropy decoder in the key viewpoint encoding and decoding module for entropy decoding to obtain the decoding features of the key viewpoint image.

[0157] When training the encoding and decoding compression network model, the entropy model in the key viewpoint encoding and decoding module determines the probability model parameters of the entropy decoder based on the prior information extracted from the sample images by the super-prior network and the uplink network. Thus, the entropy decoder is used to perform arithmetic decoding on the first image bitstream to obtain the decoding features of the key viewpoint image.

[0158] S1102_2: Input the decoding features of the key viewpoint image into the decoder in the key viewpoint encoding / decoding module for decoding to obtain the key viewpoint image.

[0159] In practice, the decoder in the key viewpoint encoding and decoding module performs operations such as convolution, residual connection, normalization and downsampling on the decoded features to obtain the key viewpoint image.

[0160] S1103: The depth encoding and decoding module in the encoding and decoding compression network model is used to independently decode the second image bitstream of each viewpoint group to obtain the depth image of each viewpoint image in the corresponding viewpoint group.

[0161] In one example, the depth image corresponding to each viewpoint image is also independently encoded and transmitted. Therefore, the decoding end can directly decode the depth images of each viewpoint image through the second image bitstream. Taking a single depth image (which can be the depth image of a reference viewpoint image or the depth image of a key viewpoint) as an example, the decoding process of the depth image is described in [link to documentation]. Figure 13 It mainly includes the following steps:

[0162] S1103_1: Input the second image bitstream into the entropy decoder in the depth codec module for entropy decoding to obtain the decoding features of the depth image.

[0163] When training the encoding and decoding compression network model, the entropy model in the deep encoding and decoding module determines the probability model parameters of the entropy decoder based on the prior information extracted from the sample images by the super-prior network. Thus, the entropy decoder is used to perform arithmetic decoding on the second image bitstream to obtain the decoding features of the depth image.

[0164] S1103_2: Input the decoding features of the depth image into the decoder in the depth encoding / decoding module for decoding to obtain the depth image.

[0165] In practice, the depth image is obtained by performing operations such as convolution, residual connection, normalization and downsampling on the decoded features through the decoder in the depth encoding and decoding module.

[0166] S1104: The reference viewpoint encoding / decoding module in the encoding / decoding compression network model is used to decode the third image bitstream of each viewpoint group to obtain the residual image of each reference viewpoint image in the corresponding viewpoint group.

[0167] Taking the decoding process of a reference viewpoint image in a viewpoint group as an example, the residual decoding process is described in [reference]. Figure 14 It mainly includes the following steps:

[0168] S1104_1: Input the third image bitstream into the entropy decoder in the reference viewpoint encoding and decoding module for entropy decoding to obtain the decoding features of the residual image.

[0169] When training the encoding and decoding compression network model, the entropy model in the reference viewpoint encoding and decoding module determines the probability model parameters of the entropy decoder based on the prior information extracted from the sample images by the super-prior network and the uplink network. Thus, the entropy decoder is used to perform arithmetic decoding on the third image bitstream to obtain the decoding features of the reference viewpoint image.

[0170] S1104_2: Input the decoding features of the residual image into the decoder in the reference viewpoint encoding / decoding module for decoding to obtain the residual image.

[0171] In practice, the decoder in the reference viewpoint encoding / decoding module performs operations such as convolution, residual connection, normalization, and downsampling on the decoded features to obtain the residual image.

[0172] S1105: A base image is obtained by performing image prediction based on at least one of the key viewpoint image of the viewpoint group to which the reference viewpoint image corresponding to the decoded residual image belongs, the key viewpoint image of the next viewpoint group, and the previous reference viewpoint image, as well as the corresponding depth image.

[0173] In one example, when performing residual decoding on the current reference viewpoint image, the key viewpoint images and corresponding depth images in the viewpoint group where the current reference viewpoint image is located, the key viewpoint images and corresponding depth images in the next viewpoint group, and the previous reference viewpoint image and corresponding depth image within the same viewpoint group can be obtained through decoding. Therefore, the decoding end can also employ... Figure 7 Image prediction is performed as shown, and will not be elaborated further here.

[0174] S1106: Perform image fusion on the base image and the residual image to obtain the corresponding reference viewpoint image.

[0175] After residual coding of the reference viewpoint image, if the quality of the decoded residual image is poor, the residual image cannot be well adapted to the predicted and compensated base image in some areas. If the decoded residual image and the predicted and compensated base image are simply superimposed to reconstruct the reference viewpoint image, it may lead to distortion in some areas.

[0176] To address the aforementioned residual coding problem, this application embodiment employs a residual fusion network in the reference viewpoint encoding / decoding module to replace the original image overlay method for reconstructing the reference viewpoint image.

[0177] like Figure 15 The diagram shown is a structural diagram of the residual fusion network provided in the embodiment of this application. The residual fusion network is mainly constructed from simple convolutional layers and residual blocks. In this way, the number of additional network parameters will not be significantly increased, while improving the image quality of residual decoding.

[0178] based on Figure 15 The residual fusion network shown illustrates the reconstruction process of the reference viewpoint as follows: Figure 16 As shown, it mainly includes the following steps:

[0179] S1106_1: Perform multi-layer convolution operations on the residual image and the base image respectively to obtain the feature vectors of the corresponding images.

[0180] First, the residual image and the base image are input into the residual fusion network in the reference viewpoint encoding and decoding module, and three convolutional operations are performed respectively to obtain the feature vectors of the residual image and the base image.

[0181] S1106_2: Fuse the feature vectors of the residual image and the feature vectors of the base image to obtain the fused vector.

[0182] Then, the feature vectors of the residual image and the feature vectors of the base image are additively fused through the FUSE layer in the residual fusion network to obtain the fused vector.

[0183] S1106_3: Perform residual and convolution operations on the fused vector to obtain the corresponding reference viewpoint image.

[0184] Finally, the fused vector is subjected to residual and convolution operations through a residual fusion network to complete the reconstruction from the feature domain to the pixel domain and obtain the corresponding reference viewpoint image.

[0185] In the embodiments of this application, at the decoding end, after independently decoding the first image stream and the third image stream, the key viewpoint image and the depth image of each viewpoint image can be directly obtained. Thus, when performing residual decoding on the current reference viewpoint image to be decoded, prediction can be performed by obtaining at least one associated viewpoint image and its corresponding depth image. Combined with the residual image decoded from the third image stream, the current reference viewpoint image is obtained. Since prediction is performed using multiple decoded viewpoint images, spatial redundancy between different viewpoints can be effectively reduced, improving image prediction quality. Simultaneously, by fusing the feature domains of the predicted compensated base image and the decoded residual image through a residual fusion network, the adaptability of the residual image and the predicted image can be effectively improved, reducing image distortion and thus enhancing the quality of reference viewpoint image decoding.

[0186] To verify the performance of the encoding and decoding compression network model deployed at the encoding and decoding ends, embodiments of this application are verified from multiple aspects.

[0187] like Figure 17 As shown, Deep Homography for Efficient Stereo Image Compression (HESIC) and SASIC, along with traditional standard High Efficiency Video Coding (HEVC), Multi Vision High Efficiency Video Coding (MV-HEVC), and 3D High Efficiency Video Coding (3D-HEVC), were used as comparative models in this application embodiment. Four test sets were selected from the MVSblended dataset as image sequences for comparison, with the number of viewpoints in each viewpoint group set to two. The relationship between the peak signal-to-noise ratio (PSNR) and compression bit rate (bits per pixel, Bpp) of each model was analyzed.

[0188] Depend on Figure 17 It is known that the binocular end-to-end compression model is only suitable for binocular test sets with small disparity and performs poorly on 6-DOF free viewpoint image test sets, while the traditional standard model performs better. Furthermore, the PSNR performance of the encoding and decoding compression network model in this embodiment is better than that of the traditional standard model due to the learning ability and prediction accuracy of the deep network, and it can basically maintain competitiveness with 3D-HEVC across various bitrate ranges.

[0189] like Figure 18The image shown is a comparison of image reconstruction quality under the MVSBlended test set provided in this application embodiment. From left to right, they are the original image, the HEVC reconstructed image, the 3D-HEVC reconstructed image, and the reconstructed image of this application embodiment. It can be seen that, regarding the tree texture in the upper image and the floor tile texture in the lower image, this application embodiment achieves superior texture reconstruction results in decoding and reconstruction.

[0190] like Figure 19 The image shows a comparison of the prediction quality of different models on the MVSBlended test set provided in this application embodiment. From left to right, they represent the prediction image based on optical flow, the prediction image based on depth information, and the prediction image enhanced with hole filling. It can be seen that the prediction effect based on optical flow is relatively blurry, while the prediction unit of this application embodiment can achieve better prediction quality, and the hole problem is greatly reduced.

[0191] Based on the same technical concept, this application provides a first terminal device that can implement the steps of the deep learning-based free viewpoint image encoding and decoding method implemented at the encoding end, and can achieve the same technical effect.

[0192] See Figure 20 The first terminal device includes a processor 2001, a memory 2002 and a communication interface 2003, wherein the communication interface 2003, the memory 2002 and the processor 2001 are connected via a bus 2004;

[0193] The communication interface 2003 is used to communicate with the second terminal device;

[0194] The memory 2002 stores a computer program, and the processor 2001 performs the following operations according to the computer program:

[0195] The free viewpoint image sequence is segmented into multiple viewpoint groups and input into a trained encoding and decoding compression network model. Each viewpoint group contains a key viewpoint image and at least one reference viewpoint image, and each viewpoint image corresponds to a depth image.

[0196] The key viewpoint encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the key viewpoint images in each viewpoint group to obtain the first image bitstream.

[0197] The depth encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the depth image corresponding to each viewpoint image to obtain the second image bitstream;

[0198] Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, for the reference viewpoint image in each viewpoint group, image prediction is performed based on the key viewpoint images and corresponding depth images in the same viewpoint group and the next viewpoint group, as well as the previous viewpoint image and corresponding depth image of the reference viewpoint image, to obtain a base image. Combined with the reference viewpoint image, a viewpoint residual image is obtained, and the viewpoint residual image is encoded to obtain a third image bitstream.

[0199] The first image stream, the second image stream, and the third image stream corresponding to each viewpoint group are transmitted to the second terminal device for decoding.

[0200] Optionally, the processor 2001 divides the free viewpoint image sequence into multiple viewpoint groups, specifically as follows:

[0201] According to the preset number of viewpoints, the consecutive free viewpoint images in the free viewpoint image sequence are divided equally to obtain multiple point groups;

[0202] For each viewpoint group, extract the feature vector of each free viewpoint image in the viewpoint group;

[0203] The similarity between two free viewpoint images is obtained based on the feature vectors of each pair of free viewpoint images. The similarity value is used to characterize the size of the overlapping area between the two free viewpoint images.

[0204] The free viewpoint image with the highest similarity to other viewpoint images is selected as the key viewpoint image, and the remaining free viewpoint images are selected as reference viewpoint images.

[0205] Optionally, the processor 2001 uses the key viewpoint encoding / decoding module in the encoding / decoding compression network model to independently encode the key viewpoint images in each viewpoint group to obtain a first image bitstream; and uses the depth encoding / decoding module in the encoding / decoding compression network model to independently encode the depth image corresponding to each viewpoint image to obtain a second image bitstream. Specifically, the operation is as follows:

[0206] The target image is input into the encoder of the target encoding / decoding module for encoding to obtain the target image features;

[0207] The target image features are input into the quantizer of the target encoding / decoding module for quantization to obtain the target quantized features;

[0208] The target quantization features are input into the entropy encoder in the target encoding / decoding module for entropy encoding to obtain the target image bitstream; wherein, the parameters of the entropy encoder are determined by the entropy model in the target encoding / decoding module based on prior information obtained from the sample images during the training of the encoding / decoding compression network model;

[0209] Wherein, when the target image is the key viewpoint image, the target encoding / decoding module is the key viewpoint encoding / decoding module, and the target image bitstream is the first image bitstream; when the target image is the depth image, the target encoding / decoding module is the depth encoding / decoding module, and the target image bitstream is the second image bitstream; when the target image is the residual image, the target encoding / decoding module is the reference viewpoint encoding / decoding module, and the target image bitstream is the third image bitstream.

[0210] Optionally, the processor 2001 employs the reference viewpoint encoding / decoding module in the encoding / decoding compression network model. For the reference viewpoint image in each viewpoint group, it performs image prediction based on at least one of the key viewpoint images in the same viewpoint group, the key viewpoint images in the next viewpoint group, and the previous reference viewpoint image, along with the corresponding depth image, to obtain a base image. The specific operation is as follows:

[0211] For candidate viewpoint images and corresponding depth images, the prediction unit in the reference viewpoint encoding and decoding module is used to extract the depth information of each pixel in the candidate viewpoint image from the depth image; wherein, the candidate viewpoint image is a key viewpoint image in the same viewpoint group and the next viewpoint group, as well as any viewpoint image in the previous viewpoint image;

[0212] Based on the depth information and the intrinsic and extrinsic parameters of the camera corresponding to the pre-calibrated candidate viewpoint images, obtain the 3D points of each pixel in the candidate viewpoint images in three-dimensional space.

[0213] Based on the intrinsic and extrinsic parameters of the camera corresponding to the pre-calibrated reference viewpoint image, each acquired 3D point is projected onto the reference viewpoint image to obtain the corresponding 2D projection points.

[0214] Render each 2D projection point obtained based on the reference viewpoint image to obtain a prediction image;

[0215] Based on each predicted image, a base image is obtained.

[0216] Optionally, the processor 2001 obtains a base image based on each predicted image, specifically by:

[0217] From the obtained prediction images, select the candidate prediction image with the smallest hole, and use the remaining prediction images in the prediction images to fill the hole in the candidate prediction image to obtain the image to be processed.

[0218] The image to be processed and the remaining predicted images are input into the filtering unit, and the feature vectors of the image to be processed and the remaining predicted images are extracted through the residual blocks and convolutional layers in the filtering unit.

[0219] An attention mechanism is used to learn the channel weights of the image to be processed and the other predicted images, and the learned channel weights are used to fuse the feature vectors to obtain a base image.

[0220] The aforementioned first terminal device, by encoding and transmitting one key viewpoint image and at least one reference viewpoint image in each viewpoint group after segmentation of the free viewpoint image sequence, can better explore the spatial correlation between images from different viewpoints. Furthermore, through the division of viewpoint images, each reference viewpoint image is associated with at least one of the key viewpoint image in its own viewpoint group, the key viewpoint image in the next viewpoint group, and the previous reference viewpoint image, and each viewpoint image corresponds to a depth image. In this way, image prediction can be performed using its associated viewpoint image and the corresponding depth image, and a residual image can be obtained by combining it with the original reference viewpoint image. The residual image is then encoded and transmitted. Since prediction is performed using multiple viewpoint images, spatial redundancy between different viewpoints can be effectively reduced, image encoding quality can be improved, the amount of image encoding data can be reduced, and the end-to-end transmission speed can be increased.

[0221] Based on the same technical concept, this application provides a second terminal device that can implement the steps of the deep learning-based free viewpoint image encoding and decoding method implemented at the decoding end, and achieve the same technical effect.

[0222] See Figure 21 The second terminal device includes a processor 2101, a memory 2102 and a communication interface 2103, wherein the communication interface 2103, the memory 2102 and the processor 2101 are connected via a bus 2104;

[0223] The communication interface 2103 is used to communicate with the first terminal device;

[0224] The memory 2102 stores computer program instructions, and the processor 2101 performs the following operations according to the computer program instructions:

[0225] The system receives the first image stream, second image stream, and third image stream of each of the multiple viewpoint groups corresponding to the free viewpoint image sequence sent by the first terminal device, and inputs them into the trained encoding and decoding compression network model; wherein, the first image stream is the encoding result of the key viewpoint image in each viewpoint group, the second image stream is the encoding result of the depth image corresponding to each viewpoint image in each viewpoint group, and the third image stream is the encoding result of the reference viewpoint image in each viewpoint group;

[0226] The key viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to independently decode the first image bitstream of each viewpoint group to obtain the key viewpoint image within the corresponding viewpoint group.

[0227] The depth codec module in the aforementioned codec compression network model is used to independently decode the second image stream of each viewpoint group to obtain the depth image of each viewpoint image within the corresponding viewpoint group.

[0228] Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, the third image stream of each viewpoint group is decoded to obtain the residual image of each reference viewpoint image within the corresponding viewpoint group. Based on the key viewpoint image and corresponding depth image in the viewpoint group and next viewpoint group where the decoded residual image is located, as well as the previous viewpoint image and corresponding depth image of the reference viewpoint image corresponding to the residual image, image prediction is performed to obtain a base image. The base image and the residual image are then fused to obtain the corresponding reference viewpoint image.

[0229] Optionally, the processor 2101 employs the key viewpoint encoding / decoding module in the encoding / decoding compression network model to independently decode the first image stream of each viewpoint group to obtain key viewpoint images within the corresponding viewpoint group; and employs the depth encoding / decoding module in the encoding / decoding compression network model to independently decode the second image stream of each viewpoint group to obtain depth images of each viewpoint image within the corresponding viewpoint group; and employs the reference viewpoint encoding / decoding module in the encoding / decoding compression network model to decode the third image stream of each viewpoint group to obtain residual images of each reference viewpoint image within the corresponding viewpoint group. Specifically, the operations are as follows:

[0230] The target image bitstream is input into the entropy decoder in the target codec module for entropy decoding to obtain the decoding features of the target image; wherein, the parameters of the entropy decoder are determined by the entropy model in the target codec module based on prior information obtained from the sample image during the training of the codec compression network model;

[0231] The decoding features of the target image are input into the decoder in the target encoding / decoding module for decoding to obtain the target image;

[0232] Wherein, when the target image stream is a first image stream, the target encoding / decoding module is a key viewpoint encoding / decoding module, and the target image is a key viewpoint image; when the target image stream is a second image stream, the target encoding / decoding module is a depth encoding / decoding module, and the target image is a depth image; when the target image stream is a third image stream, the target encoding / decoding module is a reference viewpoint encoding / decoding module, and the target image is a residual image of the reference viewpoint image.

[0233] Optionally, the processor 2101 performs image fusion on the base image and the residual image to obtain a corresponding reference viewpoint image, specifically as follows:

[0234] Perform multi-layer convolution operations on the residual image and the base image respectively to obtain the feature vectors of the corresponding images; fuse the feature vectors of the residual image and the feature vectors of the base image to obtain a fused vector;

[0235] The fused vector is subjected to residual and convolution operations to obtain the corresponding reference viewpoint image.

[0236] The aforementioned second terminal device independently decodes the first and third image streams to obtain key viewpoint images and depth images for each viewpoint image. Thus, for the current reference viewpoint image to be decoded, prediction can be performed by acquiring at least one associated viewpoint image and its corresponding depth image. This prediction, combined with the residual image decoded from the third image stream, yields the current reference viewpoint image. By utilizing multiple decoded viewpoint images for prediction, spatial redundancy between different viewpoints is effectively reduced, improving image prediction quality. Furthermore, fusing this prediction with the residual image effectively enhances the decoding quality of the reference viewpoint image.

[0237] It should be noted that, Figure 20 and Figure 21 This is merely an example illustrating the hardware necessary for the first and second terminal devices to execute the deep learning-based free-viewpoint image encoding / decoding method steps provided in the embodiments of this application. Not shown, the first and second terminal devices may further include one or more of the following: a display, a speaker, a microphone, a power supply, an IMU, a multi-view camera, and operation buttons.

[0238] In this embodiment, the memory may be volatile memory, such as random-access memory (RAM); the memory may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or the memory may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer. The memory may be a combination of the above-mentioned memories, but is not limited thereto. The processor may include one or more central processing units (CPUs) or a general-purpose processor, graphics processing unit (GPU), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0239] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a deep learning-based free-viewpoint image encoding / decoding method as described in the foregoing embodiments.

[0240] This application also provides a computer program product for storing a computer program that executes a deep learning-based free-viewpoint image encoding / decoding method as described in the foregoing embodiments.

[0241] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0242] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0243] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0244] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A free-viewpoint image encoding and decoding method based on deep learning, characterized in that, Applied to the encoding end, including: The free viewpoint image sequence is segmented into multiple viewpoint groups and input into a trained encoding and decoding compression network model. Each viewpoint group contains a key viewpoint image and at least one reference viewpoint image, and each viewpoint image corresponds to a depth image. The key viewpoint encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the key viewpoint images in each viewpoint group to obtain the first image bitstream. The depth encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the depth image corresponding to each viewpoint image to obtain the second image bitstream; Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, for the reference viewpoint image in each viewpoint group, the prediction unit in the reference viewpoint encoding / decoding module extracts the depth information of each pixel in the candidate viewpoint image from the depth image; wherein, the candidate viewpoint image is a key viewpoint image in the same viewpoint group and the next viewpoint group, as well as any viewpoint image in the previous viewpoint image; Based on the depth information and the intrinsic and extrinsic parameters of the camera corresponding to the pre-calibrated candidate viewpoint images, obtain the 3D points of each pixel in the candidate viewpoint images in three-dimensional space. Based on the intrinsic and extrinsic parameters of the camera corresponding to the pre-calibrated reference viewpoint image, each acquired 3D point is projected onto the reference viewpoint image to obtain the corresponding 2D projection points. Render each 2D projection point obtained based on the reference viewpoint image to obtain a prediction image. Select the candidate prediction image with the smallest hole from the obtained prediction images, and fill the hole in the candidate prediction image with the remaining prediction images to obtain the image to be processed. The image to be processed and the remaining predicted images are input into the filtering unit. The feature vectors of the image to be processed and the remaining predicted images are extracted through the residual blocks and convolutional layers in the filtering unit. An attention mechanism is used to learn the channel weights of the image to be processed and the other predicted images, and the learned channel weights are used to fuse the feature vectors to obtain a base image. By combining the reference viewpoint image, a viewpoint residual image is obtained, and the viewpoint residual image is encoded to obtain a third image stream; The first, second, and third image streams corresponding to each viewpoint group are transmitted to the decoding end for decoding.

2. The method as described in claim 1, characterized in that, The step of segmenting the free-viewpoint image sequence into multiple viewpoint groups includes: According to the preset number of viewpoints, the consecutive free viewpoint images in the free viewpoint image sequence are divided equally to obtain multiple viewpoint groups; For each viewpoint group, extract the feature vector of each free viewpoint image in the viewpoint group; The similarity between two free viewpoint images is obtained based on the feature vectors of each pair of free viewpoint images. The similarity value is used to characterize the size of the overlapping area between the two free viewpoint images. The free viewpoint image with the highest similarity to other viewpoint images is selected as the key viewpoint image, and the remaining free viewpoint images are selected as reference viewpoint images.

3. The method as described in claim 1, characterized in that, The key viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to independently encode the key viewpoint images in each viewpoint group to obtain a first image stream. The depth encoding / decoding module in the aforementioned encoding / decoding compression network model is used to independently encode the depth image corresponding to each viewpoint image to obtain a second image stream. The reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to encode the viewpoint residual image to obtain a third image stream, including: The target image is input into the encoder of the target encoding / decoding module for encoding to obtain the target image features; The target image features are input into the quantizer of the target encoding / decoding module for quantization to obtain the target quantized features; The target quantization features are input into the entropy encoder in the target encoding / decoding module for entropy encoding to obtain the target image bitstream; wherein, the parameters of the entropy encoder are determined by the entropy model in the target encoding / decoding module based on prior information obtained from the sample images during the training of the encoding / decoding compression network model; Wherein, when the target image is the key viewpoint image, the target encoding / decoding module is the key viewpoint encoding / decoding module, and the target image bitstream is the first image bitstream; when the target image is the depth image, the target encoding / decoding module is the depth encoding / decoding module, and the target image bitstream is the second image bitstream; when the target image is the residual image, the target encoding / decoding module is the reference viewpoint encoding / decoding module, and the target image bitstream is the third image bitstream.

4. A free-viewpoint image encoding and decoding method based on deep learning, characterized in that, Applied to the decoding end, including: The system receives the first image stream, second image stream, and third image stream of each of the multiple viewpoint groups corresponding to the free viewpoint image sequence sent by the encoding end, and inputs them into the trained encoding and decoding compression network model. The first image stream is the encoding result of the key viewpoint image in each viewpoint group, the second image stream is the encoding result of the depth image corresponding to each viewpoint image in each viewpoint group, and the third image stream is the encoding result of the reference viewpoint image in each viewpoint group. The key viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to independently decode the first image bitstream of each viewpoint group to obtain the key viewpoint image within the corresponding viewpoint group. The depth codec module in the aforementioned codec compression network model is used to independently decode the second image stream of each viewpoint group to obtain the depth image of each viewpoint image within the corresponding viewpoint group. The reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to decode the third image stream of each viewpoint group to obtain the residual image of each reference viewpoint image within the corresponding viewpoint group. Based on at least one of the key viewpoint image of the viewpoint group where the decoded residual image is located, the key viewpoint image of the next viewpoint group, and the previous reference viewpoint image, as well as the corresponding depth image, image prediction is performed to obtain a base image. The base image and the residual image are then fused to obtain the corresponding reference viewpoint image.

5. The method as described in claim 4, characterized in that, The process involves using the key viewpoint encoding / decoding module in the encoding / decoding compression network model to independently decode the first image stream of each viewpoint group to obtain key viewpoint images within the corresponding viewpoint group; and using the depth encoding / decoding module in the encoding / decoding compression network model to independently decode the second image stream of each viewpoint group to obtain depth images of each viewpoint image within the corresponding viewpoint group; and using the reference viewpoint encoding / decoding module in the encoding / decoding compression network model to decode the third image stream of each viewpoint group to obtain residual images of each reference viewpoint image within the corresponding viewpoint group, including: The target image bitstream is input into the entropy decoder in the target codec module for entropy decoding to obtain the decoding features of the target image; wherein, the parameters of the entropy decoder are determined by the entropy model in the target codec module based on prior information obtained from the sample image during the training of the codec compression network model; The decoding features of the target image are input into the decoder in the target encoding / decoding module for decoding to obtain the target image; Wherein, when the target image stream is a first image stream, the target encoding / decoding module is a key viewpoint encoding / decoding module, and the target image is a key viewpoint image; when the target image stream is a second image stream, the target encoding / decoding module is a depth encoding / decoding module, and the target image is a depth image; when the target image stream is a third image stream, the target encoding / decoding module is a reference viewpoint encoding / decoding module, and the target image is a residual image of the reference viewpoint image.

6. The method as described in claim 4, characterized in that, The step of fusing the base image and the residual image to obtain a corresponding reference viewpoint image includes: Perform multi-layer convolution operations on the residual image and the base image respectively to obtain the feature vectors of the corresponding images; The feature vectors of the residual image and the feature vectors of the base image are fused to obtain a fused vector; The fused vector is subjected to residual and convolution operations to obtain the corresponding reference viewpoint image.

7. A first terminal device, characterized in that, It includes a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus; The communication interface is used to communicate with the second terminal device; The memory stores a computer program, and the processor performs the following operations according to the computer program: The free viewpoint image sequence is segmented into multiple viewpoint groups and input into a trained encoding and decoding compression network model. Each viewpoint group contains a key viewpoint image and at least one reference viewpoint image, and each viewpoint image corresponds to a depth image. The key viewpoint encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the key viewpoint images in each viewpoint group to obtain the first image bitstream. The depth encoding and decoding module in the aforementioned encoding and decoding compression network model is used to independently encode the depth image corresponding to each viewpoint image to obtain the second image bitstream; Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, for the reference viewpoint image in each viewpoint group, image prediction is performed based on the key viewpoint images and corresponding depth images in the same viewpoint group and the next viewpoint group, as well as the previous reference viewpoint image and corresponding depth image of the reference viewpoint image, to obtain a base image. Combined with the reference viewpoint image, a viewpoint residual image is obtained, and the viewpoint residual image is encoded to obtain a third image bitstream. The first image stream, the second image stream, and the third image stream corresponding to each viewpoint group are transmitted to the second terminal device for decoding.

8. A second terminal device, characterized in that, It includes a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus; The communication interface is used to communicate with the first terminal device; The memory stores computer program instructions, and the processor performs the following operations according to the computer program instructions: The system receives the first image stream, second image stream, and third image stream of each of the multiple viewpoint groups corresponding to the free viewpoint image sequence sent by the first terminal device, and inputs them into the trained encoding and decoding compression network model; wherein, the first image stream is the encoding result of the key viewpoint image in each viewpoint group, the second image stream is the encoding result of the depth image corresponding to each viewpoint image in each viewpoint group, and the third image stream is the encoding result of the reference viewpoint image in each viewpoint group; The key viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model is used to independently decode the first image bitstream of each viewpoint group to obtain the key viewpoint image within the corresponding viewpoint group. The depth codec module in the aforementioned codec compression network model is used to independently decode the second image stream of each viewpoint group to obtain the depth image of each viewpoint image within the corresponding viewpoint group. Using the reference viewpoint encoding / decoding module in the aforementioned encoding / decoding compression network model, the third image stream of each viewpoint group is decoded to obtain the residual image of each reference viewpoint image within the corresponding viewpoint group. Based on the key viewpoint image and corresponding depth image in the viewpoint group and the next viewpoint group where the decoded residual image is located, as well as the previous reference viewpoint image and corresponding depth image of the reference viewpoint image corresponding to the residual image, image prediction is performed to obtain a base image. The base image and the residual image are then fused to obtain the corresponding reference viewpoint image.

Images