Light field transmission method fusing image super-resolution and selective quality mode

Abstract

The invention discloses a light field transmission method fusing image super-resolution and a selective quality mode. The method comprises the following steps: firstly, down-sampling an original light field image to generate a low-resolution image; and then, carrying out super-resolution on the down-sampling image through a super-resolution reconstruction technology so as to restore the down-sampling image to the resolution which is the same as that of the original light field, and generating a super-resolution light field image. Then, the residual error between the super-resolution light field image and the original light field image is calculated; the invention further provides a transmission mode based on the video coding technology, selective quality regulation and control are carried out on different sub-views in a light field, a high code rate is allocated for the sub-views meeting the watching habits of human eyes, and a low code rate is allocated for other sub-views. And finally, encoding the down-sampling image and the residual error into a pseudo video sequence for transmission through a selective quality transmission mode. And after transmission, decoding the two pseudo video sequences, carrying out super division on a down-sampling image, and adding the down-sampling image with a residual error to complete light field reconstruction. According to the method, by combining the image super-resolution and the selective quality mode, the data size of the light field image is remarkably reduced, meanwhile, the image quality is hardly influenced, and an innovative efficient solution is provided for light field transmission.

Description

Technical Field

[0001] The patent of this invention belongs to the field of image processing and communication technology, and in particular, it relates to an efficient method for light field image transmission, which combines image super-resolution reconstruction and selective quality control mode. Through downsampling, super-resolution reconstruction, residual calculation, and selective quality transmission technology of video encoding, the bit rate allocation of different sub-views in the light field image is optimized to significantly reduce the amount of transmitted data and bandwidth requirements while maintaining high fidelity of image quality. This technology is suitable for transmitting high-resolution light field images in a bandwidth-constrained network environment, providing an innovative solution for light field image transmission. Background Art

[0002] With the development of technologies such as virtual reality, augmented reality, and free-viewpoint video, light field images, as a new form of visual data that can truly record scene light information, have received widespread attention. Light field images capture the direction and intensity information of light in a scene, can reproduce the spatial sense and three-dimensional effect of objects, and support multi-view display and free viewpoint selection. However, light field images contain a large amount of view data, and the data volume is extremely large, which brings great challenges to storage and transmission.

[0003] Traditional image compression and transmission methods are difficult to effectively handle such a large amount of data, especially in a network environment with limited bandwidth, and it is difficult to ensure high-quality image transmission. To solve this problem, in recent years, image super-resolution technology and selective quality control strategies have gradually become important means in light field image transmission. Image super-resolution technology can achieve a near restoration of high-resolution images by reconstructing the details of low-resolution images, thereby maintaining the visual quality of the image with low data volume. The selective quality control strategy can allocate different bitrate resources to different views based on the visual characteristics perceived by the human eye, thereby ensuring the image quality of the key visual areas under limited bandwidth conditions.

[0004] Although some studies have combined image super-resolution and selective quality modes to improve the transmission efficiency of light field images, existing technologies still have deficiencies in the balance between data compression, transmission efficiency and image quality. Most methods only uniformly encode the full view and cannot dynamically adjust the bit rate according to application requirements, resulting in low bandwidth resource utilization efficiency. In addition, some technologies cannot ensure high-quality image reconstruction under low bit rate transmission conditions. Therefore, there is an urgent need for an efficient light field transmission method that combines image super-resolution with selective quality modes to reduce the bandwidth requirements of light field image transmission and ensure the quality of light field reconstruction after transmission. Summary of the invention

[0005] To solve the above technical problems, the present invention proposes an innovative method for efficient transmission of light field images, which combines image super-resolution reconstruction and selective quality control mode and is suitable for bandwidth-constrained network environments. The method uses a series of optimization steps, including downsampling, super-resolution reconstruction, residual calculation, and selective quality control, to reasonably allocate bit rates to each subview of the light field image, thereby significantly reducing the amount of transmitted data and bandwidth requirements while ensuring the visual quality of the image.

[0006] Specifically, the light field image is first converted into a low-resolution image through downsampling to reduce the amount of data; then the image resolution is restored using super-resolution reconstruction technology to approximately restore the original high-resolution details. Then, the residual between the super-resolution reconstructed image and the original image is calculated to obtain the detail information lost during the reconstruction process, providing compensation for subsequent image reconstruction. Finally, through selective quality control, different bitrates are allocated to different subviews, giving priority to ensuring the quality of the area that the human eye focuses on, and using a lower bitrate to transmit the secondary view to optimize the use of bandwidth resources.

[0007] The present invention adopts the form of pseudo video sequence for transmission, which further improves the transmission efficiency. At the receiving end, the image is super-resolution processed after decoding, and a complete high-resolution light field image is reconstructed in combination with residual information to ensure the image transmission quality under low bandwidth conditions.

[0008] The innovation of this method lies in the introduction of three selective quality control modes, including center scaling mode, cross mode and cube center mode, which achieve a better balance between the transmission efficiency and visual quality of light field images. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic flow chart of a light field transmission method for integrating image super-resolution and selective quality mode proposed in the present invention.

[0010] Figure 2 Schematic diagram of the center-scaling selective quality mode proposed by the present invention.

[0011] Figure 3 A schematic diagram of the cross-amplification transmission selective quality mode proposed by the present invention.

[0012] Figure 4 Schematic diagram of the cubic center transmission selective mass mode proposed in the present invention. DETAILED DESCRIPTION

[0013] Reference Figure 1 The present invention proposes a light field transmission method integrating image super-resolution and selective quality mode, comprising:

[0014] Step S1: Downsampling. Downsample the original light field image (such as bicubic interpolation method, etc.) to generate a low-resolution image. The purpose of this step is to reduce the initial data volume of the image, thereby effectively reducing the data requirements for subsequent transmission.

[0015] Step S2: Super-resolution reconstruction. Use super-resolution reconstruction technology (such as image super-resolution based on deep learning, light field image super-resolution, etc.) to reconstruct the downsampled low-resolution image to restore the same resolution as the original light field, thereby obtaining a high-resolution image. Through this technology, the visual details of the image can be retained as much as possible under low data volume conditions, achieving an effect close to the original image.

[0016] Step S3: Residual calculation. Compare the high-resolution image with the original image and calculate the residual between the two. These residual information contains the detail data that may be lost during the super-resolution reconstruction process and provides compensation information for subsequent reconstruction to improve the reconstruction quality of the image.

[0017] Step S4: Selective quality control. Combined with video coding technology (such as HEVC, etc.), selective quality control is performed on different sub-views of the light field image. According to the three selective quality control modes we proposed that are in line with the visual habits of the human eye, higher bitrate resources are allocated to views with higher attention to ensure the image quality of key areas. At the same time, lower bitrate resources are allocated to secondary views, thereby saving bandwidth and improving overall transmission efficiency.

[0018] Step S5: Encoding and transmission. Use video coding technology (such as HEVC, etc.) to encode the downsampled image and residual information after selective quality control into a pseudo video sequence for transmission. This low-data-volume pseudo video sequence form can be effectively transmitted in a low-bandwidth network, further optimizing bandwidth occupancy.

[0019] Step S6: Decoding and reconstruction. At the receiving end, the pseudo video sequence is decoded to obtain the downsampled image and residual information. Subsequently, the downsampled image is super-reconstructed and added to the decoded residual to reconstruct a complete high-quality light field image. This reconstruction process can effectively restore the high-resolution details of the light field image in a bandwidth-constrained network environment.

[0020] Reference Figure 2 , the center scaling selective quality mode proposed by the present invention includes:

[0021] Based on the center zoom mode, high bit rate transmission is preferentially set for the central area of ​​the light field image, and low bit rate is set for the peripheral area to reduce the overall data volume. Figure 2 The black area is where the high bitrate sub-view is allocated, and the gray area is where the low bitrate sub-view is allocated.

[0022] Reference Figure 3 , the cross-selective quality mode proposed by the present invention includes:

[0023] Subviews are selected alternately along the horizontal and vertical axes, and these key views are transmitted at a high bit rate, while the remaining views are transmitted at a low bit rate. Figure 3 The black area is where the high bitrate sub-view is allocated, and the gray area is where the low bitrate sub-view is allocated.

[0024] Reference Figure 4 , the cube-centered selective quality mode proposed by the present invention includes:

[0025] The light field sub-view is divided into several local cube areas, and the central area of ​​each cube is selected for high bit rate transmission, while the peripheral area is transmitted at a low bit rate. Figure 4 The black area is where the high bitrate sub-view is allocated, and the gray area is where the low bitrate sub-view is allocated.

Claims

1. A light field transmission method integrating image super-resolution and selective quality mode, characterized in that: The following steps are involved: (a) Downsampling the original light field image to generate a low-resolution image; (b) using super-resolution reconstruction technology to process the downsampled image to restore it to the same resolution as the original light field image, generating a super-resolution light field image; (c) calculating a residual value between the super-resolution light field image and the original light field image; (d) Using a transmission mode based on video coding technology, selective quality control is performed on different sub-views of the light field, and a higher bit rate is allocated to the sub-views that conform to the viewing habits of the human eye, and a lower bit rate is allocated to other sub-views; (e) encoding the downsampled image and the residual into a pseudo video sequence for transmission through a selective quality transmission mode; (f) After transmission decoding, the decoded downsampled image is super-resolved and added to the decoded residual to reconstruct the complete light field image.

2. The light field transmission method according to claim 1, characterized in that: The super-resolution reconstruction technology is an image super-resolution model based on deep learning to ensure the image reconstruction effect under low bit rate transmission.

3. The light field transmission method according to claim 1 or 2, characterized in that: The selective quality control is achieved by allocating different bit rates using video coding technology to meet the requirements of different sub-views on human visual effects in light field image transmission.

4. The light field transmission method according to any one of claims 1 to 3, characterized in that: The encoding of the downsampled image and the residual adopts video encoding technology to effectively reduce the amount of data transmission.

5. The light field transmission method according to any one of claims 1 to 4, characterized in that: The light field image is compressed by video coding technology during transmission to reduce bandwidth occupancy, and the quality close to the original light field is restored during decoding.

6. The light field transmission method according to any one of claims 1 to 5, characterized in that: The selective quality mode can dynamically adjust the bit rate allocation of different sub-views according to the application scenario to further optimize the balance between transmission efficiency and image quality.

Images