Light field image transmission method based on angle attention estimation

Abstract

The invention discloses a light field image transmission method based on angle attention estimation. The light field image transmission method is used for remarkably reducing data volume through differential coding when a light field image is transmitted. The method comprises the following steps: firstly, collecting user eye movement data, and processing and analyzing the user eye movement data to obtain user angle attention data; secondly, an angle attention estimation model is constructed by using the user angle attention data, so that the weight distribution of different sub-views of the light field image can be generated by using the model; according to the weight distribution, a video coding technology is adopted to carry out differential coding on the sub-views of the light field image and convert the sub-views into a pseudo video sequence, a high code rate is allocated to the high-weight sub-views so as to ensure the visual quality of a key area, and a low code rate is allocated to the low-attention sub-views so as to reduce the bandwidth demand. And then the converted pseudo video sequence is transmitted. And at a receiving end, the differentially coded light field image is recovered through direct decoding operation, and additional reconstruction steps are not needed. The method not only significantly reduces the transmission data volume, but also ensures the visual effect of the user concerned area, and is suitable for various application scenes needing efficient light field image transmission, such as virtual reality and augmented reality.

Description

Technical Field

[0001] The present invention relates to the field of image processing and transmission technology, and in particular to a light field image transmission method based on angular attention estimation. The method aims to significantly reduce the amount of light field image transmission data through differential coding technology while ensuring the visual effect of the user's attention area. The method is suitable for various application scenarios requiring efficient light field image transmission, such as virtual reality (VR) and augmented reality (AR). Background Art

[0002] Light field images provide a richer visual experience than traditional two-dimensional images because they can record the distribution information of light in multiple directions in the scene, and they show great potential in applications such as VR and AR. However, light field images contain a large amount of data, and the transmission and storage costs are high, which limits their popularity in practical applications. Traditional light field image transmission methods often adopt a uniform coding strategy, performing the same encoding process on all subviews, which not only leads to low transmission efficiency but also wastes network bandwidth. In addition, when users watch light field images, they often only pay attention to certain specific areas and pay less attention to other areas. Therefore, it is particularly necessary to develop a differentiated transmission method that performs differential encoding according to the user's area of ​​interest to reduce the amount of data while ensuring visual quality. Summary of the invention

[0003] The present invention proposes a light field image transmission method based on angular attention estimation. The method predicts the user's attention distribution in different sub-views by constructing an angular attention estimation model, and differentially encodes the sub-views of the light field image accordingly, thereby significantly reducing the amount of transmitted data while ensuring the visual effect of the user's attention area. The technical solution of the present invention includes the following steps:

[0004] Collect and analyze the user's light field angle attention data: Through user interaction, eye tracking and other means, collect the user's eye movement data in different light field scenes, and perform preprocessing and analysis to obtain the user's angle attention data in different scenes. The angle attention data reflects the importance of different sub-views of the light field image.

[0005] Constructing a light field angle attention estimation model: Based on the collected user angle attention data, an angle attention estimation model is constructed. This model uses deep learning techniques (such as multi-head attention mechanism, convolutional neural network, etc.) to extract features from the user's light field angle attention data and construct an angle attention model to improve the prediction accuracy of the user's angle attention.

[0006] Generate weight distribution of light field image subviews: Input the light field image to be transmitted into the constructed light field angle attention estimation model to generate the weight distribution of each subview of the light field image. The weight distribution reflects the user's attention to each subview and is the basis for subsequent differential encoding.

[0007] Differential coding: Based on the generated weight distribution, the subviews of the light field image are differentially coded and converted into pseudo video sequences using video coding technology. Specifically, higher bit rates are assigned to subviews with higher weights to ensure the visual quality of these key areas; lower bit rates are assigned to subviews with lower weights to effectively reduce the amount of transmitted data while ensuring the user's viewing experience. The light field image is converted into a pseudo video sequence while performing differential coding.

[0008] Transmission and decoding: The converted pseudo video sequence is transmitted. At the receiving end, the received pseudo video sequence is directly decoded without any additional image reconstruction steps to restore the light field image subview with differentiated coding characteristics. The decoded light field image subview can be directly used in application scenarios such as VR and AR without additional image reconstruction or optimization processing, thereby improving the real-time transmission and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of the process of light field image transmission based on angle attention estimation proposed in the present invention. DETAILED DESCRIPTION

[0010] Reference Figure 1 The light field image transmission method based on angle attention estimation proposed by the present invention includes:

[0011] Step S1: Data collection and analysis. Collect the user's light field eye movement data through user interaction means (such as eye tracking). Preprocess and analyze the collected data to obtain the user's angular attention distribution in different light field images, that is, the importance of different sub-views.

[0012] Step S2: Construct an angle attention estimation model. Use the processed angle attention data to construct an angle attention estimation model. The model can use deep learning techniques (such as multi-head attention mechanism, convolutional neural network, etc.) to extract features from the user's angle attention data and predict the user's attention to different sub-views of different light field images. By training the model, the prediction accuracy of the user's angle attention is improved to better adapt to the user behavior in actual scenarios.

[0013] Step S3: Input the light field image to be transmitted into the constructed angle attention estimation model to generate a weight distribution for each subview of the light field image, which reflects the user's attention to each subview. The weight distribution provides a basis for differential coding, and different bit rates are assigned to different subviews according to the user's attention level to ensure the visual quality of key areas.

[0014] Step S4: Differential coding and pseudo video conversion. According to the generated weight distribution, the subviews of the light field image are differentially coded using video coding technology (such as HEVC, etc.). A higher bit rate is allocated to the subviews of high interest to maintain image quality, while a lower bit rate is used for the subviews of low interest to save bandwidth. At the same time, the light field subviews that have undergone differential coding are converted into pseudo video sequences for easy transmission.

[0015] Step S5: Transmission and decoding. The converted pseudo video sequence is transmitted. The received pseudo video sequence is directly decoded at the receiving end, and a light field image with differentiated coding characteristics can be restored without additional reconstruction steps. This method ensures real-time performance while also ensuring the visual quality of the user's focus area. It is suitable for application scenarios such as VR and AR that have high requirements for transmission efficiency and visual quality.

[0016] The method of the present invention reduces the amount of data transmitted by light field images by utilizing the user's angular attention and maintains high quality in the area that the user focuses on, thereby improving transmission efficiency, and is particularly suitable for scenarios such as virtual reality and augmented reality.

Claims

1. A light field image transmission method based on angular attention estimation, characterized in that: The following steps are involved: (a) Collect the user's eye movement data in different light field scenes and obtain the user's angular attention from it. The angular attention reflects the user's attention to each sub-view. The angular attention estimation model is constructed based on the user's angular attention; (b) using the angular attention estimation model to generate weight distributions of different sub-views of the light field image for estimating the area of ​​user attention; (c) according to the weight distribution, differentially encoding the sub-views of the light field image and converting them into pseudo video sequences, allocating a higher bit rate to the high-weight sub-views to ensure the visual quality of the user's attention area; and allocating a lower bit rate to the low-weight sub-views to reduce the amount of data transmission; (d) Transmitting the pseudo video sequence converted by differential coding. (3) Decode the pseudo video sequence and restore the light field image.

2. The light field image transmission method according to claim 1, characterized in that: There is no limit to the method of collecting the user's eye movement data. The user's angular attention is analyzed and extracted. The angular attention reflects the user's attention level to each sub-view.

3. The light field image transmission method according to claim 1, characterized in that: The angular attention estimation model is constructed using user eye movement attention data to obtain a more accurate user angular attention weight distribution.

4. The light field image transmission method according to claim 1, characterized in that: The differential encoding step uses video encoding technology (such as HEVC, JPEG, etc.) to encode sub-views with different weights separately to further reduce bandwidth requirements.

5. The light field image transmission method according to claim 1, characterized in that: The differentially encoded light field image is directly restored through a decoding operation at the receiving end without the need for an additional reconstruction step, thereby improving the decoding efficiency of the receiving end.

6. The light field image transmission method according to claim 1, characterized in that: The light field image transmission method is suitable for application scenarios such as virtual reality and augmented reality that require efficient light field image transmission, and achieves efficient bandwidth utilization while ensuring visual effects by optimizing coding allocation.

7. The light field image transmission method according to any one of claims 1 to 7, characterized in that: By allocating higher bit rates to high-weight sub-views of the user's attention area and lower bit rates to low-weight sub-views, the amount of transmitted data is effectively reduced, ensuring the image quality of the user's attention area.

Images