Light field image coding method based on sparse mode selection

Abstract

The invention discloses a light field image coding method based on sparse mode selection, which explores the influence of the number and position of sub-aperture images selected by sparse sampling on coding efficiency, and combines the sub-aperture images and the coding efficiency into a sparse mode; according to the method, multiple light field images are pre-coded, multiple sparse modes are used for sampling during pre-coding, multiple quantization steps are adopted during compression of a pseudo video sequence formed by sub-aperture images selected by sampling, and a sparse mode rapid selection scheme is designed on the basis. The method comprises the following steps of: quickly selecting a proper sparse mode by using content characteristics and a target code rate of a light field image to be coded, performing sparse sampling, and compressing a pseudo video sequence formed by sub-aperture images selected by sampling according to the target code rate; the residual error of the sub-aperture image with relatively low quality is transmitted as an enhancement layer, and the average quality and the quality consistency of the reconstructed light field image are compensated; the method has the advantages that a proper sparse mode can be selected under different target code rates, and the coding efficiency is improved.

Description

Technical Field

[0001] The present invention relates to a light field image coding technology, and in particular to a light field image coding method based on sparse pattern selection. Background Art

[0002] Light field imaging technology records both the spatial information and the angular information of light. Light field images are usually represented as an array of sub-aperture images, and each sub-aperture image can be regarded as an image taken from a different perspective. Rich scene information supports light field image processing such as refocusing, depth estimation, and target detection, but the amount of light field image data is huge and needs to be efficiently compressed.

[0003] In recent years, light field image coding methods based on sparse reconstruction have received increasing attention. This type of method usually first performs sparse sampling on the sub-aperture image array, selects part of the sub-aperture image for compression and transmission, and uses the reconstruction algorithm at the decoder to synthesize the complete decoded image. Through the sparse sampling operation, the amount of data that needs to be transmitted can be reduced, thereby reducing the bit rate consumption. Combined with the reconstruction algorithm at the decoder, the coding performance can be improved within a certain range.

[0004] Although the light field image coding method based on sparse reconstruction has achieved certain light field image compression effects, there are still deficiencies in the research on the number and position of sparse sampling. In particular, there is still room for improvement in improving the coding efficiency by changing the sparse sampling method. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a light field image coding method based on sparse pattern selection, which can select a suitable sparse sampling method, namely a sparse pattern, under different target bit rates, thereby achieving improved coding efficiency.

[0006] The technical solution adopted by the present invention to solve the above technical problem is: a light field image encoding method based on sparse pattern selection, characterized by comprising the following steps:

[0007] Step 1: select Y light field images as the first light field images, where Y>1, and the angular resolution of the Y first light field images is consistent; then pre-encode each first light field image, and in the pre-encoding process, use X preset sparse modes to sparsely sample the sub-aperture image array represented by the first light field image, and use Z different quantization steps to compress the pseudo video sequence composed of the sub-aperture images selected during sparse sampling, and decode and reconstruct X×Z reconstructed sub-aperture image arrays corresponding to each first light field image, where the sparse mode is defined as a combination of the number and position of the sub-aperture images selected when sparse sampling is performed on the sub-aperture image array. For a fixed angular resolution, a total of X sparse modes are preset, which are marked as M1, M2, M3, ..., M X , Z>3;

[0008] Step 2: On the basis of step 1, a two-dimensional coordinate system corresponding to each first light field image and having X rate-distortion curves drawn thereon is obtained, wherein the abscissa of the two-dimensional coordinate system is the bit rate, the ordinate is the average quality, and a rate-distortion curve is a polyline obtained by connecting Z points located on the two-dimensional coordinate system by the bit rates representing the sizes of the Z coded streams and the average quality of all sub-aperture images in the Z reconstructed sub-aperture image arrays obtained by decoding and reconstruction when a sub-aperture image array represented by a first light field image is sparsely sampled using a sparse pattern and then a pseudo video sequence composed of sub-aperture images selected during sparse sampling is compressed using Z quantization step sizes;

[0009] Step 3: In a two-dimensional coordinate system corresponding to a first light field image and having X rate-distortion curves drawn thereon, the region where the X rate-distortion curves are located is divided into K rate intervals based on the horizontal axis, and the entire rate-distortion curves at the top position in each of the two adjacent rate intervals do not belong to the same rate-distortion curve. There are K-1 interval dividing points on the horizontal axis for the K rate intervals, where K is greater than or equal to N; then the sparse mode corresponding to the rate-distortion curve to which the entire rate-distortion curve at the top position in each rate interval belongs is taken as the best sparse mode under this rate interval, and the mark of the best sparse mode is recorded; then the marks of the best sparse modes under the K rate intervals are traversed in order of the code rates on the horizontal axis from small to large. If a mark appears again during the traversal process that is the same as one of the marks before this mark, all the rate intervals in the two rate intervals corresponding to the same mark are merged into one rate interval, and the best sparse mode with this mark is recorded. The pattern is used as the best sparse pattern under the merged code rate interval, and only this mark is retained for all marks between the same marks; after all marks are traversed, all retained marks are unique, and the number of code rate intervals N that finally exist is counted. If the number N is equal to X, the X-1 interval demarcation points corresponding to the X code rate intervals that finally exist are stored as a group, and then step 4 is executed; if the number N is less than X, XN code rate intervals are added between the first code rate interval and the second code rate interval that finally exist, so that the code rate interval is increased to X, and the X-1 interval demarcation points corresponding to the X code rate intervals are stored as a group, and then step 4 is executed, wherein the length of each added code rate interval is 0, and the interval demarcation point related to the added code rate interval is the same as the interval demarcation point that originally separated the first code rate interval and the second code rate interval that finally exist. The marks of the best sparse pattern under the added XN code rate intervals are {M1, M2, M3, …, M X}\Ω, Ω represents the set of the best sparse patterns under the final bit rate range, {M1, M2, M3, …, M X}\Ω represents the set {M1, M2, M3, …, M X} excludes the set Ω;

[0010] Step 4: Repeat step 3. After all the first light field images are processed, a total of Y groups of interval demarcation points are stored, each group of interval demarcation points has X-1 interval demarcation points, and the variance of all pixel values ​​in the central sub-aperture image in the sub-aperture image array represented by the first light field image corresponding to each group of interval demarcation points is calculated;

[0011] Step 5: Select a light field image to be encoded as the second light field image, the angular resolution of the second light field image is consistent with the angular resolution of the first light field image; then estimate X-1 interval demarcation points corresponding to the second light field image according to the Y groups of interval demarcation points stored in step 4 and the Y variances calculated;

[0012] Step 6: Given the target bit rate R t Under the premise of X-1 interval demarcation points corresponding to the second light field image, a suitable sparse pattern is quickly selected, marked as M c , Among them, P1, P2, P3, ..., P X-1 are X-1 interval demarcation points corresponding to the second light field image;

[0013] Step 7: Use the M c The sparse pattern of the second light field image is sparsely sampled by the sub-aperture image array represented by , and the target bit rate R is calculated. t The pseudo video sequence composed of the sub-aperture images selected during sparse sampling is compressed, and then decoded and reconstructed to obtain the reconstructed sub-aperture image array corresponding to the second light field image, and then the base layer reconstructed light field image L is obtained. rec1 ;

[0014] Step 8: Preset a threshold S; if L rec1 If the variance of the average quality of all sub-aperture images in the sub-aperture image array represented by is greater than S, the enhancement layer code stream is obtained and transmitted, and then decoded and reconstructed to obtain the enhancement layer reconstructed light field image L rec2 , then execute step 9; if L rec1 If the variance of the average quality of all sub-aperture images in the sub-aperture image array represented by is less than or equal to S, the encoding ends;

[0015] Step 9: If L rec2 If the variance of the average quality of all sub-aperture images in the sub-aperture image array is greater than S, the quantization step size is continuously reduced to re-acquire the enhancement layer code stream and transmit and decode and reconstruct it until L rec2 The variance of the average quality of all sub-aperture images in the sub-aperture image array denoted as is less than or equal to S.

[0016] In step 1, the specific process of pre-coding any first light field image is as follows:

[0017] At the encoding end, the first light field image is represented as a sub-aperture image array; then, the sub-aperture image array is sparsely sampled using X preset sparse modes; then, for each sparse mode, the sub-aperture images selected during sparse sampling are arranged in a preset fixed order to form a pseudo video sequence; finally, the pseudo video sequence obtained for each sparse mode is input into a standard video encoder for compression, and Z different quantization step sizes are used during compression to obtain Z coded bit streams with different bit rates; for the X preset sparse modes, there are a total of X×Z coded bit streams;

[0018] At the decoding end, a standard video decoder is used to decode the X×Z coded streams respectively, and each coded stream is decoded to obtain a compressed sub-aperture image; then, based on the compressed sub-aperture image obtained by decoding each coded stream, a reconstruction algorithm or a video interpolation network is used to synthesize the remaining sub-aperture images to reconstruct a complete reconstructed sub-aperture image array.

[0019] The specific process of step 2 is:

[0020] Step 2.1: When the sub-aperture image array represented by the y-th first light field image is sparsely sampled using the x-th sparse pattern and then the pseudo video sequence composed of the sub-aperture images selected during sparse sampling is compressed using the z-th quantization step size, the bit rate representing the size of the encoded bit stream is recorded as R(y,x,z), and the average quality of all sub-aperture images in the reconstructed sub-aperture image array obtained by decoding and reconstruction is recorded as D(y,x,z), where y=1,2,…,Y, x=1,2,…,X, and z=1,2,…,Z;

[0021] Step 2.2: Fix y and x to obtain Z groups of bit rate and average quality combinations, with bit rate as the horizontal coordinate and average quality as the vertical coordinate, locate Z points in the two-dimensional coordinate system; then connect the located Z points with straight lines in the order of bit rate from small to large, to obtain a broken line, and use this broken line as the rate-distortion curve obtained by compressing the pseudo video sequence composed of the sub-aperture images selected during sparse sampling with the xth sparse mode array represented by the yth first light field image;

[0022] Step 2.3: Traverse each sparse pattern and draw a rate-distortion curve in the same two-dimensional coordinate system according to the process of step 2.2. After traversing X sparse patterns, a total of X rate-distortion curves are drawn, that is, a two-dimensional coordinate system corresponding to a first light field image and having X rate-distortion curves drawn therein is obtained.

[0023] In step 5, the process of estimating the X-1 interval demarcation points corresponding to the second light field image is as follows:

[0024] Step 5.1: Represent the second light field image as a sub-aperture image array; then calculate the variance of all pixel values ​​in the central sub-aperture image in the sub-aperture image array, denoted as V in ;

[0025] Step 5.2: Arrange the Y variances calculated in step 4 in order of size. in If it is between these variances, then find the ratio V in Small nearest neighbor variance V L and V in Large nearest neighbor variance V H , then press P in =(V in -V L ) / (V H -V L )×(P H -P L )+P L Calculate the X-1 interval demarcation points corresponding to the second light field image, where P in represents the vector composed of the X-1 interval demarcation points corresponding to the second light field image in sequence, P H Indicates V H The vector formed by the X-1 interval demarcation points corresponding to the first light field image, P L Indicates V L The vectors formed by the X-1 interval demarcation points corresponding to the first light field image in sequence; if these variances are all less than V in , then the X-1 interval demarcation points corresponding to the first light field image corresponding to the maximum variance among these variances are directly used as the X-1 interval demarcation points corresponding to the second light field image; if these variances are greater than or equal to V in , then the X-1 interval demarcation points corresponding to the first light field image corresponding to the minimum variance among these variances are directly used as the X-1 interval demarcation points corresponding to the second light field image.

[0026] The specific process of step 7 is as follows:

[0027] On the encoding side, the c The sparse pattern of the second light field image is represented as the sub-aperture image array L org Perform sparse sampling; then select the sub-aperture image S during sparse sampling sel Arrange them in a preset fixed order to form a pseudo video sequence PVS1; then according to the target bit rate R t The pseudo video sequence PVS1 is input into a standard video encoder for compression, and a base layer bit stream bits1 is obtained by compression;

[0028] At the decoding end, bits1 is decoded using a standard video decoder to obtain the compressed sub-aperture image S s ' el Then, based on the compressed sub-aperture image, the remaining sub-aperture images are synthesized using a reconstruction algorithm or a video interpolation network to reconstruct a complete reconstructed sub-aperture image array, and then the reconstructed light field image is obtained as the base layer reconstructed light field image L rec1 , wherein the technique used to synthesize the remaining sub-aperture images is the same as the technique used to synthesize the remaining sub-aperture images in step 1.

[0029] In step 8, the enhancement layer code stream is obtained and transmitted, and then decoded and reconstructed to obtain the enhancement layer reconstructed light field image L rec2 The process is:

[0030] Step 8.1: Get L rec1 The average quality of all sub-aperture images in the sub-aperture image array is expressed as; then from L rec1 In the sub-aperture image array represented as , all sub-aperture images with quality lower than the average quality are selected as enhancement objects;

[0031] Step 8.2: Calculate the sub-aperture image array L for each enhanced object and the second light field image represented as org The residual images of the sub-aperture images with the same index in the image are obtained; and all the calculated residual images D are arranged in a preset order to form a pseudo video sequence PVS2;

[0032] Step 8.3: Input the pseudo video sequence PVS2 into the standard video encoder for compression to obtain the enhanced layer bit stream bits2;

[0033] Step 8.4: Decode bits2 with a standard video decoder to obtain a compressed residual image D'; then add the compressed residual image to the corresponding enhanced object to obtain the enhanced layer reconstructed light field image L rec2 .

[0034] In step 1, the difference between the maximum quantization step size and the minimum quantization step size among the Z different quantization step sizes is greater than three quarters of the difference between the upper limit and the lower limit of the quantization step size range that can be selected by the standard video encoder used when compressing the pseudo video sequence, and the Z different quantization step sizes are an arithmetic progression.

[0035] Compared with the prior art, the advantages of the present invention are:

[0036] The method of the present invention explores the influence of the number and position of sub-aperture images selected by sparse sampling on the coding efficiency, defines the combination of the two as a sparse pattern, and designs a sparse pattern fast selection scheme, which uses the content characteristics of the light field image to be encoded and the target bit rate to quickly select a suitable sparse pattern. In addition, the residual of the sub-aperture image with lower quality is transmitted as an enhancement layer to compensate for the average quality and quality consistency of the reconstructed light field image. Experimental results show that the method of the present invention can select the best sparse pattern more accurately and improve the coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flowchart of the encoding of the light field image to be encoded in the method of the present invention;

[0038] Figure 2 Schematic diagram of the six preset sparse modes, where the dark part indicates the number and position of sub-aperture images selected for sparse sampling. DETAILED DESCRIPTION

[0039] The present invention is further described in detail below with reference to the accompanying drawings.

[0040] The present invention proposes a light field image encoding method based on sparse pattern selection, which comprises the following steps:

[0041] Step 1: select Y light field images as the first light field images, where Y>1, and in this embodiment, Y=10, and the angular resolutions of the Y first light field images are consistent, such as the angular resolutions are all 9×9; then pre-encode each first light field image, and in the pre-encoding process, use X preset sparse modes to sparsely sample the sub-aperture image array represented by the first light field image, and use Z different quantization steps to compress the pseudo video sequence composed of the sub-aperture images selected during sparse sampling, and decode and reconstruct to obtain X×Z reconstructed sub-aperture image arrays corresponding to each first light field image, where the sparse mode is defined as a combination of the number and position of the sub-aperture images selected when sparse sampling is performed on the sub-aperture image array. The sparse mode is also called a sparse sampling method. For a fixed angular resolution, a total of X sparse modes are preset, which are marked as M1, M2, M3, ..., M X , for example, when the fixed angle resolution is 9×9, there are 6 preset sparse modes. Figure 2 Six preset sparse modes are shown, marked as M1, M2, M3, M4, M5, and M6. There are four sub-aperture images selected in the sparse mode M1. The positions of the four sub-aperture images in the sub-aperture image array are shown in Figure 1. Figure 2As shown, sparse mode M2 ​​adds a central sub-aperture image based on sparse mode M1, sparse mode M3 adds 4 sub-aperture images at different positions based on sparse mode M2, sparse mode M4 adds 4 sub-aperture images at different positions based on sparse mode M3, sparse mode M5 adds 12 sub-aperture images at different positions based on sparse mode M4, and the sub-aperture images selected in sparse mode M6 are all sub-aperture images, Z>3, and in this embodiment, Z=9, and the 9 different quantization step sizes are 7, 12, 17, 22, 27, 32, 37, 42 and 47 respectively.

[0042] In this specific embodiment, in step 1, the specific process of pre-encoding any first light field image is as follows:

[0043] At the encoding end, the first light field image is represented as a sub-aperture image array; then, the sub-aperture image array is sparsely sampled using X preset sparse modes; then, for each sparse mode, the sub-aperture images selected during sparse sampling are arranged in a preset fixed order to form a pseudo video sequence; finally, the pseudo video sequence obtained for each sparse mode is input into a standard video encoder for compression, and Z different quantization step sizes are used during compression to obtain Z coded bitstreams with different bit rates; for the X preset sparse modes, there are a total of X×Z coded bitstreams.

[0044] At the decoding end, a standard video decoder is used to decode the X×Z coded streams respectively, and each coded stream is decoded to obtain a compressed sub-aperture image; then, based on the compressed sub-aperture image obtained by decoding each coded stream, the existing reconstruction algorithm or the existing video interpolation network is used to synthesize the remaining sub-aperture images to reconstruct a complete reconstructed sub-aperture image array. Here, any existing reconstruction algorithm or video interpolation network can be used to synthesize the remaining sub-aperture images. In the actual implementation process, a reconstruction algorithm or video interpolation network with good synthesis effect can be selected.

[0045] In this specific embodiment, in step 1, the difference between the maximum quantization step size and the minimum quantization step size among the Z different quantization step sizes is greater than three quarters of the difference between the upper limit and the lower limit of the quantization step size range that can be selected by the standard video encoder used when compressing the pseudo video sequence, and the Z different quantization step sizes are an arithmetic progression. In actual implementation, the difference between the maximum quantization step size and the minimum quantization step size among the Z different quantization step sizes can be limited according to the situation, which is generally greater than one half to one of the difference between the upper limit and the lower limit of the quantization step size range that can be selected by the standard video encoder.

[0046] Step 2: Based on step 1, a two-dimensional coordinate system corresponding to each first light field image and having X rate-distortion curves drawn thereon is obtained, wherein the abscissa of the two-dimensional coordinate system is the bit rate, the ordinate is the average quality, and a rate-distortion curve is a broken line obtained by connecting Z points located on the two-dimensional coordinate system by the bit rates representing the sizes of the Z coded streams and the average quality of all sub-aperture images in the Z reconstructed sub-aperture image arrays obtained by decoding and reconstruction when a sub-aperture image array represented by a first light field image is sparsely sampled using a sparse pattern and then a pseudo video sequence composed of sub-aperture images selected during sparse sampling is compressed using Z quantization steps.

[0047] In this specific embodiment, the specific process of step 2 is:

[0048] Step 2.1: When the sub-aperture image array represented by the y-th first light field image is sparsely sampled using the x-th sparse pattern and then the z-th quantization step size is used to compress the pseudo video sequence composed of the sub-aperture images selected during sparse sampling, the bit rate representing the size of the encoded bit stream is denoted as R(y,x,z), and the average quality of all sub-aperture images in the reconstructed sub-aperture image array obtained by decoding and reconstruction is denoted as D(y,x,z), where y=1,2,…,Y, x=1,2,…,X, and z=1,2,…,Z.

[0049] Step 2.2: Fix y and x to obtain Z groups of combinations of bit rate and average quality, with bit rate as the horizontal coordinate and average quality as the vertical coordinate, locate Z points in the two-dimensional coordinate system; then connect the located Z points with straight lines in the order of bit rate from small to large, to obtain a broken line, and this broken line is used as the rate-distortion curve obtained by compressing the pseudo video sequence composed of the sub-aperture images selected during sparse sampling with the xth sparse mode and then using Z quantization steps.

[0050] Step 2.3: Traverse each sparse pattern and draw a rate-distortion curve in the same two-dimensional coordinate system according to the process of step 2.2. After traversing X sparse patterns, a total of X rate-distortion curves are drawn, that is, a two-dimensional coordinate system corresponding to a first light field image and having X rate-distortion curves drawn therein is obtained.

[0051] Step 3: In a two-dimensional coordinate system corresponding to a first light field image and having X rate-distortion curves drawn thereon, the region where the X rate-distortion curves are located is divided into K rate intervals based on the horizontal axis, and the entire rate-distortion curves at the top position in each of two adjacent rate intervals do not belong to the same rate-distortion curve. For the K rate intervals, there are K-1 interval demarcation points on the horizontal axis, and the first rate interval and the second rate interval are separated by the first interval demarcation point, and so on. The K-1th rate interval and the Kth rate interval are separated by the K-1th interval demarcation point, where K is greater than or equal to N; then the rate-distortion curve to which the entire rate-distortion curve at the top position in each rate interval belongs is calculated. The sparse mode corresponding to the curve is used as the best sparse mode under this bit rate interval, and the mark of the best sparse mode is recorded; then the marks of the best sparse modes under K bit rate intervals are traversed in order from small to large on the horizontal axis. If a mark appears again during the traversal process that is the same as one of the marks before this mark, all the bit rate intervals within the two bit rate intervals corresponding to the same mark are merged into one bit rate interval, and the best sparse mode with this mark is used as the best sparse mode under the merged bit rate interval. At the same time, only this mark is retained for all marks between the same marks, such as: K = 10, the marks of the best sparse modes under 10 bit rate intervals are M1, M2, M1, M2, M3, M2, M3, M4, M5, M6. After the intervals are merged, 6 code rate intervals are obtained, and the marks of the best sparse modes under the 6 code rate intervals are M1, M2, M3, M4, M5, M6 respectively. For example, K=8, the marks of the best sparse modes under the 8 code rate intervals are M3, M1, M3, M2, M4, M2, M5, M6 respectively. After the intervals are merged, 4 code rate intervals are obtained, and the marks of the best sparse modes under the 4 code rate intervals are M3, M2, M5, M6 respectively. After all the marks are traversed, all the retained marks are unique. The number N of the code rate intervals that finally exist is counted. If the number N is equal If the number N is less than X, then XN code rate intervals are added between the first code rate interval and the second code rate interval to increase the number of code rate intervals to X, and the X-1 interval demarcation points corresponding to the X code rate intervals are stored as a group, and then step 4 is executed, wherein the length of each added code rate interval is 0, the interval demarcation point associated with the added code rate interval is the same as the interval demarcation point that originally separated the first code rate interval and the second code rate interval, and the labels of the best sparse modes under the added XN code rate intervals are {M1, M2, M3, ..., M X}\Ω, Ω represents the set of the best sparse patterns under the final bit rate range, {M1, M2, M3, …, M X}\Ω represents the set {M1, M2, M3, …, M X} excludes the set Ω.

[0052] Step 4: Repeat step 3. After all the first light field images are processed, a total of Y groups of interval demarcation points are stored, each group of interval demarcation points has X-1 interval demarcation points, and the variance of all pixel values ​​in the central sub-aperture image in the sub-aperture image array represented by the first light field image corresponding to each group of interval demarcation points is calculated.

[0053] Step 5: Select a light field image to be encoded as the second light field image, and the angular resolution of the second light field image is consistent with the angular resolution of the first light field image; then estimate the X-1 interval demarcation points corresponding to the second light field image according to the Y groups of interval demarcation points stored in step 4 and the Y variances calculated.

[0054] In this specific embodiment, in step 5, the process of estimating the X-1 interval demarcation points corresponding to the second light field image is as follows:

[0055] Step 5.1: Represent the second light field image as a sub-aperture image array; then calculate the variance of all pixel values ​​in the central sub-aperture image in the sub-aperture image array, denoted as V in .

[0056] Step 5.2: Arrange the Y variances calculated in step 4 in order of size. in If it is between these variances, then find the ratio V in Small nearest neighbor variance V L and V in Large nearest neighbor variance V H , then press P in =(V in -V L ) / (V H -V L )×(P H -P L )+P L Calculate the X-1 interval demarcation points corresponding to the second light field image, where P in represents the vector composed of the X-1 interval demarcation points corresponding to the second light field image in sequence, P H Indicates V H The vector formed by the X-1 interval demarcation points corresponding to the first light field image, P L Indicates V L The vectors formed by the X-1 interval demarcation points corresponding to the first light field image in sequence; if these variances are all less than V in, then the X-1 interval demarcation points corresponding to the first light field image corresponding to the maximum variance among these variances are directly used as the X-1 interval demarcation points corresponding to the second light field image; if these variances are greater than or equal to V in , then the X-1 interval demarcation points corresponding to the first light field image corresponding to the minimum variance among these variances are directly used as the X-1 interval demarcation points corresponding to the second light field image.

[0057] The flow chart of encoding the light field image to be encoded is as follows: Figure 1 As shown, the implementation steps are steps 6 to 9.

[0058] Step 6: Given the target bit rate R t Under the premise of X-1 interval demarcation points corresponding to the second light field image, a suitable sparse pattern is quickly selected, marked as M c , Among them, P1, P2, P3, ..., P X-1 are the X-1 interval dividing points corresponding to the second light field image.

[0059] Step 7: Use the M c The sparse pattern of the second light field image is sparsely sampled by the sub-aperture image array represented by , and the target bit rate R is calculated. t The pseudo video sequence composed of the sub-aperture images selected during sparse sampling is compressed, and then decoded and reconstructed to obtain the reconstructed sub-aperture image array corresponding to the second light field image, and then the base layer reconstructed light field image L is obtained. rec1 .

[0060] In this specific embodiment, the specific process of step 7 is:

[0061] On the encoding side, the c The sparse pattern of the second light field image is represented as the sub-aperture image array L org Perform sparse sampling; then select the sub-aperture image S during sparse sampling sel Arrange them in a preset fixed order to form a pseudo video sequence PVS1; then according to the target bit rate R t The pseudo video sequence PVS1 is input into a standard video encoder for compression, and a base layer bit stream bits1 is obtained by compression.

[0062] At the decoding end, bits1 is decoded using a standard video decoder to obtain the compressed sub-aperture image S s ' el Then, based on the compressed sub-aperture image, the remaining sub-aperture images are synthesized using an existing reconstruction algorithm or an existing video interpolation network to reconstruct a complete reconstructed sub-aperture image array, and then the reconstructed light field image is obtained as the base layer reconstructed light field image L rec1, wherein the technique used to synthesize the remaining sub-aperture images is the same as the technique used to synthesize the remaining sub-aperture images in step 1.

[0063] Step 8: Preset a threshold value S. In this embodiment, S=1. If L rec1 If the variance of the average quality of all sub-aperture images in the sub-aperture image array represented by is greater than S, the enhancement layer code stream is obtained and transmitted, and then decoded and reconstructed to obtain the enhancement layer reconstructed light field image L rec2 , then execute step 9; if L rec1 If the variance of the average quality of all sub-aperture images in the sub-aperture image array represented as is less than or equal to S, the encoding ends.

[0064] In this specific embodiment, in step 8, the enhancement layer code stream is obtained and transmitted, and then decoded and reconstructed to obtain the enhancement layer reconstructed light field image L rec2 The process is:

[0065] Step 8.1: Get L rec1 The average quality of all sub-aperture images in the sub-aperture image array is expressed as; then from L rec1 In the sub-aperture image array represented as , all sub-aperture images with quality lower than the average quality are selected as enhancement objects.

[0066] Step 8.2: Calculate the sub-aperture image array L for each enhanced object and the second light field image represented as org The residual map of the sub-aperture image with the same index in the image is obtained; and then all the calculated residual maps D are arranged in a preset order to form a pseudo video sequence PVS2.

[0067] Step 8.3: Input the pseudo video sequence PVS2 into the standard video encoder for compression to obtain the enhancement layer bit stream bits2.

[0068] Step 8.4: Decode bits2 with a standard video decoder to obtain a compressed residual image D'; then add the compressed residual image to the corresponding enhanced object to obtain the enhanced layer reconstructed light field image L rec2 .

[0069] Step 9: If L rec2 If the variance of the average quality of all sub-aperture images in the sub-aperture image array is greater than S, the quantization step size is continuously reduced to re-acquire the enhancement layer code stream and transmit and decode and reconstruct it until L rec2 The variance of the average quality of all sub-aperture images in the sub-aperture image array represented by is less than or equal to S. In specific implementation, L rec2 When the variance of the average quality of is greater than S, reduce the quantization step size and follow the process in step 8 to calculate the average quality of L. rec2Based on the above, the enhancement layer code stream is re-acquired and transmitted, and then decoded and reconstructed to obtain the enhancement layer reconstructed light field image, which is re-recorded as L rec2 , then judge L rec2 Is the variance of the average quality greater than S? As long as it is greater than S, the quantization step size is continuously reduced and repeated. rec2 The encoding ends when the variance of the average quality of all sub-aperture images in the sub-aperture image array represented as is less than or equal to S.

[0070] In this embodiment, the standard video encoder used should be consistent, and the selected standard video encoder is the HEVC encoder; the standard video decoder used should be consistent, and the selected standard video decoder is the HEVC decoder.

[0071] In order to further illustrate the performance of the method of the present invention, the method of the present invention was tested.

[0072] Four real scenes "Bikes", "Danger de Mort", "StonePillars Outside", and "Fountain&Vincent 2" were selected from the EPFL light field database, and they were recorded as "I01", "I02", "I03", and "I04" respectively. Four synthetic scenes "Herbs", "Origami", "Tower", and "Town" were selected from the HCI light field database, and they were recorded as "I05", "I06", "I07", and "I08" respectively. The experiments were carried out in the HEVC reference software HM16.20. The four target bit rates R1, R2, R3, and R4 are 0.005bpp, 0.02bpp, 0.1bpp, and 0.75bpp respectively.

[0073] The method of the present invention is compared with three existing light field image coding methods to evaluate the effectiveness of the method. The first method is the rotation scanning method, in which all sub-aperture images are first scanned into a pseudo video sequence in a spiral order and then compressed by the HEVC encoder, referred to as RSM. This method has been published in DAI F, ZHANG J, MAY, et al. Lenslet image compression scheme based on subaperture images streaming[C] / / 2015 IEEE International Conference on Image Processing(ICIP), Quebec City, Canada, 2015:4733-4737. The second method is the context adaptive coding method, in which all sub-aperture images are adaptively arranged into a pseudo video sequence according to content similarity and then compressed by the HEVC encoder, referred to as CAE. This method has been published in CONCEICAO R, PORTO M, ZATT B, et al. LF-CAE: Context-adaptive encoding for lenslet light fields using HEVC[C] / / 2018 25th IEEE International Conference on Image Processing(ICIP), Athens, Greece, 2018:3174-3178. (Context-Adaptive Coding of Lenslet Light Field Using HEVC[C] / / 2018 25th IEEE International Conference on Image Processing(ICIP), Athens, Greece, 2018:3174-3178.) is disclosed; the third method is an exhaustive search method. When selecting a sparse mode, the sparse mode with the highest coding efficiency is selected by an exhaustive search method, referred to as BFS. This method is the method of the present invention without performing a pre-coding process, and only executing steps 7 to 9, and the step 7 is marked as M. c The sparsity pattern is M1, M2, M3, …, M X Various sparse patterns are traversed and executed once, while the method of the present invention directly selects a suitable sparse pattern through step 6, which can save time and computing resources.

[0074] Table 1 lists the sparse modes selected by the method of the present invention and the BFS method respectively under four target bit rates.

[0075] Table 1 Sparse modes selected by the method of the present invention and the BFS method respectively under four target bit rates

[0076]

[0077] Since the BFS method traverses all sparse patterns each time it encodes, the selected sparse pattern must be the one with the best rate-distortion performance. The method of the present invention first estimates all interval demarcation points corresponding to the light field image to be encoded, and then combines the variance of all pixel values ​​of the central sub-aperture image to quickly select a suitable sparse pattern. As can be seen from Table 1, at medium and low bit rates, the sparse pattern selected by the method of the present invention has a slight deviation, and at high bit rates, the best sparse pattern can be selected more accurately.

[0078] Table 2 shows the rate-distortion performance comparison between the method of the present invention and three existing light field image coding methods.

[0079] Table 2 Comparison of rate-distortion performance between the method of the present invention and three existing light field image coding methods

[0080]

[0081] Common BD-BR and BD-PSNR indicators are used to objectively measure the coding efficiency of the method of the present invention. Negative BD-BR represents the bit rate that the method of the present invention can save compared with the comparison method at the same quality. Positive BD-PSNR represents the quality that the method of the present invention can improve compared with the comparison method at the same bit rate. As can be seen from Table 2, compared with the RSM method, the method of the present invention can achieve an average 43.504% BD-BR saving and 1.476dB BD-PSNR improvement. Compared with the CAE method, the method of the present invention can achieve an average 39.559% BD-BR saving and 1.312dB BD-PSNR improvement. This is due to the fact that sparse sampling reduces the amount of data to be encoded. Compared with the BFS method, the method of the present invention increases the BD-BR consumption by 2.911% and the BD-PSNR decreases by 0.209dB on average. This is because there is an error in the determination of the sparse mode by the method of the present invention, but it is acceptable at the cost of losing some coding performance in exchange for a large reduction in coding time. In addition, in the I05 and I06 scenarios, the coding performance of the method of the present invention is higher than that of the BFS method. This is because when encoding these two scenarios, the enhancement layer code stream of the method of the present invention is larger, and the sub-aperture images with poor quality are accurately enhanced. It can be seen that quality enhancement can also improve coding efficiency to a certain extent.

[0082] Table 3 gives the variance of the average quality of the sub-aperture image array represented as before and after the reconstructed light field image enhancement, and the bold part indicates that the quality has been enhanced.

[0083] Table 3 Variance of the average quality of the sub-aperture image array represented by before and after the reconstructed light field image enhancement

[0084]

[0085] It can be seen from Table 3 that at low bit rates, the base layer reconstructed light field image maintains good quality consistency. As the bit rate increases, the impact of reconstruction distortion gradually increases, and the quality consistency of the base layer reconstructed light field image gradually decreases. After transmitting the enhancement layer bitstream, the quality variance of the enhancement layer reconstructed light field image is successfully controlled below 1. For R4, since the quality of the sub-aperture image in the base layer reconstructed light field image is good, even if there is a numerical fluctuation of the quality index, it is difficult for the human eye to detect, so no quality enhancement is performed.

Claims

1. A light field image coding method based on sparse pattern selection, characterized in that The following steps are involved: Step 1: select Y light field images as the first light field images, where Y>1, and the angular resolution of the Y first light field images is consistent; then pre-encode each first light field image, and in the pre-encoding process, use X preset sparse modes to sparsely sample the sub-aperture image array represented by the first light field image, and use Z different quantization steps to compress the pseudo video sequence composed of the sub-aperture images selected during sparse sampling, and decode and reconstruct X×Z reconstructed sub-aperture image arrays corresponding to each first light field image, where the sparse mode is defined as a combination of the number and position of the sub-aperture images selected when sparse sampling is performed on the sub-aperture image array. For a fixed angular resolution, a total of X sparse modes are preset, which are marked as M1, M2, M3, ..., M X , Z>3; Step 2: On the basis of step 1, a two-dimensional coordinate system corresponding to each first light field image and having X rate-distortion curves drawn thereon is obtained, wherein the abscissa of the two-dimensional coordinate system is the bit rate, the ordinate is the average quality, and a rate-distortion curve is a polyline obtained by connecting Z points located on the two-dimensional coordinate system by the bit rates representing the sizes of the Z coded streams and the average quality of all sub-aperture images in the Z reconstructed sub-aperture image arrays obtained by decoding and reconstruction when a sub-aperture image array represented by a first light field image is sparsely sampled using a sparse pattern and then a pseudo video sequence composed of sub-aperture images selected during sparse sampling is compressed using Z quantization step sizes; Step 3: In a two-dimensional coordinate system corresponding to a first light field image and having X rate-distortion curves drawn thereon, the region where the X rate-distortion curves are located is divided into K rate intervals based on the horizontal axis, and the entire rate-distortion curves at the top position in each of the two adjacent rate intervals do not belong to the same rate-distortion curve. There are K-1 interval dividing points on the horizontal axis for the K rate intervals, where K is greater than or equal to N; then the sparse mode corresponding to the rate-distortion curve to which the entire rate-distortion curve at the top position in each rate interval belongs is taken as the best sparse mode under this rate interval, and the mark of the best sparse mode is recorded; then the marks of the best sparse modes under the K rate intervals are traversed in order of the code rates on the horizontal axis from small to large. If a mark appears again during the traversal process that is the same as one of the marks before this mark, all the rate intervals in the two rate intervals corresponding to the same mark are merged into one rate interval, and the best sparse mode with this mark is recorded. The pattern is used as the best sparse pattern under the merged code rate interval, and only this mark is retained for all marks between the same marks; after all marks are traversed, all retained marks are unique, and the number of code rate intervals N that finally exist is counted. If the number N is equal to X, the X-1 interval demarcation points corresponding to the X code rate intervals that finally exist are stored as a group, and then step 4 is executed; if the number N is less than X, XN code rate intervals are added between the first code rate interval and the second code rate interval that finally exist, so that the code rate interval is increased to X, and the X-1 interval demarcation points corresponding to the X code rate intervals are stored as a group, and then step 4 is executed, wherein the length of each added code rate interval is 0, and the interval demarcation point related to the added code rate interval is the same as the interval demarcation point that originally separated the first code rate interval and the second code rate interval that finally exist. The marks of the best sparse pattern under the added XN code rate intervals are {M1, M2, M3, …, M X }\Ω, Ω represents the set of the best sparse patterns under the final bit rate range, {M1, M2, M3, …, M X }\Ω represents the set {M1, M2, M3, …, M X } excludes the set Ω; Step 4: Repeat step 3. After all the first light field images are processed, a total of Y groups of interval demarcation points are stored, each group of interval demarcation points has X-1 interval demarcation points, and the variance of all pixel values ​​in the central sub-aperture image in the sub-aperture image array represented by the first light field image corresponding to each group of interval demarcation points is calculated; Step 5: Select a light field image to be encoded as the second light field image, the angular resolution of the second light field image is consistent with the angular resolution of the first light field image; then estimate X-1 interval demarcation points corresponding to the second light field image according to the Y groups of interval demarcation points stored in step 4 and the Y variances calculated; Step 6: Given the target bit rate R t Under the premise of X-1 interval demarcation points corresponding to the second light field image, a suitable sparse pattern is quickly selected, marked as M c , Among them, P1, P2, P3, ..., P X-1 are X-1 interval demarcation points corresponding to the second light field image; Step 7: Use the M c The sparse pattern of the second light field image is represented as a sub-aperture image array, and the target bit rate R is calculated. t The pseudo video sequence composed of the sub-aperture images selected during sparse sampling is compressed, and then decoded and reconstructed to obtain the reconstructed sub-aperture image array corresponding to the second light field image, and then the base layer reconstructed light field image L is obtained. rec1 ; Step 8: Preset a threshold S; if L rec1 If the variance of the average quality of all sub-aperture images in the sub-aperture image array represented by is greater than S, the enhancement layer code stream is obtained and transmitted, and then decoded and reconstructed to obtain the enhancement layer reconstructed light field image L rec2 , then execute step 9; if L rec1 If the variance of the average quality of all sub-aperture images in the sub-aperture image array represented by is less than or equal to S, the encoding ends; Step 9: If L rec2 If the variance of the average quality of all sub-aperture images in the sub-aperture image array is greater than S, the quantization step size is continuously reduced to re-acquire the enhancement layer code stream and transmit and decode and reconstruct it until L rec2 The variance of the average quality of all sub-aperture images in the sub-aperture image array denoted as is less than or equal to S.

2. The light field image encoding method based on sparse pattern selection according to claim 1, characterized in that In step 1, the specific process of pre-coding any first light field image is as follows: At the encoding end, the first light field image is represented as a sub-aperture image array; then, the sub-aperture image array is sparsely sampled using X preset sparse modes respectively; Then, for each sparse mode, the sub-aperture images selected during sparse sampling are arranged in a preset fixed order to form a pseudo video sequence; finally, the pseudo video sequence obtained for each sparse mode is input into a standard video encoder for compression, and Z different quantization step sizes are used during compression to obtain Z coded bit streams with different bit rates; for X preset sparse modes, there are a total of X×Z coded bit streams; At the decoding end, a standard video decoder is used to decode the X×Z coded streams respectively, and each coded stream is decoded to obtain a compressed sub-aperture image; then, based on the compressed sub-aperture image obtained by decoding each coded stream, a reconstruction algorithm or a video interpolation network is used to synthesize the remaining sub-aperture images to reconstruct a complete reconstructed sub-aperture image array.

3. The light field image encoding method based on sparse pattern selection according to claim 1 or 2, characterized in that The specific process of step 2 is: Step 2.1: When the sub-aperture image array represented by the y-th first light field image is sparsely sampled using the x-th sparse pattern and then the pseudo video sequence composed of the sub-aperture images selected during sparse sampling is compressed using the z-th quantization step size, the bit rate representing the size of the encoded bit stream is recorded as R(y,x,z), and the average quality of all sub-aperture images in the reconstructed sub-aperture image array obtained by decoding and reconstruction is recorded as D(y,x,z), where y=1,2,…,Y, x=1,2,…,X, z=1,2,…,Z; Step 2.2: Fix y and x to obtain Z groups of bit rate and average quality combinations, with bit rate as the horizontal coordinate and average quality as the vertical coordinate, locate Z points in the two-dimensional coordinate system; then connect the located Z points with straight lines in the order of bit rate from small to large, to obtain a broken line, and use this broken line as the rate-distortion curve obtained by compressing the pseudo video sequence composed of the sub-aperture images selected during sparse sampling with the xth sparse mode array represented by the yth first light field image; Step 2.3: Traverse each sparse pattern and draw a rate-distortion curve in the same two-dimensional coordinate system according to the process of step 2.

2. After traversing X sparse patterns, a total of X rate-distortion curves are drawn, that is, a two-dimensional coordinate system corresponding to a first light field image and having X rate-distortion curves drawn therein is obtained.

4. The light field image encoding method based on sparse pattern selection according to claim 3, characterized in that In step 5, the process of estimating the X-1 interval demarcation points corresponding to the second light field image is as follows: Step 5.1: Represent the second light field image as a sub-aperture image array; then calculate the variance of all pixel values ​​in the central sub-aperture image in the sub-aperture image array, denoted as V in ; Step 5.2: Arrange the Y variances calculated in step 4 in order of size. in If it is between these variances, then find the ratio V in Small nearest neighbor variance V L and V in Large nearest neighbor variance V H , then press P in =(V in -V L ) / (V H -V L )×(P H -P L )+P L Calculate the X-1 interval demarcation points corresponding to the second light field image, where P in represents the vector composed of the X-1 interval demarcation points corresponding to the second light field image in sequence, P H Indicates V H The vector formed by the X-1 interval demarcation points corresponding to the first light field image, P L Indicates V L The vectors formed by the X-1 interval demarcation points corresponding to the first light field image in sequence; if these variances are all less than V in , then the X-1 interval demarcation points corresponding to the first light field image corresponding to the maximum variance among these variances are directly used as the X-1 interval demarcation points corresponding to the second light field image; if these variances are greater than or equal to V in , then the X-1 interval demarcation points corresponding to the first light field image corresponding to the minimum variance among these variances are directly used as the X-1 interval demarcation points corresponding to the second light field image.

5. The light field image encoding method based on sparse pattern selection according to claim 2, characterized in that The specific process of step 7 is as follows: On the encoding side, the c The sparse pattern of the second light field image is represented as the sub-aperture image array L org Perform sparse sampling; Then the sub-aperture image S selected during sparse sampling is sel Arrange them in a preset fixed order to form a pseudo video sequence PVS1; then according to the target bit rate R t The pseudo video sequence PVS1 is input into a standard video encoder for compression, and a base layer bit stream bits1 is obtained by compression; At the decoding end, bits1 is decoded using a standard video decoder to obtain the compressed sub-aperture image S s ' el Then, based on the compressed sub-aperture image, the remaining sub-aperture images are synthesized using a reconstruction algorithm or a video interpolation network to reconstruct a complete reconstructed sub-aperture image array, and then the reconstructed light field image is obtained as the base layer reconstructed light field image L rec1 , wherein the technique used to synthesize the remaining sub-aperture images is the same as the technique used to synthesize the remaining sub-aperture images in step 1.

6. The light field image encoding method based on sparse pattern selection according to claim 1, characterized in that In step 8, the enhancement layer code stream is obtained and transmitted, and then decoded and reconstructed to obtain the enhancement layer reconstructed light field image L rec2 The process is: Step 8.1: Get L rec1 The average quality of all sub-aperture images in the sub-aperture image array is expressed as; then from L rec1 In the sub-aperture image array represented as , all sub-aperture images with quality lower than the average quality are selected as enhancement objects; Step 8.2: Calculate the sub-aperture image array L for each enhanced object and the second light field image represented as org The residual images of the sub-aperture images with the same index in the image are obtained; and all the calculated residual images D are arranged in a preset order to form a pseudo video sequence PVS2; Step 8.3: Input the pseudo video sequence PVS2 into the standard video encoder for compression to obtain the enhanced layer bit stream bits2; Step 8.4: Decode bits2 with a standard video decoder to obtain a compressed residual image D'; then add the compressed residual image to the corresponding enhanced object to obtain the enhanced layer reconstructed light field image L rec2 .

7. The light field image encoding method based on sparse pattern selection according to claim 1, characterized in that In step 1, the difference between the maximum quantization step size and the minimum quantization step size among the Z different quantization step sizes is greater than three quarters of the difference between the upper limit and the lower limit of the quantization step size range that can be selected by the standard video encoder used when compressing the pseudo video sequence, and the Z different quantization step sizes are an arithmetic progression.

Images