A light field image encoding and decoding method, system, device, equipment and medium

Abstract

The present invention provides a light field image encoding and decoding method, system and device, which belongs to the field of image processing technology. It mainly collects key perspectives of each spatial position from the light field image, calculates the adjacency matrix reflecting the relationship between perspectives, encodes the key perspectives and the adjacency matrix to obtain a binary code stream. The present invention compresses some key perspectives, which can greatly improve the compression rate and reduce the size of the decoding cache. Through the correlation between different perspectives, graph learning can ensure the quality of the reconstructed light field image based on the adjacency matrix and improve the rate-distortion performance. Light field images are widely used in application fields such as environmental detection and medical imaging. Since the light field image data sampled in actual scenes can reach a large magnitude, it is urgent to design a compression algorithm to reduce the storage cost and reduce the transmission bandwidth. The module and algorithm proposed in the present invention have huge industrial potential.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a light field image encoding and decoding method, system and device. Background Art

[0002] The light field is a function that records the propagation of light in space. The full light field function is a seven-dimensional function that includes position coordinates, wavelength of light, and time. In actual scenarios, we record the coordinates of the intersection of light and two specified planes to generate a four-dimensional light field function. Light fields have many applications, such as light field refocusing and depth estimation. In practical applications, the storage of light field images has become a big problem. Since light field images contain many perspective images, the baseline between each two perspectives is very small, which brings structural redundancy between different perspectives. Direct storage or transmission of the acquired light field images will bring a great cost, which limits the practical application of light fields.

[0003] Since light field images contain images from multiple perspectives, we can arrange different perspectives into a pseudo video sequence and encode them using the HEVC codec. However, the HEVC codec can only choose one encoding configuration, and in any configuration, it is impossible to take into account the structural information between all perspectives at the same time. In light field images, there is a strong structural correlation between all perspectives, and this topological relationship needs to be used to better compress light field images. Graph signal processing is to model a signal into a graph, which contains vertices and edges. Each edge has a weight, which represents the connection between the corresponding two vertices. Graph-based modeling can well express the topological relationship within the signal and is usually used in signal decomposition and reconstruction. Based on graph signal processing, Viola et al. proposed a method for light field image compression using graph learning. This method uses a graph to represent the topological structure inside the light field image, generates a graph for all perspectives, and encodes this graph using lossless compression. Since the structures between the perspectives of different spatial points in the light field are very different, a single graph is not enough to represent all the information of the light field image. Moreover, lossless compression of the graph cannot efficiently remove redundancy. These deficiencies prompted the applicant to propose a new system for light field image compression. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a light field image encoding and decoding method, system and device, which can compress only the key viewing angles of the light field image, improve the compression rate, and enhance the rate-distortion performance.

[0005] According to a first aspect of the present invention, a light field image encoding method is provided, which encodes the light field image to obtain a binary code stream, comprising:

[0006] Collect key viewpoints at each spatial position from light field images;

[0007] Encoding the key perspective to obtain a binary code stream;

[0008] Calculating an adjacency matrix reflecting the relationship between the key perspectives;

[0009] The adjacency matrix is ​​encoded to obtain a binary code stream.

[0010] Preferably, the encoding of the key viewing angle to obtain a binary code stream includes: encoding the brightness value of the key viewing angle to generate a light field image binary code stream;

[0011] The calculation of the adjacency matrix reflecting the relationship between key perspectives includes: decoding the key perspective from the binary code stream of the light field image; setting the brightness values ​​of the remaining non-key perspectives to zero to obtain a synthetic light field image; calculating the initial adjacency matrix of each perspective at each spatial position according to the synthetic light field image; clustering the initial adjacency matrices of all spatial positions to obtain a first adjacency matrix corresponding to a first clustering center; and learning a dictionary according to the first adjacency matrix.

[0012] The encoding adjacency matrix obtains a binary code stream, including: clustering the initial adjacency matrix again to obtain a second adjacency matrix corresponding to the second cluster center; sparsely encoding the second adjacency matrix according to the dictionary to obtain a sparse coefficient binary code stream.

[0013] Preferably, the clustering of the initial adjacency matrices of all spatial positions is obtained by solving a constrained optimization problem based on the adjacency matrix and the graph signal.

[0014] The adjacency matrix is ​​a symmetric matrix in which all element values ​​are not less than zero and the diagonal elements are zero.

[0015] The graph signal is based on the smoothness of the graph Laplacian matrix, the strength of correlation of each view with other views, and the F-norm of the adjacency matrix.

[0016] The graph Laplacian matrix is ​​a diagonal matrix whose diagonal elements are formed by the degree of connectivity for each viewpoint minus the adjacency matrix.

[0017] Preferably, the learning of the dictionary according to the first adjacency matrix is ​​achieved by minimizing the square error representing the first adjacency matrix based on an optimization problem of a zero-norm constraint on sparse coefficients and alternatingly optimizing the dictionary and sparse coefficients representing the first adjacency matrix.

[0018] Preferably, the sparse encoding of the second adjacency matrix according to the dictionary is achieved by solving the sparse coefficients based on the dictionary and minimizing the square error representing the second adjacency matrix based on an optimization problem of a zero norm constraint on the sparse coefficients.

[0019] According to a second aspect of the present invention, a light field image encoding system is provided, which encodes a light field image to obtain a binary code stream, comprising:

[0020] Key perspective encoding submodule: obtains key perspectives of each spatial position from the light field image sampling, encodes the brightness value of the key perspective, and generates a light field image binary code stream;

[0021] Key view decoding submodule: decoding the light field image binary code stream obtained from the key view encoding submodule to obtain the key view, setting the brightness values ​​of the remaining non-key views to zero, and obtaining a synthetic light field image;

[0022] Adjacency matrix learning submodule: generating a first adjacency matrix of each perspective at each spatial position according to the synthetic light field image, clustering the first adjacency matrices of each perspective to obtain a first adjacency matrix, and learning to obtain a dictionary;

[0023] Adjacency matrix encoding submodule: generating a second adjacency matrix of each perspective at each spatial position according to the synthetic light field image, clustering the second adjacency matrix of each perspective into a second adjacency matrix, and sparsely encoding the second adjacency matrix according to the dictionary to obtain a sparse coefficient binary code stream.

[0024] According to a third aspect of the present invention, a light field image decoding method is provided, which is used to decode a binary code stream obtained by the above light field image encoding method to reconstruct a light field image, comprising:

[0025] Decoding the binary code stream of the light field image to obtain the key viewing angle, setting the brightness values ​​of the remaining non-key viewing angles to zero, and obtaining a synthetic light field image;

[0026] Calculating an initial adjacency matrix of each viewing angle at each spatial position according to the synthetic light field image, clustering the initial adjacency matrices at all positions to obtain a first adjacency matrix corresponding to a first clustering center, and learning a dictionary according to the first adjacency matrix;

[0027] Decoding the sparse coefficient binary code stream to obtain sparse coefficients, and reconstructing a second adjacency matrix according to the dictionary and the sparse coefficients;

[0028] A light field image is reconstructed from the synthetic light field image according to the second adjacency matrix.

[0029] Preferably, the light field image is reconstructed from the synthetic light field image according to the second adjacency matrix, which is to minimize the quadratic form of the synthetic light field image based on the graph Laplacian matrix for each of the second cluster centers under a given key viewing angle; the key viewing angle is determined by a preset sampling matrix, which is generally obtained by uniform sampling. Wherein:

[0030] The graph Laplacian matrix is ​​obtained by subtracting the second adjacency matrix from the degree matrix corresponding to the viewing angle;

[0031] The degree matrix corresponding to the viewing angle is a diagonal matrix and each diagonal element is the sum of the elements of the corresponding row of the second adjacency matrix.

[0032] According to a fourth aspect of the present invention, a light field image decoding system is provided, which is used to decode a binary code stream obtained by the above light field image encoding method to obtain a reconstructed light field image, comprising:

[0033] Key view decoding submodule: Decode the key view from the binary code stream of the light field image, set the brightness value of the non-key view to zero, and obtain a synthetic light field image;

[0034] Adjacency matrix learning submodule: generating a first adjacency matrix of each spatial position perspective according to the synthetic light field image, clustering the first adjacency matrices of all position perspectives to obtain a first adjacency matrix, and learning to obtain a dictionary;

[0035] Adjacency matrix decoding submodule: decoding the sparse coefficient binary code stream to obtain sparse coefficients, and reconstructing a second adjacency matrix according to the dictionary and the sparse coefficients;

[0036] Light field image reconstruction submodule: reconstructs a light field image from the synthesized light field image according to the second adjacency matrix.

[0037] According to a fifth aspect of the present invention, a compression method for encoding and decoding a light field image is provided, comprising: encoding a light field image using the above-mentioned light field image decoding method to obtain a binary code stream; decoding the binary code stream using the light field image decoding method to reconstruct the light field image.

[0038] According to a sixth aspect of the present invention, there is provided a compression device for encoding and decoding a light field image, comprising:

[0039] Coding device: encoding the light field image using the light field image coding method according to any one of claims 1 to 5 to obtain a binary code stream;

[0040] Decoding device: decodes the binary code stream using the light field image decoding method of claim 8 to reconstruct the light field image.

[0041] According to a seventh aspect of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any one of the following methods: the above-mentioned light field image encoding method, or the above-mentioned light field image decoding method.

[0042] According to an eighth aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded by a processor and executes any one of the following methods: the above-mentioned light field image encoding method, or the above-mentioned light field image decoding method.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The light field image encoding and decoding method, system, device, electronic device and medium provided by the present invention can greatly improve the compression rate and reduce the decoding cache size by sampling and compressing some key viewing angles;

[0045] By performing graph learning through the correlation between different perspectives, the quality of reconstructed light field images can be guaranteed based on the adjacency matrix, thereby improving the rate-distortion performance.

[0046] The light field image encoding method provided by the present invention can further reduce the additional overhead for dictionary learning and sparse coding of the adjacency matrix.

[0047] Light field images are widely used in application fields such as environmental detection and medical imaging. Since the light field image data sampled in actual scenes can reach a very large level, it is urgent to design a compression algorithm to reduce storage costs and reduce transmission bandwidth. Therefore, the module and algorithm proposed in this invention have huge industrial potential.

[0048] It is applied for compression of light field images, and reduces the encoding bit rate by about 25% compared with the HEVC solution on the HCI dataset. As the number of light field images gradually increases, a bit rate reduction of about 25% can bring huge economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0050] Figure 1 A flow chart of a light field image encoding method according to a preferred embodiment of the present invention;

[0051] Figure 2 A flow chart of a light field image decoding method according to a preferred embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the structure of a light field image encoding system according to a preferred embodiment of the present invention;

[0053] Figure 4 A schematic diagram of the structure of a light field image decoding system according to a preferred embodiment of the present invention;

[0054] Figure 5 A flow chart of a light field image encoding and decoding compression method according to a preferred embodiment of the present invention;

[0055] Figure 6 The figure is a schematic diagram of the structure of a light field image encoding and decoding compression system according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0057] A light field image encoding method is provided, wherein the light field image is encoded to obtain a binary code stream, comprising:

[0058] Collect key viewpoints at each spatial position from light field images;

[0059] Encode the key perspective to obtain a binary code stream;

[0060] Compute an adjacency matrix reflecting the relationships between key perspectives;

[0061] Encode the adjacency matrix to obtain a binary code stream.

[0062] As a preferred embodiment, encoding the key perspective to obtain a binary code stream includes:

[0063] Encode the brightness value of the key viewing angle to generate a binary code stream of the light field image;

[0064] Calculate the adjacency matrix reflecting the relationship between key perspectives, including:

[0065] The key view is obtained from the binary code stream of the light field image; the brightness values ​​of the remaining non-key view are set to zero to obtain a synthetic light field image; the initial adjacency matrix of each view at each spatial position is calculated based on the synthetic light field image; the initial adjacency matrices of all spatial positions are clustered to obtain a first adjacency matrix corresponding to the first cluster center; and a dictionary is obtained by learning based on the first adjacency matrix.

[0066] Encode the adjacency matrix to obtain a binary code stream, including:

[0067] The initial adjacency matrix is ​​clustered again to obtain a second adjacency matrix corresponding to the second cluster center; the second adjacency matrix is ​​sparsely encoded according to the dictionary to obtain a sparse coefficient binary code stream.

[0068] As a preferred embodiment, Figure 1 FIG. 1 is a flow chart of a light field image encoding method according to a preferred embodiment of the present invention. Figure 1 , the light field image encoding method includes:

[0069] S101 obtains key viewing angles at various spatial positions from light field image sampling, encodes brightness values ​​of the key viewing angles, and generates a light field image binary code stream;

[0070] S102: decoding the binary code stream of the light field image to obtain a key viewing angle, setting the brightness values ​​of the remaining non-key viewing angles to zero, and obtaining a synthetic light field image;

[0071] S103 calculates the initial adjacency matrix of each viewing angle of each spatial position according to the synthetic light field image, clusters the initial adjacency matrices of all positions to obtain a first adjacency matrix corresponding to a first cluster center, and obtains a dictionary according to the first adjacency matrix;

[0072] S104 clusters the initial adjacency matrix again to obtain a second adjacency matrix corresponding to the second cluster center, and performs sparse coding on the second adjacency matrix according to the dictionary to obtain a sparse coefficient binary code stream.

[0073] As a preferred embodiment, the initial adjacency matrix of all spatial positions is clustered, which is obtained by solving a constrained optimization problem based on the adjacency matrix and the graph signal; wherein:

[0074] The adjacency matrix is ​​a symmetric matrix with all elements not less than zero and diagonal elements being zero;

[0075] The graph signal is based on the smoothness of the graph Laplacian matrix, the correlation strength of each view with other views, and the F-norm of the adjacency matrix;

[0076] The graph Laplacian matrix is ​​a diagonal matrix with diagonal elements of the degree of connectivity for each viewpoint minus the adjacency matrix.

[0077] As a preferred embodiment, learning a dictionary based on the first adjacency matrix is ​​based on an optimization problem of a sparse coefficient zero norm constraint, and is achieved by alternately optimizing the dictionary and sparse coefficients representing the first adjacency matrix and minimizing the square error representing the first adjacency matrix.

[0078] As a preferred embodiment, sparse coding of the second adjacency matrix according to the dictionary is achieved by solving the sparse coefficients based on the dictionary and minimizing the square error representing the second adjacency matrix based on an optimization problem of zero norm constraint of the sparse coefficients.

[0079] Based on the same basic concept, another embodiment of the present invention further provides a light field image decoding method, which is used to decode the binary code stream obtained by the above light field image encoding method to reconstruct the light field image. Figure 2 This is a flow chart of a light field image decoding method according to a preferred embodiment of the present invention. Figure 2 ,include:

[0080] S201 decodes the binary code stream of the light field image to obtain a key viewing angle, sets the brightness values ​​of the remaining non-key viewing angles to zero, and obtains a synthetic light field image;

[0081] S202: calculating the initial adjacency matrix of each viewpoint at each spatial position according to the synthetic light field image, clustering the initial adjacency matrices at all positions to obtain a first adjacency matrix corresponding to a first cluster center, and obtaining a dictionary according to the first adjacency matrix;

[0082] S203 decodes the sparse coefficient binary code stream to obtain sparse coefficients, and reconstructs a second adjacency matrix according to the dictionary and the sparse coefficients;

[0083] S204 reconstructs a light field image from the synthesized light field image according to the second adjacency matrix.

[0084] As a preferred embodiment, reconstructing the light field image from the synthetic light field image according to the second adjacency matrix is ​​to minimize the quadratic form of the synthetic light field image based on the graph Laplacian matrix for each second cluster center under a given key viewing angle; wherein:

[0085] The graph Laplacian matrix is ​​obtained by subtracting the second adjacency matrix from the degree matrix corresponding to the view angle;

[0086] The degree matrix corresponding to the viewing angle is a diagonal matrix and each diagonal element is the sum of the elements of the corresponding row of the second adjacency matrix.

[0087] Based on the same basic concept, another embodiment of the present invention further provides a light field image coding system. The light field image is coded to obtain a binary code stream.

[0088] Figure 3It is a light field image encoding system of a preferred embodiment of the present invention, including: a key perspective encoding submodule S110: sampling the light field image to obtain the key perspective of each spatial position, encoding the brightness value of the key perspective, and generating a light field image binary code stream. A key perspective decoding submodule S120: decoding the light field image binary code stream to obtain the key perspective, setting the brightness value of the non-key perspective to zero, and obtaining a synthetic light field image. An adjacency matrix learning submodule S130: generating a first adjacency matrix of each spatial position angle block according to the synthetic light field image, clustering the first adjacency matrices of all position angle blocks to obtain a second adjacency matrix, and learning to obtain a dictionary. An adjacency matrix encoding submodule S140: clustering the first adjacency matrix into a third adjacency matrix, sparsely encoding the third adjacency matrix according to the dictionary, and obtaining a sparse coefficient binary code stream.

[0089] like Figure 3 As shown, the original light field image is input into the key view encoding submodule to obtain the light field image binary code stream. The code stream is input into the key view decoding submodule to obtain the decoded key view. The decoded key view is input into the adjacency matrix learning submodule to obtain the adjacency matrix, and the obtained adjacency matrix is ​​input into the adjacency matrix encoding submodule to obtain the sparse coefficient binary code stream.

[0090] Based on the same basic concept, in another embodiment of the present invention, a light field image decoding system is provided, which is used to decode the binary code stream obtained by the above method to obtain a reconstructed light field image.

[0091] Figure 4 It is a structural schematic diagram of a light field image decoding system of a preferred embodiment of the present invention, including: a key perspective decoding submodule S210: decoding the light field image binary code stream to obtain the key perspective, setting the brightness value of the non-key perspective to zero, and obtaining a synthetic light field image. An adjacency matrix learning submodule S220: generating a first adjacency matrix for each spatial position angle block based on the synthetic light field image, clustering the first adjacency matrix into a second adjacency matrix, and obtaining a dictionary based on the second adjacency matrix learning. An adjacency matrix decoding submodule S230: decoding the sparse coefficient binary code stream to obtain sparse coefficients, and reconstructing a third adjacency matrix based on the dictionary and the sparse coefficients. A light field image reconstruction submodule S240: reconstructing the light field image from the synthetic light field image based on the second adjacency matrix.

[0092] like Figure 4 As shown, the light field image binary code stream is input into the key view decoding submodule to obtain the decoded key view. The decoded key view is input into the adjacency matrix learning submodule to obtain the dictionary. The sparse coefficient binary code stream and dictionary are input into the adjacency matrix decoding submodule to obtain the decoded adjacency matrix, which is then input into the light field image reconstruction submodule to obtain the reconstructed light field image.

[0093] Based on the same basic concept, another embodiment of the present invention provides a compression method for encoding and decoding a light field image, comprising: encoding the light field image using a light field image decoding method to obtain a binary code stream; decoding the binary code stream using a light field image decoding method to reconstruct the light field image.

[0094] As a compression method for light field image encoding and decoding in a preferred embodiment of the present invention, refer to Figure 5 , a flow chart of a compression method for light field image encoding and decoding, comprising:

[0095] S0 acquires a light field image;

[0096] S1 light field image coding: Sample the key viewpoints of each spatial position from the light field image, calculate the adjacency matrix reflecting the relationship between the viewpoints, encode the key viewpoints and the adjacency matrix to obtain a binary code stream;

[0097] S2 Light Field Image Decoding: Decode key viewpoints and adjacency matrices from binary bitstreams to reconstruct light field images.

[0098] Based on the same basic concept, in another embodiment of the present invention, a compression device for encoding and decoding a light field image is provided, comprising an encoding device: encoding the light field image using the above-mentioned light field image encoding method to obtain a binary code stream; and a decoding device: decoding the binary code stream using the above-mentioned light field image decoding method to reconstruct the light field image.

[0099] As a compression device for light field image encoding and decoding in a preferred embodiment of the present invention, refer to Figure 6 , a structural diagram of a compression device for encoding and decoding light field images, and also includes an acquisition module.

[0100] Based on the same basic concept, in another embodiment of the present invention, an electronic device is provided, the electronic device includes a processor and a memory, the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement any of the following methods: the above-mentioned light field image encoding method, or the above-mentioned light field image decoding method.

[0101] Based on the same basic concept, in another embodiment of the present invention, a storage medium is provided, in which at least one instruction, at least one program, code set or instruction set is stored, and the at least one instruction, at least one program, code set or instruction set is loaded by a processor and executes any one of the following methods: the above-mentioned light field image encoding method, or the above-mentioned light field image decoding method.

[0102] Based on the above light field image encoding and decoding method, the present invention also provides a preferred embodiment, specifically including:

[0103] S0: Get light field image: light field image Record the brightness of a beam at any angle of view at any spatial position, where (p, q) represents the angle of view and (s, t) represents the spatial position. Represents the angle block at position (s, t), records the brightness of all viewing angle beams at position (s, t), and uses The aperture image represents the viewing angle (p, q), and records the brightness of the viewing angle (p, q) beam at each position.

[0104] S1: Light field image encoding: Based on the light field image Aperture images at different viewing angles Sampling to get the key perspective

[0105] At each position (s, t), calculate the adjacency matrix A that reflects the relationship between perspectives s，t , encoding key perspectives and the adjacency matrix A s，t Get the binary code stream.

[0106] Furthermore, light field image encoding includes:

[0107] S101: Critical View Sampling via Mask Implementation, Mask When the view (p, q) is sampled as the key view, When , the view (p, q) is regarded as a non-critical view and discarded, that is, Key perspectives Encode to obtain a binary code stream of the light field image;

[0108] S102: Decoding the binary code stream of the light field image to obtain the key viewing angle Set the brightness values ​​of the remaining non-critical viewing angles to zero:

[0109]

[0110] Based on these aperture images Obtaining a synthetic light field image

[0111] S103: Synthesizing the light field image Calculate the angle block for each spatial position The initial adjacency matrix A s，t , the initial adjacency matrix A at the spatial position (s, t) s，t The element in row i and column j of Reflects the viewing angle (p) at position (s, t)i ,q i ) and (p j ,q j ), the value can be calculated:

[0112]

[0113] Here σ is a scale parameter used to control the correlation.

[0114] Cluster the initial adjacency matrix of all positions to obtain the first cluster centers with the number of K1, where the i-th first cluster center corresponds to a first adjacency matrix By taking the angle block belonging to the i-th first cluster center The graph signal X i To perform graph learning optimization, we obtain:

[0115]

[0116]

[0117] here is a diagonal matrix with each diagonal element being The first adjacency matrix corresponding to the K1 first cluster centers is Converting from 2D matrix to 1D vector The target training data Y1 is obtained in conjunction with the optimization problem to learn the dictionary And solve for the sparse coefficients

[0118]

[0119] Here, i is the index of the cluster center, is the sparsity coefficient of each cluster, The zero norm of The maximum value of is constrained to be T1, and λ is the Lagrangian parameter that balances the reconstruction error and sparsity.

[0120] S104: Initial adjacency matrix A for all spatial positions (s, t) s，t Clustering is performed again to obtain the second cluster centers with the number of K2, where the i-th second cluster center corresponds to a second adjacency matrix The second adjacency matrix corresponding to the K2 second cluster centers Converting from 2D matrix to 1D vector The target training data Y2 is obtained by combining the dictionary Solve the optimization problem to obtain sparse coefficients

[0121]

[0122] here, yes The sparse coefficient of T2 is Sparsity constraints. Encoding is performed to generate a sparse coefficient binary code stream.

[0123] Furthermore, light field image decoding includes:

[0124] S201: Decoding the binary stream of the light field image to obtain the key perspective The brightness values ​​of the remaining non-critical viewing angles are set to zero to obtain a synthetic light field image. Same method as S22;

[0125] S202: Synthesizing the light field image Calculate the angle block for each spatial position The initial adjacency matrix A s，t , cluster the initial adjacency matrix of all positions to obtain the first adjacency matrix corresponding to the first cluster center with the number of K1 According to the first adjacency matrix Learn to get a dictionary The method is the same as S103;

[0126] S203: Decoding the sparse coefficient binary code stream to obtain the sparse coefficient According to the dictionary and the sparse coefficient Reconstruct the second adjacency matrix

[0127] S204: Reconstruct the second adjacency matrix From synthetic light field images Reconstruct the light field image. For the angle block belonging to the i-th second cluster center, according to The key perspective is vectorized into a one-dimensional vector and obtained by The vectorized angle block X corresponding to the desired reconstructed light field image i ,satisfy here is the mask mentioned above. Therefore, the vectorized angle block X i The reconstruction is:

[0128]

[0129] here, Solve to obtain Reconstruct the light field image.

[0130] Based on the above light field image encoding and decoding method, the embodiment of the present invention is also applied to a data set example. The data set is an HCI data set including Buddha, Buddha2, MonasRoom, Medieval, Statue, and Papillon. The results are shown in Table 1 below.

[0131] Table 1: Comparison of this scheme with HEVC and GLA based on BD-PSNR and BD-BR on the HCI dataset

[0132]

[0133] In Table 1, Buddha, Buddha2, MonasRoom, Medieval, Statue, and Papillon are light field images in the HCI dataset. Table 1 lists the BD-PSNR and BD-BR comparisons of the codec proposed in this scheme with the HEVC codec and the GLA codec. It can be seen from Table 1 that compared with the method based on the HEVC codec, the method in this scheme reduces the bit rate by 25.32% on average and improves the PSNR by 0.6dB. Compared with the GLA method, the method in this scheme reduces the bit rate by 43.80% on average and improves the PSNR by 1.53dB.

[0134] Implementation Effect

[0135] The present invention compresses light field images and reduces the encoding bit rate by about 25% compared with the HEVC solution on the HCI data set. As the number of light field images gradually increases, a bit rate reduction of about 25% can bring huge economic benefits.

[0136] The present invention proposes a light field compression system, including: light field image acquisition, light field image encoding, and light field image coding. Light field images are widely used in application fields such as environmental detection and medical imaging. Since the light field image data sampled in actual scenes can reach a large amount, it is urgent to design a compression algorithm to reduce storage costs and reduce transmission bandwidth. Therefore, the module and algorithm proposed in the present invention have huge industrial potential.

[0137] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A light field image encoding method, encoding a light field image to obtain a binary code stream, characterized in that: include: Collect key viewpoints at each spatial position from light field images; Encoding the key perspective to obtain a binary code stream; Calculating an adjacency matrix reflecting the relationship between the key perspectives; Encoding the adjacency matrix to obtain a binary code stream; The key perspective is encoded to obtain a binary code stream, including: Encoding the brightness value of the key viewing angle to generate a light field image binary code stream; The calculation of the adjacency matrix reflecting the relationship between key perspectives includes: Decoding the binary code stream of the light field image to obtain a key viewing angle; The brightness values ​​of the remaining non-critical viewing angles are set to zero to obtain a synthetic light field image; Calculating an initial adjacency matrix of each viewing angle at each spatial position according to the synthetic light field image; Clustering the initial adjacency matrices of all spatial positions to obtain a first adjacency matrix corresponding to the first cluster center; Learning a dictionary according to the first adjacency matrix; The encoding adjacency matrix is ​​used to obtain a binary code stream, including: Clustering the initial adjacency matrix again to obtain a second adjacency matrix corresponding to the second cluster center; The second adjacency matrix is ​​sparsely encoded according to the dictionary to obtain a sparse coefficient binary code stream.

2. The light field image encoding method according to claim 1, characterized in that: The clustering of the initial adjacency matrix of all spatial positions is obtained by solving a constrained optimization problem based on the adjacency matrix and the graph signal, where: The adjacency matrix is ​​a symmetric matrix in which all element values ​​are not less than zero and the diagonal elements are zero; The graph signal is based on the smoothness of the graph Laplacian matrix, the correlation strength of each view with other views, and the F-norm of the adjacency matrix; The graph Laplacian matrix is ​​a diagonal matrix whose diagonal elements are formed by the degree of connectivity for each viewpoint minus the adjacency matrix.

3. The light field image encoding method according to claim 1, characterized in that: Learning a dictionary based on the first adjacency matrix is ​​an optimization problem based on the zero norm constraint of the sparse coefficients, and is achieved by minimizing the square error representing the first adjacency matrix based on alternating optimization of the dictionary and the sparse coefficients representing the first adjacency matrix.

4. The light field image encoding method according to claim 1, characterized in that: Sparse coding of the second adjacency matrix according to the dictionary is based on an optimization problem of a zero norm constraint on sparse coefficients, and is achieved by solving the sparse coefficients based on the dictionary and minimizing the square error representing the second adjacency matrix.

5. A light field image coding system, which encodes a light field image to obtain a binary code stream, characterized in that: include: Key perspective encoding submodule: obtains key perspectives of each spatial position from the light field image sampling, encodes the brightness value of the key perspective, and generates a light field image binary code stream; Key view decoding submodule: decoding the light field image binary code stream obtained from the key view encoding submodule to obtain the key view, setting the brightness values ​​of the remaining non-key views to zero, and obtaining a synthetic light field image; Adjacency matrix learning submodule: generating a first adjacency matrix of each perspective at each spatial position according to the synthetic light field image, clustering the first adjacency matrices of each perspective to obtain a first adjacency matrix, and learning to obtain a dictionary; Adjacency matrix encoding submodule: generating a second adjacency matrix of each perspective at each spatial position according to the synthetic light field image, clustering the second adjacency matrices of each perspective into a second adjacency matrix, and sparsely encoding the second adjacency matrix according to the dictionary to obtain a sparse coefficient binary code stream.

6. A light field image decoding method, used to decode the binary code stream obtained in any one of claims 1 to 4 to reconstruct a light field image, characterized in that: include: The key viewing angle is obtained by decoding the binary code stream of the light field image, and the brightness values ​​of the remaining non-key viewing angles are set to zero to obtain a synthetic light field image; Calculating an initial adjacency matrix of each viewing angle at each spatial position according to the synthetic light field image, clustering the initial adjacency matrices at all positions to obtain a first adjacency matrix corresponding to a first clustering center, and learning a dictionary according to the first adjacency matrix; Decoding the sparse coefficient binary code stream to obtain sparse coefficients, and reconstructing a second adjacency matrix according to the dictionary and the sparse coefficients; A light field image is reconstructed from the synthetic light field image according to the second adjacency matrix.

7. The light field image decoding method according to claim 6, characterized in that: Reconstructing the light field image from the synthetic light field image according to the second adjacency matrix is ​​to minimize the quadratic form of the synthetic light field image based on the graph Laplacian matrix for each of the second cluster centers under a given key viewing angle, where: The graph Laplacian matrix is ​​obtained by subtracting the second adjacency matrix from the degree matrix corresponding to the viewing angle; The degree matrix corresponding to the viewing angle is a diagonal matrix and each diagonal element is the sum of the elements of the corresponding row of the second adjacency matrix.

8. A light field image decoding system, used for decoding the binary code stream obtained in any one of claims 1 to 5 to obtain a reconstructed light field image, characterized in that: include: Key view decoding submodule: Decode the key view from the binary code stream of the light field image, set the brightness value of the non-key view to zero, and obtain a synthetic light field image; Adjacency matrix learning submodule: generating a first adjacency matrix of each spatial position perspective according to the synthetic light field image, clustering the first adjacency matrices of all position perspectives to obtain a first adjacency matrix, and learning to obtain a dictionary; Adjacency matrix decoding submodule: decoding the sparse coefficient binary code stream to obtain sparse coefficients, and reconstructing a second adjacency matrix according to the dictionary and the sparse coefficients; Light field image reconstruction submodule: reconstructs a light field image from the synthesized light field image according to the second adjacency matrix.

9. A compression method for encoding and decoding a light field image, characterized in that: include: Encoding the light field image using the light field image decoding method according to any one of claims 1 to 4 to obtain a binary code stream; The binary code stream is decoded using the light field image decoding method of claim 6 to reconstruct the light field image.

10. A compression device for encoding and decoding a light field image, characterized in that: include: Coding device: encoding the light field image using the light field image coding method according to any one of claims 1 to 4 to obtain a binary code stream; Decoding device: decodes the binary code stream using the light field image decoding method of claim 6 to reconstruct the light field image.

11. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any one of the following methods: - the light field image encoding method according to claims 1 to 4, or, -The light field image decoding method according to claim 6.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes any one of the following methods: - the light field image encoding method according to claims 1 to 4, or, -The light field image decoding method according to claim 6.

Images