Method, computer program product and system for representing visual information

Abstract

The present disclosure relates to a computer-implemented method for representing a data set including at least one n-dimensional data element representing visual information, the method comprising obtaining (210) the data set, obtaining (220) a dictionary set including a plurality of dictionaries, each dictionary including at least one basis function (102), assigning (230) each at least one data element to a dictionary, wherein the set of basis functions represents an m-dimensional transform domain, transforming (240) at least one data element to the transform domain using a corresponding dictionary of basis functions, wherein each data element is defined by an associated set of coefficients, sparsely (250) the set of coefficients, and forming (260) a representation of the visual information including the coefficient data set and the corresponding dictionary.

Description

Technical Field

[0001] The present disclosure relates to representing visual information, such as light field data. Background Art

[0002] Over the past decade, the field of computational photography, particularly light field and multi-view imaging, has emerged and matured as a new paradigm in imaging and video technology. These techniques enable a range of novel applications, from advanced multi-dimensional image processing to movie editing, glasses-free 3D display systems, single-sensor light field cameras, spectral imaging, and appearance capture. The challenges of using these new visual information formats are related to the size of the data generated. Two important issues that arise due to the large amount of data are the efficient storage of the data and the infrastructure required to transfer the captured data from the sensor system. Data compression and image compression are two mature research areas dedicated to addressing the challenges of processing and storing large data volumes. Summary of the invention

[0003] A very important and still unsolved challenge inherent in capturing, storing, and processing high-dimensional data is dealing with very large data sizes.

[0004] The present invention relates to a solution to this problem.

[0005] The invention relates to a computer-implemented method for representing a data set comprising n-dimensional data elements representing visual information, the method comprising the steps of obtaining the visual information, obtaining a dictionary set, the dictionary comprising a set of basis functions, assigning the data elements to the dictionary, transforming the data elements to a transform domain represented by a set of coefficients using a corresponding set of basis functions, sparsely ...

[0006] One advantage of the method described above is that the compression rate of visual information can be improved.

[0007] One advantage of the method described above is that the reconstruction error of the compressed visual information can be reduced.

[0008] One advantage of the method as described above is improved compressed perception of visual information.

[0009] The invention as defined herein enables improved selective reconstruction of compressed visual information.

[0010] The step of obtaining a data set comprising data elements of visual information may include obtaining multi-dimensional visual information. The data element comprises at least one data point. The visual information may include data elements and / or data points based on sensor data and possible interpolation of the sensor data. The visual information may include data elements and / or data points based on computer-generated images and possible interpolation of the computer-generated images.

[0011] The method can be particularly useful in the capture, storage and processing of very large data sets, such as high-dimensional data such as perspective data. The method can be used to compress any kind of multi-dimensional data. The method can use non-local clustering. The method can be used for non-local clustering of multi-dimensional data sets. The method allows for efficient local reconstruction of compressed, possibly multi-dimensional data.

[0012] The step of obtaining a dictionary set including basis functions may include learning a dictionary set. The dictionary set includes a set of basis functions that represent n-dimensional data elements of visual information in an m-dimensional transform domain. The dimension m of the transform domain may be equal to or greater than 2. The dimension m of the transform domain may be equal to or greater than the dimension n of the visual information. The present disclosure relates to a method for transforming between arbitrary dimensions. The basis function set may independently represent visual information in its various dimensions in the transform domain. Transforming the data elements of the visual information into a multi-dimensional transform domain may enable reconstruction of specific data elements, such as reconstructing a single frame of a video, rather than reconstructing the entire video. Visual information including n-dimensional data elements representing visual information may be assigned to a dictionary respectively. Data elements may be assigned based on sparsity. Assigning data elements based on sparsity allows all data elements within a specific sparsity range to be transformed using a dictionary, the dictionary containing a set of basis functions suitable for transforming data elements within the sparsity range to the transform domain, wherein the data elements are represented by associated sets of coefficients.

[0013] The term transform domain as used herein refers to the representation of visual information by a set of coefficients and a corresponding set of basis functions. -2πiwx The Fourier transform of can realize the well-known transform, which transforms the time function into the sum of sine waves of different frequencies, each sine wave represents a frequency component. The frequency components in the frequency domain represent the time function. Similarly, in the present disclosure, the coefficient set represents the visual information in the transform domain defined by the basis function of each dictionary.

[0014] The term visual information refers to both sensor data and computer-generated images.

[0015] As used herein, the term "data element" refers to a sub-block of visual information that includes at least one data point. An image sub-block is also referred to as an "image block" or a "window". A sub-block can be a small image, such as a 10x10 pixel area.

[0016] As used herein, the term "data point" refers to the smallest unit of data, such as the red, green, or blue value of a pixel.

[0017] The term coefficient set comprises a set of values ​​that, if plugged into a set of basis functions, would describe a sub-block of visual information.

[0018] At least a portion of the obtained dictionary set can be created as part of the step of obtaining the dictionary set, and / or pre-created by using pre-clustered training visual information of the same dimension n as the training process of the visual information. The training process is intended to create a dictionary to form a coefficient set that is as sparse as possible for the pre-clustering of the training visual information. The training visual information can be of the same type as the visual information, for example, it can be both a live video and an animated video. The training process of creating the dictionary set can be a machine learning-assisted process. The existing set can be expanded and / or reduced. The process of creating the set can be a process of expanding the existing dictionary set using other dictionaries. The process of creating the set can be a process of limiting the existing dictionary set by deleting the existing dictionary. It is expected that the dictionary set trained on a certain type of visual information will show a high degree of sparsity in the coefficient set representing similar visual information in the transform domain. The creation of a sparse coefficient set and the sparsity of the set are important factors for effective compression and sampling. The dictionary set will typically contain multiple basis functions, but its memory footprint is very low compared to the size of the data set, so the encoding and decoding efficiency can be greatly improved.

[0019] When it comes to the step of assigning each data element to a dictionary, the data elements can be assigned to the dictionary based on sparsity. The method includes the step of transforming each data element to a transform domain, wherein each data element is represented by a dictionary of basis functions and a coefficient set. The coefficient set representing the data element in the transform domain defined by the basis function set of the corresponding dictionary includes coefficient values. Below a certain absolute value, the effect of coefficient values ​​close to zero on the data point value of the corresponding reconstructed data element is negligible. It is expected that a large number of coefficient sets with coefficient values ​​close to zero are generated by training a basis function set of a dictionary set for learning visual information including visual information data elements of similar type and sparsity to the data elements to be transformed. The present disclosure utilizes the fact that the visual information feature is only locally sparse.

[0020] The method includes a thinning step, wherein coefficient values ​​close to zero are set to zero. The threshold below which the absolute value of the coefficient value is truncated to zero can be based on a predetermined value determined during the process of creating a dictionary set. The threshold below which the absolute value of the coefficient value is truncated can be based on at least one normalized absolute value of the contribution of the coefficient value to the data point in the data element. After the truncation step, each coefficient set should contain a large number of zero values, and the coefficient set is considered to be sparse. The more coefficients that are sparse to zero, the smaller the size of the compressed coefficient set is expected to be. The more coefficients that are sparse to zero, the more information of the visual information is lost, and a larger reconstruction error is expected. Due to the anti-correlation between small size and information loss, the thinning step can be adjusted based on the type of visual information being thinned.

[0021] The step of forming the representation of the visual information may comprise compressing at least one set of coefficients. Compressing at least one set of coefficients may comprise using a compression algorithm.

[0022] The method may include a reconstruction step, including selecting a data element and selecting at least one dimension for reconstructing data points in the data element, and reconstructing at least one data point in the selected at least one dimension from a corresponding set of coefficients of a basis function and a dictionary. Reconstructing a single frame of a 2D video data set may require reconstructing a large number of data points in a large number of data elements, but reconstructing a complete data set may require a significantly large number of computations. The method may be arranged to transform a data set including key frames and reconstruct a data set including intermediate frames. Intermediate frames are frames between known key frames, for example, if a 20fps video of a key frame is transformed and reconstructed into a 60fps video, additional frames will be generated.

[0023] The reconstruction error of the method may be calculated by reconstructing at least a portion of the data element and comparing at least a portion of the reconstructed data element with the corresponding original data element. To calculate the reconstruction error, all data points of the data element may be reconstructed for comparison.

[0024] The present disclosure also relates to a computer program for representing a data set comprising n-dimensional data elements representing visual information. The computer program comprises routines for executing a method according to the present disclosure. In one example, the computer program is an application on a sensor system.

[0025] The present disclosure also relates to a computer program product. The computer program product includes a program code stored on a readable data storage medium for representing a data set including n-dimensional data elements representing visual information. The data storage medium may be non-volatile. The program code is configured to perform a method according to the present disclosure. The computer program product may control a sensor system. The computer program product may be an application of a sensor system.

[0026] The present disclosure also relates to a system for representing a data set including n-dimensional data elements representing visual information, the system comprising a processor, a memory including a computer program product, at least one visual information generating device, wherein the processor is arranged to: store data in the memory, control at least one visual information generating device, receive visual information from at least one visual information generating device, run a computer program product arranged to perform the above steps, and provide a representation of the visual information based on a sparse coefficient set. The system can be a personal computer. The system can include a sensor system. The system can be a sensor system. The system can be a compression sensor system, which is arranged to provide a significantly compressed representation of the captured visual information. The system can be a compressed light field camera. The processor can be arranged to control at least one visual information generating device. At least one visual information generating device can be a camera. At least one visual information device can be a database including visual information. The system can include hardware for wireless communication between the processor and at least one of the at least one visual information generating devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An example of a dictionary set including basis functions is shown.

[0028] Figure 2 An example of a method for representing visual information is shown.

[0029] Figure 3 An example of a method for learning a dictionary set is shown.

[0030] Figure 4 An example of a system for representing visual information is shown. DETAILED DESCRIPTION

[0031] In all drawings, the same reference numerals refer to the same parts, concepts and / or elements. Therefore, unless otherwise expressly stated, statements about a reference numeral in one drawing will also apply to the same reference numeral in other drawings.

[0032] Figure 1 An example of a dictionary set 100 including basis functions 102 is shown.

[0033] The set 100 includes at least one dictionary 101. The set 100 may include multiple dictionaries. Each dictionary 101 includes at least one basis function 102. Each dictionary 101 may include multiple basis functions 102, forming a multi-dimensional dictionary set 100.

[0034] The visual information includes at least one data element 110, which includes at least one data point 111. The data element 110 of dimension n can be transformed by a dictionary 101 of its assigned basis functions 102 to form a sparse coefficient set 120 representing the data element 110 in the transform domain. The sparse coefficients 120 can be saved to a memory such as a database or a solid-state drive. The sparse coefficient set 120 and the corresponding dictionary 101 can be used to form a reconstructed data element 130. When reconstructing, it is not necessary to reconstruct all data points 131 of the data element 130. At least one data point 131 along at least one dimension in the data element 130 can be selected for reconstruction. By comparing the data element 110 or at least one data point 111 with the corresponding reconstructed data element 130 or at least one reconstructed data point 131, a reconstruction error can be calculated.

[0035] The dictionary set 100 may be created by training the dictionary set on training visual information of a similar type of visual information to the visual information to be compressed, such as a live 2D video or an animated 2D video. The training visual information may include at least a portion of the visual information to be compressed. The dictionary set 100 may be trained by dividing the training visual information into pre-clusters based on sparsity and learning a pre-clustered set of dictionaries for each pre-cluster. The obtained pre-clustered set of dictionaries is learned so that the pre-clustered set of dictionaries represents a pre-cluster of the training visual information in a transform domain with a sparse set of coefficients. The learned pre-clustered set of dictionaries may be combined to form the dictionary set 100.

[0036] Figure 2 An example of a method 200 for representing visual information is shown.

[0037] The method 200 includes a step 210 of obtaining visual information including at least one data element 110. The visual information may be multi-dimensional. The visual information may include a computer-generated image. The visual information may be obtained from a non-volatile storage medium. The visual information may be continuously obtained from a sensor system. The visual information may be obtained from at least one camera that captures a physical scene. The visual information may be an interpolated data element 110 and / or data point 111 based on sensor data. The visual information may be an interpolated data element 110 and / or data point 111 based on a computer-generated image.

[0038] The method comprises a step 220 of obtaining a dictionary set 100, the dictionary comprising at least one set of basis functions 102. The dictionary set 100 comprises a set of basis functions 102, which are arranged to transform data elements 110 of visual information into a transform domain. The set of basis functions 102 may transform the visual information into a transform domain of equal or higher dimensionality. The dimension m of the transform domain may be equal to or greater than 2. The set of basis functions 102 and the corresponding set of coefficients 120 may independently represent the visual information in the transform domain in its respective dimensionality. The step 220 of obtaining the dictionary set 100 may comprise creating and / or training and / or learning and / or modifying the dictionary set 100.

[0039] The method includes a step 230 of assigning each data element 111 to a dictionary 101. The step 230 of assigning each data element 111 to a dictionary may assign the data element 111 to fewer dictionaries 101 than there are dictionaries 101 in the set 100. The assignment of the data element 111 to the dictionary 101 may be based on sparsity. The assignment of the data element 111 to the dictionary 101 may include machine learning.

[0040] The method comprises a step 240 of transforming the data elements 111 of the visual information into a transform domain, wherein each coefficient set 120 and the corresponding set of basis functions 102 describe the data elements 111 of the visual information. For a dictionary set 100 created based on training visual information of similar type and sparsity to the visual information to be transformed by the basis function set 102, it is desirable to generate a coefficient set 120 having a large number of coefficients 120 with values ​​close to zero.

[0041] The method includes a step 250 of sparse coefficient sets 120. The step 250 of sparse coefficient sets is an irreversible step because information is permanently lost. The step 250 of sparseness can allow increased potential to compress the coefficient set. The correlation between sparsity, reconstruction error and potential data compression rate is expected, however, small reconstruction error and large data compression rate are desired. The step 250 of sparse coefficient sets 120 may include setting the values ​​of coefficients 120 to near zero to zero. The step 250 of sparse coefficient sets 120 may include setting the values ​​of coefficients 120 that have a lower relative contribution to the data elements of visual information to zero. A threshold for setting the value of coefficients 120 to near zero to zero can be defined by a tolerable amount of error.

[0042] The method includes a step 260 of forming a representation of visual information, the visual information including a coefficient data set and a corresponding dictionary. The coefficient data set may include at least one coefficient set. The step 260 of forming a representation of visual information may include compressing at least one coefficient set. The compression of at least one coefficient set may include a compression algorithm suitable for compressing sparse data.

[0043] Figure 3An example of a method 300 for learning a dictionary set 100 is shown. The step 220 of obtaining the dictionary set 100 in the method 200 for representing visual information may include the method 300 for learning a dictionary set or steps thereof.

[0044] The method 300 for learning the dictionary set 100 comprises a step 310 of obtaining a training data set comprising training data elements representing training visual information, a step 320 of pre-clustering the training visual information into at least one pre-cluster, a step 330 of initializing the pre-cluster set of dictionaries for each pre-cluster, a step 340 of clustering the at least one pre-cluster by assigning each training data element to a dictionary in a corresponding pre-cluster set, a step 350 of training the at least one pre-cluster set to minimize the reconstruction error and maximize the sparsity of the coefficients of the corresponding training data elements, a step 360 of updating the clusters and iterating the training step 350 until at least one condition is satisfied, and a step 370 of combining at least one of the at least one pre-cluster set to form the dictionary set 100.

[0045] The method 300 for learning a dictionary set 100 may be a machine learning assisted process. The step 310 of obtaining a training data set may include a training data set having the same dimension and type as the data elements 111 of the visual information to be represented by the learned dictionary set 100 in a later stage. The step 320 of pre-clustering the training visual information may be based on sparsity. The step 320 of pre-clustering the training visual information may include machine learning. The step 330 of initializing a pre-cluster set of dictionaries for each pre-cluster may include initializing the pre-cluster set based on the sparsity of the corresponding pre-cluster. The step 340 of clustering at least one pre-cluster by assigning each training data element to a dictionary may be based on sparsity. The step 340 of clustering at least one pre-cluster by assigning each training data element to a dictionary may include machine learning. The step 350 of training at least one pre-cluster set may be arranged to minimize the computational requirements for encoding and decoding. The step 360 of updating the clustering and iterating the training step 350 may be arranged to minimize the number of dictionaries to which the training data elements are assigned. The step 370 of combining at least one of the at least one pre-clustered sets to form a dictionary set 100 may include combining at least one pre-clustered set with at least one dictionary from at least one other dictionary set 100 .

[0046] Figure 4An example of a system 400 for representing visual information is shown. The system 400 for representing a data set comprises n-dimensional data elements 110 representing visual information, comprises a processor 410, a memory 420 comprising a computer program product, at least one visual information generating device 430, wherein the processor 410 is arranged to store data on the memory 420, receive visual information from the at least one visual information generating device 430, run a computer program product arranged to perform the steps according to any one of claims 1 to 15, and provide a representation of the visual information based on the sparse coefficient set 120. The system may be a personal computer. The system may include a sensor system. The system may be a sensor system. The processor 410 may be arranged to control the at least one visual information generating device 430. The at least one visual information generating device 430 may be at least one camera. The at least one visual information generating device 430 may be a database comprising visual information. The system may include hardware for wireless communication between the processor 410 and the at least one visual information generating device 430.

[0047] The memory 420 is arranged to store the Figure 2 and Figure 3 The computer program product may be at least partially executed on the processor 410. The computer program product may include routines for controlling any one of the at least one visual information generating device 430. The computer program product may include routines for wireless communication between the processor and the at least one visual information generating device 430.

Claims

1. A computer-implemented method for representing a data set comprising at least one n-dimensional data element (110) representing visual information, the method comprising: obtaining (210) the data set, obtaining (220) a dictionary set (100) comprising a plurality of dictionaries (101), each dictionary comprising at least one basis function (102), assigning (230) each data element of the at least one data element (110) to a dictionary (101), wherein the set of basis functions (102) represents an m-dimensional transform domain, transforming (240) at least one data element (110) into a transform domain using a corresponding dictionary (101) of basis functions (102), wherein each data element (110) is defined by an associated set of coefficients (120), Sparse (250) the coefficient set (120), forming (260) a representation of the visual information comprising the set of coefficients (120) and the corresponding dictionary (101), wherein the obtained dictionary set (100) comprises a dictionary set formed by combining at least one pre-cluster set, wherein each pre-cluster set is obtained from a training data set of training visual information comprising training data elements, wherein the training visual information is divided into pre-clusters based on sparsity, wherein the pre-clusters are clustered by assigning each training data element (110) to a dictionary of basis functions (102) in a corresponding set of pre-clusters of the dictionary of each pre-cluster, and wherein each set of pre-clusterings is trained to minimize reconstruction error and maximize sparsity of a set (120) of training coefficients for training data elements (110), and wherein the clusterings are updated and the training is iterated until at least one condition is satisfied, The set acquisition step (220) further includes a set learning step (300), wherein the set learning step includes: obtaining (310) a training data set of training visual information including training data elements, Dividing the training visual information into (320) pre-clusters, Initialize (330) a pre-cluster set of dictionaries for each pre-cluster, clustering the pre-clusters by assigning each training data element to a dictionary of basis functions in the corresponding set of pre-clusters (340), training (350) each set of pre-clustering to minimize the reconstruction error and maximize the sparsity of the set of training coefficients for training data elements, updating (360) the clustering and iterating the training until at least one condition is satisfied, At least one of the at least one pre-clustered sets is combined (370) to form a dictionary set (100).

2. The method according to claim 1, wherein: The transform domain dimension m is at least 2, and wherein the set of basis functions (102) independently represents each data element (110) in each of its dimensions in the transform domain.

3. The method according to claim 1 or 2, wherein: The transform domain dimension m is equal to or greater than the dimension n of the visual information, and wherein the set of basis functions (102) independently represents each data element (110) in the transform domain at each of its dimensions.

4. The method according to claim 1 or 2, further comprising: The reconstruction step comprises: selecting a data element (130), selecting at least one dimension for reconstructing a data point (131) in the data element (130), and reconstructing at least one data point (131) in the selected at least one dimension from a corresponding coefficient set (120) of a basis function and a dictionary (101).

5. The method according to claim 1 or 2, wherein: The obtained data set includes interpolated sensor data elements (110) and / or data points (111) based on the sensor data.

6. The method according to claim 1 or 2, wherein: The obtained data set includes interpolated data elements (110) and / or data points (111) based on the computer-generated image.

7. The method according to claim 4, wherein: The obtained data set comprises key frames, and the reconstruction step is arranged to generate intermediate frames.

8. The method according to claim 1 or 2, wherein: The threshold for the sparse coefficients is defined by the amount of tolerable error in lossy compression.

9. The method according to claim 1 or 2, wherein: Forming (260) the representation of the visual information includes compressing at least one sparse coefficient set (120).

10. The method according to claim 1, wherein: The at least one condition is a reconstruction error below a certain value and / or a sparsity value above a certain value.

11. The method according to claim 1 or 10, wherein: The learning step (300) is based on machine learning.

12. A computer-implemented method (300) for learning a dictionary set (100), comprising the steps of obtaining a training data set (310) comprising training data elements representing n-dimensional training visual information, partitioning the training visual information into at least one pre-cluster based on sparsity (320), creating a pre-cluster set for each pre-cluster of training visual information, and combining at least one of the at least one pre-cluster set to form the dictionary set (100) (370), wherein: The step of creating at least one pre-cluster set for each pre-cluster comprises: a set of pre-clusters including a dictionary of basis functions for each pre-cluster is initialized (330), clustering (340) at least one pre-cluster of the training visual information by assigning each training data element to a dictionary of a corresponding set of pre-clusters, training (350) each pre-clustered set to minimize reconstruction error and maximize sparsity of the set of training coefficients for the training data elements, The clustering is updated (360) and the training of the set of pre-clustering clusters is iterated until at least one condition is satisfied.

13. The method according to claim 12, wherein: The at least one condition is a reconstruction error below a certain value and / or a sparsity value above a certain value.

14. The method according to claim 12 or 13, wherein: The step of creating the at least one pre-clustered set is based on machine learning.

15. Use of a computer-implemented method according to any one of the preceding claims in compressed sensing, wherein: The visual information is continuously obtained from the sensor system.

16. A computer program product for executing the method according to any one of claims 1 to 14.

17. A system (400) for representing a data set comprising n-dimensional data elements (110) representing visual information, the system comprising: Processor (410), Memory (420), at least one visual information generating device (430), Wherein, the processor (410) is arranged to: storing data in said memory (420), receiving visual information from at least one visual information generating device (430), running a computer program arranged to perform the steps of the method according to any one of claims 1 to 14, and A representation of visual information based on a sparse coefficient set (120) is provided.

18. The system of claim 17, wherein: At least one visual information generating device (430) is a sensor system that captures a physical scene.

19. The system according to claim 17 or 18, wherein: At least one visual information generating device (430) is a database providing visual information.

20. The system according to claim 17 or 18, wherein: At least one visual information generating device (430) is a computer that provides a computer-generated image.

21. The system according to claim 17 or 18, wherein: The processor (410) is arranged to control at least one of the at least one visual information generating device (430).

Images