Imaging device and measurement coding device

Abstract

To enable efficiently compressing an image while suppressing deterioration of image quality in use of compressed sensing.SOLUTION: A compressed sensing unit uses a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequence ordered Walsh-Hadamard) matrix in which respective rows of a NoH (Naturally ordered Hadamard) matrix are arranged in sequence as a measurement matrix to perform compressed sensing processing. The respective rows of the CoWH matrix have mutually different projection patterns when projected onto an image block. The CoWH matrix is configured such that the number of consecutive elements of the same value in the projection pattern decreases from the upper row to the lower row in the CoWH matrix.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an imaging device and a measurement coding device. [Background technology]

[0002] The Nyquist-Shannon sampling theorem states that the sampling rate (SR) must be at least twice the frequency of the signal, but conventional imaging technology faces implementation limitations. Recently developed CMOS image sensors (CIS) have resolutions of 4K, 8K, and 16K, and are expensive and consume high power. In addition, they require huge memory space to store large amounts of image data.

[0003] To reduce the cost of storing and transmitting image data, multimedia compression tools such as JPEG / JPEG2000, AVC (Advanced Video Coding), HEVC (High-Efficiency Video Coding), and VVC (Versatile Video Coding) are widely used. Software and hardware implementations can reach up to 10 times the complexity with each generation, which is the result of CIS design failures.

[0004] All data compression algorithms reduce spatial and temporal redundancies in the original signal. They effectively compress image data in terms of a sparse vector of coefficients using a common transformation basis such as the Discrete Cosine Transform (DCT), a variant of the Fourier Transform, or the Discrete Wavelet Transform (DWT), which is lossy.

[0005] When performing lossy compression to meet low bitrate, the question arises as to why it is necessary to collect high-dimensional and redundant measurements in the first place. Now, instead of collecting and compressing high-dimensional measurements, we can take the compressed measurements and derive the sparsest high-dimensional signal that matches the measurements. This is called "compressive sensing" or "compressive sampling", which will be abbreviated as "CS" in the following.

[0006] Such compressed sensing is a new perspective that has the potential to revolutionize data collection and processing, enabling several enhanced approaches in signal processing, data compression, solving inverse problems, and designing radiating systems, radar, imaging through walls, and Synthetic Aperture Radar (SAR) antennas.

[0007] Block-based computational imaging (BCI) using compressed sensing has been gaining attention in recent years. This paradigm will enable next-generation image sensors to achieve superior performance in terms of shorter readout latency and lower power consumption.

[0008] Nevertheless, there is an increasing need for new encoding algorithms for minimal processing BCI measurements. In general, compressed sensing is built on two main fundamental conditions, consisting of sparsity and incoherence.

[0009] An important condition for the application of CS in a given application is that the signal property, when represented in a particular orthogonal transform basis, must be sparse. This property means that only a small number of coefficients contain most of the signal information and can be expressed as

[0010]

number

[0011]

number

[0012]

number

[0013]

number

[0014] The theory of compressed sensing asserts that incoherence is key, related to the quality of the measurement and the way the signal is sampled. Thus, the measurement matrix used to sample X

[0015]

number

[0016] With various distributions such as Bernoulli, Gaussian, and binary

[0017]

number

[0018]

number

[0019]

number

[0020] To ensure good recovery of the signal, a property of the matrix called the Restricted Isometry Property (RIP) is important. The following formula can be used to determine the lower bound dimension of m for the non-uniform variance measurement matrix:

[0021]

number

[0022]

number

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029] Therefore, using a convex optimization technique called l1 minimization, we can recover the measurements in a classical way:

[0030]

number

[0031]

number

[0032] As greedy-based restoration algorithms, OMP (Orthogonal Matching Pursuit), its extensions Stepwise OMP, A*Omp, CoSaMP, and TwIST2 have been proposed for compressed sensing.

[0033] Compared to l1 minimization, greedy-based restoration algorithms minimize a series of subspace problems and are generally faster. However, they require higher prior knowledge and good measurements, and some applications require reduced SR at the sensing device, so it is not practical to obtain such information.

[0034] Scalar quantization (SQ) provided a simple measurement coding approach (see non-patent literature 1, 2). It has been established that SQ coding is very inefficient in terms of rate-distortion (RD) performance by information theory. Compared to traditional coding, SQ coding produces low bitrate utilization but results in high reconstruction errors. Therefore, complex reconstruction algorithms are required to predict the corrupted encoded measurements, such as QIHT (Quantized Iterative Hard Thresholding), QCoSaMP (Quantized Compressed Sampling Matching Pursuit), and AOP-QIHT (Adaptive Outlier Pursuit For Quantized Iterative Hard Thresholding). The quality degradation is finally displayed in the reconstructed image / video.

[0035] DPCM (Differential Pulse-Code Modulation) is also a simple approach applied to reduce the bit rate on the transmission channel (see Non-Patent Document 3). In DPCM coding, one previous coded block is used to predict the current block. However, the previous coded block may contain irrelevant information that is not related to the current block, resulting in an unnecessary increase in the bit rate. Nevertheless, DPCM coding provides a practical coding method.

[0036] The coding method in Non-Patent Document 4 performs DCT coding on BCI measurements. This approach improves coding efficiency over SQ and DPCM.

[0037] In the study of Non-Patent Document 5, we proposed an alternative approach from the state-of-the-art coding schemes investigated in the quantizer dead zone. In Non-Patent Document 5, we stated that the quality degradation mainly comes from the quantizer, and the distortion is related to both quantization and SR in the measurement coding scheme. Therefore, in this approach, we designed system parameters including the selection of the measurement matrix, the trade-off between quantization and SR, and a restoration algorithm using deep learning. Furthermore, an effective method can jointly control the quantization step and SR to achieve near-optimal quality at any bit rate.

[0038] The directional coding approach was first introduced in Non-Patent Document 6. SDPC (Spatially Directional Predictive Coding) is an efficient measurement coding approach that achieves SQ and DPCM coding with high recovery probability and low bit rate. In this work, a random Gaussian is used as the measurement matrix to sample natural images. However, since a Gaussian measurement matrix is ​​applied, it results in unstable utilization of the bit rate. Importantly, this kind of measurement matrix could not be used for BCI that only supports binary.

[0039] In Non-Patent Document 7, a partial modification of the measurement matrix to generate intra prediction candidates is proposed by combining the concept of intra prediction borrowed from H.264 / AVC with measurement coding (Modified SRM). It is a four-way intra prediction approach using a partially modified random binary measurement matrix, and the work mainly focuses on transmitting the coded measurements to the endpoint. A specific sampling pattern is embedded in the measurement matrix to obtain boundary information while sensing. In Non-Patent Document 7, the obtained information is used to

[0040]

number

[0041] [Non-Patent Document 1] A. Zymnis, S. Boyd and E. Candes, "Compressed Sensing With Quantized Measurements," in IEEE Signal Processing Letters, vol. 17, no. 2, pp. 149-152, Feb. 2010, doi: 10.1109 / LSP.2009.2035667 [Non-Patent Document 2] W. Dai and O. Milenkovic. Information Theoretical and Algorithmic Approaches to Quantized Compressive Sensing. IEEE Transactions on Communications, 59(7):1857-1866, 2011 [Non-Patent Document 3] S. Mun and JE Fowler, "DPCM for quantized block-based compressed sensing of images," 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), Bucharest, 2012, pp. 1424-1428 [Non-Patent Document 4] Yifu Zhang, Shunliang Mei, Quqing Chen and Zhibo Chen, "A novel image / video coding method based on Compressed Sensing theory," 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, NV, 2008, pp. 1361-1364, doi: 10.1109 / ICASSP.2008.4517871 [Non-Patent Document 5] C. Zhao, H. Bai and Y. Zhao, "Optimized Measurements Coding for Compressive Sensing Reconstruction Network," 2018 14th IEEE International Conference on Signal Processing (ICSP), Beijing, China, 2018, pp. 596-599, doi: 10.1109 / ICSP.2018.8652427 [Non-Patent Document 6] J. Zhang, D. Zhao and F. Jiang, "Spatially directional predictive coding for block-based compressive sensing of natural images," 2013 IEEE International Conference on Image Processing, Melbourne, VIC, 2013, pp. 1021-1025, doi: 10.1109 / ICIP.2013.6738211 [Non-Patent Document 7] J. Zhou, D. Zhou, L. Guo, Y. Takeshi and S. Goto, "Measurement-domain intra prediction framework for compressively sensed images," 2017 IEEE International Symposium on Circuits and Systems (ISCAS), Baltimore, MD, 2017, pp. 1-4, doi: 10.1109 / ISCAS.2017.8050262 Summary of the Invention [Problem to be solved by the invention]

[0042] An object of the present invention is to provide an imaging device and a measurement encoding device that are capable of efficiently compressing images while suppressing degradation of image quality when using compressed sensing. [Means for solving the problem]

[0043] The imaging device according to the first aspect includes an image sensor unit that outputs an image in units of image blocks, a compressive sensing unit that performs compressive sensing processing to output m measurement values by applying a measurement matrix of m rows and n columns (m < n) to the pixel values of n pixels constituting the image block, and an analog / digital conversion unit that outputs measurement data by performing analog / digital conversion processing on the m measurement values. The compressive sensing unit performs the compressive sensing processing using a Continuously ordered Walsh-Hadamard (CoWH) matrix based on a Sequency ordered Walsh-Hadamard (SoWH) matrix in which each row of a Naturally ordered Hadamard (NoH) matrix is arranged in sequence order as the measurement matrix. Each row of the CoWH matrix has a different projection pattern when projected onto the image block. The CoWH matrix is configured such that the number of consecutive elements having the same value in the projection pattern decreases as going from the upper row to the lower row in the CoWH matrix.

[0044] The measurement encoding device according to the second aspect includes the imaging device according to the first aspect, and an intra prediction unit that outputs predicted measurement data obtained by predicting the measurement data of the image block using any one of a plurality of intra prediction modes including a vertical prediction mode using an upper adjacent block adjacent to the upper side of the image block and a horizontal prediction mode using a left adjacent block adjacent to the left side of the image block, and a quantization unit that performs quantization processing on residual data representing the difference between the measurement data of the image block and the predicted measurement data output by the intra prediction unit.

Advantages of the Invention

[0045] According to the present invention, there can be provided an imaging device and a measurement encoding device that can efficiently compress an image while suppressing deterioration in image quality when compressive sensing is used.

Brief Description of the Drawings

[0046] [Figure 1]FIG. 1 is a diagram illustrating a configuration of a measurement transmission system according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of a measurement encoding device according to an embodiment. [Diagram 3] FIG. 4 is a diagram illustrating an operation of a compressed sensing unit according to the embodiment. [Figure 4] FIG. 2 is a diagram illustrating vertical prediction modes and horizontal prediction modes according to an embodiment. [Diagram 5] FIG. 1 is a diagram illustrating an example of the configuration of an imaging apparatus according to an embodiment. [Figure 6] FIG. 13 is a diagram illustrating a projection pattern of a measurement matrix according to an embodiment. [Figure 7] FIG. 11 is a diagram illustrating a generation algorithm of a CoWH matrix according to the embodiment. [Figure 8] This figure shows a comparison of the restoration results using l1 minimization and the inverse fast Walsh-Hadamard transform (IFWHT) when using the NoH matrix, the SoWH matrix, and the CoWH matrix, respectively. [Figure 9] FIG. 2 is a diagram illustrating an example of the configuration of a portion of an encoding device according to an embodiment. [Figure 10] FIG. 2 is a diagram illustrating a configuration of a measurement decoding device according to an embodiment. [Figure 11] FIG. 2 is a diagram for explaining the operation and effect of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0047] A measurement transmission system according to an embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0048] (1) Measurement transmission system configuration First, the configuration of the measurement transmission system according to this embodiment will be described. Fig. 1 is a diagram showing the configuration of a measurement transmission system 1 according to this embodiment. As shown in Fig. 1, the measurement transmission system 1 has a measurement encoding device 100 and a measurement decoding device 200.

[0049] The measurement coding device 100 generates image data by image sensing and compresses the image data by compressed sensing. The measurement coding device 100 further compresses the measurement data compressed by compressed sensing by compression coding, and transmits the compressed coded data (bit stream) via a transmission path. For example, the transmission path is a wireless transmission path. The transmission path may include a wired transmission path.

[0050] The measurement decoding device 200 receives the compressed encoded data from the measurement encoding device 100, and decodes the received encoded data into measurement data. The measurement decoding device 200 then restores the original image data from the measurement data using a restoration algorithm, and outputs the image data to, for example, a display.

[0051] The measurement transmission system 1 may constitute a remote monitoring system. For example, the measurement encoding device 100 may be a battery-powered sensing device, and the measurement decoding device 200 may be provided in a server. The server may collect image data from the measurement encoding device 100 and perform remote monitoring of the monitoring target. Such a sensing device is required to consume low power in order to extend its operating time.

[0052] Although only one measurement coding device 100 is shown in FIG. 1, the measurement decoding device 200 may receive encoded data from a plurality of measurement coding devices 100.

[0053] In addition, the transmission of encoded data from the measurement encoding device 100 to the measurement decoding device 200 is not limited to being performed via a transmission path, and the encoded data may be transmitted indirectly from the measurement encoding device 100 to the measurement decoding device 200 via a storage medium.

[0054] (2) Measurement encoding device Next, the measurement coding device 100 according to this embodiment will be described.

[0055] (2.1) Overall configuration of the measurement coding device FIG. 2 is a diagram showing the configuration of the measurement encoding apparatus 100 according to the present embodiment.

[0056] As shown in FIG. 2, the measurement encoding apparatus 100 includes an imaging apparatus 110, an encoding apparatus 120, and a transmission unit 130. The imaging apparatus 110, the encoding apparatus 120, and the transmission unit 130 may be configured separately or integrally. The measurement encoding apparatus 100 may include a battery (not shown).

[0057] The imaging apparatus 110 includes an image sensor unit 111, a compressive sensing unit 112, and an analog / digital (A / D) conversion unit 113.

[0058] The image sensor unit 111 outputs an image in units of image blocks. The image sensor unit 111 includes a complementary metal oxide semiconductor (CMOS) image sensor (CIS). However, the image sensor unit 111 may include a charge-coupled device (CCD) image sensor.

[0059] The compressive sensing unit 112 applies a measurement matrix of m rows and n columns (m < n) to the pixel values of n pixels constituting the image block output by the image sensor unit 111 to output m measurement values. Such processing corresponds to conversion processing from the pixel domain to the measurement domain.

[0060] FIG. 3 is a diagram showing the operation of the compressive sensing unit 112 according to the present embodiment. As shown in FIG. 3, one image is divided into a plurality of image blocks, and the image blocks are acquired in raster scan order. Each image block consists of pixel values for n pixels (n pixels). The compressive sensing unit 112 acquires pixel values from each image block in raster scan order.

[0061] In FIG. 3, a pixel value sequence X1 corresponding to the image block B1, a pixel value sequence X2 corresponding to the image block B2, a pixel value sequence X3 corresponding to the image block B3, and an image block B i corresponding pixel value sequence X i are illustrated. The pixel value sequence is n pixel values (x1 to xn )

[0062] The compressed sensing unit 112 multiplies each pixel value column X by a measurement matrix with m rows and n columns, and outputs the result. As a result, each pixel value column X is represented by m measurement values ​​(y1 to y m ) into a measurement value sequence Y consisting of m and n. Here, m is smaller than n, so the amount of data is compressed.

[0063] 2, the A / D conversion unit 113 performs A / D conversion processing on the m measurement values ​​Y output by the compressive sensing unit 112, thereby outputting measurement data Y, which is a digital signal. Since the number of measurement values ​​is reduced (compressed) by compressed sensing, the power consumption of the A / D conversion unit 113 can be kept low.

[0064] The measurement data Y of each image block output by the A / D conversion unit 113 is further compressed by compression encoding using intra prediction by the encoding device 120. The encoding device 120 has an intra prediction unit 121, a residual data generation unit 122, a quantization unit 123, a Huffman encoding unit 124, a Huffman decoding unit 125, a dequantization unit 126, and a synthesis unit 127.

[0065] The intra prediction unit 121 predicts the measurement data Y of the current image block using one of a plurality of intra prediction modes to generate predicted measurement data Y p The multiple intra prediction modes include a vertical prediction mode using an upper adjacent block adjacent to the upper side of the current image block and a horizontal prediction mode using a left adjacent block adjacent to the left side of the current image block, as shown in Figure 4. The multiple prediction modes further include an average prediction mode that calculates an average of prediction measurement data of the vertical prediction mode and prediction measurement data of the horizontal prediction mode as prediction measurement data.

[0066] The intra prediction unit 121 includes a reference data generation unit 1211 , a prediction data generation unit 1212 , and a selection unit 1213 .

[0067] The reference data generator 1211 generates reference data for an upper adjacent block and reference data for a left adjacent block to obtain predicted measurement data.

[0068] The predicted data generating unit 1212 uses the reference data generated by the reference data generating unit 1211 to generate predicted measurement data Y v and the predicted measurement data Y in the horizontal prediction mode h and the predicted measurement data Y avg and generate.

[0069] The selection unit 1213 selects the predicted measurement data Y v and the predicted measurement data Y in the horizontal prediction mode h and the predicted measurement data Y avg For each of the predicted measurement data Y, the degree of similarity between the predicted measurement data Y and the current image block is calculated, and the predicted measurement data Y having the highest degree of similarity is selected. p to the residual data generating unit 122. As an index value representing the similarity, for example, the sum of absolute differences (SAD) can be used.

[0070] However, if the degree of similarity between any of the predicted measurement data and the measurement data Y of the current image block is low, the selection unit 1213 may output the measurement data Y of the current image block as is to the quantization unit 123. For example, the selection unit 1213 may select the predicted measurement data Y v and the predicted measurement data Y in the horizontal prediction mode h and the predicted measurement data Y avg If the SADs of these two are greater than the threshold, the measurement data Y of the current image block is output to the quantization unit 123 as is.

[0071] The residual data generator 122 calculates the measured data Y of the current image block and the predicted measured data Y output by the intra-prediction unit 121. p Residual data Y that represents the difference with r Generate the residual data Y rto the quantization unit 123.

[0072] The quantization unit 123 converts the residual data Y r (or the measurement data Y of the current image block), and q to the Huffman coding unit 124.

[0073] The Huffman coding unit 124 converts the quantized data Y q The Huffman coding unit 124 performs Huffman coding on the input image to generate coded data, and outputs the coded data to the transmitting unit 130 and the Huffman decoding unit 125. Note that the coding is not limited to Huffman coding, and entropy coding other than Huffman coding may be used.

[0074] The transmission unit 130 transmits the coded data output by the Huffman coding unit 124 to the measurement decoding device 200 via a transmission path. The transmission unit 130 may include a wireless transmitter that transmits a wireless signal including the coded data. The transmission unit 130 also transmits mode information indicating the prediction mode selected by the selection unit 1213, included in the coded data.

[0075] The Huffman decoding unit 125 performs Huffman decoding processing on the coded data output by the Huffman coding unit 124 to obtain quantized data Y q and restore the restored quantized data Y q to the inverse quantization unit 126.

[0076] The inverse quantization unit 126 converts the restored quantized data Y q By performing inverse quantization on the residual data Y r and restore the restored residual data Y deq is output to the synthesis unit 127.

[0077] The synthesis unit 127 synthesizes the restored residual data Y deq and predicted measurement data Y p The measured data Y is restored by combining the above and the measured data Y, and the restored measured data is output to the reference data generation unit 1211 of the intra prediction unit 121.

[0078] In the exemplary configuration of the encoding device 120 shown in FIG. 2, the encoded data output by the Huffman encoding unit 124 is input to the Huffman decoding unit 125. However, the Huffman decoding unit 125 may be omitted. In this case, instead of inputting the encoded data output by the Huffman encoding unit 124 to the Huffman decoding unit 125, the quantized data Y q is input to the inverse quantization unit 126.

[0079] (2.2) Imaging equipment Next, the imaging device 110 according to this embodiment will be described.

[0080] 5 is a diagram showing an example of the configuration of an imaging device 110 according to this embodiment. The imaging device 110 according to this embodiment is a CMOS image sensor that uses compressed sensing, that is, a CS-CIS.

[0081] As shown in FIG. 5, the imaging device 110 includes a pixel array 111a, a row selector 111b, a column selector 111c, a matrix operation trigger unit 112a, a matrix operation unit 112b, and an A / D converter 113.

[0082] The pixel array 111a, the row selection unit 111b, and the column selection unit 111c constitute the image sensor unit 111 shown in Fig. 2. The pixel array 111a has a plurality of pixels arranged in an array. Each pixel receives light and outputs a pixel value, which is an electrical signal. The pixel array 111a is divided into blocks each consisting of n pixels. The row selection unit 111b and the column selection unit 111c select pixels from which pixel values ​​are to be read, and read the pixel values ​​from the selected pixels.

[0083] The matrix calculation trigger unit 112a and the matrix calculation unit 112b constitute the compressive sensing unit 112 shown in Fig. 2. The matrix calculation trigger unit 112a triggers a matrix calculation on a pixel value sequence X consisting of n pixels. The matrix calculation unit 112b performs a matrix calculation on the pixel value sequence X using a measurement matrix, and outputs a measurement value sequence Y consisting of m measurement values ​​to the A / D conversion unit 113. The A / D conversion unit 113 converts the measurement value sequence Y into a digital signal, and outputs measurement data Y.

[0084] FIG. 6 is a diagram showing a projection pattern of a measurement matrix according to this embodiment. Here, the size of an image block is 8 pixels x 8 pixels, that is, the value of n is 64. Also, the value of m is 8 (there are eight patterns from low frequency to high frequency in the figure), that is, eight patterns used when compressing to 12.5% ​​(m / n=8 / 64=12.5%), which converts from n=64 to eight measured values ​​(m=8), are shown. In FIG. 6, each white area represents an element value of "1" in the measurement matrix, and each black area represents an element value of "0" in the measurement matrix.

[0085] As shown in Fig. 6, the compressed sensing unit 112 performs compressed sensing processing using a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequency ordered Walsh-Hadamard) matrix in which each row of a NoH (Naturally ordered Hadamard) matrix is ​​arranged in sequential order as a measurement matrix. Note that sequence refers to the frequency at which values ​​change in one row of the matrix.

[0086] Each row of the CoWH matrix has a different projection pattern when projected onto an image block. The CoWH matrix is ​​configured such that the number of consecutive elements with the same value in the projection pattern decreases from the top row to the bottom row in the CoWH matrix.

[0087] For example, the top row (row #1) of the CoWH matrix has 64 consecutive pixels of the element "1" in the projection pattern. The measurement calculated using row #1 is called y1.

[0088] In the second row (row #2) and the third row (row #3) from the top of the CoWH matrix, the element "1" is consecutive for 32 pixels and the element "0" is consecutive for 32 pixels in the projection pattern. The measurement value calculated using row #2 is called y2, and the measurement value calculated using row #3 is called y3.

[0089] In the fourth row from the top (row #4) of the CoWH matrix, the element "1" appears consecutively for 16 pixels and the element "0" appears consecutively for 32 pixels in the projection pattern.

[0090] In the fifth row from the top (row #5) of the CoWH matrix, the element "1" appears consecutively for 16 pixels, and the element "0" appears consecutively for 16 pixels in the projection pattern.

[0091] In the sixth row from the top (row #6) of the CoWH matrix, the element "1" appears consecutively for 16 pixels and the element "0" appears consecutively for 32 pixels in the projection pattern.

[0092] In the seventh row from the top (row #7) of the CoWH matrix, the element "1" appears consecutively for 16 pixels, and the element "0" appears consecutively for 16 pixels in the projection pattern.

[0093] The eighth row from the top (row #8) of the CoWH matrix has an area of ​​8 consecutive pixels of element "1" and an area of ​​16 consecutive pixels of element "1" in the projection pattern. It also has an area of ​​8 consecutive pixels of element "0" and an area of ​​16 consecutive pixels of element "0".

[0094] The CoWH matrix has rows constituting a first projection pattern that obtains all pixel values ​​of an image block, rows constituting a second projection pattern that obtains pixel values ​​of the upper half of the image block, and rows constituting a third projection pattern that obtains pixel values ​​of the left half of the image block. This makes it possible to smoothly perform intra prediction of vertical prediction mode and horizontal prediction mode in the measurement domain. Specifically, the topmost row (row #1) in the CoWH matrix constitutes the first projection pattern, and the second and third rows (rows #2 and #3) from the top of the CoWH matrix constitute the second and third projection patterns.

[0095] In addition, by adopting CS theory to CIS, the measurement matrix

[0096]

number

[0097] The continuous order Walsh-Hadamard (CoWH) of this embodiment is an optimization of the binary sequential order Walsh-Hadamard (SoWH) which can be derived by the natural order Hadamard (NoH). Note that the columns of the NoH matrix and the SoWH matrix are orthogonal. These matrices are the n-th order matrices

[0098]

number

[0099]

number

[0100]

number

number

[0101]

number

[0102]

number

[0103]

number

[0104]

number

[0105] The sequential rows of the Walsh-Hadamard matrix are derived from the NoH matrix by applying bit reversal and gray-code permutation. This results in

[0106]

number

[0107] In this embodiment, the projection pattern of the SoWH measurement matrix is ​​reorganized by grouping low-frequency coefficients at the top of the measurement matrix. In the CoWH arrangement, the projection pattern is organized to increase the number of connected elements, as shown in Figure 6(c). This can improve the measurement quality while sensing. The related components correspond to groups of consecutive pixels with corresponding values, which influences the restoration algorithm and inverse transformation in terms of richer prior knowledge.

[0108] Also, the good continuous pattern of the CoWH matrix shown in Fig. 6(c) allows for an extension of the lower bound of SR compared to NoH and SoWH matrices, which are usually limited by 50% for good visual quality.

[0109] The CoWH matrix is ​​generated by the algorithm shown in Figure 7. Specifically, let i and j be

number

number

number

number

number

[0110] A (the CoWH matrix) satisfies the RIP with probability at least

[0111]

number

[0112] Here, β is a positive constant.

[0113] For hardware implementation, this expanded matrix A is binarized from its complementary matrix to binary. The sampling mask (projection pattern) of the CoWH matrix is ​​arranged gradually from low-frequency coefficients to high-frequency coefficients compared to the NoH and SoWH matrices.

[0114] Figure 8 shows the comparison of the restoration results using l1 minimization and inverse fast Walsh-Hadamard transform (IFWHT) for the NoH, SoWH, and CoWH matrices, respectively. As shown in Figure 8, the CoWH matrix outperforms the NoH and SoWH matrices in terms of visual quality at the same SR. Note that Figure 8(a) shows the constructed measurement matrix, Figure 8(b) shows the restored image, and Figure 8(c) shows the result of 400% zoom. Note that n = 65536 (256 pixels x 256 pixels), m = 8192 (compressed to 12.5%), and B (i.e., the size of the block used in BCI) = 8.

[0115] (2.3) Encoding device Next, a description will be given of the encoding device 120 according to this embodiment. Fig. 9 is a diagram showing an example of the configuration of a portion of the encoding device 120 according to this embodiment.

[0116] 9, the reference data generation unit 1211, the prediction data generation unit 1212, and the selection unit 1213 configure the intra prediction unit 121 shown in Fig. 2. An input buffer 141 is provided on the input side of the intra prediction unit 121. An output buffer 142 is provided on the output side of the quantization unit 123.

[0117] The reference data generation unit 1211 includes a package splitter 1211a that separates measurement values ​​y1, y2, and y3 from the restored measurement data Y output by the synthesis unit 127, a subtractor 1211b that subtracts measurement value y2 from measurement value y1, a subtractor 1211c that subtracts measurement value y3 from measurement value y1, a register 1211d that stores the output of subtractor 1211b, and a register 1211e that stores the output of subtractor 1211c.

[0118] Here, the measurement value y1 is a measurement value obtained by row #1 constituting a projection pattern that acquires all pixel values ​​of an image block in the CoWH matrix, and corresponds to the sum of all pixel values ​​of the image block.

[0119] The measurement value y2 is a measurement value obtained by row #2 constituting a projection pattern that obtains pixel values ​​in the left half of the image block in the CoWH matrix, and corresponds to the sum of the pixel values ​​in the left half of the image block.

[0120] Measurement value y3 is a measurement value obtained by row #3 constituting a projection pattern that obtains pixel values ​​in the upper half of the image block in the CoWH matrix, and corresponds to the sum of the pixel values ​​in the upper half of the image block.

[0121] Therefore, the output of subtractor 1211b, which subtracts measurement value y2 from measurement value y1, corresponds to the sum of pixel values ​​in the right half of the image block, which is stored in register 1211d. This sum of pixel values ​​in the right half of the image block is used as reference data in the horizontal prediction mode shown in Figure 4. Register 1211d only needs to store reference data corresponding to the left adjacent block of the current image block, and its size is set to 16 bits here.

[0122] On the other hand, the output of the subtractor 1211c, which subtracts the measurement value y3 from the measurement value y1, corresponds to the sum of the pixel values ​​of the lower half of the image block, which is stored in the register 1211e. This sum of the pixel values ​​of the lower half of the image block is used as reference data in the vertical prediction mode shown in Figure 4. The register 1211e stores reference data for each image block that belongs to the row one row above the row to which the current image block belongs. If the width of the image is W and the size of the image block is B, the register 1211e stores reference data for W / B blocks.

[0123] The predicted data generating unit 1212 calculates a pixel value (average value) per pixel from the sum of the pixel values ​​of the lower half of the image block output by the register 1211e, and multiplies this average value by the transpose of the CoWH matrix.

[0124]

number

[0125] In addition, the prediction data generating unit 1212 calculates a pixel value (average value) per pixel from the sum of the pixel values ​​in the right half of the image block output by the register 1211d, and multiplies this average value by the transpose of the CoWH matrix.

[0126]

number

[0127] Furthermore, the predicted data generating unit 1212 generates the predicted measurement data Y v and predicted measurement data Y h The average of the predicted measurement data Y avg is required.

[0128] Such an intra prediction method can be expressed as follows.

[0129] Vertical prediction mode:

[0130]

number

[0131]

number

[0132]

number

[0133] The selection unit 1213 selects the final intra-prediction mode prediction measurement data Y p To select the minimum difference between the predicted candidate and Y, we use SAD as follows:

[0134]

number

[0135]

number

[0136] However, if a predicted candidate is not selected, the selection unit 1213 selects the predicted measurement data Y p Assign to zero.

[0137] Then, the residual data generating unit 122 generates the residual data Y r Calculate.

[0138]

number

[0139] The quantization unit 123 performs scalar quantization to quantize the residual data Y r Quantize the quantized data Y q Output.

[0140]

number

[0141] However, B p is the bit depth parameter, e.g. 4 bits.

[0142] The inverse quantization unit 126 performs the following calculation.

[0143]

number

[0144] Among these processes, quantization is lossy, while intra prediction and entropy coding are lossless.

[0145] (3) Measurement Decoding Device Next, a description will be given of the measurement decoding device 200 according to this embodiment. Fig. 10 is a diagram showing the configuration of the measurement decoding device 200 according to this embodiment.

[0146] As shown in FIG. 10, the measurement decoding device 200 includes a receiving section 210, a decoding device 220, and a reconstruction section 230.

[0147] The receiving unit 210 receives the coded data from the measurement coding device 100 via a transmission path, and outputs the received coded data to the decoding device 220. The coded data includes mode information indicating the prediction mode selected by the measurement coding device 100. The receiving unit 210 also outputs the mode information to the decoding device 220.

[0148] The decoding device 220 includes an intra prediction unit 221, a Huffman decoding unit 222, an inverse quantization unit 223, and a synthesis unit 224. The intra prediction unit 221 includes a reference data generation unit 2211 and a prediction data generation unit 2212.

[0149] The reference data generator 2211 is configured similarly to the reference data generator 1211 of the measurement encoding device 100, and the predicted data generator 2212 is configured similarly to the reference data generator 1211 of the measurement encoding device 100. However, the predicted data generator 2212 generates predicted measurement data Y p Generate the predicted measurement data Y p is output to the synthesis unit 224.

[0150] The Huffman decoding unit 222 performs Huffman decoding processing on the encoded data output by the receiving unit 210 to obtain quantized data Y q and restore the restored quantized data Y q The inverse quantization unit 223 outputs the signal.

[0151] The inverse quantization unit 223 converts the restored quantized data Y q By performing inverse quantization on the residual data Y r and restore the restored residual data Y deqis output to the synthesis unit 224.

[0152] The synthesis unit 224 synthesizes the restored residual data Y deq and predicted measurement data Y p The measured data Y is restored by combining the above and the above, and the restored measured data is output to the restoration section 230.

[0153] The restoration unit 230 restores pixel values ​​X (image block) of the original n pixels from the measurement data Y using a restoration algorithm and outputs the restored pixel values. The CS theory asserts that m-dimensional measurement data is sufficient to restore the measurement of the original signal through an efficient restoration algorithm. For example, the restoration unit 230 restores pixel values ​​X (image block) of the original n pixels using the l1 algorithm and IFWFT.

[0154] (4) Actions and Effects According to this embodiment, a lightweight coding framework for a new high-resolution BCI can be realized, including an optimized binary structural measurement matrix (SRM) based on the Walsh-Hadamard transform (WHT). In addition, spatial redundancy can be reduced by performing intra prediction using the features of the CoWH matrix in the measurement domain. Therefore, images can be efficiently compressed for storage and transmission.

[0155] In the following, we present simulation results for efficient storage and transmission of the coded measurements. The quantitative metrics are peak-to-noise-ratio (PSNR), structural similarity index measure (SSIM) and bits-per-pixel (bpp).

[0156] We use a 4K dataset in the color space Beauty, ReadySetGo, Bosphorus, and HoneyBee defined in A. Mercat, M. Viitanen, and J. Vanne, “UVG dataset: 50 / 120fps 4K sequences for video codec analysis and development,” in Proc. ACM Multimedia Syst. Conf., Istanbul, Turkey, June 2020, where n = 24883200 (3840 × 2160 × 3), SR ∈ {4 / 4, 1 / 8, 1 / 16, 1 / 32, 1 / 64}, and B = 16.

[0157] Table 1 below shows a comparison of minimum process BCI measurements in terms of PSNR (dB), SSIM, and bpp without coding. The comparison of minimum process BCI measurements in Table 1 shows that the CoWH matrix provides quality degradation characteristics of an average of 5-8% PSNR and 0.2-0.5 SSIM for each SR reduction.

[0158] [Table 1]

[0159] Furthermore, when SR is equal to 1 / 16, the CoWH matrix provides a better lower bound with fewer undersampling artifacts compared to 1 / 32 and 1 / 64. In the following, we fix SR to 1 / 16 to evaluate the coding performance and compare it with state-of-the-art work.

[0160] Furthermore, we prove the hypothesis that increasing the relationship between each projected pattern results in higher visual quality, but at the expense of loss of edge detail.

[0161] Furthermore, the intra prediction-based measurement coding (IPMC) by the measurement coding device 100 according to the present embodiment is compared with state-of-the-art work on spatially coding minimal process BCI measurements. For these comparisons, the state-of-the-art codes were selected, but the quantization methods and parameters were changed to be the same to make the coding performance comparison fair. SQ was used and quantization was performed as a baseline.

[0162]

number

[0163] [Table 2]

[0164] As shown in the simulation results in Table 2, IPMC achieved an overall PSNR of 19.96% dB, SSIM of 20.25%, and SQ of 10.34% bpp.

[0165] Compared to DPCM, it achieved a PSNR of 14.50%, SSIM of 9.19%, and bpp of 8.77%, outperforming DPCM in all aspects.

[0166] Compared to similar coding such as SDPC, the PSNR was 14.37%, the SSIM was 9.19%, and the bpp was 7.19%.

[0167] Furthermore, compared to Modified SRM with more advanced sampling and coding, the present embodiment outperforms it by 11.73% in PSNR, 13.09% in SSIM, and 6.74% in bpp. Therefore, Modified SRM is not very efficient because it modifies binary SRM to obtain information from neighboring blocks and finally generates intra prediction candidates as H.264.

[0168] Thus, the IPMC according to the present embodiment exhibits an overall superior coding performance compared to state-of-the-art work, with PSNR of 15.06%, SSIM of 18.74% and bpp of 3.7%.

[0169] Fig. 11 shows an example of details zoomed 400% using the HoneyBee dataset. In Fig. 11, (a) is SQ (baseline), (b) is DPCM, (c) is SDCP, (d) is Modified SRM, and (e) is the present embodiment. As shown in Fig. 11, the results cropped using the HoneyBee dataset show that when SR is 1 / 16, the present embodiment provides richer visual quality compared to the state-of-the-art research.

[0170] Thus, according to the overall performance in Table 2, this embodiment provides all-around trade-off performance compared to state-of-the-art research.

[0171] Furthermore, we implemented the IPMC algorithm in an FPGA, where the resolution is 3820x2140, the total logic utilization is 3,421, and the maximum operating frequency is 102.65MHz. The architecture consumed 28.32mW of power. This resulted in the possibility of real-time encoding 100 times faster than the software implementation, as summarized in Table 3.

[0172] [Table 3]

[0173] (5) Other embodiments A program for causing a computer to execute each process according to the above-described embodiment may be provided. The program may be recorded in a computer-readable medium. Using the computer-readable medium, it is possible to install the program in the computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

[0174] Circuits for executing the various processes performed by the measurement coding device 100 (coding device 120) may be integrated, and the measurement coding device 100 (coding device 120) may be configured as a semiconductor integrated circuit (chip set, SoC).

[0175] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes, etc. are possible without departing from the spirit of the invention. [Explanation of symbols]

[0176] 1: Measurement transmission system 100: Measurement coding device 110: Imaging equipment 111: Image sensor section 111a: pixel array 111b: Line selection section 111c: Column selection section 112: Compressed sensing unit 112a: Matrix operation trigger section 112b: Matrix operation section 113: A / D conversion section 120: Encoding device 121: Intra prediction unit 122: Residual data generation unit 123: Quantization section 124: Huffman coding section 125: Huffman Decoding Unit 126: Inverse quantization section 127: Composite section 130: Transmitter 141: Input buffer 142: Output buffer 200: Measurement decoder 210: Receiving unit 220: Decoding device 221: Intra prediction unit 222: Huffman Decoding Unit 223: Inverse quantization section 224: Composite section 230: Restoration section 1211: Reference data generation unit 1211a: Package splitter 1211b : Subtractor 1211c :Subtractor 1211d : Register 1211e : Register 1212: Prediction data generation unit 1213: Selection section 2211: Reference data generation unit 2212: Prediction data generation unit

Claims

1. an image sensor unit that outputs images in units of image blocks; a compressed sensing unit that performs a compressed sensing process by applying a measurement matrix with m rows and n columns (m<n) to pixel values ​​of n pixels that constitute the image block to output m measurement values; an analog / digital conversion unit that outputs measurement data by performing analog / digital conversion processing on the m measurement values, The compressed sensing unit performs the compressed sensing process by using, as the measurement matrix, a CoWH (Continuously ordered Walsh-Hadamard) matrix based on a SoWH (Sequency ordered Walsh-Hadamard) matrix in which each row of a NoH (Naturally ordered Hadamard) matrix is ​​arranged in sequential order; Each row of the CoWH matrix has a different projection pattern when projected onto the image block, An imaging apparatus, comprising: a CoWH matrix configured such that the number of consecutive elements with the same value in the projection pattern decreases from upper rows to lower rows in the CoWH matrix.

2. The CoWH matrix is a row constituting a first projection pattern for acquiring all pixel values ​​of the image block; rows constituting a second projection pattern for acquiring pixel values ​​of an upper half of the image block; and a third projection pattern of rows that capture pixel values ​​in a left half of the image block.

3. the topmost row in the CoWH matrix constitutes the first projection pattern; The imaging apparatus of claim 2 , wherein the second and third rows from the top of the CoWH matrix constitute the second and third projection patterns.

4. An imaging device according to claim 2 or 3; an intra prediction unit that outputs predicted measurement data obtained by predicting the measurement data of the image block using one of a plurality of intra prediction modes including a vertical prediction mode using an upper adjacent block adjacent to the upper side of the image block and a horizontal prediction mode using a left adjacent block adjacent to the left side of the image block; a quantization unit that performs a quantization process on residual data representing a difference between the measurement data of the image block and the predicted measurement data output by the intra prediction unit.

5. The intra prediction unit, with respect to the vertical prediction mode, obtaining a total pixel value for a lower half of the upper adjacent block in the pixel domain by subtracting measurements obtained by applying the second projection pattern to the upper adjacent block from measurements obtained by applying the first projection pattern to the upper adjacent block; obtaining an average of the pixel values ​​of the lower half of the upper adjacent block by dividing the total pixel value of the lower half by the number of valid pixels; and obtaining predicted measurement data for the vertical prediction mode from the average based on the CoWH matrix.

6. The intra prediction unit, with respect to the horizontal prediction mode, obtaining a sum of pixel values ​​for a right half of the left adjacent block in the pixel domain by subtracting measurements obtained by applying the third projection pattern to the left adjacent block from measurements obtained by applying the first projection pattern to the left adjacent block; obtaining an average of the pixel values ​​of the right half of the left adjacent block by dividing the sum of the pixel values ​​of the right half by the number of valid pixels; 6. The measurement coding device according to claim 4, further comprising a process of obtaining predicted measurement data for the horizontal prediction mode from the average based on the CoWH matrix.

7. The plurality of intra prediction modes further include an average prediction mode that calculates an average of prediction measurement data of the vertical prediction mode and prediction measurement data of the horizontal prediction mode as prediction measurement data; The measurement encoding device described in any one of claims 4 to 6, characterized in that the intra prediction unit calculates a similarity between the measurement data of the image block and each of the prediction measurement data of the vertical prediction mode, the prediction measurement data of the horizontal prediction mode, and the prediction measurement data of the average prediction mode, and selects the prediction measurement data with the highest similarity.

8. an encoding unit that encodes the residual data quantized by the quantization unit; A transmitter that transmits residual data encoded by the encoding unit to a decoding side, The measurement encoding device of claim 7 , wherein the transmission unit further transmits mode information indicating a prediction mode of the predicted measurement data selected by the intra prediction unit.

Images