Systems and methods for mobile zoom digital cameras using compressed sensing

Abstract

In some aspects, methods and systems for compressing a video stream, comprising partitioning captured images into a plurality of pixel blocks. Further, for each pixel block of a captured image, computing any of: a plurality of block pixel sums according to a plurality of block compression masks, wherein a number of said block pixel sums is lesser than a number of pixels included in said set of pixels; or, a block pixel sum according to a block compression mask, wherein the block compression mask is any of the following: (i) associated with said captured image, wherein at least two captured images are each associated with distinct block compression masks; (ii) corresponds a noncontiguous subset of pixels. Further, for each block compression mask, combining the block pixel sums corresponding said block compression mask, to obtain a compressed sum image. In other aspects, methods and systems for decompressing the compressed video streams.

Description

[0001] SYSTEMS AND METHODS FOR MOBILE ZOOM DIGITAL CAMERAS USING COMPRESSED SENSING CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to and claims priority from US Provisional Patent Applications No. 63 / 560,711 filed March 3, 2024, 63 / 649,741 filed May 20, 2024, and 63 / 700,713 filed September 29, 2024, all of which are incorporated herein by reference in their entirety. TECHNOLOGICAL FIELD

[0003] The presently disclosed subject matter generally relates to the field of digital imaging. More particularly, the presently disclosed subject matter relates to the field of compressed imaging.

[0004] BACKGROUND One technique of compressed imaging is pixel binning. Pixel binning is a photograph-capturing technique where adjacent pixels of an image sensor are averaged. The average values constitute pixels of lower resolution images. For example, 2-by-2, 3- by-3, or 4-by-4 blocks of adjacent pixels may be averaged.

[0005] Pixel binning technique is used by image sensors for increasing signal-to-noise ratio of a captured image (e.g., for photography in low light intensity scenarios), and / or in order to capture images in lower resolution than the maximum resolution possible by the image sensor. In other words, the pixel binning technique can function as a method of image compression. Therefore, pixel binning is used in compressed imaging.

[0006] Lower resolution is usually required in bandwidth constrained environments (e.g., limited image data transition rate [from sensor to application processor] or memory availability or limited network capacity). An image sensor set to capture images using the pixel binning technique may be referred to as being in “lower pixel resolution mode” (LRM). An image sensor set to capture images not using the pixel binning technique may be referred to as being in “higher pixel resolution mode” (HRM).

[0007] A plurality of methods may be used to complement the low resolution of images captured in LRM. For example, when performing digital zoom, super resolution techniques (also known as SR) may in conjunction be applied, for increasing resolution (also known as Image Quality, or IQ) of the zoomed image.

[0008] GENERAL DESCRIPTION

[0009] The zoom-factor (ZF) achievable using state-of-the-art methods is only about 1.5, due to the loss of image quality. This is caused by the averaging of the adjacent pixels, resulting loss of scene information.

[0010] In other compressed imaging techniques, there is a deficiency that the data-structure encoding the compressed image, is not an image by itself. That is, data pieces included in the data-structure cannot be juxtaposed, or processed with only basic processing techniques, so as to be displayed on a screen, and be interpreted as an image whether by a human’s eye or by a computerized-vision system. Recovering an image or even parts of an image, from a compressed image, may require intensive processing.

[0011] As size of captured images grows (e.g., reaching 200 megapixels), the limited zoom-factor implies a growing memory / network requirement, that may be difficult to provide. The problem is exacerbated for video streams, that can comprise from hundreds of frames to tens of thousands of frames.

[0012] Solving these problems is therefore especially important for the market of smartphones, where many photographs and videos are captured, reduced availability of memory (both primary and secondary) and reduced network capacity (e.g., network connections are metered) are frequently encountered, and photographs are desired in both high resolution and in low resolution (e.g., different devices and / or different viewing conditions).

[0013] The present disclosure provides systems and methods as presented in the claims.

[0014] According to a first aspect, the presently disclosed subject matter provides a method for compressing a video stream. The video stream includes a sequence of images captured by an image sensor. The images having a first resolution. The method includes, for each captured image, partitioning the captured image into a plurality of pixel blocks, each pixel block including a contiguous set of pixels. For each captured image, the method further includes for each pixel block of the captured image, computing a plurality of block pixel sums according to a plurality of block compression masks. A number of the block pixel sums is lesser than a number of pixels included in the set of pixels. For each captured image, the method further includes for each block compression mask, combining the block pixel sums corresponding the block compression mask, to obtain a compressed sum image. The method is performed so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

[0015] According to some embodiments of this aspect, at least one block compression mask corresponds to noncontiguous subsets of pixels.

[0016] According to some embodiments of this aspect, at least two captured images are each associated with a distinct plurality of block compression masks.

[0017] According to a second aspect, the presently disclosed subject matter provides a method for compressing a video stream. The video stream includes a sequence of images captured by an image sensor. The images having a first resolution. The method includes, for each captured image, partitioning the captured image into a plurality of pixel blocks, each pixel block including a contiguous set of pixels. For each captured image, the method further includes for each pixel block of the captured image, computing at least one block pixel sum according to at least one block compression mask. The at least one block compression mask is associated with the captured image. At least two captured images are each associated with distinct block compression masks. For each captured image, the method further includes combining the block pixel sums to obtain a compressed sum image. The method is performed so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

[0018] According to some embodiments of this aspect, the at least one block compression mask corresponds to noncontiguous subsets of pixels.

[0019] According to some embodiments of this aspect, the at least one block compression mask includes a plurality of block compression masks. The method includes, for each pixel block, computing a plurality of block pixel sums according to the plurality of block compression masks. A number of the block pixel sums is lesser than a number of pixels included in the set of pixels. For each block compression mask, the method further includes combining the block pixel sums corresponding the block compression mask, to obtain a compressed sum image.

[0020] According to a third aspect, the presently disclosed subject matter provides a method for compressing a video stream. The video stream includes a sequence of images captured by an image sensor. The images having a first resolution. The method includes, for each captured image, partitioning the captured image into a plurality of pixel blocks, each pixel block including a contiguous set of pixels. For each captured image, the method further includes for each pixel block of the captured image, computing at least one block pixel sum according to at least one block compression mask. The block compression mask corresponds a noncontiguous subset of pixels. For each captured image, the method further includes combining the block pixel sums to obtain a compressed sum image. The method is performed so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

[0021] According to some embodiments of this aspect, the at least one block compression mask includes a plurality of block compression masks. The method includes, for each pixel block, computing a plurality of block pixel sums according to the plurality of block compression masks. A number of the block pixel sums is lesser than a number of pixels included in the set of pixels. For each block compression mask, the method further includes combining the block pixel sums corresponding the block compression mask, to obtain a compressed sum image.

[0022] According to some embodiments of this aspect, at least two captured images are each associated with distinct block compression masks.

[0023] A method for compressing a video stream, according to any one of the first to third aspects of the presently disclosed subject matter, can optionally comprise one or more of features (i) to (xxiv) below, in any technically possible combination or permutation: i. being implemented by a compressed image sensor. ii. an association of block compression masks to captured images is periodic. iii. a period is two frames or three frames. iv. at least one block compression mask is a binary mask. v. all block compression masks are binary masks. vi. at least two block compression masks correspond complementary subsets of pixels. vii. the plurality of block pixel sums include an average and a difference of sums of the complementary subsets of pixels, so as to obtain a sequence of compressed average images and a sequence of compressed difference images. viii. for each pixel block, a number of pixels in the pixel subset is equal to half a number of pixels included in the pixel block. ix. at least one block compression mask corresponds a subset of pixels that includes any one of: a left half of a pixel block, a right half of a pixel block, an upper half of a pixel block, and a lower half of a pixel block. x. at least one block compression mask corresponds a subset of pixels that includes any one of: an upper right quarter and a lower left quarter of a pixel block, and an upper left quarter and a lower right quarter of a pixel block. xi. at least two block compression masks correspond random subsets of pixels. xii. each pixel block includes the same number of pixels. xiii. each pixel block is square. xiv. a number M of pixels included in each pixel block is in the range of 4 pixels to 36 pixels. xv. any of the following: (a) M=4 and a number of pixels N included in each captured image is in the range of 45 million to 55 million, (b) M=9 and N is in the range of 95 million to 120 million, (c) M=16 and N is in the range of 180 million to 220 million, (d) M=36 and N is in the range of 350 million to 450 million. xvi. a pixel block size equals a predefined downscale ratio. xvii. for at least one captured image, each block is associated with an identical block compression mask. xviii. including optimizing the block compression mask by minimizing at least one of: maximum coherence between distinct block compression masks, and average coherence between distinct block compression masks. xix. a number of pixel blocks is in the range of 10 million to 14 million. xx. the captured images include at least two color-channels. Preferably, the captured images include at least three color-channels. More preferably, the captured images include at least a red channel, a green channel, and a blue channel. xxi. the second resolution is 4k resolution. xxii. including performing preliminarily downsampling of at least one captured image, xxiii. including maximizing any one of: peak signal to noise ratio, and structured image similarity measure, thereby optimizing a balance between reconstruction quality and compression rate. xxiv. including quantization of pixel values.

[0024] According to a fourth aspect, the presently disclosed subject matter provides an image decompression method. The decompression method being for decompressing a compressed image sequence. The compressed image sequence being compressed using the image compression method according to any one of the first to third aspects of the presently disclosed subject matter. The (decompression) method includes applying a neural network to the compressed image sequence, so as to obtain at least one reconstructed image. The at least one reconstructed image having a resolution higher than a resolution of the compressed image sequence.

[0025] According to some embodiments of this aspect, the method includes processing between 1 to 16 compressed images for each reconstructed image. Preferably, the method includes processing any one of 3 images, 5 images, and 7 images, for each reconstructed image.

[0026] According to some embodiments of this aspect, the neural network includes a vision transformer. Preferably, the vision transformer is a spatial-temporal transformer.

[0027] According to some embodiments of this aspect, the method includes applying a motion estimation algorithm to the compressed image sequence.

[0028] According to some embodiments of this aspect, the method includes applying any one of: video frame interpolation, and video super resolution.

[0029] According to some embodiments of this aspect, the method includes applying video frame interpolation, and applying video super resolution. According to a fifth aspect, the presently disclosed subject matter provides a system. The system includes a compressed image sensor. The compressed image sensor being configured for capturing a sequence of images. The compressed image sensor being further configured for compressing the sequence of images, by a compression method according to any one of the first to third aspects of the presently disclosed subject matter.

[0030] According to some embodiments of this aspect, the system further includes decompression processing circuitry. The decompression processing circuitry is configured to perform the decompression method according to the fourth aspect of the presently disclosed subject matter.

[0031] According to a sixth aspect, the presently disclosed subject matter provides a camera module. The camera module includes the system according to the fifth aspect of the presently disclosed subject matter.

[0032] According to a seventh aspect, the presently disclosed subject matter provides a mobile device. The mobile device includes a camera module according to the sixth aspect of the presently disclosed subject matter. The mobile device is preferably a smartphone or a tablet computer.

[0033] In the present disclosure, the following terms and their derivatives may be understood according to the below explanations:

[0034] The term “lower pixel resolution mode” (LRM) may, depending on context, refer to an image sensor set to capture images using the pixel binning technique, or to an image captured using the pixel binning technique.

[0035] The term “higher pixel resolution mode” (HRM) may, depending on context, refer to an image sensor set to capture images not using the pixel binning technique, or to an image captured without using the pixel binning technique.

[0036] The term “frame” may be a synonym for the term “image”.

[0037] BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to better understand the subject matter that is disclosed herein and to exemplify how it may be carried out in practice, embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: Fig. 1 shows a block diagram schematically illustrating compression methods according to the present disclosure.

[0039] Fig. 2A-2C schematically illustrate properties of block compression masks used in methods according to the present disclosure.

[0040] Fig. 3A-3D schematically illustrate combinations of properties for block compression masks used in methods according to the present disclosure.

[0041] Fig. 3E schematically illustrates periodic association of block compression masks to captured images.

[0042] Fig. 4A-4E schematically illustrate examples of block compression masks used in methods according to the present disclosure.

[0043] Fig. 5 schematically illustrates down-sampling.

[0044] Fig. 6A-6B shows a block diagram schematically illustrating decompression methods according to the present disclosure.

[0045] Fig. 7 shows a block diagram schematically illustrating a device incorporating camera systems implementing methods according to the present disclosure.

[0046] DETAILED DESCRIPTION OF EMBODIMENTS

[0047] With reference to Fig. 1, a block diagram, schematically illustrating compression methods according to the present disclosure, is shown. Steps included in the methods are illustrated by rectangles. Objects that may be used as input for steps of the methods, or may be output of steps of the methods, are illustrated by rounded-corners rectangles.

[0048] A method 100 may include a step 110 of capturing a sequence of images of a scene 105. In other words, a video stream 115 may be captured in step 110. The images captured by the image sensor may have a first resolution.

[0049] In some embodiments, the method 100 may include a step 117 of performing preliminarily down-sampling of at least one captured image. Down-sampling is described hereinbelow in relation to Fig. 5. The method 100 may include, for each captured image, a step 120 of partitioning the captured image into a plurality of pixel blocks 125. Each pixel block may include a contiguous set of pixels.

[0050] The method 100 may include a step 130 of summing values of pixels, so as to obtain a plurality of pixel block sums 135. The summing may be according to at least one block compression mask 127. That is, according to at least one predefined table of weights, where each pixel may be assigned a weight. In other words, for each pixel block, at least one weighted sum of pixels included in that pixel block may be computed.

[0051] Summing according to more than one mask may be described as “dynamical binning” (DB).

[0052] The method 100 may include a step 140 of combining the block pixel sums 135. For each captured image, block pixel sums corresponding the captured image may be combined, so as to obtain a compressed sum image. For example, the block pixel sums may be stored in a data structure describing an image (e.g., a rectangular array, a bitmap data structure). In some embodiments, the step 140 may include processing (i.e., manipulation) of the block pixel sums. For example, the data structure describing an image may be according to the JPEG format. The step 140 may include applying a discrete cosine transform to the block pixel sums.

[0053] The method 100 may be performed so as to obtain a sequence of compressed sum images 145 having a second resolution lower than the first resolution. In other words, the method 100 may be performed so as to compress video stream 115, where the compressed data may be a video stream.

[0054] The method 100 may be implemented in software or in hardware, or by a combination thereof. For brevity, when implementations by hardware and / or by a combination of hardware and software, may both be referred, the implementation may be referred to as implementation by a compressed image sensor.

[0055] As an example of implementation in hardware, the method 100 may be implemented by a compressed image sensor, where the compressed image sensor may not be communicating with a programmable controller (e.g., a programmable processor).

[0056] As an example of implementation in a combination of hardware and software, the method 100 may be implemented by a compressed image sensor, where the compressed image sensor may be communicating with a programmable controller (e.g., a programmable processor), so as to receive commands from the controller.

[0057] As an example of implementation in software, the method 100 may be implemented post-capture by a processor, where the video stream may be stored in memory.

[0058] It is noted that in some embodiments, step 110 may be omitted from method 100. In other words, step 110 may be performed without of compressing the captured images, and the rest of the steps of method 100 may be deferred to a later time. For example, the method 100 may applied by an image-database management program on stored images, in preparation to be sent to a client over a slow network connection.

[0059] In some embodiments, the method 100 may include a step 150 of optimizing the block compression mask 127. Optimizing the block compression mask 127 may be achieved by minimizing at least one of: maximum coherence between distinct block compression masks, and average coherence between distinct block compression masks. In some embodiments, the step 150 of optimizing the block compression mask 127 may be static. That is, the step 150 may be performed once, before processing any captured image. In some embodiments, the step 150 may be performed dynamically. That is, compressed sum images may be processed so as to obtain optimal block compression masks 127. For example, a computerized vision algorithm may be applied so as to estimate where important features may be positioned. Block compression masks 127 may be computed so as to keep as much information on these positions.

[0060] In some embodiments, the step 150 may include maximizing any one of: peak signal to noise ratio, and structured image similarity measure, thereby optimizing a balance between reconstruction quality and compression rate.

[0061] In some embodiments, a number of pixels included in a pixel block may vary. In some embodiments, each pixel block may include the same number of pixels. For example, rectangles having the same height and width. In other words, rectangles of X-pixels by Y- pixels, where X, Y are predefined. For example, 5-by-7 rectangular pixel blocks.

[0062] In some embodiments, each pixel block may be square. For example, each pixel block may be a 2-by-2 pixel block, may be a 3-by-3 pixel block, or may be a 4-by-4 pixel block. In some embodiments, a number M of pixels included in each pixel block is in the range of 4 pixels to 36 pixels. In some embodiments, a pixel block size may be equal to a predefined downscale ratio. That is, the ratio of the first resolution to the second resolution, may be equal to the pixel block size. In yet other words, for a X X X pixel block size, the second resolution may be X times less than the first resolution. For example, the pixel block size may be 5-by-5, and the downscale ratio may be 5.

[0063] It is noted that the method 100 is applicable both for monochrome images and for colored images. In other words, the method 100 is applicable for images captured using an image sensor having a single channel, and for images captured using an image sensor having a plurality of channels. In some embodiments, the captured images may include at least two color-channels, preferably at least three color-channels. More preferably, in some embodiments, the captured images may include at least a red channel, a green channel, and a blue channel. In some embodiments, the method 100 may include demosaicing, as known in the art. In a brief description, demosaicing is the process of interpolating values of pixels for different channels. Demosaicing may be required when sensor elements included in the image sensor are configured to detect (sense) different channels (e.g., different colors of visible-light). Therefore, not every pixel may provide information on every detectable channel, and interpolation may be required. For example, if the image sensor includes a Bayer color filter array.

[0064] Some specific embodiments are notable. In some embodiments, a number of pixels included in a pixel block M may be 4, and a number of pixels N, included in each captured image, may be in the range of 45 million to 55 million pixels, e.g., 48MP or 50MP. In some embodiments, M may be 9, and N may be in the range of 95 million to 120 million pixels, e.g., 108MP. In some embodiments, M may be 16, and N may be in the range of 180 million to 220 million pixels, e.g., 200MP. In some embodiments, M may be 36, and N may be in the range of 350 million to 450 million pixels. In some embodiments, a number of pixel blocks may be in the range of 10 million to 14 million pixel blocks, e.g., 12MP or 12.5MP. In some embodiments, the second resolution may be a 4k resolution. It is to be noted that these embodiments are non-limiting.

[0065] With reference to Figs. 2A-2C, properties of block compression masks, that may be used in methods according to the present disclosure, are schematically illustrated.

[0066] As illustrated in Fig. 2A, in some embodiments, different captured images may be associated with different block compression masks. In other words, the at least one block compression mask may include a plurality of block compression masks. Each captured image may be associated with one block compression mask. At least two captured images may each be associated with distinct block compression masks. In yet other words, captured image 1 may be associated with block compression mask , captured image 2 may be associated with block compression mask M2, and so on until captured image K may be associated with block compression mask MK. At least two block compression masks Ma, Mp may be different (i.e., Ma#= Mp). It is noted that the plurality of block compression masks may not all be different. In other words, there can be two distinct captured images associated with the same block compression mask. In yet other words, there can be block compression masks

[0067] As illustrated in Fig. 2B, in some embodiments, the at least one block compression mask may include a plurality of block compression masks Each captured image may be associated with the plurality of block compression masks

[0068] As illustrated in Fig. 2C, in some embodiments, the at least one block compression mask may correspond noncontiguous subset of pixels. In some embodiments, all the block compression masks may correspond noncontiguous subset of pixels. Noncontiguous subsets of pixels are described hereinbelow in relation to Fig. 4A.

[0069] With reference to Figs. 3A-3D, combinations of properties of block compression masks, that may be used in methods according to the present disclosure, are schematically illustrated. Generally, any two features described in relation to each of Figs. 2A-2C may be combined.

[0070] As illustrated in Fig. 3A, the features illustrated in Figs. 2A, 2C may be combined. That is, in some embodiments, different captured images may be associated with different block compression masks. In other words, the at least one block compression mask may include a plurality of block compression masks. Each captured image may be associated with one block compression mask. At least two captured images may each be associated with distinct block compression masks. Further, at least one block compression mask may correspond to noncontiguous subsets of pixels. In some embodiments, all the block compression masks may correspond to noncontiguous subsets of pixels.

[0071] As illustrated in Fig. 3B, the features illustrated in Figs. 2B, 2A may be combined. That is, in some embodiments, at least two captured images may be each associated with a distinct plurality of block compression masks. In other words, captured image 1 may be associated with a first plurality of block compression masks "'M , captured image

[0072] 2 may be associated with a second plurality of block compression masks and so on until captured image K may be associated with a A'-th plurality of block compression

[0073] As illustrated in Fig. 3C, the features illustrated in Figs. 2B, 2C may be combined. That is, in some embodiments, the at least one block compression mask may include a plurality of block compression masks Each captured image may be associated with the plurality of block compression masks Further, at least one block compression mask included in the plurality of block compression masks may correspond to noncontiguous subsets of pixels. In some embodiments, all the block compression masks may correspond to noncontiguous subsets of pixels.

[0074] As illustrated in Fig. 3D, the features illustrated in Figs. 2A, 2B, 2C may be combined. That is, in some embodiments, at least two captured images may be each associated with a distinct plurality of block compression masks. In other words, captured image 1 may be associated with a first plurality of block compression masks "'M , captured image 2 may be associated with a second plurality of block compression masks and so on until captured image K may be associated with a A'-th plurality of block compression masks At least two pluralities of block compression masks M^a>••• may be different. In other words, there is at least one block compression mask not included in the plurality of block compression masks Further, at least one block compression mask, included in any plurality of block compression masks, may correspond to noncontiguous subsets of pixels. In some embodiments, at least one plurality of block compression masks may consist of block compression masks that may correspond to noncontiguous subsets of pixels. In some embodiments, all the block compression masks may correspond noncontiguous subset of pixels.

[0075] In embodiments where a captured image may be associated with a plurality of block compression masks (e.g., embodiments illustrated in Figs. 2B, 3B, 3D), in step 130 the method 100 (illustrated in Fig. 1) may include, for each pixel block of a captured image, computing a plurality of block pixel sums according to the plurality of block compression masks associated with the captured image. A number of the block pixel sums may be lesser than a number of pixels included in the set of pixels. In some embodiments, the number of block compression masks, included in the plurality of block compression masks associated with the captured image, may be lesser than the number of pixels included in the set of pixels. In step 140 the method 100 may include, for each block compression mask, combining the block pixel sums corresponding to the block compression mask, to obtain a compressed sum image. In other words, for each captured image, block pixel sums corresponding different block compression masks may not be combined together. In yet other words, for each captured image, a plurality of compressed sum images may be computed, where each compressed sum image may consist of block pixel sums associated with a single block compression mask.

[0076] In embodiments where at least two captured images may be each associated with a distinct plurality of block compression masks (e.g., embodiments illustrated in Figs. 3B, 3D), at least two pluralities of block compression masks may include a different number of block compression masks. It is noted that the pluralities of block compression masks may not all be different. In other words, there can be two distinct captured images associated with the same plurality of block compression masks.

[0077] In some embodiments, for at least one captured image, each pixel block may be associated with an identical block compression mask. In other words, for at least one captured image, all pixel blocks corresponding to the at least one captured image, may be associated with the same block compression mask.

[0078] With reference to Fig. 3E, periodic association of block compression masks to captured images is schematically illustrated.

[0079] In some embodiments, where different captured images may be associated with different block compression masks, e.g., embodiments illustrated in Figs. 2A, 3A, 3B, or 3D, an association of block compression masks to captured images may be periodic. In other words, an associating function, that may output a block compression mask according to an input of a position of a captured image in the sequence of captured images (i.e., first, second, third, etc.), may cycle through the block compression masks in a specific order. In yet other words, all captured images, being of positions in the sequence of captured images having the same reminder after division by the number of block compression masks, may be associated with the same block compression mask. The number of block compression masks may be referred to as a period.

[0080] For example, in some embodiments, a period may be two frames (i.e., captured images). All captured images, being of even positions in the sequence of captured images (i.e., the 2nd, 4th, 6thetc.), may be associated with a first block compression mask. All captured images, being of odd positions in the sequence of captured images (i.e., the 1st, 3rd, 5thetc.), may be associated with a second block compression mask.

[0081] In a different example, in some embodiments, a period may be three frames. All captured images, being of positions in the sequence of captured images divisible by three with reminder one (i.e., the 1st, 4th, 7thetc.), may be associated with a first block compression mask. All captured images, being of positions in the sequence of captured images divisible by three with reminder two (i.e., the 2nd, 5th, 8thetc.), may be associated with a second block compression mask. All captured images, being of positions in the sequence of captured images divisible by three with zero reminder (i.e., the 3rd, 6th, 9thetc.), may be associated with a third block compression mask.

[0082] It is noted that block compression masks, associated with captured images having distinct reminder (e.g., 0 and 1), can be identical. In other words, the definition of a period (i.e., the number of block compression masks) includes multiplicities of block compression masks. For example, in embodiments where a period may be three frames, two block compression masks may be equal.

[0083] In embodiments where different captured images may be associated with different pluralities of block compression masks, e.g., embodiments illustrated in Figs. 3B, or 3D, an association of pluralities of block compression masks to captured images may be periodic. In other words, in the definition of “periodic” and “period” and in the examples, the reference may be to pluralities of block compression masks instead of individual block compression masks.

[0084] With reference to Figs. 4A-4D examples of block compression masks, used in methods according to the present disclosure, are schematically illustrated.

[0085] In some embodiments, at least one block compression mask may be a binary mask. That is, there may be only two weights, a zero weight and a non-zero weight. In other words, the masks may define a selection of a subset of pixels. It is noted that the non-zero weight may not necessarily be unity. For example, the non-zero weight may be a reciprocal of the number of pixels selected. Thereby, the block pixel sum may be equal to the average value of the selected pixels. In some embodiments, different non-zero weights may be used in different embodiments so as to provide a scaled average value of the selected pixels. In some embodiments, all block compression masks may be binary masks.

[0086] Throughout the examples illustrated in Figs. 4A-4B (numbered 400 through 465), pixels having a non-zero weight (i.e., pixels being selected) are shaded and marked with an “A” letter. Pixels having a zero weight (i.e., pixels being not selected) are not shaded, and are marked with an “B” letter. It is noted that in Fig. 4B, each square can correspond to a single pixel or to a sub-block of pixels, e.g., a 2-by-2 sub-block included in a 4-by-4 pixel block, or a 3-by-3 sub-block included in a 6-by-6 pixel block.

[0087] In some embodiments, for each pixel block, a number of pixels in the pixel subset may be equal to half a number of pixels included in the pixel block. For example, pixel subsets 400, 405 and 410. In embodiments where a pixel block may include an odd number of pixels, half a number of pixels may be defined according to rounding-up. For example, for a 5-by-5 pixel block, half a number of pixels included in the pixel block may refer to 13 pixels.

[0088] In some embodiments, the at least one block compression mask may correspond to a subset of pixels that may include any one of: a left half of a pixel block 450, a right half of a pixel block 455, an upper half of a pixel block 460, and / or a lower half of a pixel block 465. In some embodiments, the at least one block compression mask may correspond a subset of pixels that may includes any one of: an upper right quarter and a lower left quarter of a pixel block 440, and / or an upper left quarter and a lower right quarter of a pixel block 445. In some embodiments, a number of pixels in the pixel subset may not be equal to half a number of pixels included in the pixel block. For example, the at least one block compression mask may correspond a subset of pixels that may includes any one of: an upper right quarter of a pixel block 420, a lower right quarter of a pixel block 425, a lower left quarter of a pixel block 430, an upper left quarter of a pixel block 435, and / or combination of any three block compression masks thereof.

[0089] It is noted that the pixel subset may not be required to be of specific structure. In some embodiments, at least two block compression masks may correspond random subsets of pixels. For example, block compression mask 415.

[0090] In some embodiments, at least two block compression masks may correspond to complementary subsets of pixels. That is, for at least one pair of block compression masks, the selection of pixels may be opposite. In other words, all pixels that may be marked with an “A” letter in a first block compression mask, may be marked with a “B” letter in a second block compression mask, and vice versa. Examples of block compression masks corresponding complementary subsets of pixels include: block compression masks 440 and 445, block compression masks 450 and 455, and block compression masks 460 and 465.

[0091] In some embodiments, where block compression masks corresponding complementary subsets of pixels may be included, the plurality of block pixel sums may include an average and a difference of sums of the complementary subsets of pixels. In other words, in step 130 the method 100 (illustrated in Fig. 1) may include computing an average and a difference of sums of the complementary subsets of pixels. Computing the average and a difference of sums of the complementary subsets of pixels may be in order to obtain a sequence of compressed average images and a sequence of compressed difference images. The sequence of compressed average images and the sequence of compressed difference images may be included in the sequence of compressed sum images. In some embodiments, the sequence of compressed sum images may consist of the sequence of compressed average images and the sequence of compressed difference images.

[0092] As indicated herein above, in some embodiments, the at least one block compression mask may correspond to noncontiguous subsets of pixels. A contiguous subset of pixels may be defined as a subset of pixels included in a single pixel block, where any pixel included in the subset of pixels, may be connected to any other pixel included in the subset of pixels, by a path traversing only pixels included in the subset of pixels, where the path consists of only left / right / up / down movements to a nearby pixel.

[0093] A subset of pixels included in a single pixel block, that may not be contiguous, may be referred to as noncontiguous. In other words, a noncontiguous subset of pixels may be defined as a subset of pixels included in a single pixel block, where at least one pixel, included in the subset of pixels, may not be connected to at least one other pixel included in the subset of pixels, by a path traversing only pixels included in the subset of pixels, where the path consists of only left / right / up / down movements to a nearby pixel. It is noted that a noncontiguous subset of pixels can include a contiguous sub-subset of pixels.

[0094] Fig. 4A illustrates examples of noncontiguous subsets of pixels 400 405 410 415. Pixels included in a noncontiguous subset of pixels are shaded and marked with an “A” letter.

[0095] In noncontiguous subset of pixels 400, any path between two different pixels included in the subset 400, traversing only pixels included in the subset of pixels, requires only diagonal movements, that are unallowed by the definition. In noncontiguous subset of pixels 405, a path between two different pixels included in the subset 405, traversing only pixels included in the subset of pixels, may have only left / right / up / down movements for some pairs of pixels. However, for some pairs of pixels, diagonal movements are required. In noncontiguous subset of pixels 410, a path between two different pixels included in the subset 410, for some pairs of pixels, cannot traverse only pixels included in the subset of pixels 410.

[0096] For noncontiguous subsets of pixels 400, 405 and 410, a complimentary subset of pixels is also a noncontiguous subset of pixels. However, such property is not mandatory. For example, for noncontiguous subset of pixels 415, a complimentary subset of pixels is not a noncontiguous subset of pixels.

[0097] Figs. 4C-4E schematically illustrate further examples of block compression masks, used in methods according to the present disclosure.

[0098] Fig. 4C schematically illustrates examples of “tetromino” partitions of a pixel block 470 475. The tetromino shape may be described as “T-shaped”. A different description of the tetromino shape may be: up to rotation, three of the four pixels form a row, and a fourth pixel is adjacent to a middle pixel (of the row of pixels) so that the middle pixel and the fourth pixel form a two-pixel column. Generally, in some embodiments, a pixel block may include a tetromino-shaped subset of pixels.

[0099] For example, a 4-by-4 pixel block may be partitioned into four four-pixel subsets, each having a tetromino shape (e.g., according to any one of partitions 470475). Different subsets are marked by different letters and shadings. Partition 475 may be a mirror image of partition 470.

[0100] Block compression masks may follow (i.e., be defined according to) a tetromino partition. Generally, weights assigned to pixels may correspond the tetromino-shaped subsets. In some embodiments, all the pixels may have a non-zero weight, where the different subsets may represent different weights. In some embodiments, only some of the pixels may be summed, where the selection of pixels may correspond the subsets (e.g., only pixels in “A” subset may be summed, or only pixels in “B” and “D” subsets may be summed).

[0101] Figs. 4D-4E schematically illustrate exemplary sets of block compression masks 480485, that include block compression masks following tetromino partitions. The sets of block compression masks 480 485 may each consist of binary block compression masks. The sets of block compression masks 480 485 may each consist of complementary block compression masks. Throughout the masks illustrated in Figs. 4D-4E, pixels being selected are not shaded, and are marked with a “1” digit. Pixels being not selected are shaded, and are marked with a “0” digit.

[0102] A block compression mask having an “a” appended to its associated referencenumeral may be complementary to a block compression mask having an “b” appended to its associated reference-numeral. For example, block compression mask 481a may be complementary to block compression mask 481b. When both block compression masks of a complementary pair may be referenced hereinbelow, only the reference numeral may be indicated. For example, “block compression masks 481” may refer to both block compression mask 481a and to block compression mask 481b.

[0103] Block compression masks 481 482 483 may be included in both sets of block compression masks 480 485. Block compression masks 486 487 488 may be included in block compression masks set 485. Block compression masks 481 482 483 may follow partition 470. Block compression masks 486487488 may follow partition 475. The block compression masks included in the sets of block compression masks 480 485 maybe described, using the marking of Fig. 4C, as follows:

[0104] I. In block compression mask 481a (481b) only subsets A, C (B, D) may be summed.

[0105] II. In block compression mask 482a (482b) only subsets A, B (C, D) may be summed.

[0106] III. In block compression mask 483a (483b) only subsets A, D (B, C) may be summed.

[0107] IV. In block compression mask 486a (486b) only subsets C, D (A, B) may be summed.

[0108] V. In block compression mask 487a (487b) only subsets A, C (B, D) may be summed.

[0109] VI. In block compression mask 488a (488b) only subsets B, C (A, D) may be summed.

[0110] The sets of block compression masks 480 485 may each be used in methods according to the present disclosure (e.g., method 100) for efficient x4 downscaling of images, using up to two block pixel sums per frame. Thereby, compression-ratio of about one-eight (i.e., 8 times less data), or even more than about one-sixteenth (i.e., 16 times less data), may be achieved.

[0111] In some embodiments, block compression masks may consist of (i.e., exclusively be) block compression masks included in the sets of block compression masks 480485. In some embodiments, where an association of block compression masks to captured images may be periodic (i.e., as described hereinabove in relation to Fig. 3E), block compression masks set 480 may be used with a period of three, where each captured image may be associated with one of the block compression masks 481 482 483. In variants of these embodiments, block compression masks set 480 may be used with a period of six, where each captured image may be associated with one of the block compression masks 481a 481b 482a 482b 483a 483b. In some other embodiments, block compression masks set 485 may be used with a period of six, where each captured image may be associated with one of the block compression masks 481 482 483 486 487 488. In variants of these embodiments, block compression masks set 485 may be used with a period of twelve, where each captured image may be associated with one of the block compression masks 481a 481b 482a 482b 483a 483b 486a 486b 487a 487b 488a 488b.

[0112] It is noted that the order of the block compression masks illustrated in Figs. 4D-4E is non limiting, and the order of the block compression masks, included in the sets of block compression masks 480 485, may be permuted. Moreover, the order of the block compression mask pairs, included in the sets of block compression masks 480485, may in principle be random. Further, block compression masks, associated with different pairs of complementary block compression masks, may be mixed.

[0113] In some embodiments, for some captured images, only one block compression mask of any pair of complementary block compression masks 481 482483486487488 may be used, while for some captured images, both block compression mask of any pair of complementary block compression masks 481 482483486487488 may be used. In other words, for some frames, only one block pixel sum may be computed, while for other frames, two block pixel sums may be computed. For example, for a first frame a block pixel sum may be computed according to block compression mask 481a, but not according to block compression mask 481b. For a second frame, block pixel sums may be computed according to both block compression masks 482. In some embodiments, block compression masks of a complementary pair may be both associated with some frames, while only one mask of the same complementary pair may be both associated with other frames.

[0114] With reference to Fig. 5, down-sampling is schematically illustrated. Downsampling may be described as preliminary binning. That is, instead of processing individual pixels (i.e., step 120 of partitioning and step 130 of summing in method 100), blocks of pixels may be processed. For example, a pixel array 510 may include a plurality of pixels. 2-by-2 sub-blocks of pixels may be averaged, so as to obtain an effective pixel array 520, having a number of effective pixels being one fourth of the pixels included in the pixel array 510. The effective pixel array 520 may then be partitioned, and (effective-) pixel blocks may be summed according to at least one block compression mask to obtain block pixel sums, and the block pixel sums may be combined to obtain a compressed sum image.

[0115] Down-sampling may be required, for example, in applications where image processing or image data transmission facilities may be constrained. For example, in applications where images are captured in a very high frame rate, such as “slow-motion”.

[0116] In some embodiments, the method 100 may include a step of quantizing pixel values, as known in the art. In a brief description, quantizing pixel values is the process of reducing the amount of values a pixel may potentially take. For example, an image sensor may have 16-bit resolution (e.g., values measured by pixels may range from 0 to 65535). Pixel values may be quantized, so as to have 8-bit resolution (e.g., quantized values may range from 0 to 255). In embodiments where pixel values may be continuous (i.e., analog), quantizing pixel values may include digitizing the pixel values.

[0117] Quantizing pixel values may be beneficial by reducing volume of data to be processed, without reducing spatial resolution of images. For example, in hardware implementations, quantizing pixel values may reduce the complexity of processing circuitry, such as reducing the size of registers and / or summing circuitry. Quantizing pixel values may also enable a more robust design of the processing circuitry, less footprint of the processing circuitry, and / or less power consumption by the processing circuitry. In software implementations and mixed hardware-software implementations, reducing the volume of data may also benefit by reducing the required bandwidth (e.g., for transmitting the data between the image sensor and the processing circuitry).

[0118] With reference to Figs. 6A-6B, block diagrams, schematically illustrating decompression methods 600 6600 according to the present disclosure, are shown. Method 6600 may be a variant of method 600. Method elements (e.g., steps, manipulated objects) common to both methods 600 6600 are numbered with the same reference numerals. It is noted that general reference to method 600, or to common method-elements, implies reference to both methods 600 6600.

[0119] Generally, a method 600 may be used to decompress a compressed image sequence 610 compressed using image compression methods as described herein above (e.g., method 100 illustrated in Fig. 1).

[0120] Generally, the method 600 may include a step 620 of applying a reconstruction algorithm to the compressed image sequence, so as to obtain at least one reconstructed image 630 having a resolution higher than a resolution of the compressed image sequence 610.

[0121] In some embodiments, the at least one reconstructed image 630 may be a reconstruction of a specific region of interest included in a compressed image. In other words, the at least one reconstructed image 630 may not be a reconstruction of a whole compressed image. For example, in digital zoom applications, the at least one reconstructed image 630 may be a reconstruction of a region to which the zoom may be applied.

[0122] In some embodiments (e.g., method 6600), the reconstruction algorithm may include a step 675 of applying a matrix being a generalized inverse of a matrix representing the block compression mask(s) used in the compression methods as described herein above. For example, the reconstruction algorithm may include a step 670 of an initial reconstruction. Applying the generalized inverse matrix 675 may be performed in the initial reconstruction 670.

[0123] In some embodiments, the method 600 may include a step 640 of applying a neural network to the compressed image sequence 610. In some embodiments, the neural network may include a vision transformer, preferably a spatial-temporal transformer 650. In other embodiments (e.g., method 6600), the neural network may include a convolutional neural network (CNN) 645. In some embodiments, the neural network may include both a spatial- temporal transformer 650 and a convolutional neural network 645.

[0124] For each reconstructed image, the reconstruction algorithm may process at least one compressed image. In some embodiments, the reconstruction algorithm may process between 1 to 16 compressed images, for each reconstructed image. Preferably, in some embodiments, the reconstruction algorithm may process any of 3 compressed images, 5 compressed images, and / or 7 compressed images, for each reconstructed image.

[0125] It is noted that compressed images, that may be processed so as to compute the at least one reconstructed image 630, may not necessarily be consecutive. In other words, if a first compressed image has an index j and n frames may be processed, a last compressed image may have an index larger-than-or-equal to j + n.

[0126] It is noted that in embodiments where an association of block compression masks to captured images may be periodic (i.e., as described hereinabove in relation to Fig. 3E), the compressed images that may be processed so as to compute the at least one reconstructed image 630, may not necessarily correspond a period, nor a number thereof necessarily be equal to the periodicity. For example, in embodiments that may use exclusively any of the block compression mask sets 480 485 (schematically illustrated in Fig. 4E) with a period of six, the number of compressed images that may be processed, may be seven.

[0127] In some embodiments, the method 600 may include a step 660 of applying a motion estimation algorithm to the compressed image sequence. A motion estimation algorithm may be applied so as to refine an output of the reconstruction algorithm. In some embodiments, the method 6600 may include a step 680 of video frame interpolation. That is, at least one decompressed frame, having no corresponding compressed frame included in the compressed image sequence 610, may be computed. The step 680 of video frame interpolation may provide reconstructed images that may correspond original sequence of frames, thereby completing potentially missing frames.

[0128] A step 680 of video frame interpolation may be beneficial, for example, if a video compression method (e.g., method 100) may reduce the number of frames (e.g., omit frames and / or combine frames), or in other words, the compressed image sequence 610 may include less frames than an original sequence of frames (e.g., sequence of images of a scene 105 described in relation to method 100).

[0129] In some embodiments, the method 6600 may include video frame interpolation, and video super resolution, simultaneously.

[0130] As indicated herein above, the method 100 may be implemented in software or in hardware, or by a combination thereof. In other words, a system may include a compressed image sensor, may be configured for capturing a sequence of images and configured for compressing the sequence of images by a compression method as described herein above.

[0131] As an example of implementation in hardware, the compressed image sensor may include an image sensor, circuitry configured to read pixels of the image sensor, and circuitry configured to perform the summing. For example, a plurality of op-amps in summing- amplifier configuration, with voltage-dependent resistors. The voltagedependent resistors may provide selection of different block compression masks. In some embodiments, a zero weight may be implemented via an in-pixel shutter (IPS). For example, via CMOS transmission-gates, that may block selected pixels. In some embodiments, the compressed image sensor may include multiplexers. Multiplexers may provide selecting different pixels in a pixel block. Multiplexers may be used, for example, to implement block compression masks corresponding to complementary subsets of pixels. Signals to the summing circuitry (e.g., the voltage -dependent resistors, CMOS transmission-gates, and / or multiplexers) may be provided by a non-programmable hardware (e.g., a non-programmable ASIC).

[0132] As an example of a mixed hardware-software implementation, signals to the summing circuitry may be provided by a programmable controller, for example, a central processing unit of a computer or a smartphone. For example, a programmable compressed image sensor, or a compressed image sensor communicating with a programmable processor, so as to receive commands from the processor.

[0133] As an example of a software implementation, pixel values may be digitized and be sent to a programmable processor so as to be summed according to the block compression masks, where the weights of the block compression masks may be stored in a memory.

[0134] In some embodiments, the system may further be configured for performing image decompression. That is, the system may include decompression processing circuitry configured to perform a decompression method as described herein above (e.g., method 600). In some embodiments, the system may be included in a camera module. In some embodiments, the camera module may be incorporated (included) in a mobile device, preferably a smartphone or a tablet computer.

[0135] As another example, a computer provided with an image manipulation program, incorporating instructions to perform compression and decompression methods as described herein above. Optionally, the computer may include a camera (e.g., a webcam). It is noted that methods 100 600 may benefit not only camera modules. For example, methods 100, 600 may benefit image-processing utilities and file management utilities, that may compress and decompress images as needed.

[0136] It is noted that methods 100, 600 may be combined in a single process. For example, for performing digital zoom. A video stream may be captured and compressed via method 100. Selected frames (compressed images) may be decompressed via method 600 so as to perform digital zoom.

[0137] Another example is for transmitting image data over a network. A server may perform method 100 to compress a video stream, and transmit the compressed video stream (compressed sequence of images) to a client. The client may perform the method 600 in order to decompress the compressed video stream.

[0138] Fig. 7 shows schematically exemplary embodiments of a mobile device (for example, a smartphone) 700 that may be configured to perform methods disclosed herein. Mobile device 700 may include a first camera 710, having a first camera field-of-view FOVi and including a first lens 712 and a first image sensor 714. First image sensor 714 may be a compressed imaging sensor, configured to perform methods for generating a video stream of compressed input images as disclosed herein. Optionally, mobile device 700 may further include a second camera 720, having a second camera field-of-view FOV2 and including a second lens and a second image sensor. In some examples, first camera 710 may be a Wide camera with FOVi= 60-100 deg and an effective focal length (EFL) of EFLi = 3mm- 12.5mm. In some examples, optional second camera 720 may be an Ultra-Wide camera with FOV2= 100-180 deg and EFL2 = 1.5mm-7.5mm. In other examples, optional second camera 720 may be a Tele camera with FOV2= 10-60 deg and EFL2 = 7.5mm - 40mm.

[0139] In some embodiments, mobile device 700 may further include an application processor (AP) 730. AP 730 may include an IPS control 732 configured to provide IPS pixel masking control data, and may include a dynamic binning (DB) control 734 configured to provide DB pixel masking control data. In other embodiments, compressed image sensor 714 may provide IPS masking control data and / or DB masking control data.

[0140] Mobile device 700 may further include a screen 740 for displaying information. Screen 740 may be a touchscreen, configured to detect a particular location that a user touches, for example for zooming into a scene. Mobile device 700 may further include a memory 750, e.g., for storing image data, for storing calibration data between first camera 710 and second camera 720.

[0141] Mobile device 700 may further include several additional sensors to capture additional information. For example, an additional sensor may be a microphone or even a directional microphone, a location sensor such as GPS, an inertial measurement unit (IMU) etc.

[0142] In other examples, methods disclosed herein may be fully performed in postcapture, i.e., after a video stream of input images is already captured. In some examples, the methods may be performed at AP 730 to compress a non-compressed video stream captured in a HRM. For compressing the video stream of input images, step 110 of method 100 may be performed by AP 730. This means that compression configurations such as described in relation to Figs. 2A-5 are applied at AP 730 to generate a compressed video stream of input images for example having a pixel resolution as obtained in the LRM. In some examples, IPS control 732 may be configured to block particular pixels and sum only the active (non-blocked) pixels as for example shown in Figs. 4A-4B in post-capture. In other examples, DB control 734 may be configured to sum individual pixels as for example shown in Figs. 2A-3E in post-capture. An advantage of this approach is that the generated compressed video stream of input images has a smaller file size as the original video stream of input images, which is beneficial for storing it on a device like mobile device 700 or for sharing it between mobile devices. It is noted that a pixel resolution of each image of the video stream of images is converted from a HRM pixel resolution to a lower pixel resolution, for example to the LRM pixel resolution. After storing or transferring the compressed video stream, a single output image or a video stream of output images, with elevated pixel resolution, may be generated as in step 620, for example, by a neural- network module 736.

[0143] Having described and illustrated the principles of the disclosed technology with reference to the illustrated embodiments, it will be recognized that the illustrated embodiments can be modified in arrangement and detail without departing from such principles. The technologies from any example can be combined with the technologies described in any one or more of the other examples.

[0144] Therefore, casting into a language of clauses, the present disclosure provides methods and systems according to, but not limited to, the following clauses:

[0145] Clause 1: A method for compressing a video stream comprising a sequence of images captured by an image sensor, wherein: i. said images having a first resolution; ii. the method being implemented by a compressed image sensor; iii. the method comprising, for each captured image:

[0146] • partitioning the captured image into a plurality of pixel blocks, each pixel block comprising a contiguous set of pixels;

[0147] • for each pixel block of said captured image, computing a plurality of block pixel sums according to a plurality of block compression masks, wherein a number of said block pixel sums is lesser than a number of pixels included in said set of pixels;

[0148] • for each block compression mask, combining the block pixel sums corresponding said block compression mask, to obtain a compressed sum image; so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution. Clause 2: The method according to clause 1, wherein at least one block compression mask corresponds to noncontiguous subsets of pixels.

[0149] Clause 3: The method according to any one of clauses 1 to 2, wherein at least two captured images are each associated with a distinct plurality of block compression masks. Clause 4: A method for compressing a video stream comprising a sequence of images captured by an image sensor, wherein: i. said images having a first resolution; ii. the method being implemented by a compressed image sensor; iii. the method comprising, for each captured image:

[0150] • partitioning the captured image into a plurality of pixel blocks, each pixel block comprising a contiguous set of pixels;

[0151] • for each pixel block of said captured image, computing at least one block pixel sum according to at least one block compression mask associated with said captured image, wherein at least two captured images are each associated with distinct block compression masks;

[0152] • combining the block pixel sums to obtain a compressed sum image; so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

[0153] Clause 5: The method according to clause 4, wherein said at least one block compression mask corresponds to noncontiguous subsets of pixels.

[0154] Clause 6: The method according to any one of clauses 4 to 5, wherein said at least one block compression mask comprises a plurality of block compression masks; the method comprises, for each pixel block, computing a plurality of block pixel sums according to said plurality of block compression masks, wherein a number of said block pixel sums is lesser than a number of pixels included in said set of pixels; and, for each block compression mask, combining the block pixel sums corresponding said block compression mask, to obtain a compressed sum image.

[0155] Clause 7: A method for compressing a video stream comprising a sequence of images captured by an image sensor, wherein: i. said images having a first resolution; ii. the method being implemented by a compressed image sensor; iii. the method comprising, for each captured image:

[0156] • partitioning the captured image into a plurality of pixel blocks, each pixel block comprising a contiguous set of pixels;

[0157] • for each pixel block of said captured image, computing at least one block pixel sum according to at least one block compression mask, wherein said block compression mask corresponds a noncontiguous subset of pixels;

[0158] • combining the block pixel sums to obtain a compressed sum image; so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

[0159] Clause 8: The method according to clause 7, wherein said at least one block compression mask comprises a plurality of block compression masks; the method comprises, for each pixel block, computing a plurality of block pixel sums according to said plurality of block compression masks, wherein a number of said block pixel sums is lesser than a number of pixels included in said set of pixels; and, for each block compression mask, combining the block pixel sums corresponding said block compression mask, to obtain a compressed sum image.

[0160] Clause 9: The method according to any one of clauses 7 to 8, wherein at least two captured images are each associated with distinct block compression masks.

[0161] Clause 10: The method according to any one of clauses 3 to 6, or according to clause 9, wherein an association of block compression masks to captured images is periodic.

[0162] Clause 11: The method according to clause 10, wherein a period is two frames or three frames.

[0163] Clause 12: The method according to any one of the preceding clauses, wherein at least one block compression mask is a binary mask.

[0164] Clause 13: The method according to clause 12, wherein all block compression masks are binary masks.

[0165] Clause 14: The method according to any one of clauses 12 to 13, wherein at least two block compression masks correspond complementary subsets of pixels.

[0166] Clause 15: The method according to clause 14, wherein said plurality of block pixel sums comprise an average and a difference of sums of said complementary subsets of pixels, so as to obtain a sequence of compressed average images and a sequence of compressed difference images.

[0167] Clause 16: The method according to any one of clauses 12 to 15 wherein for each pixel block, a number of pixels in the pixel subset is equal to half a number of pixels included in said pixel block.

[0168] Clause 17: The method according to one of clauses 12 to 16, wherein at least one block compression mask corresponds a subset of pixels that includes any one of: a left half of a pixel block, a right half of a pixel block, an upper half of a pixel block, and a lower half of a pixel block.

[0169] Clause 18: The method according to any one of clauses 12 to 17, wherein at least one block compression mask corresponds a subset of pixels that includes any one of: an upper right quarter and a lower left quarter of a pixel block, and an upper left quarter and a lower right quarter of a pixel block.

[0170] Clause 19: The method according to any one of clauses 12 to 18, wherein at least two block compression masks correspond random subsets of pixels.

[0171] Clause 20: The method according to any one of the preceding clauses, wherein each pixel block comprises the same number of pixels.

[0172] Clause 21: The method according to clause 20, wherein each pixel block is square.

[0173] Clause 22: The method according to clause 21, wherein a number M of pixels included in each pixel block is in the range of 4 pixels to 36 pixels.

[0174] Clause 23: The method according to clause 22, wherein any of the following:

[0175] • M = 4 and a number of pixels N included in each captured image is in the range of 45 million to 55 million;

[0176] • M = 9 and N is in the range of 95 million to 120 million;

[0177] • M = 16 and N is in the range of 180 million to 220 million; or

[0178] • M = 36 and N is in the range of 350 million to 450 million.

[0179] Clause 24: The method according to any one of clauses 21 to 23, wherein a pixel block size equals a predefined downscale ratio.

[0180] Clause 25: The method according to any one of clauses 21 to 24, wherein for at least one captured image, each block is associated with an identical block compression mask. Clause 26: The method according to any one of the preceding clauses, comprising optimizing said block compression mask by minimizing at least one of: maximum coherence between distinct block compression masks, and average coherence between distinct block compression masks.

[0181] Clause 27: The method according to any one of the preceding clauses, wherein a number of pixel blocks is in the range of 10 million to 14 million.

[0182] Clause 28: The method according to any one of the preceding clauses, wherein said captured images comprise at least two color-channels, preferably at least three colorchannels, more preferably, at least a red channel, a green channel, and a blue channel.

[0183] Clause 29: The method according to any one of the preceding clauses, wherein said second resolution is 4k resolution.

[0184] Clause 30: The method according to any one of the preceding clauses, comprising performing preliminarily downsampling of at least one captured image.

[0185] Clause 31: The method according to any one of the preceding clauses, comprising maximizing any one of: peak signal to noise ratio, and structured image similarity measure, thereby optimizing a balance between reconstruction quality and compression rate.

[0186] Clause 32: The method according to any one of the preceding clauses, comprising quantization of pixel values.

[0187] Clause 33: An image decompression method for decompressing a compressed image sequence compressed using the image compression method according to any one of the preceding clauses, the method comprising applying a neural network to said compressed image sequence so as to obtain at least one reconstructed image having a resolution higher than a resolution of said compressed image sequence.

[0188] Clause 34: The method according to clause 33, comprising processing between 1 to 16 compressed images for each reconstructed image, preferably any one of 3 images, 5 images, and 7 images.

[0189] Clause 35: The method according to any of clauses 33 to 34, wherein said neural network includes a vision transformer, preferably a spatial-temporal transformer.

[0190] Clause 36: The method according to any one of clauses 33 to 35, comprising applying a motion estimation algorithm to said compressed image sequence. Clause 37: The method according to any one of clauses 33 to 36, comprising applying any one of: video frame interpolation, and video super resolution.

[0191] Clause 38: The method according to clause 37, comprising applying video frame interpolation, and applying video super resolution. Clause 39: A system comprising a compressed image sensor configured for capturing a sequence of images and configured for compressing said sequence of images by a compression method according to any one of clauses 1 to 32.

[0192] Clause 40: The system according to clause 39, further comprising decompression processing circuitry configured to perform the decompression method according to any one of clauses 33 to 38.

[0193] Clause 41: A camera module comprising the system according to any one of clauses 39 to 40.

[0194] Clause 42: A mobile device including the camera module according to clause 41, preferably a smartphone or a tablet computer.

Claims

CLAIMS:

1. A method for compressing a video stream comprising a sequence of images captured by an image sensor, wherein: i. said images having a first resolution; ii. the method being implemented by a compressed image sensor; iii. the method comprising, for each captured image:• partitioning the captured image into a plurality of pixel blocks, each pixel block comprising a contiguous set of pixels;• for each pixel block of said captured image, computing a plurality of block pixel sums according to a plurality of block compression masks, wherein a number of said block pixel sums is lesser than a number of pixels included in said set of pixels;• for each block compression mask, combining the block pixel sums corresponding said block compression mask, to obtain a compressed sum image; so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

2. The method according to claim 1, wherein at least one block compression mask corresponds to noncontiguous subsets of pixels.

3. The method according to any one of claims 1 to 2, wherein at least two captured images are each associated with a distinct plurality of block compression masks.

4. A method for compressing a video stream comprising a sequence of images captured by an image sensor, wherein: i. said images having a first resolution; ii. the method being implemented by a compressed image sensor; iii. the method comprising, for each captured image:• partitioning the captured image into a plurality of pixel blocks, each pixel block comprising a contiguous set of pixels;• for each pixel block of said captured image, computing at least one block pixel sum according to at least one block compression mask associated with said captured image, wherein at least two captured images are each associated with distinct block compression masks;• combining the block pixel sums to obtain a compressed sum image; so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

5. The method according to claim 4, wherein said at least one block compression mask corresponds to noncontiguous subsets of pixels.

6. The method according to any one of claims 4 to 5, wherein said at least one block compression mask comprises a plurality of block compression masks; the method comprises, for each pixel block, computing a plurality of block pixel sums according to said plurality of block compression masks, wherein a number of said block pixel sums is lesser than a number of pixels included in said set of pixels; and, for each block compression mask, combining the block pixel sums corresponding said block compression mask, to obtain a compressed sum image.

7. A method for compressing a video stream comprising a sequence of images captured by an image sensor, wherein: i. said images having a first resolution; ii. the method being implemented by a compressed image sensor; iii. the method comprising, for each captured image:• partitioning the captured image into a plurality of pixel blocks, each pixel block comprising a contiguous set of pixels;• for each pixel block of said captured image, computing at least one block pixel sum according to at least one block compression mask, wherein said block compression mask corresponds a noncontiguous subset of pixels;• combining the block pixel sums to obtain a compressed sum image; so as to obtain a sequence of compressed sum images having a second resolution lower than the first resolution.

8. The method according to claim 7, wherein said at least one block compression mask comprises a plurality of block compression masks; the method comprises, for each pixel block, computing a plurality of block pixel sums according to said plurality of block compression masks, wherein a number of said block pixel sums is lesser than a number of pixels included in said set of pixels; and, for each block compression mask, combining theblock pixel sums corresponding said block compression mask, to obtain a compressed sum image.

9. The method according to any one of claims 7 to 8, wherein at least two captured images are each associated with distinct block compression masks.

10. The method according to any one of claims 3 to 6, or according to claim 9, wherein an association of block compression masks to captured images is periodic.

11. The method according to claim 10, wherein a period is two frames or three frames.

12. The method according to any one of the preceding claims, wherein at least one block compression mask is a binary mask.

13. The method according to claim 12, wherein all block compression masks are binary masks.

14. The method according to any one of claims 12 to 13, wherein at least two block compression masks correspond complementary subsets of pixels.

15. The method according to claim 14, wherein said plurality of block pixel sums comprise an average and a difference of sums of said complementary subsets of pixels, so as to obtain a sequence of compressed average images and a sequence of compressed difference images.

16. The method according to any one of claims 12 to 15 wherein for each pixel block, a number of pixels in the pixel subset is equal to half a number of pixels included in said pixel block.

17. The method according to one of claims 12 to 16, wherein at least one block compression mask corresponds a subset of pixels that includes any one of: a left half of a pixel block, a right half of a pixel block, an upper half of a pixel block, and a lower half of a pixel block.

18. The method according to any one of claims 12 to 17, wherein at least one block compression mask corresponds a subset of pixels that includes any one of: an upper right quarter and a lower left quarter of a pixel block, and an upper left quarter and a lower right quarter of a pixel block.

19. The method according to any one of claims 12 to 18, wherein at least two block compression masks correspond random subsets of pixels.

20. The method according to any one of the preceding claims, wherein each pixel block comprises the same number of pixels.

21. The method according to claim 20, wherein each pixel block is square.

22. The method according to claim 21, wherein a number M of pixels included in each pixel block is in the range of 4 pixels to 36 pixels.

23. The method according to claim 22, wherein any of the following:• M = 4 and a number of pixels N included in each captured image is in the range of 45 million to 55 million;• M = 9 and N is in the range of 95 million to 120 million;• M = 16 and N is in the range of 180 million to 220 million; or• M = 36 and N is in the range of 350 million to 450 million.

24. The method according to any one of claims 21 to 23, wherein a pixel block size equals a predefined downscale ratio.

25. The method according to any one of claims 21 to 24, wherein for at least one captured image, each block is associated with an identical block compression mask.

26. The method according to any one of the preceding claims, comprising optimizing said block compression mask by minimizing at least one of: maximum coherence between distinct block compression masks, and average coherence between distinct block compression masks.

27. The method according to any one of the preceding claims, wherein a number of pixel blocks is in the range of 10 million to 14 million.

28. The method according to any one of the preceding claims, wherein said captured images comprise at least two color-channels, preferably at least three color-channels, more preferably, at least a red channel, a green channel, and a blue channel.

29. The method according to any one of the preceding claims, wherein said second resolution is 4k resolution.

30. The method according to any one of the preceding claims, comprising performing preliminarily downsampling of at least one captured image.

31. The method according to any one of the preceding claims, comprising maximizing any one of: peak signal to noise ratio, and structured image similarity measure, thereby optimizing a balance between reconstruction quality and compression rate.

32. The method according to any one of the preceding claims, comprising quantization of pixel values.

33. An image decompression method for decompressing a compressed image sequence compressed using the image compression method according to any one of the preceding claims, the method comprising applying a neural network to said compressed image sequence so as to obtain at least one reconstructed image having a resolution higher than a resolution of said compressed image sequence.

34. The method according to claim 33, comprising processing between 1 to 16 compressed images for each reconstructed image, preferably any one of 3 images, 5 images, and 7 images.

35. The method according to any of claims 33 to 34, wherein said neural network includes a vision transformer, preferably a spatial-temporal transformer.

36. The method according to any one of claims 33 to 35, comprising applying a motion estimation algorithm to said compressed image sequence.

37. The method according to any one of claims 33 to 36, comprising applying any one of: video frame interpolation, and video super resolution.

38. The method according to claim 37, comprising applying video frame interpolation, and applying video super resolution.

39. A system comprising a compressed image sensor configured for capturing a sequence of images and configured for compressing said sequence of images by a compression method according to any one of claims 1 to 32.

40. The system according to claim 39, further comprising decompression processing circuitry configured to perform the decompression method according to any one of claims 33 to 38.

41. A camera module comprising the system according to any one of claims 39 to 40.

42. A mobile device including the camera module according to claim 41, preferably a smartphone or a tablet computer.

Images