Method for transmitting encoded image data

Abstract

A method for transmitting encoded image data for a video conference comprises: acquiring a sequence of images for the video conference from a camera of a first participant of the video conference; processing each image of the sequence through an encoding convolutional neural network, CNN, so as to generate a first plurality of channels of encoded image data comprising a map having a spatial resolution lower than a spatial resolution of a corresponding acquired image, each pixel of each map comprising a discrete value from a range of possible values. For at least some images of the sequence of images, a binary quantization of each discrete value for each pixel of each map is performed to provide, for each channel of the images, a respective binary valued map which is transmitted to other participant(s) of the video conference as representative of a respective image of the sequence of images.

Description

[0001] Method for transmitting encoded image data

[0002] Field

[0003] The present invention relates to a method for transmitting encoded image data.

[0004] Background

[0005] SWIFT: Adaptive Video Streaming with Layered Neural Codecs (https: / / www.usenix.org / conference / nsdi22 / presentation / dasari) discloses i) a layered encoder that learns to encode a video frame into layered codes by purely encoding residuals from previous layers without introducing any cross-layer compression overheads, ii) a decoder that can fuse together a subset of these codes (based on availability) and decode them all in one go, and, iii) an adaptive bit rate (ABR) protocol that synergistically adapts video quality based on available network and client-side compute capacity.

[0006] It is an object of the present invention to provide an improved neural network based codec, especially which can take into account its context of operation for providing high quality data with a lower bit rate budget than conventional codecs.

[0007] Summary

[0008] According to the present invention there is provided a method for transmitting encoded image data for a video conference according to claim 1.

[0009] Embodiments of the present invention use a neural network for the compression of video images for video conferencing.

[0010] A particular characteristic of video conferencing is that typically, there is very little movement or change of scene from frame to frame and this lends itself to extremely efficient encoding with a strong generative element employed in a decoder in order to produce high quality decoded images for any client of the video conferencing system.

[0011] Embodiments employ a binary rounding regularisation layer to improve compression of video images. Masking is employed in end-to-end training of a system comprising an encoder for implementing the method and a corresponding decoder in order to force the decoder to train to generate high quality output based on minimal encoded image data.

[0012] In some systems comprising an encoder for implementing the method and a corresponding decoder, a video frame is reconstructed within the decoder based on an encoded difference between two adjacent encoded video frames.

[0013] In some embodiments, compression rates of between 90% and 98% of input frame size can be achieved with minimal impact on video quality.

[0014] Brief Description of the Drawings

[0015] Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0016] Figure 1 shows a method for transmitting encoded image data according to an embodiment of the present invention;

[0017] Figure 2 shows a method for decoding encoded image data according to an embodiment of the present invention;

[0018] Figure 3 illustrates a discriminator network employed in training a system comprising an encoder and a decoder according to an embodiment of the present invention;

[0019] Figure 4 illustrates components of a system according to a second embodiment of the present invention; and

[0020] Figure 5 illustrates a system for encoding and decoding of image data according to the second embodiment of the present invention.

[0021] Detailed Description of the Embodiments

[0022] Referring now to Figure 1, there is illustrated a method 10 for transmitting encoded image data according to an embodiment of the present invention. According to the method, a sequence of input images 12, only one of which is shown, is acquired from an image acquisition device (not shown) for a client in a video conferencing system. The image acquisition device can comprise an integrated webcam within a personal computer, smartphone, tablet device or a dedicated conferencing terminal or the image acquisition device can comprise a stand-alone camera either directly connected or connected through a wired and / or wireless network connection to a separate computing device. In the latter case, encoding of the acquired image can be implemented either in the image acquisition device or in the connected computing device.

[0023] In the illustrated embodiment, each acquired image 12 comprise a multi-plane chromatic image, for example, including R, G and B color planes. However, it will be appreciated that in alternative embodiments, either a monochrome image or a different image format may be employed.

[0024] In a first block of layers 11 of an encoder network implementing the method 10, a first convolutional layer 14 comprises a number of channels, in this case, 64, each employing a 2x2 kernel, applied with a stride of 2, which combine the or each layer of the input image 12 and output 64 feature maps, each reduced to half the size of the input image. This convolution is followed by batch normalization (BN) and an activation function, in this case, ReLU. It will be appreciated, however, that in variants of this embodiment, different activation functions such as PReLU can be employed.

[0025] After ReLU, a further 64-channel 2x2 convolution kernel 16 is applied to each respective feature map provided from the ReLU function and a second batch normalization, BN, is then applied to each of these feature maps.

[0026] The feature maps provided by the first ReLU layer and the second batch normalization are combined before being provided to a second ReLU activation function. This combination can comprise any of: concatenation, addition or aggregation. The second ReLU activation function provides a set of 64 feature maps 18 halved in resolution vis-a-vis the input image The structure of the block of layers 11 is repeated in a second block of layers 11', which take the set of feature maps 18 to produce a second set of feature maps 18', halved in resolution vis-a-vis the feature maps 18, but where the number or channels has doubled to 128.

[0027] The second set of feature maps 18' are provided to a third block of layers 11" which again halve the resolution of the feature maps 18' and double the number of channels to 256 to produce a third set of feature maps 18".

[0028] Note that in variations of the illustrated embodiment, the number of blocks 11, 11' and 11" may vary, each block may not comprise the same structure as the other blocks and the types and characteristics of each layer employed in each block, for example, the kernel size, stride etc. may vary.

[0029] So, for example, while in the embodiment, each block employs a residual connection between the first ReLU layer and the output of the second batch normalization layer, other connections are also possible. It will also be seen that, in the embodiment, encoding of a given image is based only on that image. In variations of the embodiment, the outputs of intermediate hidden layers within the encoder, for example, one or more of the convolutional layers 14, 14', 14" can be connected to corresponding layers within an instance of the encoder processing a successively acquired image, in a recurrent framework, or even with corresponding layers within respective instances of the encoder processing both previously and successively acquired images, in a bi-directionally recurrent framework. In such cases, LSTM layers can be employed to facilitate the interconnection of instances of the encoder.

[0030] In general, however, these blocks 11, 11' and 11" provide an encoding function, reducing the spatial resolution of the input image.

[0031] In the embodiment, the third set of feature maps 18" is provided to a pooling layer, in this case a 2x2 MaxPooling layer 20 which again halves the resolution of the feature maps, but does not increase the number of channels. In variants of the embodiment, other pooling functions, such as average pooling could be employed. This is followed by a further 2x2 convolutional layer 22 with a stride of 2 and which again halves the resolution of its input feature maps and doubles the number of channels to 512.

[0032] The 512 channels are then subjected to a sigmoid activation function 24 which produces a value within a range for each of the 8x8 pixels of the 512 channels of the input feature maps.

[0033] When in normal operation, the feature maps output by the sigmoid activation function 24 are subject to a binary quantization 26 so that the final encoded image information provided by process 10 comprises 512 8x8 binary valued feature maps 30.

[0034] On the other hand, when training the encoder network, a decision is made to periodically mask some of the output channels of the output feature maps. So, for example, input images can be chosen at random or periodically and if so, a masking index (MSI) is chosen in the range 256-512, step 32. For each least significant map of the output feature maps 30 with an index greater than or equal to MSI, their pixel values are set to 0 (or an equivalent null value), step 34. Once subjected to this rounding, the masked feature maps 30 are output as normal.

[0035] Note that while shown before rounding, the quantization step 26, can be performed after masking on only those feature maps which have not been masked during training.

[0036] It will be seen that the process 10 reduces a 256x256x3 byte image to an 8x8x512 bit image compressing 1.5Mb to 32k - a compression of 98%.

[0037] In variations of the embodiment, the input image need not be 256x256 and for example might not be square and this can either be compressed to 8x8x512 or a differently sized set of binary feature maps using the above teaching.

[0038] Turning now to Figure 2, there is a shown a method 40 for decoding the binary feature maps 30 produced by the encoding process 10 of Figure 1. The method 40 employs a sequence of deconvolutional layers 42-1...42-5, each of which increases the resolution of its input feature maps and reduces the number of channels. The deconvolutional layers 42-1...42-5 are interleaved with a number of batch normalization (BN) layers 44-1...44-3 and followed with a final deconvolutional layer 46 which in the present example outputs a final decoded image 48 corresponding in resolution and form to the originally acquired image 12.

[0039] Note that this need not be the case, and in variations of the embodiment, variants of the decoder network 40 might produce higher or lower resolution images or images with different color planes or forms than that of the originally acquired image. These networks would of course need to be trained separately, but possibly using the same training data.

[0040] In any case, once decoded, the final image 48 can be provided for rendering as required on a client device (not shown) participating in a video conference with the user of the client device from which the encoded image information was acquired. Again, this can be any of the types of devices employed by the user of the client device from which the original acquired and encoded image was acquired and as long, as this has a suitable display, it need not be the same as the client device from which the original acquired and encoded image was acquired. As such, while one client device of the video conference might acquire a high- resolution color image, another client of the video conference might render a decoded lower resolution monochrome version of that image, while another client of the video conference might even decode and render a higher super-resolution version of the originally acquired image - depending on their device and display capabilities.

[0041] Referring now to the training of the networks for implementing the encoding and decoding processes 10, 40, in an embodiment of the invention, this is done on an end-to-end basis. Thus, in any given training epoch a batch of training images are acquired and provided through the encoding and decoding processes 10, 40 and the decoded images are then provided to a discriminator network 50, Figure 3. The network 50 effectively comprises an encoding classifier which reduces the decoded images through a number of convolutional layers 50-1...50-3, each followed by a respective batch normalization (BN) and activation function, in this case, LeakyReLU with a parameter 0.2, to a classification close to 0 for poor, synthetically generated images and close to 1 for real acquired images. Residual connections 52-1...52-3 from each convolutional layer to after their respective activation functions are provided to generate a dense low-spatial resolution feature set 54 which is then subjected to a sigmoid activation function 56 outputting the final classification. As the processes 10, 40 and 50 are trained, then decoded images should produce increasingly higher valued classifications which appear more real and indeed as the network 50 trains, it should produce classifications which indicate real acquired images are real and that poorly generated images are not real. The differences between these classifications produced by the discriminator and the ground truth for the training images provide the basis for a loss function which drives the updating of weights for successive training epochs.

[0042] Turning now to a second embodiment of the present invention, Figure 4, in which encoded image information N (8,8,512), N+l (8,8,512) for successively acquired image frames N, N+l is combined by a delta generator sub-model 60 to produce encoded image data 62 representing a difference between the two image frames. So, rather than transmitting the N+l (8,8,512) encoded image information 30 produced by the encoding process 10 as representing image frame N+l, more compressed data can be transmitted instead. In more detail, in this case, the encoded image information N (8,8,512) and N+l (8,8,512) for each frame is combined by concatenation 64. In the embodiment, the final binarized encoded information produced by the encoding process 10 is combined, however, in variants of the embodiment, encoded image information produced by the sigmoid function 24 prior to binary quantization could be used.

[0043] The combined information is passed through sequence of convolutional layers 66-1...66-4, each reducing the number of channels from the initial 512 until a final 16 channels. In the embodiment, the spatial resolution of the input image information is maintained, but in variants of the embodiment, this could be varied with the final number of channels being adjusted accordingly. The 16 channels are passed through a sigmoid function 68 before being subjected to binary quantization 70 which provides the final delta (8,8,16) binary output 62.

[0044] The decision to transmit either the initially encoded (8x8x512) image information or the delta (8,8,16) information for a frame N+l, can be based on one or more criteria. So, for example, initially encoded (8x8x512) image information can be sent periodically or at least at a minimum frequency. Alternatively, or in addition, initially encoded (8x8x512) image information can be sent if an appearance between one image frame and the next differs by more than a given amount. This difference can be measured either between unencoded image frames or between the encoded image information for successively acquired image frames. In any case, it will be appreciated that initially encoded (8x8x512) image information rather than delta (8,8,16) information for a first image frame of a sequence needs to be transmitted.

[0045] In a receiver decoder, the delta (8,8,16) information for a frame N+l is combined with the encoded (8x8x512) image information for a frame N in a reconstruction sub-model 80 in order to synthesize decoded (8,8,512) image information for frame N+l. When processing the delta (8,8,16) information for a frame N+l, the encoded (8x8x512) image information for a frame N can be obtained in a number of ways. If this has previously been received directly, this can be stored for later access by the decoder for use in processing the next frame. Alternatively, a previously decoded image frame N can be encoded using an instance of the encoding process lOd, to produce encoded information for frame N. Otherwise, if delta (8,8,16) information has been acquired for the previous image frame N, and this was reconstructed to synthesize decoded (8,8,512) image information using the sub-model 80, this can be used as the synthesized decoded (8,8,512) image information for frame N, as indicated by line 30'.

[0046] Initially, in the reconstruction sub-model 80, the delta (8,8,16) information for frame N+l is passed through a number of deconvolution layer 82-1...82-3 to increase the number of channels from 16 to 512 to match those of the encoded or synthesized decoded (8,8,512) image information for frame N. This deconvolved information is combined by concatenation 84 with the encoded or synthesized decoded (8,8,512) image information from frame N. In the embodiment, the combined information is passed through a further convolutional layer 86 and a sigmoid function 88, before being subjected to binary quantization 90 to produce synthesized decoded (8,8,512) image information for frame N+l.

[0047] In any case, when either encoded (8x8x512) image information for a given frame or synthesized decoded (8,8,512) image information for the frame is available within the decoder, this can be provided to an instance of the decoding process 40d+l (illustrated in Figure 2) to provide a fully decoded image frame N+l which can be rendered as described previously. With reference now to Figure 5, systems which implement the delta-generator and reconstruction sub-models 60, 80 can either be trained end-to-end while both the encoder and decoder and discriminator networks 10, 40 and 50 are being trained. Alternatively, the encoder and decoder and discriminator networks 10, 40 and 50 can be trained first as described above in relation to Figure 3 and locked, before then training the delta-generator and reconstruction sub-models 60, 80.

[0048] Note that embodiments comprising only the encoder 10 and decoder 40 can be trained using batches of still image frames, whereas embodiments employing the delta-generator 60 and reconstruction model 80 require sequences of image frames.

[0049] In any case, it will be seen that an encoder can either provide a 32k (8,8,512) representation or a lk (8,8,16) representation of any given frame and whether a 32k (8,8,512) representation or a lk (8,8,16) representation is received, this can be decoded and rendered by any client of the video conference.

[0050] It will be appreciated that in a video conference client connected to the image acquisition device providing acquired images, it may be desirable to display a preview of the acquired image sequence. In this case, either the originally acquired image and / or a synthesized version derived from the encoded (8,8,512) or (8,8,16) image information for the acquired image frame can be displayed, so that the user of the image acquisition device can assess the quality of the acquired image and / or image as they will be seen by other participants of the video conference.

[0051] It will be appreciated that still further variations of the embodiments described above are possible. For example, in some implementations, users can be allowed to specify varying quality levels for the video conference. For the highest level of quality in the example provided above, all 8x8x512 channels of encoded image information would be transmitted for non-delta image frames. For progressively lower levels of quality, an increasing number of the least significant channels would not be transmitted, up to for example 256 channels, so potentially halving the information transmitted for non-delta frames. At the decoder, information for these un-transmitted channels is simply set to zero and decoding progresses as normal. Nonetheless, the fully decoded image can still be of reasonable quality, in particular because of the manner in which the decoder has been trained based on partially masked data as described above.

Claims

Claims:

1. A method for transmitting encoded image data for a video conference comprising: acquiring a sequence of images for the video conference from a camera of a first participant of the video conference; processing each image of the sequence through a plurality of layers of an encoding convolutional neural network, CNN, so as to generate a first plurality of channels of encoded image data, encoded image data for each channel comprising a map having a spatial resolution lower than a spatial resolution of a corresponding acquired image, each pixel of each map comprising a discrete value from a range of possible values; for at least some images of the sequence of images, performing a binary quantization of each discrete value for each pixel of each map to provide, for each channel of each of said at least some images, a respective binary valued map; and for said at least some images of the sequence of images, transmitting the plurality of binary valued maps to at least one other participant of the video conference as representative of a respective image of the sequence of images.

2. The method of claim 1, wherein said processing includes applying a sigmoid activation function to an output of a final convolutional layer of said CNN to provide said encoded image data.

3. The method of claim 1, wherein said processing includes, at least once, pooling an output of an intermediate convolutional layer of said CNN.4.The method of claim 3 wherein said pooling comprises one of max pooling or average pooling.

5. The method of claim 1 wherein said processing includes applying batch normalization after at least one convolutional layer of said CNN.G.The method claim 1 wherein said processing includes combining an output of at least one convolutional layer of said CNN with an output of at least one successive convolutional layer of said CNN having an equal number of channels of maps having the same spatial resolution.

7. The method of claim 6 wherein said combining comprises one of concatenation, aggregation or convolution.

8. The method of claim 1 wherein one or more convolutional layers of said CNN provide an increased number of output channels by comparison to an output of a previous convolutional layer.

9. The method of claim 1 wherein a first convolutional layer of said CNN provides 64 output channels and a final convolutional layer of said CNN provides 512 output channels.

10. The method of claim 1 wherein each acquired image comprises either a monochrome image or a multi-plane image in one of RGB, LAB, YCC or YUV format.

11. The method of claim 1, wherein sequence of images comprises a first image and a second successively acquired image, and wherein the method comprises the following steps: a) combining encoded image data for each channel of the first image with encoded image data for each corresponding channel of the second successively acquired image to provide a plurality of combined maps; b) processing the combined maps through a plurality of convolutional layers of a second CNN, so as to generate a second plurality of channels of encoded image data, encoded image data for each channel comprising a map having a spatial resolution equal to each of said first and second encoded image data and said second plurality of channels being fewer than said first plurality of channels, each pixel of each map comprising a discrete value from a range of possible values; c) performing a binary quantization of each discrete value for each pixel of each map of said second plurality of channels to provide a respective binary valued map; andd) transmitting the plurality of binary valued maps to said at least one other participant of the video conference as representative of said second image.

12. The method of claim 11 wherein said encoded image data is combined after being subjected to said binary quantization.

13. The method of claim 11, wherein the first image comprises one of said at least some images of the sequence of images.

14. The method of claim 11, further comprising repeating steps a)-d) for each successively acquired pair of images starting until a condition is met before choosing a successively acquired image as one of said at least some images of the sequence of images.

15. The method of claim 14 wherein said condition comprises either: a difference between a pair of successively acquired images being determined to exceed a threshold; or a predetermined number of images having been acquired since a last image was selected as one of said at least some images of the sequence of images.

16. A transmitter for transmitting encoded image data for a video conference, the transmitter configured to perform the method of any of the preceding claims.

17. A system comprising a transmitter according to claim 16 and a receiver configured to: for each of at least some images of a sequence of images acquired by a transmitter of another participant of the video conference, receive a plurality of binary valued maps as representative of a respective image of the sequence of images; process said binary valued maps through a plurality of layers of a decoding CNN, so as to generate one or more channels of decoded image data, decoded image data for each channel comprising a map having a spatial resolution higher than a spatial resolution of a binary valued map, each pixel of each decoded map comprising a discrete value from a range of possible values for a color plane of a decoded image; andprovide the decoded image data for rendering on a display of the first participant of the video conference.

18. A method for training the system of claim 17 comprising: providing, to the transmitter, a set of training video conference images; processing each training image through the plurality of layers of the encoding CNN of the transmitter so as to generate a plurality of channels of encoded image data; randomly selecting a number of least significant channels and masking at least some of the pixels in the maps belonging to the selected channels; after the masking, performing a binary quantization of each non-masked value for each pixel of each map to provide, for the channels of each of said training images, a plurality of respective binary valued maps; transmitting the plurality of binary values maps to the receiver as representative of respective images of the set of training video conference images; process said binary valued maps through a plurality of layers of a decoding CNN of the receiver, so as to generate one or more channels of decoded image data; processing decoded image data for a plurality of images of the set of training video conference images through a discriminator to generate a measure of said images being originally acquired images or decoded image data; processing image data for a plurality of originally acquired images of the set of training video conference images through the discriminator to generate a measure of said images being originally acquired images or decoded image data; and adjusting the weights of the layers of the encoding CNN, the decoding CCN and the discriminator to reduce a loss function for the classifications produced by said discriminator.

19. The method of claim 18 wherein said masking comprises setting the values of selected pixels in the maps belonging to the selected channels to 0.

Images