Task-dependent layered image and video encoder

Abstract

Embodiments of invention relates to an image encoding device and an image decoding device. An an encoded semantic segmentation image s' of an input image x is generated based on a semantic segmentation deep neural network (120) and a lossless encoder (130). An encoded coarse image c' of the input image x is generated based on a first lossy encoder (140) and a semantic based distortion map m. An estimated image x' of the input image x is generated based on a semantic segmentation image s, a coarse image c and the semantic based distortion map m in a refinement deep neural network (150). An encoded residual image r' is generated based on the estimated image x', the input image x and a second lossy encoder (160). Furthermore, embodiments of the invention also relate to a first and second communication devices, and also to corresponding methods and a computer program.

Description

[0001] TASK-DEPENDENT LAYERED IMAGE AND VIDEO ENCODER

[0002] Technical Field

[0003] Embodiments of the invention relate to source coding of an input image in an image encoding device and to generation of a reconstructed image of the input image in an image decoding device. Furthermore, embodiments of the invention also relate to a first communication device and to a second communication device, and to corresponding methods and a computer program.

[0004] Background

[0005] As the amount of data generated both by humans and machines increases exponentially, there is demand for higher and higher wireless data rates. Also, there is a risk that future wireless systems, including e.g., intemet-of-things (loT) systems, autonomous vehicles, surveillance systems, augmented reality solutions and / or virtual reality solutions, will generate such amounts of wireless data that usage of traditional source-channel coding techniques will overload the wireless communication systems with data. To be able to handle the increasing amounts of data to be transferred in wireless the communication systems, a shift in the way information is encoded will be needed.

[0006] Alternative source coding approaches useful for image and / or video information are thus needed. Such alternative source coding approaches may enhance the accuracy and efficiency of human to machine (H2M) and / or machine to machine (M2M) communications. The image and / or video source coding approaches may therefore utilize the semantics of the data, i.e. the meaning behind the bits, to improve the source coding. Machine learning (ML), and in particular deep learning (DL), have shown to be capable to process various intelligent tasks, such as the task of extracting semantic information out of input data, and may thus be utilized when exploiting the semantics of the image and / or video data to be encoded. When taking the semantics into consideration at source coding, the semantic fidelity has more significance than error rate measurements used for conventional source coding, for example conventional bit error rate or symbol error rate measurements.

[0007] Summary

[0008] An objective of embodiments of the invention is to provide a solution which mitigates or solves the drawbacks and problems of conventional solutions. Another objective of embodiments of the invention is to provide a source coding with enhanced accuracy and efficiency.

[0009] The above and further objectives are solved by the subject matter of the independent claims. Further embodiments of the invention can be found in the dependent claims.

[0010] According to a first aspect of the invention, the above mentioned and other objectives are achieved with an image encoding device configured to: generate a semantic segmentation image s of an input image x based on a semantic segmentation deep neural network, SS DNN; encode the semantic segmentation image s based on a lossless encoder to generate an encoded semantic segmentation image s'; encode the input image x based on a first lossy encoder, a corresponding first lossy decoder and a semantic based distortion map m to generate an encoded coarse image c and a coarse image c; generate an estimated image x of the input image x based on the semantic segmentation image s, the coarse image c and the semantic based distortion map m in a refinement deep neural network, refinement DNN; generate a residual image r based on the estimated image x' and the input image x; and encode the residual image r based on a second lossy encoder to generate an encoded residual image r' .

[0011] Hereby, a semantic-based and task-dependent source encoding and decoding is provided. Thus, a source encoding and a decoding of image or video data are provided, that are based on the semantics of the image or video data to be stored or transmitted, i.e. based on the content / meaning of the image or video data, and which is dependent on a specific task / goal of the image or video data. The encoding and decoding are task-, application- and resourcedependent in order to enable semantic based compression and reconstruction, and in order to enable semantic based communications for future wireless networks.

[0012] An advantage of the image encoding device is that it efficiently encodes image and video data using semantic content of the image and video data, respectively, with the goal of achieving high performance and sufficient perceptual quality for a computer vision (CV) task at the image decoder. The image encoding device provides improved accuracy and efficiency compared to conventional solutions, as it avoids unnecessarily spending bits on irrelevant regions of the input image. The usage of the semantic based distortion map m masks out areas / objects of the input image x that are of less interest, and instead focusses the encoding on the task- dependent / semantic-based information. Thus, to base the encoding on the semantic based distortion map m makes the source encoding task-dependent and thereby the compression more efficient. In other words, the utilization of the semantic based distortion map m in the encoding device results in an improved compression ratio, without compromising the semantic information.

[0013] Furthermore, the image encoding device also provides a trade-off between coding rate and visual / perceptual quality. An adaptive source coding for adaptive rate transmission is provided, which is crucial e.g. for wireless communications. The image encoding device provides a flexible encoding rate that may be optimized based on variable channel conditions, the application and / or a distortion target. The distortion target refers to a targeted or acceptable level of distortion that is allowed for the compression and reconstruction of an image or video.

[0014] The utilized combination of artificial intelligence (Al) based blocks and classical / standard compression blocks provides for an efficient and adaptive semantic-based and / or taskdependent source coding scheme, which focusses on particular areas of an image or a video, based on the specific task and / or goal. The image encoding device thus partly utilizes standard image compression as well as lossless and lossy encoding algorithms. Such standard engineered compression algorithms have over many years been proven to work well, are easy to use, and offer flexibility in tuning the compression ratio, which will have a positive effect on the overall performances of the codec. It has also been shown that, for a range of compression rates, the image encoding device, which partly uses corresponding classical / standard decompression blocks, outperforms conventional encoders.

[0015] By the provided efficient source encoding, the image encoding device reduces the payload over a communication system transferring the encoded images, e.g. used by various 5G applications or future 6G applications. As much of the data traffic of such communication systems today involves transmission of image and video data, the proposed image encoding device, which efficiently encodes this type of data based on its semantic content, reduces the total amount of data transmitted over the communication system. The image encoding device comprises a refinement deep neural network, which is capable of generating a high-resolution image. The refinement deep neural network could be implemented in different ways, e.g. as a conditional generative adversarial network (cGAN) or a denoising diffusion probabilistic models (DDPM) neural network. Moreover, the semantic segmentation deep neural network can be an off-the-shelf semantic segmentation network. Also, the standard image or video codec can be a classical / standard image compression algorithm, such as e.g. JPEG2000, or a classical video codec, such as e.g. H.265. The first and second lossy encoders may also be standard encoders. The compression performance of the image encoding device and the reconstruction performance at the corresponding image decoding device depend on the total combination of these blocks, and also depend on which one of the off-the-shelf algorithms that are used. The compression and reconstruction performances may therefore be adapted for a specific application.

[0016] The image encoding device reduces the training burden in relation to conventional deep semantic segmentation-based layered image compression schemes utilizing neural networks.

[0017] The image encoding device provides a source coding which makes a scalable three layer transmission possible, where the transmission can be made in a hierarchical manner dependent e.g. on the application. Thus, the transmission may comprise a first fundamental layer only, or the first fundamental layer and a second enhancement layer, or the first fundamental layer, the second enhancement layer and the third refined layer. More in detail, by providing three possible layers of information, i.e. the first fundamental image encoding layer provided by the encoded semantic segmentation image s', the second enhancem ent / coarse image encoding layer provided by also the encoded coarse image c , and the third refined image encoding layer provided by also the compressed version r' of the residual image, image / video files may be encoded by using the three layered image / video codec to reduce the encoding rate and / or to offer a semantic-based and / or task-dependent source encoding. This could be useful as a valuable feature for many applications, such as e.g. for internet of things (loT), H2M applications and M2M applications.

[0018] In an implementation form of an image encoding device according to the first aspect, the image encoding device is configured to store the encoded semantic segmentation image s' . An advantage with this implementation form is that only the encoded semantic segmentation image s' is stored, which may be seen as a first fundamental image encoding layer. Such a first fundamental image encoding layer comprises the semantic map, i.e. semantic segmentation image s, which comprises the entire semantic information of the image that can be used as a general base for every application and / or user.

[0019] In an implementation form of an image encoding device according to the first aspect, the image encoding device is configured to store the encoded coarse image c .

[0020] An advantage with this implementation form is that both the encoded semantic segmentation image s' and the encoded coarse image c are stored, which may be seen as a second enhanced / coarse image encoding layer, i.e. a compressed image / video representation of the input image / video that acts as an enhancement layer to the first fundamental layer, is also provided. Such a second coarse image encoding layer together with the first fundamental layer provides a quality acceptable for some applications and / or users, for example if a coarse reconstruction of the image preserving the semantic meaning is required, which may be used by image decoding devices for some users. Thus, either of the first fundamental image encoding layer alone, or both of the first fundamental image encoding layer and the second coarse image encoding layer may here be provided by the image encoding device, adaptable to the present application. The first fundamental image encoding layer and the second coarse image encoding layer can be reconstructed to an image plausible to a human observer. Thus, the resulting image will seem reasonable for a human being, i.e. will make sense as an image possible to be perceived by a human as an image conveying a certain meaning.

[0021] In an implementation form of an image encoding device according to the first aspect, the image encoding device is configured to store the encoded residual image r' .

[0022] An advantage with this implementation form is all of the encoded semantic segmentation image s', the encoded coarse image c , and the encoded residual image r' are stored, which may be seen as a third refined image encoding layer, including also a compressed version of the residual, that acts as an extra enhancement layer to the first fundamental layer and the second coarse image encoding layer, is also provided. Such a refined image encoding layer provides a quality demanded by some applications and / or users, for example, if a refined reconstruction of the image is required, and may be thus used by image decoding devices for some users. Thus, either of the first fundamental image encoding layer alone, both of the first fundamental image encoding layer and the second coarse image encoding layer, or all of the first fundamental image encoding layer, the second coarse image encoding layer and the third refined image encoding layer may here be provided by the image encoding device, adaptable to the application utilizing the image encoding device. The third refined image encoding layer can here provide extra application-dependent details.

[0023] In an implementation form of an image encoding device according to the first aspect, the semantic based distortion map m is determined based on a task associated with the input image x and the semantic segmentation image s.

[0024] An advantage with this implementation form is that the task, comprising e.g. a set of labels, is used for generating the semantic based distortion map m. The semantic based distortion map m is like a filter, which masks out areas / objects of the input image x that are not of interest, or are of less interest, and expresses the task-dependent and semantic-based information of the input image x. The semantic based distortion map m does not need to be transmitted or encoded. The semantic based distortion map m may instead be obtained from a semantic task filter by using as input the attributes of a task, which is known to both the image encoding device and the image decoding device, and the semantic segmentation image s. By usage of the semantic based distortion map m, it is possible to adapt the encoding rates in the first and second lossy encoders. The semantic based distortion map m is also used as input in the refinement DNN to generate an estimated image x' of the input image x. Since the semantic based distortion map m is associated with the task, the utilization of the semantic based distortion map m in the image encoding device makes the provided source coding task-dependent.

[0025] An adaptive semantic-based and task-dependent feature of the compression scheme is provided by the utilization of the semantic based distortion map m, where the meaning which the image / video conveys is utilized depending on a desired task, i.e. depending on the semanticbased distortion, such that the encoding is focussed on the relevant parts / regions of the image / video, and such that the irrelevant parts / regions are masked out. This increases the image / video compression without affecting the semantic of the image, since the compression can be optimized depending on the distortion target. In an implementation form of an image encoding device according to the first aspect, the task comprises a set of labels and / or a sub-image of interest of the input image x.

[0026] An advantage with this implementation form is that tasks, i.e. attributes, that are known to both the image encoding device and the image decoding device, may comprise e.g. a set of labels and / or an area or sub-image of interest. The corresponding parts of the image being indicated by the tasks could e.g. be encoded with a high resolution. As non-limiting examples, a task could for example in a self-driving car application be e.g. a car, a traffic light, a traffic sign, a person, or another crucial attribute. In self-driving cars, it is important not to distort pixels related to the tasks / attributes / labels that are crucial for the applications of the self-driving cars, such as e.g. cars, traffic lights, traffic signs, people etc., while the rest of the scene, such as e.g. trees, the sky, buildings etc., can be partially synthesized by a neural network. The image encoder device will, by utilizing these tasks, encode the parts / regions of the residual image corresponding to those crucial labels with a relatively high rate. The rest of the scene, i.e. the less crucial parts / regions, are masked out and synthesized by the neural network and the compact version of the image. Hereby, a task-dependent and efficient source encoding is provided.

[0027] In an implementation form of an image encoding device according to the first aspect, the semantic based distortion map m comprises a heat-map distortion of the input image x.

[0028] An advantage with this implementation form is that it provides a semantic-based distortion measure for the input image x. This task-dependent distortion measure is based on the specific task / goal, which focuses on particular areas of an image or a video, and makes the encoding more efficient.

[0029] In an implementation form of an image encoding device according to the first aspect, the residual image r is a difference image between the estimated image x' and the input image x.

[0030] An advantage with this implementation form is that the residual image r provides the extra task- and application-dependent details, i.e. the labels / areas of interest that enhance the second coarse and the first fundamental layers such that a visually appealing representation of the original image is produced, and such that a certain target distortion is achieved. A sum of the encoding rate of the residual image r' and the encoding rate of the coarse image c may be optimized together with the distortion depending on the requirements in order to achieve an efficient task-dependent image / video compression.

[0031] In an implementation form of an image encoding device according to the first aspect, the input image x is for a video application.

[0032] An advantage with this implementation form is that the image encoding device may be used for a large number of video applications, such as e.g. self-driving cars, video surveillance, virtual or augmented reality applications. A video is a sequence of images, i.e. a sequence of frames, that can be processed frame by frame in order to generate corresponding sequences of semantic segmentation images s, estimated images x' and residual images r, that are then converted to video files. Standard video codecs can here be used to generate the encoded coarse video / images c' , and to provide encoded semantic segmentation video / images s' and residual video / images r', such that a task-dependent video compression is implemented.

[0033] According to a second aspect of the invention, the above mentioned and other objectives are achieved with a first communication device for a communication system, the first communication device comprising an image encoding device as herein described, the first communication device being configured to: transmit the encoded semantic segmentation image s' over a first communication channel to a second communication device.

[0034] The first communication device comprises the herein described image encoding device and thus provides the above mentioned advantages of the image encoding device. Further, an advantage of the first communication device is that efficiently encoded image and video data may by usage of the first communication device be transmitted to the second communication device, which reduces the total payload of the communication system. Also, by using semantic content of the image and video data together with the task, as well as application and resource requirements, a trade-off between coding rate and visual / perceptual quality is provided. Also, an adaptive transmission at optimized rate is provided, which is crucial especially for wireless communications. Further, the hierarchical three-layer encoding provided by the image encoding device provides a scalable transmission depending on the application. Hereby, semantic-based communications are enabled, usable for current and future wireless networks. By transmission of the first fundamental image encoding layer provided by the encoded semantic segmentation image s', the encoding rate may be reduced dramatically at the same time as a semantic-based encoding is provided, i.e. the entire semantic information of the image is transmitted.

[0035] In an implementation form of a first communication device according to the second aspect, the first communication device is configured to: transmit the encoded coarse image c to the second communication device.

[0036] An advantage with this implementation form is that both the encoded semantic segmentation image s' and the encoded coarse image c may be transmitted, which may be seen as, in addition to the first fundamental image encoding layer, the second coarse image encoding layer, i.e. a compressed image representation of the input image, being transmitted. Such a second coarse image encoding layer provides a quality acceptable if a coarse reconstruction of the image is required. Either only the first fundamental image encoding layer, or both the first fundamental image encoding layer and the second coarse image encoding layer may here be transmitted, which may be chosen based on the application.

[0037] In an implementation form of a first communication device according to the second aspect, the first communication device is configured to: transmit the encoded residual image r' to the second communication device.

[0038] An advantage with this implementation form is that all of the encoded semantic segmentation image s', the encoded coarse image c , and the encoded residual image r' may be transmitted to the second communication device, which may be seen as, in addition to the first fundamental image encoding layer and the second coarse image encoding layer, a third refined image encoding layer, including also a compressed version of the residual, being transmitted. Such a refined image encoding layer provides a quality demanded, e.g. extra application-dependent details if a refined reconstruction of the image is required at the second communication device. Either of the first fundamental image encoding layer alone, both of the first fundamental image encoding layer and the second coarse image encoding layer, or all of the first fundamental image encoding layer, the second coarse image encoding layer and the third refined image encoding layer, may here be transmitted, which may be chosen based on the application. In an implementation form of a first communication device according to the second aspect, the first lossy encoder has a first code rate Rcand the second lossy encoder has a second code rate Rr, and wherein the image encoding device is configured to: determine a sum of the first code rate Rcand the second code rate Rr.

[0039] An advantage with this implementation form is that it provides the possibility to trade-off between the first Rcand second Rrcode rates, and to sweep over a large range of rate and distortion pairs. Hereby, adaptivity and flexibility for the encoding and decoding of images or video are provided, as well as a trade-off between code rate and visual / perceptual quality. This might be especially important for wireless communication systems having varying channel conditions.

[0040] In an implementation form of a first communication device according to the second aspect, the sum of the first code rate Rcand the second code rate Rris determined based on one or more of an image target distortion for the input image x, an estimate of the first communication channel, available channel resources of the first communication channel, and an image requirement set by an application associated with the input image x.

[0041] Hereby, it is possible to sweep over a large range of rate and distortion pairs, which provides adaptivity and flexibility, being especially important for wireless communication systems.

[0042] In an implementation form of a first communication device according to the second aspect, the first communication device is configured to: transmit a signal over a second communication channel to the second communication device, the signal indicating one or more of: the task associated with the input image x, the refinement DNN, a decoder corresponding to the lossless encoder, the first lossy decoder corresponding to the first lossy encoder, and a second lossy decoder corresponding to the second lossy encoder.

[0043] An advantage with this implementation form is that the image decoding device in the second communication device is provided, via the second communication channel, which may be a control / knowledge sharing channel different from the first communication channel, with various information useful for efficiently generating a reconstructed image x of the input image x.

[0044] According to a third aspect of the invention, the above mentioned and other objectives are achieved with an image decoding device configured to: obtain an encoded semantic segmentation image s' of an input image x; decode the encoded semantic segmentation image s' based on a lossless decoder to generate a semantic segmentation image s of the input image x; generate an estimated image x' of the input image x based on the semantic segmentation image s and a semantic based distortion map m in a refinement DNN; and generate a reconstructed image x of the input image x based on the estimated image x'.

[0045] An advantage with this implementation form is that the image decoding device is provided with knowledge of the semantic segmentation image s and also has knowledge of the task attributes such that it can generate the semantic based distortion map m. The semantic based distortion map m should be known both to the image encoding device and to the image decoding device. Hereby, an estimated image x' of the input image x can be generated based on the semantic segmentation image s provided by the first fundamental image encoding layer and the semantic based distortion map m in a refinement deep neural network.

[0046] The image decoding device efficiently decodes image and video data using semantic content of the image and video data, respectively, with the goal of achieving high performance and sufficient perceptual quality for a CV task. Furthermore, the image decoding device provides a trade-off between coding rate and visual / perceptual quality.

[0047] The utilized combination of Al based blocks and classical / standard decompression blocks provides for an efficient and adaptive semantic-based and / or task-dependent image decoding scheme, which focusses on particular areas of interest in an image or a video, based on the specific task and / or goal. The image decoding device comprises the refinement deep neural network, which is capable of generating a high-resolution image, and which may be implemented as, e.g. as a conditional generative adversarial network (cGAN) or a denoising diffusion probabilistic models (DDPM) neural network. In an implementation form of an image decoding device according to the third aspect, the image decoding device is configured to: obtain an encoded coarse image c of the input image x; decode the encoded coarse image c based on a first lossy decoder to generate a coarse image c of the input image x; and generate the estimated image x of the input image x further based on the coarse image c of the input image x.

[0048] By obtaining the encoded semantic segmentation image s' of the first fundamental image encoding layer and the encoded coarse image c of the second coarse image encoding layer, the image decoding device is provided with knowledge of the semantic segmentation image s and the coarse image c. The image decoding device can then synthesize the estimated missing details and thereby provide an enhanced estimated image x' of the input image x.

[0049] In an implementation form of an image decoding device according to the third aspect, the image decoding device is configured to: obtain an encoded residual image r' of the input image x ; decode the encoded residual image r' of the input image x based on a second lossy decoder to generate a reconstructed residual image of the input image x; and generate the reconstructed image x of input image x further based on the reconstructed residual image of the input image x.

[0050] By obtaining the encoded semantic segmentation image s' of the first fundamental image encoding layer, the encoded coarse image c of the second coarse image encoding layer and the encoded residual image r' of the third refined image encoding layer, the image decoding device may utilize knowledge of the semantic segmentation image s, the coarse image c and a reconstructed residual image to produce an even more visually appealing representation of the original image.

[0051] In an implementation form of an image decoding device according to the third aspect, the reconstructed image x is an addition of the estimated image x' and the reconstructed residual image . By adding the reconstructed residual image to the estimated image x', the important parts / regions / labels, having been encoded with a high rate in the image encoding device, are included in the reconstructed image x. As mentioned above, less important parts / regions / labels of the image / video, such as e.g. the sky, trees etc., are, by utilization of the semantic based distortion map m, masked out to be synthesized by the neural network as a compact lower rate version of the image. However, more important parts / regions / labels, such as e.g. persons, cars etc., are encoded included in the encoded residual image r' at a relatively higher rate.

[0052] In an implementation form of an image decoding device according to the third aspect, the the semantic based distortion map m is determined based on a task associated with the input image x and the semantic segmentation image s.

[0053] As mentioned above, the task is used for generating the semantic based distortion map m. Hereby, areas / objects of the input image x that are not of interest, or are of less interest, may be masked out, whereas the task-dependent / semantic-based information is expressed. Hereby, the actual content / meaning of the image / video is utilized depending on a desired task, i.e. on the semantic based distortion map m, such that the encoding is focussed on the relevant parts / regions of the image / video, paying less attention to the irrelevant parts / regions. This increases the image / video compression efficiency, while the compression can be optimized depending on the distortion target.

[0054] In an implementation form of an image decoding device according to the third aspect, the the task comprises a set of labels and / or a sub-image of interest of the input image x.

[0055] An advantage with this implementation form is that tasks, i.e. attributes, being known to both the image encoding device and the image decoding device, may comprise e.g. a set of labels and / or an area or sub-image of interest, such that the corresponding parts of the image being indicated by the tasks could e.g. be encoded with a high resolution. As non-limiting examples, a task could be e.g. a car, a traffic light, a traffic sign, a person, or another attribute in a selfdriving car application.

[0056] In an implementation form of an image decoding device according to the third aspect, the semantic based distortion map m comprises a heat-map distortion of the input image x. An advantage with this implementation form is that it provides a semantic-based distortion measure for the input image x. This task-dependent distortion measure is based on the specific task / goal, which focuses on particular areas of an image or a video.

[0057] In an implementation form of an image decoding device according to the third aspect, the the input image x is for a video application.

[0058] Hereby, the image decoding device may be used for a large number of video applications, such as e.g. self-driving cars, video surveillance, or virtual reality applications. A video is a sequence of images, i.e. a sequence of frames, that can be processed frame by frame in order to generate corresponding sequences of semantic segmentation images s, estimated images x' and / or residual images r, that are then converted to video files. Standard video codecs can be used to decode the first fundamental video / images encoding layer, the second coarse video / images encoding layer and / or the third refined video / images encoding layer being input to the video / image decoding device.

[0059] According to a fourth aspect of the invention, the above mentioned and other objectives are achieved with a second communication device for a communication system, the second communication device comprising an image decoding device as herein described, the second communication device being configured to: receive the encoded semantic segmentation image s' of the input image x over a first communication channel from a first communication device.

[0060] By the received transmission of the the encoded semantic segmentation image s', the image decoding device is provided with knowledge of the semantic segmentation image s. The image decoding device also has knowledge of the task attributes such that it can generate the semantic based distortion map m. Hereby, an estimated image x' of the input image x can be generated based on the semantic segmentation image s provided by the first fundamental image encoding layer and the semantic based distortion map m in a refinement deep neural network.

[0061] Thus, the task attributes should be known both to the image encoding device of the first communication device and to the image decoding device of the second communication device, and may be synchronized via a second communication channel, possibly being a control / knowledge sharing channel different from the first communication channel. Hereby, the semantic based distortion map m may be generated by the image encoding device and / or by the image decoding device, such that an estimated image x' can be generated based on the semantic segmentation image s and the semantic based distortion map m in a refinement deep neural network of the image encoding device and / or of the image decoding device.

[0062] In an implementation form of a second communication device according to the fourth aspect, the second communication device is configured to: receive an encoded coarse image c of the input image x from the first communication device.

[0063] By the received transmission of the encoded semantic segmentation image s' of the first fundamental image encoding layer and the encoded coarse image c of the second coarse image encoding layer, the image decoding device is provided with knowledge of the semantic segmentation image s and the coarse image c. The image decoding device can then synthesize the estimated missing details and thereby provide an enhanced estimated image x' of the input image x.

[0064] In an implementation form of a second communication device according to the fourth aspect, the second communication device is configured to: receive an encoded residual image r' of the input image x from the first communication device.

[0065] By the received transmission of the encoded semantic segmentation image s' of the first fundamental image encoding layer, the encoded coarse image c of the second coarse image encoding layer and the encoded residual image r' of the third refined image encoding layer, the image decoding device may utilize knowledge of the semantic segmentation image s, the coarse image c and a reconstructed residual image f to produce an even more visually appealing representation of the original image.

[0066] In an implementation form of a second communication device according to the fourth aspect, the second communication device is configured to: receive a signal over a second communication channel from the first communication device, the signal indicating one or more of: the task associated with the input image x, the refinement DNN, the lossless decoder, the first lossy decoder, and the second lossy decoder.

[0067] An advantage with this implementation form is that the image decoding device of the second communication device is provided, via the second communication channel, with various information useful for efficiently generating a reconstructed image x of the input image x.

[0068] According to a fifth aspect of the invention, the above mentioned and other objectives are achieved with a method for for an image encoding device, the method comprising: generating a semantic segmentation image s of an input image x based on a semantic segmentation deep neural network, SS DNN, (120); encoding the semantic segmentation image s based on a lossless encoder to generate an encoded semantic segmentation image s'; encoding the input image x based on a first lossy encoder, a corresponding first lossy decoder and a semantic based distortion map m to generate an encoded coarse image c and a coarse image c; generating an estimated image x of the input image x based on the semantic segmentation image s, the coarse image c and the semantic based distortion map m in a refinement deep neural network, refinement DNN; generating a residual image r based on the estimated image x' and the input image x; and encoding the residual image r based on a second lossy encoder to generate an encoded residual image r' .

[0069] The method according to the fifth aspect can be extended into implementation forms corresponding to the implementation forms of the image encoding device according to the first aspect. Hence, an implementation form of the method comprises the feature(s) of the corresponding implementation form of the image encoding device.

[0070] The advantages of the methods according to the fifth aspect are the same as those for the corresponding implementation forms of the image encoding device according to the first aspect.

[0071] According to a sixth aspect of the invention, the above mentioned and other objectives are achieved with a method for an image decoding device, the method comprising: obtaining an encoded semantic segmentation image s' of an input image x;decoding the encoded semantic segmentation image s' based on a lossless decoder to generate a semantic segmentation image s of the input image x; generating an estimated image x' of the input image x based on the semantic segmentation image s and a semantic based distortion map m in a refinement DNN; and generating a reconstructed image x of the input image x based on the estimated image x'.

[0072] The method according to the sixth aspect can be extended into implementation forms corresponding to the implementation forms of the image decoding device according to the second aspect. Hence, an implementation form of the method comprises the feature(s) of the corresponding implementation form of the image decoding device.

[0073] The advantages of the methods according to the sixth aspect are the same as those for the corresponding implementation forms of the image decoding device according to the second aspect.

[0074] Embodiments of the invention also relates to a computer program, characterized in program code, which when run by at least one processor causes the at least one processor to execute any method according to embodiments of the invention. Further, embodiments of the invention also relate to a computer program product comprising a computer readable medium and the mentioned computer program, wherein the computer program is included in the computer readable medium, and may comprises one or more from the group of read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically erasable PROM (EEPROM), hard disk drive, etc.

[0075] Further applications and advantages of embodiments of the invention will be apparent from the following detailed description.

[0076] Brief Description of the Drawings

[0077] The appended drawings are intended to clarify and explain different embodiments of the invention, in which:

[0078] - Fig. 1 shows an image encoding device according to embodiments of the invention; - Fig. 2 shows a flow chart of a method for an image encoding device according to embodiments of the invention;

[0079] - Fig. 3 shows a first communication device according to embodiments of the invention;

[0080] - Fig. 4 shows an image decoding device according to embodiments of the invention;

[0081] - Fig. 5 shows a flow chart of a method for an image decoding device according to embodiments of the invention;

[0082] - Fig. 6 shows a first communication device according to embodiments of the invention;

[0083] - Fig. 7 shows a communication system according to embodiments of the invention;

[0084] - Fig. 8 shows an image encoding device according to embodiments of the invention;

[0085] - Fig. 9 shows an image decoding device according to embodiments of the invention; and

[0086] - Fig. 10 shows a signaling diagram according to embodiments of the invention.

[0087] Detailed Description

[0088] There is generally a need to reduce the payload of various wireless applications by using enhanced encoding approaches, such that the amount of data to be transmitted over the communication systems, especially over wireless communication systems, is reduced. Since a large part of the transmitted data involves transmission of image and / or video data, an accurate and efficient encoding scheme would reduce the total amount of data transmitted by the communication systems conveying the image and / or video data.

[0089] A deep semantic segmentation-based layered image compression (DSSLIC) scheme is previously known, which utilizes a semantic content of the image data. The already known DSSLIC scheme offers a layered image encoding. The DSSLIC encoder outputs three layers of information: a semantic segmentation map layer, a residual layer and a compact layer. The encoder uses three different neural networks (NNs), which are SegNet, CompNet and FineNet, for generating the semantic segmentation map, the compressed representation, and the fine details of the image, respectively. A semantic segmentation network is here used as the SegNet for generating the semantic segmentation map. A convolutional neural network (CNN) and a generative adversarial network (GAN) are used for the CompNet and the FineNet, respectively, which are jointly trained. To encode the three layers of information, classic lossless and lossy encoders are used in the DSSLIC encoder. In particular, the semantic segmentation map and the compressed image are here encoded with a free lossless image format (FLIF) encoder, while the residual is encoded with a better portable graphics (BPG) encoder. The known DSSLIC encoding scheme is reported to outperform traditional coding schemes, such as joint picture expert group (JPEG), JPEG200, BPG and an image format for the web (WebP), on most widely considered datasets, e.g., ADE20K, Kodak, CityScapes.

[0090] However, the known DSSLIC encoding scheme has some drawbacks related to its performance, as it provides a suboptimal accuracy and efficiency. Also, in some situations, the DSSLIC encoding scheme unnecessarily spends bits by encoding irrelevant regions of the input image, and also has a fixed output quality irrespective of varying channel conditions. The known DSSLIC encoding scheme is further burdened by extensive training of the convolutional neural network.

[0091] Therefore, it is herein presented a solution mitigating or fully solving the drawbacks of conventional solutions. A general aspect of the solution is to provide an enhanced image and / or video encoding scheme utilizing a semantic content of the image and / or video data to be encoded, which provides for a more efficient and accurate source coding.

[0092] Fig. 1 schematically shows a block scheme of an image encoding device 100 configured to encode an input image x according to some herein described embodiments. Such an image encoding device 100 may be called a Task-Dependent Layered Image and Video (LIV) Encoder. The image encoding device 100 is configured to generate a semantic segmentation image s of an input image x based on a semantic segmentation deep neural network (SSDNN) 120. The image encoding device 100 is further configured to encode the semantic segmentation image s based on a lossless encoder 130 to generate an encoded semantic segmentation image s'. In this document, semantic segmentation refers to a deep learning algorithm, i.e. a deep neural network, which associates a label or category with each pixel in an image.

[0093] The image encoding device 100 is further configured to encode the input image x based on a first lossy encoder 140, a corresponding first lossy decoder 142 and a semantic based distortion map m to generate an encoded coarse image c and a coarse image c. Thus, the first lossy encoder 140 generates the encoded coarse image c based on the input image x and the semantic based distortion map m, and the first lossy decoder 142 generates the coarse image c based on the encoded coarse image c . The semantic based distortion map m may be provided to the first lossy encoder 140 by a semantic task filter 152. The image encoding device 100 is further configured to generate in the refinement deep neural network (DNN) 150 an estimated image x of the input image x based on the semantic segmentation image s, the coarse image c and the semantic based distortion map m. The refinement deep neural network 150 is here provided with the semantic segmentation image s from the semantic segmentation deep neural network 120, is provided with the coarse image c from the first lossy decoder 142, and is provided with the semantic based distortion map m from the semantic task filter 152. The refinement deep neural network 150 may be any block / network / algorithm, e.g., a conditional generative adversarial network (cGAN) or a denoising diffusion probabilistic network (DDPM) neural network, capable of reconstructing the estimated high-resolution image x from the coarse image c, i.e. from its low-resolution compressed version, the semantic segmentation image s, and the semantic based distortion map m. The refinement deep neural network 150 thus generates an image of fine details, which is then added to the coarse image c to produce an estimated image x of the input image x. The refinement deep neural network 150 hereby directly generates the estimated image x' of the input image x.

[0094] The image encoding device 100 is further configured to generate a residual image r based on the estimated image x' and the input image x, e.g. by using a differential unit 154 subtracting the input image x from the estimated image x'. The differential unit 154 is provided with the estimated image x' from the refinement deep neural network 150, and is provided with the input image from the input 158. The image encoding device 100 is further configured to encode the residual image r based on a second lossy encoder 160 to generate an encoded residual image r'. The second lossy encoder 160 is provided with the residual image r from the differential unit 154.

[0095] The image encoding device 100 may, according to various embodiments, further be configured to store the encoded semantic segmentation image s', and to possibly, in addition to the encoded semantic segmentation image s', also store the encoded coarse image c or both of the encoded coarse image c and the encoded residual image r' in an encoding memory 180, as explained below. Thus, the image encoding device outputs three layers of information. The first fundamental image encoding layer provided by the encoded semantic segmentation image s' may also be described as the semantic segmentation map layer. The second coarse image encoding layer provided by the additional encoded coarse image c may also be described as a compact / compressed layer of the input image x. The third refined image encoding layer provided by the additional compressed version r' of the residual image image / video data may also be described as a residual layer.

[0096] Fig. 2 shows a flow chart of a corresponding method 200 which may be executed by the image encoding device 100, such as the one shown in Fig. 1, possibly implemented in a first communication device 600, such as the one shown in Fig. 3. It should be noted that the method steps illustrated in Fig. 2 and described herein do not necessarily have to be executed in the order illustrated in Fig. 2. The steps may essentially be executed in any suitable order, or in parallel, as long as the physical requirements and the information needed to execute each step is available when the step is executed.

[0097] The method 200 comprises generating 202 a semantic segmentation image s of an input image x based on a semantic segmentation deep neural network 120. The method 200 further comprises encoding 204 the semantic segmentation image s based on a lossless encoder 130 to generate an encoded semantic segmentation image s'. The method 200 further comprises encoding 206 the input image x based on a first lossy encoder 140, a corresponding first lossy decoder 142 and a semantic based distortion map m to generate an encoded coarse image c and a coarse image c. The method 200 further comprises generating 208 an estimated image x' of the input image x based on the semantic segmentation image s, the coarse image c and the semantic based distortion map m in a refinement deep neural network 150. The method 200 further comprises generating 210 a residual image r based on the estimated image x' and the input image x. The method 200 further comprises encoding 212 the residual image r based on a second lossy encoder 160 to generate an encoded residual image r'.

[0098] The image encoding device 100 and the corresponding method 200 encodes the input image x based on the semantic based distortion map m, which makes the encoding scheme taskdependent, i.e. dependent on the task input. The semantic based distortion map m defines the areas / objects of interest as well as trades off the visual quality and the encoding rate, and as a result the compression ratio. Hereby, unnecessarily spending bits by encoding areas / objects and / or regions of the input image that are irrelevant for the specific task and application is avoided. It should be noted that the image encoding device 100 and the corresponding method 200 differ from the above mentioned known DSSLIC scheme by the use of the semantic based distortion map m. Thus, the image encoding device 100 and the corresponding method 200 utilize the task input, i.e. utilize the semantic based distortion map m, to provide a taskdependent encoding scheme, whereas the DSSLIC scheme is not task-depending, i.e. does not base its encoding on any parameter corresponding to the semantic based distortion map m.

[0099] The image encoding device 100 and the corresponding method 200 utilize a combination of Al and standard / classical compression blocks for providing better compression results and faster training of the blocks. Hereby, the burden of extensive training of neural networks, such as e.g. extensive training of the convolutional neural network in the known DSSLIC encoding scheme, is avoided.

[0100] The image encoding device 100 and the corresponding method 200 provide an adaptivity of the encoding, i.e. an optimization of the encoding depending e.g. on channel conditions, requirements, tasks and / or applications. The image encoding device 100 and the corresponding method 200 further provide a scalability by hierarchically storing and / or transmitting the first, second or third image encoding layers depending on the requirements and the image decoding device 300. Hereby, the output quality may be adapted based on varying channel conditions.

[0101] Fig. 3 schematically shows a first communication device 600 according to an embodiment of the invention. In the embodiment shown in Fig. 3, the first communication device 600 comprises a processor 102, a transceiver 104 and a memory 106. The processor 102 is coupled to the transceiver 104 and the memory 106 by communication means 108 known in the art. The first communication device 600 further comprises a communication interface 110 coupled to the transceiver 104, which means that the first communication device 600 is configured for communications in a communication system 500 e.g., with a second communication device 800, as shown in Fig. 7. The herein described image encoding device 100 may be implemented in the first communication device 600, more precisely may be implemented in the processor 102 of the first communication device 600. The processor 102 may be referred to as one or more general-purpose CPU, one or more digital signal processor (DSP), one or more application-specific integrated circuit (ASIC), one or more field programmable gate array (FPGA), one or more programmable logic device, one or more discrete gate, one or more transistor logic device, one or more discrete hardware component, or one or more chipsets. The memory 106 may be a read-only memory, a random access memory (RAM), or a non-volatile RAM (NVRAM). The transceiver 104 may be a transceiver circuit, a power controller, or an interface providing capability to communicate with other communication modules or communication devices. The transceiver 104, memory 106 and / or processor 102 may be implemented in separate chipsets or may be implemented in a common chipset. That the first communication device 600 is configured to perform certain actions can in this disclosure be understood to mean that the first communication device 600 comprises suitable means, such as e.g., the processor 102 and the transceiver 104, configured to perform the actions.

[0102] According to embodiments illustrated in Fig. 3, the first communication device 100 comprises the herein described image encoding device 100. As explained above, the image encoding device 100 is configured to generate a semantic segmentation image s of an input image x based on a semantic segmentation deep neural network 120. The image encoding device 100 is further configured to encode the semantic segmentation image s based on a lossless encoder 130 to generate an encoded semantic segmentation image s'.

[0103] The image encoding device 100 is further configured to encode the input image x based on a first lossy encoder 140, a corresponding first lossy decoder 142 and a semantic based distortion map m to generate an encoded coarse image c and a coarse image c. The image encoding device 100 is further configured to generate an estimated image x' of the input image x based on the encoded semantic segmentation image s, the coarse image c and the semantic based distortion map m in a refinement deep neural network 150. The image encoding device 100 is further configured to generate a residual image r based on the estimated image x' and the input image x.

[0104] The image encoding device 100 is further configured to encode the residual image r based on a second lossy encoder 160 to generate an encoded residual image r'. Thus, the image encoding device 100 is configured to provide the encoded semantic segmentation image s', the encoded coarse image c and the encoded residual image r', and may be configured to store the encoded semantic segmentation image s' , and to possibly, in addition to the encoded semantic segmentation image s' , also store the encoded coarse image c or both of the encoded coarse image c and the encoded residual image r', in the encoding memory 180.

[0105] The first communication device 600 is further configured to transmit at least the encoded semantic segmentation image s' over a first communication channel 510 to a second communication device 800. The first communication device 600 may also be further configured to, in addition to the encoded semantic segmentation image s’ , transmit the encoded coarse image c or both of the encoded coarse image c and the encoded residual image r' to the second communication device 800.

[0106] Fig. 4 schematically shows a block scheme of an image decoding device 300. Such an image decoding device 100 may be called a Task-Dependent LIV decoder. The image decoding device 300 is configured to obtain an encoded semantic segmentation image s' of an input image x, where the encoded semantic segmentation image s' may have been previously stored and / or transmitted.

[0107] The image decoding device 300 is further configured to decode the encoded semantic segmentation image s' based on a lossless decoder 132 to generate a semantic segmentation image s of the input image x. The image decoding device 300 is further configured to generate an estimated image x of the input image x based on the semantic segmentation image s and a semantic based distortion map m in a refinement deep neural network 150. Thus, the refinement deep neural network 150 is provided with the semantic segmentation image s by the lossless decoder 132. The refinement deep neural network 150 may also be provided with the semantic based distortion map m by a semantic task filter 152.

[0108] The image decoding device 300 is further configured to generate a reconstructed image x of the input image x based on the estimated image x'. The reconstructed image x of the input image x may here be generated in an addition unit 156, as explained in detail below. The input image x may according to some embodiments for example be associated with a video application. As explained below, the image decoding device 300 may, according to some embodiments, be configured to, in addition to the encoded semantic segmentation image s', obtain the encoded coarse image c or both of the encoded coarse image c and the encoded residual image r', and to base the generation of the reconstructed image x also on the obtained encoded coarse image c or also on both of the obtained encoded coarse image c and the obtained encoded residual image r', respectively.

[0109] Fig. 5 shows a flow chart of a corresponding method 400 which may be executed by a image decoding device 300, such as the one shown in Fig. 4, possibly implemented in a second communication device 800, such as the one shown in Fig. 6. It should be noted that the method steps illustrated in Fig. 5 and described herein do not necessarily have to be executed in the order illustrated in Fig. 5. The steps may essentially be executed in any suitable order, or in parallel, as long as the physical requirements and the information needed to execute each step is available when the step is executed.

[0110] The method 400 comprises obtaining 402 an encoded semantic segmentation image s' of an input image x. The method 400 further comprises decoding 404 the encoded semantic segmentation image s' based on a lossless decoder 132 to generate a semantic segmentation image s of the input image x. The method 400 further comprises generating 406 an estimated image x' of the input image x based on the semantic segmentation image s and a semantic based distortion map m in a refinement deep neural network 150. The method 400 further comprises generating 408 a reconstructed image x of the input image x based on the estimated image x'.

[0111] The image decoding device 300 and the corresponding method 400 decodes the input image x based on the semantic based distortion map m, which makes the decoding scheme taskdependent, i.e. dependent on the task input. The semantic based distortion map m defines the areas / objects of interest as well as trades off the visual quality and the encoding rate, and as a result the compression ratio. It should be noted that the image decoding device 300 and the corresponding method 400 differ from the above mentioned known DSSLIC scheme by the use of the semantic based distortion map m. The DSSLIC scheme is not task-depending, i.e. does not base its encoding and decoding on any parameter corresponding to the semantic based distortion map m. The image decoding device 300 and the corresponding method 400 utilize a combination of Al and standard / classical compression blocks. Hereby, the burden of extensive training of neural networks is avoided.

[0112] Fig. 6 shows a second communication device 800 for a communication system 500 according to an embodiment of the invention. In the embodiment shown in Fig. 6, the second communication device comprises a processor 302, a transceiver 304 and a memory 306. The processor 302 is coupled to the transceiver 304 and the memory 306 by communication means 308 known in the art. The second communication device 800 further comprises a communication interface 310 coupled to the transceiver 304, which means that the second communication device 800 is configured for communications in a communication system 500.

[0113] The processor 302 may be referred to as one or more general-purpose CPU, one or more digital signal processor (DSP), one or more application-specific integrated circuit (ASIC), one or more field programmable gate array (FPGA), one or more programmable logic device, one or more discrete gate, one or more transistor logic device, one or more discrete hardware component, or one or more chipsets. The memory 306 may be a read-only memory, a random access memory (RAM), or a non-volatile RAM (NVRAM). The transceiver 304 may be a transceiver circuit, a power controller, or an interface providing capability to communicate with other communication modules or communication devices. The transceiver 304, memory 306 and / or processor 302 may be implemented in separate chipsets or may be implemented in a common chipset. That second communication device 800 is configured to perform certain actions can in this disclosure be understood to mean that the second communication device 800 comprises suitable means, such as e.g., the processor 302 and the transceiver 304, configured to perform the actions.

[0114] According to embodiments of the invention, the second communication device 800 is configured to receive the encoded semantic segmentation image s' of the input image x over a first communication channel 510 from a first communication device 600. The second communication device 800 comprises the herein described image decoding device 300. As explained above, the image decoding device 300 is configured to obtain the encoded semantic segmentation image s' of the input image x. The image decoding device 300 is further configured to decode the encoded semantic segmentation image s' based on the lossless decoder 132 to generate the semantic segmentation image s of the input image x. The image decoding device 300 is further configured to generate an estimated image x' of the input image x based on the semantic segmentation image s and the semantic based distortion map m in a refinement deep neural network 150. The image decoding device 300 is further configured to generate the reconstructed image x of the input image x based on the estimated image x'.

[0115] As explained below, the second communication device 800 may, according to some embodiments, be configured to, in addition to the encoded semantic segmentation image s', receive the encoded coarse image c or both of the encoded coarse image c and the encoded residual image r', such that the image decoder device 300 may, in addition to the encoded semantic segmentation image s', base the generation of the reconstructed image x also on the obtained encoded coarse image c or also on both of the obtained encoded coarse image c and the obtained encoded residual image r'.

[0116] Fig. 7 schematically illustrates a communication system 500, over which the first communication device 600 may transmit at least the encoded semantic segmentation image s' over a first communication channel 510 to the second communication device 800. According to some embodiments explained below, the first communication device 600 may also, in addition to the encoded semantic segmentation image s', transmit an encoded coarse image c or both an encoded coarse image c and an encoded residual image r' to the second communication device 800 over the first communication channel 510. According to some embodiments explained below, the first communication device 600 may further transmit a signal 620 to the second communication device 800 over a second communication channel 520. The first communication device 600 may according to various embodiments be a mobile device or a base station device. The second communication device 800 may according to various embodiments be a mobile device or a base station device. Thus, the encoded image may be transmitted in any direction in Fig. 7, either directly between two mobile devices or between one base station device and a mobile device, or may be transmitted indirectly between such entities, via one or more intermediate devices and / or networks.

[0117] Fig. 8 schematically illustrates a block scheme for some embodiments of the image encoding device 100. As explained in connection with Fig 1, the image encoding device 100 is configured to encode an input image x, which according to some embodiments is associated with a video application. A semantic segmentation image s of the input image x is created based on a semantic segmentation deep neural network 120. The semantic segmentation image s is then encoded based on a lossless encoder 130 to generate an encoded semantic segmentation image s', which may be stored in the encoder memory 180 and / or transmitted over a first communication channel 510 to an image decoding device 300.

[0118] The input image x is also encoded such that the first lossy encoder 140 generates a encoded coarse image c at a first code rate Rcbased on the input image x and the semantic based distortion map m, and such that the first lossy decoder 142 generates the coarse image c based on the encoded coarse image c . The semantic based distortion map m may be provided to the first lossy encoder 140 by a semantic task filter 152. The first lossy encoder 140 may according to some embodiments use a classical compression algorithm, such as e.g. BPG or JPEG2000, whereby the input image x is compressed into a low dimensional version c of the input image x at the first code rate Rc.

[0119] The image encoding device 100 is further configured to generate an estimated image x of the input image x based on the semantic segmentation image s, the coarse image c and the semantic based distortion map m in the refinement deep neural network 150. The refinement deep neural network 150 is here provided with the encoded semantic segmentation image s from the semantic segmentation deep neural network 120, the coarse image c from the first lossy decoder 142, and the semantic based distortion map m from the semantic task filter 152.

[0120] According to an embodiment, the semantic based distortion map m is determined in the semantic task filter 152 based on a task associated with the input image x and on the semantic segmentation image s. The task associated with the input image x and the semantic segmentation image s are then provided to the semantic task filter 152, such that it can determine the semantic based distortion map m based on these parameters. The task associated with the input image x may, according to some embodiments, comprise a set of labels and / or a sub-image / area of interest, for which the corresponding parts of the input image x should be encoded with a high resolution.

[0121] According to an embodiment, the semantic based distortion map m is provided to the first lossy encoder 140. It is hereby possible to adapt a first code rate Rcof the first lossy encoder 140 based on the semantic based distortion map m. According to an embodiment, the semantic based distortion map m, which is used by the first lossy encoder 140, comprises a heat-map distortion of the input image x. An image distortion heat-map, or intensity-map, represents a visualization, comprising e.g. pixels, of the areas of an image that are most important or relevant for a particular task, e.g. for object detection or any other herein mentioned task. Thus, each pixel of the image heat-map distortion indicates the intensity of importance of the corresponding pixel in the original image.

[0122] A residual image r is further generated based on the estimated image x' and the input image x by subtracting the input image x from the estimated image x' in a differential unit 154. Thus, according to an embodiment, the residual image r is a difference image between the estimated image x' and the input image x, which complies with a target distortion. The target distortion refers to a targeted or acceptable level of distortion that is allowed during the process of compressing an image or video. For example, in lossy compression algorithms, the data size of the original image is reduced by the compression, which produces some distortion in the encoded image that should be below the target distortion. The differential unit 154 is here provided with the estimated image x' from the refinement deep neural network 150, and with the input image x from the input 158. The residual image r is provided to the second lossy encoder 160 from the differential unit 154, and the encoded residual image r' is then generated in the second lossy encoder 160. According to some embodiments, the second lossy encoder 160 generates the encoded residual image r' at a second code rate Rr. The second code rate Rrhas an impact on the amount of information and the energy content of the encoded residual image r'. According to some embodiments, the semantic based distortion map m is provided to the second lossy encoder 160 by the semantic task filter 152. By providing the semantic based distortion map m to the second lossy encoder 160, it is possible to adapt the second code rate Rrof the second lossy encoder 160 based on the semantic based distortion map m.

[0123] The image encoding device 100 may further be configured to store the encoded semantic segmentation image s', and possibly to, in addition to the encoded semantic segmentation image s', also store the encoded coarse image c or both the encoded coarse image c and the encoded residual image r' in an encoding memory 180.

[0124] Thus, the image encoding device 100 generates the encoded semantic segmentation image s', the encoded coarse image c , and the encoded residual image r' . The encoded semantic segmentation image s', the encoded coarse image c', and the encoded residual image r' may then, according to various embodiments, be stored in a hierarchical fashion, with the encoded semantic segmentation image s' having the highest ranking. Thus, either only the encoded semantic segmentation image s' is stored, or the encoded semantic segmentation image s' is stored in together with the encoded coarse image c , or all of the semantic segmentation image s' , the encoded coarse image c , and the encoded residual image r' are stored.

[0125] For encoding video, the input video is processed frame by frame, i.e., image by image. A sequence of semantic segmentation images s may here be created by the semantic segmentation deep neural network 120, which may be an off-the-shelf semantic segmentation network. In parallel, the input video is compressed using a classical off-the-shelf video codec 140, 142, e.g., H265, to create corresponding coarse / compressed video / images c. The coarse / compressed video / images c are fed frame-by-frame, i.e. image by image, to the refinement deep neural network 150. The refinement deep neural network 150 creates a sequence of fine details residual images r, that are converted to a residual video / images. The residual video / images is created in a corresponding way as above described for the image encoding device 100, and can be encoded as a video / images r' by a lossy video encoder of choice. Using the above described three layers of information for the encoded video, i.e. by generating the encoded semantic segmentation video / images s', the encoded coarse video / images c , and the encoded residual video / images r', video / images files can be encoded such that the encoding rate is reduced, and such that a task-dependent and semantic-based encoding being a valuable feature for may applications, such as e.g. internet of things applications, is offered at the same time.

[0126] As mentioned above, the first communication device 600 is configured to transmit the encoded semantic segmentation image s' over a first communication channel 510 to a second communication device 800. According to various embodiments, the first communication device 600 is configured to transmit the encoded semantic segmentation image s' , the encoded coarse image c , and the encoded residual image r' in a hierarchical fashion, with the encoded semantic segmentation image s' having the highest ranking. Thus, only the encoded semantic segmentation image s' may be transmitted to the second communication device 800, or the encoded coarse image c is also transmitted in addition to the encoded semantic segmentation image s' , or all of the semantic segmentation image s' , the encoded coarse image c , and the encoded residual image r' are transmitted. As mentioned above, according to an embodiment, the first lossy encoder 140 has a first code rate Rcand the second lossy encoder 160 has a second code rate Rr, where the first code rate Rcand / or the second code rate Rrmay be adapted based on the semantic based distortion map m. The image encoding device 100 is then configured to determine a sum of the first code rate Rcand the second code rate Rr. The sum of the first code rate Rcand the second code rate Rrmay be determined based on an image target distortion for the input image x. The target distortion corresponds to an optimal point of operation, e.g. a smallest code rate for which the resulting distortion is smaller than or equal to the target distortion. Thus, if the sum of the first code rate Rcand the second code rate Rris determined based on this target distortion, an optimal encoding point is achieved in respect of the actual resulting distortion.

[0127] According to an embodiment, the sum may also be determined based on an estimate of the first communication channel 510. The channel estimate is here an estimation or a measure of the quality of the first communication channel 510 between the first communication device 600 and the second communication device 800. If the quality of the first communication channel 510 is good, higher code rates may be used for the transmission, if the application allows it. However, if the quality of the first communication channel 510 is poor, the application may not allow higher code rates, wherefore lower code rates are then used for the transmission.

[0128] The sum may also be determined based on available channel resources of the first communication channel 510, where the available channel resources may be dependent on a number of users, such that each user may be allocated less resources if there are many users. For wireless communication, the available resources typically comprise the restricted radio resources available for a user.

[0129] The sum may also be determined based on an image requirement set by an application associated with the input image x. For example, if the application is for loT control, the image requirements of that application allow a high compression ratio corresponding to a lower code rate, whereas if the application is for H2M, the image requirements of that application may only allow a lower compression ratio corresponding to a higher code rate. The sum may further be determined based on combination of one or more of these parameters. Since the first code rate Rcat which the encoded coarse image c is encoded by the the first lossy encoder 140 has an impact on the amount of information and the energy content of the residual image r, there is a tradeoff between the first code rate Rcand the second code rate Rr. For example, if the first code rate Rc, at which the encoded coarse image c is encoded, is very small, i.e. if the encoded coarse image c is too coarse, it is natural to assume that the residual image r will comprise more energy, and will require more bits to achieve the target distortion. On the other hand, if the encoded coarse image c is encoded at a high first code rate Rc, it is to be expected that the residual image r will comprise less energy, and that encoding of the encoded residual image r in the second lossy encoder 160 will require less bits to achieve the target distortion at the expense of using more bits for generating the encoded coarse image c .

[0130] Therefore, given a certain target distortion, the second code rate Rris a function of the first code rate Rc, and the overall rate Rc+Rrcan, according to some embodiments, be optimized with respect the first code rate Rcand the refinement deep neural network 150, which in turn should be retrained for different values of the first code rate Rc. This optimization can be achieved by selecting a finite set of possible values for the first code rate Rcand retraining a refinement deep neural network for each first code rate Rc. Also, for each of the first code rates Rc, a large range of second code rates Rrand empirical pairs of Rc+Rrand corresponding distortions may be be generated. Then, sweeping over operating points for these pairs of Rc+Rrand corresponding distortions is performed, and the pair Rc+Rrwith the smallest rate for which the distortion is not larger than the given target distortion is selected as the operating point to be used.

[0131] It is an advantageous feature that rate adaptation depending on the channel conditions and / or depending on the application requirements may be provided by the image encoding device 100. Specifically for wireless applications, an optimization of pairs of Rc+Rrand corresponding distortions is very important, because the used operation point, and therefore also the actually used total rate, depends on the possibly varying channel conditions, resources etc. In addition, an application dependent flexibility is provided, since the rate may be adapted based application requirements as well. Thus, the image encoding device 100 may be task and / or application dependent since it may be adapted based on input parameters, such as the labels / classes / areas of interest that specify the semantic based distortion map m, and on the application requirements, e.g. rate and distortion parameters, that constrain the operation points.

[0132] According to some embodiments, the first communication device 600 is configured to transmit a signal 530 over a second communication channel 520 to the second communication device 800, as illustrated in Fig. 7. The transmitted signal 530 may then indicate to the second communication device 800 the task associated with the input image x, the refinement deep neural network 150, a decoder 132 corresponding to the lossless encoder 130, the first lossy decoder 142 corresponding to the first lossy encoder 140, and / or a second lossy decoder 162 corresponding to the second lossy encoder 160.

[0133] Fig. 9 schematically illustrates a block scheme for some embodiments of the image decoding device 300. As explained in connection with Fig 4, the image encoding device 100 is configured to encode an input image x, which according to some embodiments is associated with a video application. The image decoding device 300 is configured to obtain at least an encoded semantic segmentation image s' of the input image x, where the encoded semantic segmentation image s' may have been previously stored and / or transmitted.

[0134] The encoded semantic segmentation image s' is provided to a lossless decoder 132 and is decoded based on the lossless decoder 132 to generate a semantic segmentation image s of the input image x. The semantic segmentation image s is then provided to a refinement deep neural network 150, and an estimated image x' of the input image x is generated based on the semantic segmentation image s and a semantic based distortion map m in the refinement deep neural network 150. The semantic based distortion map m is provided to the refinement deep neural network 150 by a semantic task filter 152, such that the generation of the estimated image x' is task-dependent.

[0135] According to an embodiment, the semantic based distortion map m is determined by the semantic task filter 152 based on a task associated with the input image x and the semantic segmentation image s. Thus, the task and the semantic segmentation image s are provided to the semantic task filter 152, which provides the semantic based distortion map m to the refinement deep neural network 150. As mentioned above, the task associated with the input image x may, according to some embodiments, comprise a set of labels and / or a sub-image of interest of the input image x. According to an embodiment, the semantic based distortion map m comprises a heat-map distortion of the input image x, which is utilized by the refinement deep neural network 150 for generating the estimated image x' of the input image x.

[0136] The image decoding device 300 is further generates a reconstructed image x of the input image x based on the estimated image x'. The reconstructed image x of the input image x may here be generated in an addition unit 156.

[0137] The image decoding device 300 may, according to some embodiments, be configured to, in addition to the encoded semantic segmentation image s', obtain the encoded coarse image c , or both of the encoded coarse image c and the encoded residual image r ', and to base the generation of the reconstructed image x also on, in addition to the encoded semantic segmentation image s', the obtained encoded coarse image c , or on both of the obtained encoded coarse image c and the obtained encoded residual image r'.

[0138] More in detail, if the encoded coarse image c is obtained, e.g. retrieved from a memory or received as a transmission, the image decoder device 300 is configured to decode the encoded coarse image c based on a first lossy decoder 142 to generate a coarse image c of the input image x, and to generate the estimated image x further based on, in addition to the encoded semantic segmentation image s' , the coarse image c of the input image x. The coarse image c of the input image x is here provided to the refinement deep neural network 150, which generates the estimated image x' of the input image x based also on the coarse image c of the input image x. Thus, the refinement deep neural network 150 then generates the estimated image x' based on the semantic segmentation image s, a semantic based distortion map m, and the coarse image c of the input image x.

[0139] If both of the encoded coarse image c and the encoded residual image r' are obtained, e.g. retrieved from a memory or received as a transmission, in addition to the encoded semantic segmentation image s' , the image decoder device 300 is configured to decode the encoded coarse image c based on a first lossy decoder 142 to generate a coarse image c of the input image x, to decode the encoded residual image r' of the input image x based on a second lossy decoder 162 to generate a reconstructed residual image of the input image x, and to generate the reconstructed image x of input image x further based on, in addition to on the semantic segmentation image s, the coarse image c of the input image x and the reconstructed residual image of the input image x. The coarse image c of the input image x is here provided to the refinement deep neural network 150, which generates the estimated image x' of the input image x based also on, in addition to on the semantic segmentation image s, the coarse image c of the input image x. The reconstructed residual image of the input image x is provided to the addition unit 156, which is configured to generate the reconstructed image x as an addition of the estimated image x' and the reconstructed residual image .

[0140] Thus, at the image decoding device 300, the decoder has perfect knowledge of the semantic segmentation image / map s, due to the lossless encoding of encoded semantic segmentation image s' in the lossless encoder 130 in the image encoding device 100. Also, the semantic based distortion map m is known to both the image encoding device 100 and the image decoding device 300. The semantic based distortion map m does not need to be transmitted or encoded. The semantic based distortion map m may be obtained from a semantic task filter 152 by using as input the attributes of a task, which is known to both the image encoding device 100 and the image decoding device 300, and the semantic segmentation image s. The semantic based distortion map m is used as input in the refinement DNN 150 to generate an estimated image x' of the input image x. Since the semantic based distortion map m is associated with the task, the utilization of the semantic based distortion map m in the image decoding device 300 makes it task-dependent. The image decoding device 300 also has perfect knowledge of the coarse / compressed image c, meaning that the coarse / compressed image c in the image decoding device 300 is identical to the the coarse / compressed image c in the image encoding device 100 since the same first lossy decoder 142 is used in the in the image decoding device 300 as in the image encoding device 100. Hence, the decoder can synthesize the estimated missing details as well as generate an estimated image x' of the input image x. Together with the lossy encoded residual image r', the image decoding device 300 can finally produce a visually appealing reconstructed image x of the input image x.

[0141] Thus, according to an embodiment, the second communication device 800 is configured to receive, in addition to the encoded semantic segmentation image s', an encoded coarse image c of the input image x from the first communication device 100. The image decoding device 300 is then configured to decode the encoded coarse image c based on the first lossy decoder 142 to generate the coarse image c of the input image x, and to generate the estimated image x of the input image x based on the semantic segmentation image s and the coarse image c of the input image x.

[0142] According to an embodiment, the second communication device 800 is configured to receive, in addition to the encoded semantic segmentation image s' and the encoded coarse image c of the input image x, an encoded residual image r' of the input image x from the first communication device 100. The image decoding device 300 is then configured to decode the encoded residual image r' of the input image x based on the second lossy decoder 162 to generate the reconstructed residual image f of the input image x, and to generate the reconstructed image x of the input image x based on the semantic segmentation image s, the coarse image c of the input image x and the reconstructed residual image f of the input image x. Specifically, the reconstructed residual image f of the input image x is provided to the addition unit 156, in which the reconstructed residual image f is added to the estimated image x' to generate the reconstructed image x.

[0143] According to an embodiment, the second communication device 800 is further configured to receive a signal 530 over a second communication channel 520 from the first communication device 600. This signal 530 indicates the task associated with the input image x, the refinement deep neural network 150, the lossless decoder 132, the first lossy decoder 142, and / or the second lossy decoder 162 to the second communication device 800. The image decoder device 300 is configured to obtain the information indicated in the signal 530, and to base the generating of the reconstructed image x of input image x on this information. The hereby obtained task is used to generate the semantic based distortion map m. The hereby obtained refinement DNN is used to generate the estimated image x' as explained above, based on the semantic segmentation image s and the coarse image c provided by the hereby obtained lossless decoder 132 and first lossy decoder 142, respectively. The hereby obtained second lossy decoder 162 generates the reconstructed residual image f being added to the estimated image x' to generate the reconstructed image x. Fig. 10 shows a signaling diagram illustrating the interaction between the first communication device 600 and the second communication device 800 according to various embodiments of the invention.

[0144] In step I in Fig. 10, the image encoding device 100 of the first communication device 600 generates a semantic segmentation image s of an input image x and encodes the semantic segmentation image s to generate an encoded semantic segmentation image s', as explained above.

[0145] In step II, the image encoding device 100 of the first communication device 600 generates an encoded coarse image c , as explained above.

[0146] In step III in Fig. 10, the image encoding device 100 of the first communication device 600 generates a residual image r and encodes the residual image r to generate an encoded residual image r', as explained above.

[0147] The first communication device 600 transmits at least the encoded semantic segmentation image s' over a first communication channel 510 to the second communication device 800.

[0148] According to some embodiments, first communication device 600 transmits, in addition to the encoded semantic segmentation image s' , also the encoded coarse image c over the first communication channel 510 to the second communication device 800.

[0149] According to some embodiments, first communication device 600 transmits, in addition to the encoded semantic segmentation image s' , both of the encoded coarse image c and the encoded residual image r' over the first communication channel 510 to the second communication device 800.

[0150] According to some embodiments, the image encoding device 100 and / or the first communication device 600 determines in step IV in Fig. 10 a signal 530 indicating the task associated with the input image x, the refinement deep neural network 150, a decoder 132 corresponding to the lossless encoder 130, the first lossy decoder 142 corresponding to the first lossy encoder 140, and / or a second lossy decoder 162 corresponding to the second lossy encoder 160. The first communication device 600 then transmits the signal 530 over a second communication channel 520 to the second communication device 800.

[0151] In step V, the second communication device 800 receives the encoded semantic segmentation image s' of the input image x over the first communication channel 510 from the first communication device 600, and the image decoding device 300 generates a reconstructed image x of the input image x based on the received encoded semantic segmentation image s', as explained above.

[0152] According to an embodiment, the second communication device 800 receives in step V, in addition to the encoded semantic segmentation image s', also the encoded coarse image c . The image decoding device then generates the reconstructed image x of the input image x based on both the received encoded semantic segmentation image s' and the received encoded coarse image c , as explained above.

[0153] According to an embodiment, the second communication device 800 receives in step V, in addition to the encoded semantic segmentation image s' , both the encoded coarse image c and the encoded residual image r'. The image decoding device then generates the reconstructed image x of the input image x based on all of the received encoded semantic segmentation image s' , the received encoded coarse image c , and the received encoded residual image r', as explained above.

[0154] According to some embodiments, the second communication device 800 in step V further receives the signal 530 indicating the task associated with the input image x, the refinement deep neural network 150, a decoder 132 corresponding to the lossless encoder 130, the first lossy decoder 142 corresponding to the first lossy encoder 140, and / or a second lossy decoder 162 corresponding to the second lossy encoder 160. The image decoding device then generates the reconstructed image x of the input image x based the information provided by the signal 530.

[0155] The herein presented image encoding device 100 and image decoding device 300 aim at defining the architecture of an efficient and adaptive semantic-based and task-dependent source encoding and decoding scheme as part of first communication device 600 and a second communication device 800 of a communication system 500, in order to reduce the communication payload of the communication system 500. In particular, the source coding of the image encoding device 100 and the image decoding device 300 comprise combinations of different artificial intelligence based sub-blocks, e.g. various forms of neural networks, and classical compression / decompression and lossless / lossy encoding / decoding sub-blocks, in order to achieve a semantic-based and task-dependent compression / encoding and reconstruction / decoding scheme for images and videos.

[0156] The herein presented image encoding device 100 and image decoding device 300 could be utilized in essentially all potential products that comprise an image / video source coding block. The first communication device 600 may be e.g. a user terminal, a vehicle, a robot, a machine in general, or even a base station, where the source of information to be encoded is an image or a video produced by a camera or being stored in a server. The second communication device 800, which may be e.g. a user terminal, a vehicle, a robot, a machine in general, or even a base station, comprises the image decoding device 300, which decodes the compressed images / videos and maps, such that it can synthesize the original image / video or just take a specific action.

[0157] Consequently, the herein presented image encoding device 100 and image decoding device 300 can be placed either at a terminal side or at a base station side. However, the image encoding device 100 and the image decoding device 300 need to share some common knowledge, e.g. the task associated with the input image x and the refinement DNN, through an additional knowledge sharing channel in order to be able to reconstruct the original image / video. By sharing this common knowledge, the image encoding device 100 and an image decoding device 300 are fully compatible. The semantic-based nature of the coding scheme results in a certain level of performance / complexity and visually unique representations of the original image / video.

[0158] The image encoding device 100 and image decoding device 300 provide the possibility of incorporating semantic-based distortion measures into the source encoding, and in particular to incorporate heat-map distortion into the encoding and decoding processes. Hereby, the encoding and decoding processes are focused on particular areas of interest in the images, which could e.g. be as areas of interest in images captured by e.g., surveillance cameras, traffic cameras, or augmented / virtual reality (AR / VR) applications. The image encoding device 100 and the image decoding device 300 also make it possible to incorporate standard lossy image and video coding into the source encoding and decoding, such that the schemes are compatible with any standard player application. The image encoding device 100 and the image decoding device 300 also make it possible to adapt the source encoding by sweeping over a large range of pairs of rate and distortion, via the optimization of the compression of the encoded coarse image c .

[0159] As non-limiting examples, a number of possible applications for the image encoding device 100 and the image decoding device 300 are hereafter mentioned. One such example is monitoring behavior for Al-based self-driving cars. It is for this application important not to distort pixels related to the labels that are crucial for the self-driving car applications, such as e.g., cars, traffic light, traffic signs, people etc., while the rest of the scene e.g., trees, sky, buildings, etc., can be partially synthesized by a neural network. The image encoding device 100 will here use a suitable semantic based distortion map m, which is generated based on the specific task, to mask out anything but these crucial labels, and will then encode the regions of these crucial labels with the encoded residual image r' at a relatively high rate. The rest of the scene, i.e. the less important labels, are synthesized by the neural network and the compact version of the image, i.e. by the encoded coarse image c . As a result, a considerable reduction in rate is achieved using the image encoding device 100 and the image decoding device 300.

[0160] Another possible application for the image encoding device 100 and the image decoding device 300 is video surveillance at a public place for security reasons. Here, a set of labels of interest, being related to the semantic based distortion map m which is generated based on the specific task, may be defined, e.g., people, animals, bags etc. The image encoding device 100 should then transmit the pixels related to the labels of interest with high rate to the security center, while the rest of the scene can be synthesized by the neural network. In this case, the sequence of encoded semantic segmentation images s' can be obtained using an off-the shelf semantic segmentation algorithm encoded with the FLIF algorithm. The compressed version on the video, i.e. the sequence of encoded coarse images c', and the encoding of the residual video, i.e. the sequence of encoded residual images r', may be obtained using the H.265 algorithm.

[0161] Another possible application for the image encoding device 100 and the image decoding device 300 is augmented / virtual reality or metaverse applications. In such applications, it is important not to distort pixels related to the semantic segmentation image s and also to, related to the semantic based distortion map m which is generated based on the specific task, focus on certain specific areas, here being the field of view of the user. The image encoding device 100 should then transmit the pixels related to the labels and the area of interest with high rate by encoding them with high resolution, while the rest of the scene can be synthesized by the neural network. The utilization of the image encoding device 100 and the image decoding device 300 can achieve great reduction in rate, as explained above for other embodiments.

[0162] Thus, the image encoding device 100 and the image decoding device 300 refer to basic components of a transmitter-receiver communication system, where semantic image / video source coding is utilized. They may generally be applied to any image / video transmission related application, such as above mentioned self-driving cars, surveillance or AR / VR applications, of any communication system / network, either wireless or wired, where semantic fidelity is of main interest. The image encoding device 100 and the image decoding device 300 can then provide a reduction of the communication payload depending on the specific goal. The image encoding device 100 provides an object based or area of interest based encoding, which is reconstructed by the image decoding device 300, such that actions can be taken at the second communication device 800.

[0163] Semantic coding has been proposed as a potential technology for 6G networks. Also, wifi or other fixed networks can utilize it, as well as in-car or in-factory local network for local decision making. Finally, this image / video source coding scheme can be used as a semantic compression algorithm to store compressed versions of images / videos focusing on specific classes of interest, while a semantic viewer / player could reconstruct the image / video using the original areas of interest and synthetic generated parts.

[0164] Furthermore, any method according to embodiments of the invention may be implemented in a computer program, having code means, which when run by processing means causes the processing means to execute the steps of the method. The computer program is included in a computer readable medium of a computer program product. The computer readable medium may comprise essentially any memory, such as previously mentioned a read-only memory (ROM), a programmable read-only memory (PROM), an erasable PROM (EPROM), a flash memory, an electrically erasable PROM (EEPROM), or a hard disk drive. Moreover, it should be realized that the image encoding device 100 and the image decoding device 300 comprise the necessary communication capabilities in the form of e.g., functions, means, units, elements, etc., for performing or implementing embodiments of the invention. Embodiments of other such means, units, elements and functions are: processors, memory, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selecting units, switches, interleavers, de-interleavers, modulators, demodulators, inputs, outputs, antennas, amplifiers, receiver units, transmitter units, DSPs, MSDs, TCM encoder, TCM decoder, power supply units, power feeders, communication interfaces, communication protocols, etc. which are suitably arranged together for performing the solution.

[0165] Therefore, the processor(s) of the imager encoding device 100 and the imager decoding device 300 may comprise, e.g., one or more instances of a central processing unit (CPU), a processing unit, a processing circuit, a processor, an application specific integrated circuit (ASIC), a microprocessor, or other processing logic that may interpret and execute instructions. The expression “processor” may thus represent a processing circuitry comprising a plurality of processing circuits, such as e.g., any, some or all of the ones mentioned above. The processing circuitry may further perform data processing functions for inputting, outputting, and processing of data comprising data buffering and device control functions, such as call processing control, user interface control, or the like.

[0166] Finally, it should be understood that the invention is not limited to the embodiments described above, but also relates to and incorporates all embodiments within the scope of the appended independent claims.

Claims

CLAIMS1. An image encoding device (100) configured to: generate a semantic segmentation image s of an input image x based on a semantic segmentation deep neural network, SS DNN, (120); encode the semantic segmentation image s based on a lossless encoder (130) to generate an encoded semantic segmentation image s'; encode the input image x based on a first lossy encoder (140), a corresponding first lossy decoder (142) and a semantic based distortion map m to generate an encoded coarse image c and a coarse image c; generate an estimated image x of the input image x based on the semantic segmentation image s, the coarse image c and the semantic based distortion map m in a refinement deep neural network, refinement DNN, (150); generate a residual image r based on the estimated image x' and the input image x; and encode the residual image r based on a second lossy encoder (160) to generate an encoded residual image r' .

2. The image encoding device (100) according to claim 1, configured to: store the encoded semantic segmentation image s'.

3. The image encoding device (100) according to claim 2, configured to: store the encoded coarse image c .

4. The image encoding (100) according to claim 3, configured to: store the encoded residual image r' .

5. The image encoding device (100) according to any one of the preceding claims, wherein the semantic based distortion map m is determined based on a task associated with the input image x and the semantic segmentation image s.

6. The image encoding device (100) according to claim 5, wherein the task comprises a set of labels and / or a sub-image of interest of the input image x.

7. The image encoding device (100) according to claim 5 or 6, wherein the semantic based distortion map m comprises a heat-map distortion of the input image x.

8. The image encoding device (100) according to any one of the preceding claims, wherein the residual image r is a difference image between the estimated image x' and the input image x.

9. The image encoding device (100) according to any one of the preceding claims, wherein the input image x is for a video application.

10. A first communication device (600) for a communication system (500), the first communication device (600) comprising an image encoding device (100) according to any one of the preceding claims, the first communication device (600) being configured to: transmit the encoded semantic segmentation image s' over a first communication channel (510) to a second communication device (800).

11. The first communication device (600) according to claim 10, configured to: transmit the encoded coarse image c to the second communication device (800).

12. The first communication device (600) according to claim 11, configured to: transmit the encoded residual image r' to the second communication device (800).

13. The first communication device (600) according to any one of claims 10 to 12, wherein the first lossy encoder (140) has a first code rate Rcand the second lossy encoder (160) has a second code rate Rr, and wherein the image encoding device (100) is configured to: determine a sum of the first code rate Rcand the second code rate Rr.

14. The first communication device (600) according to claim 13, wherein the sum of the first code rate Rcand the second code rate Rris determined based on one or more of: an image target distortion for the input image x, an estimate of the first communication channel (510), available channel resources of the first communication channel (510), and an image requirement set by an application associated with the input image x.

15. The first communication device (600) according to any one of claims 10 to 14, configured to: transmit a signal (530) over a second communication channel (520) to the second communication device (800), the signal (530) indicating one or more of the task associated with the input image x, the refinement DNN (150), a decoder (132) corresponding to the lossless encoder (130), the first lossy decoder (142) corresponding to the first lossy encoder (140), and a second lossy decoder (162) corresponding to the second lossy encoder (160).

16. An image decoding device (300) configured to: obtain an encoded semantic segmentation image s' of an input image x; decode the encoded semantic segmentation image s' based on a lossless decoder (132) to generate a semantic segmentation image s of the input image x; generate an estimated image x' of the input image x based on the semantic segmentation image s and a semantic based distortion map m in a refinement DNN (150); and generate a reconstructed image x of the input image x based on the estimated image x'.

17. The image decoding device (300) according to claim 16, configured to: obtain an encoded coarse image c of the input image x; decode the encoded coarse image c based on a first lossy decoder (142) to generate a coarse image c of the input image x; and generate the estimated image x of the input image x further based on the coarse image c of the input image x.

18. The image decoding device (300) according to claim 17, configured to: obtain an encoded residual image r' of the input image x ; decode the encoded residual image r' of the input image x based on a second lossy decoder (162) to generate a reconstructed residual image f of the input image x; and generate the reconstructed image x of input image x further based on the reconstructed residual image f of the input image x.

19. The image decoding device (300) according to claim 18, wherein the reconstructed image x is an addition of the estimated image x' and the reconstructed residual image f .

20. The image decoding device (300) according to any one of claims 16-19, wherein the semantic based distortion map m is determined based on a task associated with the input image x and the semantic segmentation image s.

21. The image decoding device (300) according to any one of claims 16 to 20, wherein the task comprises a set of labels and / or a sub-image of interest of the input image x.

22. The image decoding device (300) according to claim 20 or 21, wherein the semantic based distortion map m comprises a heat-map distortion of the input image x.

23. The image decoding device (300) according to any one of claims 16 to 22, wherein the input image x is for a video application.

24. A second communication device (800) for a communication system (500), the second communication device (800) comprising an image decoding device (300) according to any one of claims 16 to 23, the second communication device (800) being configured to: receive the encoded semantic segmentation image s' of the input image x over a first communication channel (510) from a first communication device (600).

25. The second communication device (800) according to claim 24, configured to: receive an encoded coarse image c of the input image x from the first communication device (100).

26. The second communication device (800) according to claim 25, configured to: receive an encoded residual image r' of the input image x from the first communication device (100).

27. The second communication device (300) according to any one of claims 24 to 26, configured to: receive a signal (530) over a second communication channel (520) from the first communication device (100), the signal (530) indicating one or more of: the task associated with the input image x, the refinement DNN (150), the lossless decoder (132), the first lossy decoder (142), and the second lossy decoder (162).

28. A method (200) for an image encoding device (100), the method (200) comprising: generating (202) a semantic segmentation image s of an input image x based on a semantic segmentation deep neural network, SS DNN, (120); encoding (204) the semantic segmentation image s based on a lossless encoder (130) to generate an encoded semantic segmentation image s'; encoding (204) the input image x based on a first lossy encoder (140), a corresponding first lossy decoder (142) and a semantic based distortion map m to generate an encoded coarse image c and a coarse image c; generating (206) an estimated image x of the input image x based on the semantic segmentation image s, the coarse image c and the semantic based distortion map m in a refinement deep neural network, refinement DNN, (150); generating (208) a residual image r based on the estimated image x' and the input image x; and encoding (210) the residual image r based on a second lossy encoder (160) to generate an encoded residual image r'.

29. A method (400) for an image decoding device (300), the method (400) comprising: obtaining (402) an encoded semantic segmentation image s' of an input image x; decoding (404) the encoded semantic segmentation image s' based on a lossless decoder (132) to generate a semantic segmentation image s of the input image x; generating (406) an estimated image x' of the input image x based on the semantic segmentation image s and a semantic based distortion map m in a refinement DNN (150); and generating (408) a reconstructed image x of the input image x based on the estimated image x'.

30. A computer program with a program code for performing a method according to any one of claims 28-29 when the computer program runs on a computer.

Images