Task-related hierarchical image and video encoder

By combining semantic segmentation-based deep neural networks and lossless and lossy encoders, task-related image and video codes are generated, solving the problems of data overload and low efficiency in traditional coding methods, and realizing efficient and accurate wireless communication.

CN121079975APending Publication Date: 2025-12-05HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380097666.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-12
Publication Date
2025-12-05

AI Technical Summary

Technical Problem

Traditional image and video source coding methods pose a risk of data overload in wireless communication systems and fail to effectively utilize the semantic content of image and video data, resulting in unnecessary bit consumption and low coding efficiency.

Method used

We employ a semantic segmentation-based deep neural network to generate semantic segmentation images, and combine lossless and lossy encoders. We utilize a fine-grained deep neural network to generate estimated and residual images, and mask irrelevant regions through semantic distortion maps to achieve task-relevant and efficient encoding.

Benefits of technology

It improves the accuracy and efficiency of image and video coding, reduces the amount of data transmitted in wireless communication systems, and provides adaptive coding rates and flexible compression ratios to adapt to different channel conditions and application requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121079975A_ABST
    Figure CN121079975A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to an image encoding apparatus and an image decoding apparatus. An encoded semantically segmented image of an input image is generated based on a semantic segmentation deep neural network (120) and a lossless encoder (130). An encoded coarse image of the input image is generated based on a first lossy encoder (140) and the semantic-based distortion map. An estimated image of the input image is generated based on a semantic segmented image in a fine deep neural network (150), a coarse image, and the semantic-based distortion map. Based on the estimated image, the input image and a second lossy encoder (160) generate an encoded residual image. Furthermore, embodiments of the invention also relate to a first communication device and a second communication device, and also to corresponding methods and computer programs.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to source coding of an input image in an image encoding device and generation of a reconstructed image of the input image in an image decoding device. Furthermore, embodiments of the present application also relate to a first communication device and a second communication device, and corresponding methods and computer programs. BACKGROUND

[0002] With the exponential growth of data produced by humans and machines, the demand for wireless data rates is increasing. Furthermore, future wireless systems, including internet-of-things (IoT) systems, autonomous vehicles, surveillance systems, augmented reality schemes and / or virtual reality schemes will produce massive amounts of wireless data, using traditional source channel coding techniques will put wireless communication systems at risk of data overload. In order to be able to handle the increasing demand for data transmission in wireless communication systems, the way information is encoded needs to be changed.

[0003] Therefore, there is a need for alternative source coding methods for image and / or video information. Such alternative source coding methods can improve the accuracy and efficiency of human to machine (H2M) and / or machine to machine (M2M) communication. Thus, image and / or video source coding methods can exploit the semantics of data, i.e. the meaning of bits to improve source coding. Machine learning (ML), in particular deep learning (DL) has proven to be able to handle a variety of intelligent tasks, such as extracting semantic information from input data, and can thus be applied when exploiting the semantics of image and / or video data to be encoded. When considering semantics in source coding, semantic fidelity is of greater importance than error rate metrics used in traditional source coding, such as traditional bit error rate or symbol error rate. SUMMARY

[0004] It is an object of embodiments of the present application to provide a solution which alleviates or solves the drawbacks and problems of traditional solutions.

[0005] It is a further object of embodiments of the present application to provide a source coding which can improve accuracy and efficiency.

[0006] The above and other objects are achieved by the subject matter of the independent claims. Further embodiments of the present application are provided by the dependent claims.

[0007] According to a first aspect of the present application, the above and other objects are achieved by an image encoding device for: Generate input image based on semantic segmentation deep neural network (SSDNN) semantic segmentation image ; The semantic segmentation image is based on a lossless encoder. Encode to generate encoded semantic segmentation images ; Based on the first lossy encoder, the corresponding first lossy decoder, and the semantically based distortion graph For the input image Encode to generate a coarse encoded image. and rough image ; The semantic image segmentation based on a refinement deep neural network (refinement DNN) The rough image and the semantic-based distortion map Generate the input image Estimated image ; Based on the estimated image and the input image Generate residual image ; The residual image is processed based on the second lossy encoder. Encode to generate encoded residual image ´.

[0008] Therefore, a semantically based and task-related source encoding and decoding method is provided. This method is based on the semantics of the image or video data to be stored or transmitted—that is, on the content / meaning of the image or video data—and is related to the specific task / objective of the image or video data. The encoding and decoding are task-, application-, and resource-related to enable semantic-based compression and reconstruction, and to facilitate semantic-based communication for future wireless networks.

[0009] The advantage of the image encoding device is that it efficiently encodes image and video data using their semantic content separately, thereby achieving high performance and sufficient perceptual quality at the image decoder to meet the needs of computer vision (CV) tasks. Compared to traditional schemes, this image encoding device improves accuracy and efficiency because it avoids unnecessary bit consumption in irrelevant regions of the input image. Semantic-based distortion map. The use of the semantic distortion map masks less interesting regions or objects in the input image and instead focuses the encoding on task-relevant or semantic-based information. Thus, the semantic distortion map is encoded based on semantics, making the source coding task-relevant, thereby improving the compression efficiency. In other words, the use of the semantic distortion map in the encoding device allows for higher compression ratios without compromising the semantic information.

[0010] Furthermore, the image encoding device is also able to trade-off between encoding rate and visual / perceptual quality. An adaptive source coding for adaptive rate transmission is provided, which is crucial for wireless communication. The image encoding device provides a flexible encoding rate that can be optimized based on variable channel conditions, applications, and / or distortion targets. Distortion targets refer to the allowed or acceptable distortion levels for the compression and reconstruction of images or videos.

[0011] The combination of AI-based blocks and classic / standard compression blocks provides an efficient and adaptive semantic-based and / or task-relevant source coding scheme that focuses on specific regions of an image or video based on a specific task and / or target. Thus, the image encoding device partly leverages standard image compression as well as lossless and lossy encoding algorithms. These well-established standard engineering compression algorithms work well, are easy to use, and have flexibility in adjusting the compression ratio, which positively impacts the overall performance of the codec. It has also been shown that the image encoding device partly uses the corresponding classic / standard decompression modules, which outperform traditional encoders in terms of performance at a range of compression rates.

[0012] By providing an efficient source coding, the image encoding device reduces the payload on the communication system that transmits the encoded images (e.g., used by various 5G applications or future 6G applications). Since a large portion of today’s data traffic on such communication systems involves the transmission of image and video data, the proposed image encoding device efficiently encodes this type of data based on its semantic content, thereby reducing the overall amount of data transmitted through the communication system.

[0013] The image encoding device comprises a fine deep neural network, which is capable of generating high resolution images. The fine deep neural network can be implemented in different ways, for example, as a conditional generative adversarial network (cGAN) or a denoising diffusion probabilistic model (DDPM) neural network. Furthermore, an off-the-shelf semantic segmentation network can be used as the semantic segmentation deep neural network. Moreover, the standard image or video codec can be a classical / standard image compression algorithm, for example, JPEG2000, or a classical video codec, for example, H.265. The first and second lossy encoders can also be standard encoders. The compression performance of the image encoding device and the reconstruction performance of the corresponding image decoding device depend on the total combination of these blocks and also on which off-the-shelf algorithm is used. Thus, the compression performance and the reconstruction performance can be adapted to a specific application.

[0014] The image encoding device reduces the training burden in comparison to a conventional deep semantic segmentation based hierarchical image compression scheme using neural networks.

[0015] The image encoding device provides source coding such that a scalable three-layer transmission is possible, wherein the transmission can be performed in a layered manner depending on the application. Thus, the transmission can comprise only the first base layer, or the first base layer and the second enhancement layer, or the first base layer, the second enhancement layer and the third fine layer. In more detail, by providing three possible layers of information, i.e. the encoded semantic segmentation image The first base image encoding layer is provided by the encoded coarse image The second enhancement / coarse image encoding layer is provided, and the third fine image encoding layer is provided by the compressed version of the residual image The three-layer image / video codec can be used to encode image / video files to reduce the encoding rate and / or to provide semantic-based and / or task-dependent source coding. This can be a practical and high-value feature for many applications, such as internet of things (IoT), H2M applications and M2M applications.

[0016] In an implementation form of the image encoding device according to the first aspect, the image encoding device is configured to store the encoded semantic segmentation image .

[0017] This implementation form has the advantage that only the encoded semantic segmentation image which can be considered as a first basic image coding layer. Such a first basic image coding layer comprises a semantic map, i.e. a semantic segmentation image The semantic map comprises the full semantic information of the image and can be used as a common basis for each application and / or user.

[0018] In an implementation form of the image encoding device according to the first aspect, the image encoding device is configured to store the encoded coarse image .

[0019] The advantage of this implementation form is that the encoded semantic segmentation image and the encoded coarse image are stored, which can be considered as a second enhancement / coarse image coding layer, i.e. a compressed image / video representation of the input image / video as an enhancement layer of the first basic layer is also provided. Such a second coarse image coding layer in combination with the first basic layer can provide an acceptable quality for some applications and / or users, e.g. in case of a coarse image reconstruction where the semantic meaning has to be preserved, the image decoding device can use this layer in combination for certain users. Thus, the first basic image coding layer can be provided by the image encoding device alone or the first basic image coding layer and the second coarse image coding layer can be provided to adapt to the current application requirements. The first basic image coding layer and the second coarse image coding layer can reconstruct to an image which is reasonable for a human observer. Thus, the generated image will appear reasonable to a human, i.e. can be understood as an image which can be perceived by a human and which conveys a certain meaning.

[0020] In an implementation form of the image encoding device according to the first aspect, the image encoding device is configured to store the encoded residual image .

[0021] The advantage of this implementation form is that the encoded semantic segmentation image , the encoded coarse image and the encoded residual image This can be seen as a third fine image coding layer (comprising a compressed version of the residual) provided as an additional enhancement layer together with the first base layer and the second coarse image coding layer. Such a fine image coding layer provides a quality required by some applications and / or users, e.g. in case of a fine image reconstruction is required, and thus can be used by image decoding devices of some users. Thus, the first base image coding layer can be provided by the image coding device alone, the first base image coding layer and the second coarse image coding layer can be provided, or the first base image coding layer, the second coarse image coding layer and the third fine image coding layer can be provided, to adapt to the application requirements of using the image coding device. The third fine image coding layer can provide additional application dependent details.

[0022] In an implementation form of the image coding device according to the first aspect, the semantic-based distortion map is determined based on a task associated with the input image and the semantic segmentation image. .

[0023] An advantage of this implementation form is that the task comprising a set of labels is used to generate the semantic-based distortion map . The semantic-based distortion map acts like a filter, which masks areas / objects of the input image that are not or less of interest and expresses task-related and semantic-based information of the input image . The semantic-based distortion map does not need to be transmitted or coded. The semantic-based distortion map can be obtained from the semantic task filter by using the properties of the task and the semantic segmentation image as input, which are known to both the image coding device and the image decoding device. By using the semantic-based distortion map , the coding rate can be adjusted in the first and second lossy encoders. The semantic-based distortion map is also used as input in the fine DNN to generate an estimated image of the input image . Since the semantic-based distortion map is associated with the task, using the semantic-based distortion map in the image coding device makes the provided source coding task-dependent.

[0024] By exploiting the semantic-based distortion map ​​to provide an adaptive semantic-based and task-dependent feature of the compression scheme, where the meaning conveyed by the image / video depends on the desired task, i.e. on the semantic-based distortion, such that the encoding focuses on relevant parts / regions of the image / video and such that irrelevant parts / regions are masked. This improves the compression performance of the image / video without affecting the semantics of the image, as the compression can be optimized according to the distortion target.

[0025] In an implementation form of the image encoding device according to the first aspect, the task comprises a set of labels and / or a sub-image of interest of the input image .

[0026] An advantage of this implementation form is that the task (i.e. the property) known by both the image encoding device and the image decoding device can comprise a set of labels and / or regions or sub-images of interest. For example, the corresponding parts of the image indicated by the task can be encoded with high resolution. As a non-limiting example, in an autonomous car application, the task can be the car, the traffic light, the traffic sign, the pedestrian or other key properties. In an autonomous car, it is important not to distort the key pixels related to the task / property / label, e.g. the car, the traffic light, the traffic sign, the pedestrian, etc., while other parts of the scene, e.g. trees, sky, buildings, etc., can be synthesized by the neural network part. The image encoding device will encode the parts / regions of the residual image corresponding to those key labels at a relatively high rate by exploiting these tasks. The rest of the scene, i.e. the less critical parts / regions, are masked and synthesized by the neural network and the compressed version of the image. Thus, a task-dependent and efficient source coding is provided.

[0027] In an implementation form of the image encoding device according to the first aspect, the semantic-based distortion map comprises a heat map distortion of the input image .

[0028] An advantage of this implementation form is that it provides a semantic-based distortion measure for the input image . This task-dependent distortion measure focuses on specific regions of the image or video based on a specific task / target and makes the encoding more efficient.

[0029] In an implementation form of the image encoding device according to the first aspect, the residual image is a difference image between the estimated image and the input image .

[0030] An advantage of this implementation form is that the residual image providing additional task-dependent and application-dependent details, i.e. augmenting the labels / attention regions of the second coarse layer and the first base layer, resulting in visually appealing representations of the original image and achieving specific target distortions. The residual image The sum of the encoding rate of the residual image The sum of the encoding rate of the coarse image

[0031] In one implementation of the image encoding device according to the first aspect, the input image is used for video applications.

[0032] An advantage of this implementation is that the image encoding device can be used for a large number of video applications, such as autonomous driving cars, video surveillance, virtual or augmented reality applications. Video is a sequence of images (i.e. a sequence of frames), which can be processed frame-wise to generate corresponding sequences of semantic segmentation images estimation images and residual images which are then converted into a video file. Standard video codecs can be used to generate the encoded coarse video / images and to provide encoded semantic segmentation video / images and residual video / images ´, resulting in task-dependent video compression.

[0033] According to a second aspect of the present application, the above and other objects are achieved by a first communication device for a communication system, the first communication device comprising the image encoding device described herein, the first communication device being configured to: transmit the encoded semantic segmentation image to a second communication device over a first communication channel.

[0034] The first communication device comprises the image encoding device described herein and thus provides the above-mentioned advantages of the image encoding device. In addition, the first communication device has the advantage that it can be used to transmit efficiently encoded image and video data to the second communication device, which reduces the overall payload of the communication system. Furthermore, by incorporating the semantic content of the image and video data, the task itself, and the application and resource requirements, a trade-off between encoding rate and visual / perceptual quality can be achieved. In addition, adaptive transmission of optimized rate is provided, which is particularly important for wireless communication. Furthermore, the layered three-layer encoding provided by the image encoding device provides scalable transmission depending on the application. Thus, semantic-based communication is achieved, which can be used for current and future wireless networks.

[0035] by transmitting the encoded semantic segmentation image The provision of the first base image encoding layer allows for a significant reduction of the encoding rate, i.e. the transmission of the overall semantic information of the image, while providing a semantic-based encoding.

[0036] In an implementation form of the first communication device according to the second aspect, the first communication device is configured to: transmit the encoded coarse image to the second communication device.

[0037] This implementation form has the advantage that the encoded semantic segmentation image and the encoded coarse image can be transmitted, which can be seen as the transmission of a second coarse image encoding layer in addition to the first base image encoding layer, i.e. a compressed image representation of the input image. Such a second coarse image encoding layer provides an acceptable quality if a coarse reconstruction of the image is required. Either the first base image encoding layer can be transmitted only, or the first base image encoding layer and the second coarse image encoding layer can be transmitted, in particular based on the application.

[0038] In an implementation form of the first communication device according to the second aspect, the first communication device is configured to: transmit the encoded residual image to the second communication device.

[0039] This implementation form has the advantage that the encoded semantic segmentation image , the encoded coarse image and the encoded residual image can be transmitted to the second communication device, which can be seen as the transmission of a third fine image encoding layer (including a compressed version of the residual) in addition to the first base image encoding layer and the second coarse image encoding layer. Such a fine image encoding layer provides the required quality, e.g. additional application-dependent details, if a fine image reconstruction is required at the second communication device. Either the first base image encoding layer can be transmitted alone, the first base image encoding layer and the second coarse image encoding layer can be transmitted, or the first base image encoding layer, the second coarse image encoding layer and the third fine image encoding layer can be transmitted, in particular based on the application.

[0040] In an implementation form of the first communication device according to the second aspect, the first lossy encoder has a first encoding rate , the second lossy encoder has a second encoding rate , wherein the image encoding device is configured to: determining the first encoding rate and the second encoding rate sum.

[0041] An advantage of this implementation is that it provides the possibility to trade-off between the first encoding rate and the second encoding rate and enables searching over a wide range of rate-distortion pairs. Thus, adaptability and flexibility of encoding and decoding of images or videos, as well as trade-off between encoding rate and visual / perceptual quality, is provided. This is especially important for wireless communication systems with different channel conditions.

[0042] In an implementation form of the first communication device according to the second aspect, the first encoding rate and the second encoding rate are determined based on one or more of the following: an image target distortion of the input image , an estimation of the first communication channel, available channel resources of the first communication channel, and image requirements set by an application associated with the input image .

[0043] Thus, searching over a wide range of rate-distortion pairs is possible, which provides adaptability and flexibility that is especially important for wireless communication systems.

[0044] In an implementation form of the first communication device according to the second aspect, the first communication device is configured to: transmit a signal to the second communication device over a second communication channel, the signal indicating one or more of the following: the task associated with the input image , the fine DNN, a decoder corresponding to the lossless encoder, the first lossy decoder corresponding to the first lossy encoder, a second lossy decoder corresponding to the second lossy encoder.

[0045] An advantage of this implementation is that various useful information is provided to the image decoding device in the second communication device over the second communication channel (which can be a control / knowledge sharing channel different from the first communication channel), which is used for efficiently generating the reconstructed image of the input image .

[0046] According to a third aspect of the present invention, the above mentioned and other objectives are achieved by an image decoding device configured to: obtain an encoded semantic segmentation image of an input image ; based on the lossless decoder on the encoded semantic segmentation image decoded to generate the input image based on the semantic segmentation image ; based on the semantic segmentation image and a semantic-based distortion map in a fine DNN to generate an estimated image of the input image ; based on the estimated image to generate a reconstructed image of the input image . An advantage of this implementation is that the image decoding device is provided with knowledge of the semantic segmentation image and also with knowledge of the task properties, so that the image decoding device can generate the semantic-based distortion map

[0047] . The semantic-based distortion map should be known to both the image encoding device and the image decoding device. Thus, an estimated image of the input image can be generated based on the semantic segmentation image provided by the first base image encoding layer and the semantic-based distortion map in a fine deep neural network . The image decoding device efficiently decodes image and video data using semantic content of the image and video data, respectively, with the goal to achieve high performance and sufficient perceptual quality for CV tasks. Furthermore, the image decoding device provides a trade-off between encoding rate and visual / perceptual quality.

[0048] The combination of AI-based blocks and classic / standard decompression blocks provides an efficient and adaptive semantic-based and / or task-dependent image decoding scheme that focuses on specific regions of the image or video based on the specific task and / or goal. The image decoding device comprises a fine deep neural network that is capable of generating high-resolution images, e.g., the fine deep neural network can be implemented as a conditional generative adversarial network (cGAN) or a denoising diffusion probabilistic model (DDPM) neural network.

[0049] The combination of AI-based blocks and classic / standard decompression blocks provides an efficient and adaptive semantic-based and / or task-dependent image decoding scheme that focuses on specific regions of the image or video based on the specific task and / or goal. The image decoding device comprises a fine deep neural network that is capable of generating high-resolution images, e.g., the fine deep neural network can be implemented as a conditional generative adversarial network (cGAN) or a denoising diffusion probabilistic model (DDPM) neural network.

[0050] ​​In one implementation of the image decoding device according to the third aspect, the image decoding device is used for: Obtain the input image Coarse image after encoding ; Based on the first lossy decoder, the encoded coarse image Decode to generate the input image rough image ; Further based on the input image The rough image Generate the input image The estimated image .

[0051] By obtaining the encoded semantic segmentation image from the first basic image coding layer and the encoded coarse image of the second coarse image encoding layer Provide the semantic segmentation image to the image decoding device. and the rough image The image decoding device can then synthesize the estimated missing details to provide the input image. Enhanced estimated image .

[0052] In one implementation of the image decoding device according to the third aspect, the image decoding device is used for: Obtain the input image Encoded residual image ; The input image is processed based on the second lossy decoder. The encoded residual image Decode to generate the input image The reconstructed residual image ; Further based on the input image The reconstructed residual image Generate the input image The reconstructed image .

[0053] By obtaining the encoded semantic segmentation image from the first basic image coding layer The encoded coarse image of the second coarse image encoding layer and the encoded residual image of the third fine image coding layer The image decoding device can utilize the semantic segmentation image , the coarse image and the reconstructed residual image to generate a more visually appealing representation of the original image.

[0054] In an implementation form of the image decoding device according to the third aspect, the reconstructed image is an addition of the estimated image and the reconstructed residual image .

[0055] By adding the reconstructed residual image to the estimated image , important parts / regions / labels encoded at a high rate in the image encoding device are included in the reconstructed image . As mentioned above, by masking out less important parts / regions / labels of an image / video, e.g. sky, trees, etc., using the semantic based distortion map , they are then synthesized by a neural network into a compact low rate version of the image. However, more important parts / regions / labels, e.g. pedestrians, cars, etc., are encoded at a relatively higher rate in the encoded residual image ´.

[0056] In an implementation form of the image decoding device according to the third aspect, the semantic based distortion map is determined based on a task associated with the input image and the semantic segmentation image .

[0057] As mentioned above, the task is used to generate the semantic based distortion map . Thus, areas / objects of the input image that are not or less worth attention can be masked out, while task related / semantic based information is expressed. Thus, depending on the desired task, i.e. depending on the semantic based distortion map , the actual content / meaning of the image / video is utilized such that the encoding is focused on relevant parts / regions of the image / video, while less attention is paid to irrelevant parts / regions. This improves the image / video compression efficiency, while the compression can be optimized according to the distortion target.

[0058] In an implementation form of the image decoding device according to the third aspect, the task comprises a set of labels and / or a sub-image of interest of the input image .

[0059] The advantage of this implementation is that the task (i.e., attribute) known to both the image encoding device and the image decoding device can include a set of labels and / or regions or sub-images of interest, such that the corresponding portion of the image indicated by the task can be encoded at high resolution. As a non-limiting example, the task could be other attributes in applications involving cars, traffic lights, traffic signs, pedestrians, or autonomous vehicles.

[0060] In one implementation of the image decoding device according to the third aspect, the semantically based distortion image Including the input image The heat map is distorted.

[0061] The advantage of this implementation is that it provides the input image... It provides a semantic-based distortion metric. This task-related distortion metric is based on a specific task / objective and focuses on a specific region of an image or video.

[0062] In one implementation of the image decoding device according to the third aspect, the input image For video applications.

[0063] Therefore, the image decoding device can be used in a wide range of video applications, such as autonomous vehicles, video surveillance, or virtual reality applications. Video is a sequence of images (i.e., a sequence of frames), which can be processed frame by frame to generate corresponding semantically segmented images. Sequence, estimated image Sequence and / or residual images The sequences are then converted into video files. Standard video codecs can be used to decode the first basic video / image coding layer, the second coarse video / image coding layer, and / or the third fine video / image coding layer input to the video / image decoding device.

[0064] According to a fourth aspect of the invention, the above and other objectives are achieved using a second communication device for a communication system, the second communication device including the image decoding device described herein, the second communication device being used for: The input image is received from the first communication device via the first communication channel. The encoded semantic segmentation image .

[0065] By receiving the transmitted encoded semantic segmentation image The semantic segmentation image is provided to the image decoding device. The image decoding device also possesses knowledge of task attributes, enabling it to generate the semantically-based distortion image. The semantic segmentation image provided by the first base image encoding layer can thus be used to generate an estimated image of the input image and the semantic-based distortion map in the fine depth neural network The input image The estimated image .

[0066] The task attribute is thus known to both the image encoding device of the first communication device and the image decoding device of the second communication device and can be synchronized over a second communication channel, possibly a control / knowledge sharing channel different from the first communication channel. The semantic-based distortion map can thus be generated by the image encoding device and / or the image decoding device, such that an estimated image of the input image and the semantic-based distortion map can be generated based on the semantic segmentation image .

[0067] In an implementation form of the second communication device according to the fourth aspect, the second communication device is configured to receive from the first communication device an encoded coarse image of the input image .

[0068] By receiving the transmitted encoded semantic segmentation image of the first base image encoding layer and the encoded coarse image of the second coarse image encoding layer the image decoding device is provided with knowledge of the semantic segmentation image and the coarse image . The image decoding device can then synthesize the missing details of the estimated image, thereby providing an enhanced estimated image of the input image .

[0069] In an implementation form of the second communication device according to the fourth aspect, the second communication device is configured to receive from the first communication device an encoded residual image of the input image .

[0070] By receiving the transmitted encoded semantic segmentation image of the first base image encoding layer , the encoded coarse image of the second coarse image encoding layer and the encoded residual image of the third fine image encoding layer ​The image decoding device can utilize the semantic segmentation image , the coarse image , and the knowledge of the reconstructed residual image to generate a more visually appealing representation of the original image.

[0071] In an implementation form of the second communication device according to the fourth aspect, the second communication device is configured to receive a signal from the first communication device over a second communication channel, the signal indicating one or more of: the task associated with the input image , the fine DNN, the lossless decoder, the first lossy decoder, and the second lossy decoder.

[0072] This implementation form has the advantage that the second communication device is provided with various useful information for the image decoding device in the second communication device to efficiently generate a reconstructed image of the input image over the second communication channel.

[0073] According to a fifth aspect of the present invention, the above mentioned and other objectives are achieved by a method for an image encoding device, the method comprising: generating a semantic segmentation image of an input image based on a semantic segmentation deep neural network (SS DNN) (120); encoding the semantic segmentation image based on a lossless encoder to generate an encoded semantic segmentation image ; encoding the input image based on a first lossy encoder, a corresponding first lossy decoder, and a semantic based distortion map to generate an encoded coarse image and a coarse image ; generating an estimated image of the input image based on the semantic segmentation image , the coarse image , and the semantic based distortion map ; generating a residual image based on the estimated image and the input image . ; encoding the residual image based on the second lossy encoder to generate an encoded residual image

[0074] The method according to the fifth aspect can be extended to an implementation corresponding to the implementation of the image encoding device according to the first aspect. Thus, one implementation of the method comprises one or more features of the corresponding implementation of the image encoding device.

[0075] The advantages of the method according to the fifth aspect are the same as the advantages of the corresponding implementation of the image encoding device according to the first aspect.

[0076] According to a sixth aspect of the present application, the above mentioned and other objects are achieved by a method for an image decoding device, the method comprising: obtaining an input image ; decoding the encoded semantic segmentation image based on a lossless decoder to generate a semantic segmentation image of the input image ; ; generating an estimated image of the input image based on the semantic segmentation image and a semantic-based distortion map in a fine DNN generating a reconstructed image of the input image based on the estimated image .

[0077] The method according to the sixth aspect can be extended to an implementation corresponding to the implementation of the image decoding device according to the second aspect. Thus, one implementation of the method comprises one or more features of the corresponding implementation of the image decoding device.

[0078] The advantages of the method according to the sixth aspect are the same as the advantages of the corresponding implementation of the image decoding device according to the second aspect.

[0079] ​​​​​​​​​Embodiments of the present application also relate to a computer program, characterized in that program code which, when executed by at least one processor, causes the at least one processor to perform any of the methods provided by embodiments of the present application. Embodiments of the present application also relate to a computer program product including a computer-readable medium and the computer program embodied in said computer-readable medium, and one or more of the following: a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), a flash memory, an electrically erasable PROM (EEPROM), a hard disk drive, etc.

[0080] Other applications and advantages of embodiments of the present application will be readily apparent to those skilled in the art from the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0081] The accompanying drawings, which are included to provide a further understanding of the various embodiments of the application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. Figure 1 An image encoding apparatus according to an embodiment of the present application is shown; Figure 2 A flowchart of a method for an image encoding apparatus according to an embodiment of the present application is shown; Figure 3 A first communication apparatus according to an embodiment of the present application is shown; Figure 4 An image decoding apparatus according to an embodiment of the present application is shown; Figure 5 A flowchart of a method for an image decoding apparatus according to an embodiment of the present application is shown; Figure 6 A first communication apparatus according to an embodiment of the present application is shown; Figure 7 A communication system according to an embodiment of the present application is shown; Figure 8 An image encoding apparatus according to an embodiment of the present application is shown; Figure 9 An image decoding apparatus according to an embodiment of the present application is shown; Figure 10 A signaling diagram according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0082] It is often desirable to reduce the payload of various wireless applications by using enhanced encoding methods to reduce the amount of data transmitted through a communication system, and in particular, a wireless communication system. As a significant portion of the data transmitted involves the transmission of image and / or video data, an accurate and efficient encoding scheme will reduce the total amount of data transmitted by a communication system that transmits image and / or video data.

[0083] A previously known deep semantic segmentation-based layered image compression (DSSLIC) scheme utilizes the semantic content of image data. The known DSSLIC scheme provides layered image coding. The DSSLIC encoder outputs three layers of information: a semantic segmentation map layer, a residual layer, and a compact layer. The encoder uses three different neural networks (NNs), namely SegNet, CompNet, and FineNet, for generating the semantic segmentation map, the compact representation, and the fine details of the image, respectively. The semantic segmentation network is used as SegNet to generate the semantic segmentation map. Convolutional neural networks (CNNs) and generative adversarial networks (GANs) are used for CompNet and FineNet, respectively, and the two networks are jointly trained. To encode the three layers of information, classical lossless and lossy encoders are used in the DSSLIC encoder. Specifically, the semantic segmentation map and the compact image are encoded using a free lossless image format (FLIF) encoder, while the residual is encoded using a better portable graphic (BPG) encoder. The known DSSLIC encoding scheme is reported to outperform traditional encoding schemes, such as joint picture expert group (JPEG), JPEG200, BPG, and image format for the web (WebP), on most widely considered datasets, such as ADE20K, Kodak, and CityScapes.

[0084] However, the known DSSLIC encoding scheme has some drawbacks in terms of its performance as it provides sub-optimal accuracy and efficiency. Furthermore, in certain cases, the DSSLIC encoding scheme unnecessarily spends bits to encode irrelevant regions of the input image and also has a fixed output quality regardless of varying channel conditions. The known DSSLIC encoding scheme further burdens due to extensive training of convolutional neural networks.

[0085] Accordingly, it is proposed herein a scheme that mitigates or completely addresses the drawbacks of the conventional schemes. A general aspect of the scheme is to provide an enhanced image and / or video encoding scheme that leverages the semantic content of the image and / or video data to be encoded, which provides a more efficient and more accurate source coding.

[0086] Figure 1 A block scheme of an image encoding device 100 for encoding an input image according to some embodiments described herein is shown. Such an image encoding device 100 can be referred to as a task-related layered image and video (LIV) encoder. The image encoding device 100 is configured to generate a semantic segmentation image of the input image based on a semantic segmentation deep neural network (SSDNN) 120. The image encoding device 100 is further configured to encode the semantic segmentation image based on a lossless encoder 130 to generate an encoded semantic segmentation image . In this context, semantic segmentation refers to a deep learning algorithm, i.e. a deep neural network, that associates a label or class with each pixel in an image.

[0087] The image encoding device 100 is further configured to encode the input image based on a first lossy encoder 140, a corresponding first lossy decoder 142 and a semantic-based distortion map to generate an encoded coarse image and a coarse image . Accordingly, the first lossy encoder 140 generates the encoded coarse image based on the input image and the semantic-based distortion map and the first lossy decoder 142 generates the coarse image based on the encoded coarse image . The semantic-based distortion map may be provided to the first lossy encoder 140 by a semantic task filter 152.

[0088] Image encoding device 100 is also used for semantic segmentation of images in a fine-grained deep neural network (DNN) 150. Rough image and semantically based distortion graphs Generate input image Estimated image Provides semantic segmentation images from the semantic segmentation deep neural network 120 to the fine-grained deep neural network 150. A rough image from the first lossy decoder 142 and semantically based distortion graphs from semantic task filter 152. The fine-grained deep neural network 150 can be any block / network / algorithm, such as a conditional generative adversarial network (cGAN) or a denoising diffusion probabilistic network (DDPM) neural network, capable of processing coarse images. (i.e., from its low-resolution compressed version), semantic segmentation image and semantically based distortion graphs Reconstructed estimated high-resolution image Therefore, a fine-grained deep neural network 150 generates images with fine details, which are then added to a coarse image. To generate the input image Estimated image A sophisticated deep neural network 150 directly generates the input image. Estimated image .

[0089] Image encoding device 100 is also used for image estimation and input image Generate residual image For example, the input image is processed through the difference unit 154. From the estimated image Subtract from the middle. This provides the difference unit 154 with the estimated image from the fine-grained deep neural network 150. The image encoding device 100 also provides an input image from input 158. The image encoding device 100 is further used to process the residual image based on the second lossy encoder 160. Encode to generate encoded residual image The residual image from the differential unit 154 is provided to the second lossy encoder 160. .

[0090] According to various embodiments, the image encoding device 100 can also be used to store encoded semantic segmentation images. And in addition to the encoded semantic segmentation image In addition, it is also possible to separately encode the coarse image. Stored or to be encoded coarse images and encoded residual image All of them are stored in the encoding memory 180, as described below.

[0091] Therefore, the image encoding device outputs three layers of information. The encoded semantic segmentation image. The provided first basic image encoding layer can also be described as a semantic segmentation layer. (Additional encoding of the coarse image) The provided second coarse image encoding layer can also be described as the input image. A compact / compressed layer. An additional compressed version of the image / video data of the residual image. The provided third fine image coding layer can also be described as a residual layer.

[0092] Figure 2 It shows that it can be encoded by an image encoding device 100 (e.g. Figure 1 The flowchart of the corresponding method 200 executed (shown) indicates that the image encoding device 100 may be in the first communication device 600 (e.g., Figure 3 Implemented as shown. It should be noted that... Figure 2 The methods and steps shown and described in this article do not necessarily have to be followed. Figure 2 The steps are executed in the order shown. These steps can be executed in virtually any suitable order, or in parallel, provided that the physical requirements and the information needed to execute each step are available at the time of execution.

[0093] Method 200 includes: generating 202 input images based on a semantic segmentation deep neural network 120. semantic segmentation image Method 200 also includes: semantic segmentation of the image based on the lossless encoder 130. Encode 204 to generate encoded semantic segmentation image Method 200 further includes: a first lossy encoder 140, a corresponding first lossy decoder 142, and a semantically based distortion map. For the input image Encode 206 to generate a coarse encoded image. and rough image Method 200 also includes: semantic image segmentation based on a fine-grained deep neural network 150. Rough image and a semantic-based distortion map generating 208 an estimated image of the input image . The method 200 further comprises generating 210 a residual image based on the estimated image and the input image . The method 200 further comprises encoding 212 the residual image based on the second lossy encoder 160 to generate an encoded residual image ´.

[0094] The image encoding device 100 and the corresponding method 200 encode the input image based on a semantic-based distortion map , which makes the encoding scheme task-dependent, i.e. related to the task input. The semantic-based distortion map defines regions / objects of interest and trade-offs between visual quality and encoding rate, which influences the compression ratio. Thus, unnecessary bit encoding of regions / objects and / or specific regions of the input image, which are not relevant for the specific task and application, is avoided. It should be noted that the image encoding device 100 and the corresponding method 200 differ from the known DSSLIC scheme described above in that a semantic-based distortion map is used. Thus, the image encoding device 100 and the corresponding method 200 utilize the task input, i.e. the semantic-based distortion map , to provide a task-dependent encoding scheme, whereas the DSSLIC scheme is not task-dependent, i.e. does not encode based on any parameters corresponding to the semantic-based distortion map .

[0095] The image encoding device 100 and the corresponding method 200 utilize a combination of AI and standard / classic compression blocks to provide better compression results and faster block training. Thereby, the burden of training large neural networks, e.g. convolutional neural networks as in the known DSSLIC encoding scheme, is avoided.

[0096] The image encoding device 100 and the corresponding method 200 provide an adaptive encoding, i.e. the encoding is optimized according to channel conditions, requirements, tasks and / or applications, etc. The image encoding device 100 and the corresponding method 200 also provide scalability by hierarchically storing and / or transmitting the first image encoding layer, the second image encoding layer or the third image encoding layer according to requirements and an image decoding device 300. Thus, the output quality can be adjusted based on varying channel conditions.

[0097] Figure 3 A first communication device 600 according to an embodiment of the present application is shown. In Figure 3In the illustrated embodiment, the first communication device 600 comprises a processor 102, a transceiver 104 and a memory 106. The processor 102 is coupled to the transceiver 104 and the memory 106 by communication means 108 known in the art. The first communication device 600 further comprises a communication interface 110 coupled to the transceiver 104, which means that the first communication device 600 is configured to communicate in the communication system 500, e.g. with the second communication device 800, as illustrated in Figure 7 The image encoding device 100 described herein can be implemented in the first communication device 600, more precisely in the processor 102 of the first communication device 600.

[0098] The processor 102 can be referred to as one or more general-purpose CPUs, one or more digital signal processors (DSP), one or more application-specific integrated circuits (ASIC), one or more field programmable gate arrays (FPGA), one or more programmable logic devices, one or more discrete gates, one or more transistor logic devices, one or more discrete hardware components, one or more chipsets. The memory 106 can be a read-only memory, a random access memory (RAM) or a non-volatile RAM (NVRAM). The transceiver 104 can be a transceiver circuit, a power controller or an interface providing the ability to communicate with other communication modules or communication devices. The transceiver 104, the memory 106 and / or the processor 102 can be implemented in separate chipsets or in a common chipset. In the present disclosure, it can be understood that the first communication device 600 is configured to perform certain operations means that the first communication device 600 comprises suitable means for performing these operations, e.g. the processor 102 and the transceiver 104.

[0099] According to Figure 3 In the illustrated embodiment, the first communication device 100 comprises the image encoding device 100 described herein. As described above, the image encoding device 100 is configured to generate a semantic segmentation image of an input image based on the semantic segmentation deep neural network 120. The image encoding device 100 is further configured to encode the semantic segmentation image based on the lossless encoder 130 to generate an encoded semantic segmentation image .

[0100] The image encoding device 100 is further configured to encode the input image based on the first lossy encoder 140, the corresponding first lossy decoder 142 and the semantic-based distortion map to generate a coarse image and the coarse image . The image encoding device 100 is further configured to generate an estimated image of the input image based on the encoded semantic segmentation image in the fine depth neural network 150, the coarse image and the semantic-based distortion map . The image encoding device 100 is further configured to generate a residual image based on the estimated image and the input image .

[0101] The image encoding device 100 is further configured to encode the residual image based on the second lossy encoder 160 to generate an encoded residual image . Thus, the image encoding device 100 is configured to provide the encoded semantic segmentation image , the encoded coarse image and the encoded residual image and can be configured to store the encoded semantic segmentation image and, in addition to the encoded semantic segmentation image , to store the encoded coarse image separately in the encoding memory 180 or to store the encoded coarse image and the encoded residual image .

[0102] The first communication device 600 is further configured to transmit at least the encoded semantic segmentation image to the second communication device 800 via the first communication channel 510. In addition to the encoded semantic segmentation image , the first communication device 600 can be configured to transmit the encoded coarse image to the second communication device 800 or to transmit the encoded coarse image and the encoded residual image to the second communication device 800.

[0103] Figure 4 A block scheme of the image decoding device 300 is shown. Such an image decoding device 100 can be referred to as a task-related LIV decoder. The image decoding device 300 is configured to obtain an encoded semantic segmentation image of an input image The encoded semantic segmentation image It may have been stored and / or transmitted previously.

[0104] Image decoding device 300 is also used for encoding semantic segmentation images based on lossless decoder 132. Decode to generate the input image semantic segmentation image The image decoding device 300 is also used for semantic image segmentation based on a sophisticated deep neural network 150. and semantically based distortion graphs Generate input image Estimated image Therefore, the lossless decoder 132 provides semantic segmentation images for the fine-grained deep neural network 150. Semantic-based distortion maps can also be provided to the fine-grained deep neural network 150 via the semantic task filter 152. .

[0105] Image decoding device 300 is also used for image estimation Generate input image Reconstructed image Input image Reconstructed image This can be generated by the addition unit 156, as detailed below. According to some embodiments, the input image... It can be associated with video applications.

[0106] As described below, according to some embodiments, in addition to the encoded semantic segmentation image In addition, the image decoding device 300 can also be used to acquire the encoded coarse image. Or obtain a rough encoded image. and encoded residual image ´, and based on the acquired encoded coarse image Or, simultaneously based on the acquired encoded coarse image and the acquired encoded residual image To generate reconstructed images .

[0107] Figure 5 It shows that it can be generated by an image decoding device 300 (e.g. Figure 4 The flowchart of the corresponding method 400 executed (shown) indicates that the image decoding device 300 may be in the second communication device 800 (e.g., Figure 6 Implemented as shown. It should be noted that... Figure 5 The methods and steps shown and described in this article do not necessarily have to be followed. Figure 5The steps are executed in the order shown. These steps can be executed in virtually any suitable order, or in parallel, provided that the physical requirements and the information needed to execute each step are available at the time of execution.

[0108] Method 400 includes acquiring 402 input images. Encoded semantic segmentation image Method 400 further includes: processing the encoded semantic segmentation image based on the lossless decoder 132. Decode 404 to generate the input image semantic segmentation image Method 400 also includes: semantic image segmentation based on a fine-grained deep neural network 150. and semantically based distortion graphs Generate 406 input images Estimated image Method 400 also includes: based on the estimated image Generate 408 input images Reconstructed image .

[0109] Image decoding device 300 and corresponding method 400 based on semantic-based distortion image For the input image Decoding is performed, making the decoding scheme task-specific, i.e., task-input-specific. Semantic-based distortion graph. The region / object of interest is defined, and a trade-off between visual quality and coding rate is made, thus affecting the compression ratio. It should be noted that the image decoding device 300 and the corresponding method 400 differ from the known DSSLIC schemes mentioned above in that they use a semantically based distortion map. The DSSLIC scheme is task-independent, meaning its encoding and decoding are not based on semantically based distortion graphs. Any corresponding parameter.

[0110] The image decoding device 300 and the corresponding method 400 utilize a combination of AI and standard / classical compression blocks. Therefore, the burden of extensive training of neural networks is avoided.

[0111] Figure 6 A second communication device 800 for a communication system 500 according to an embodiment of the present invention is shown. Figure 6In the illustrated embodiment, the second communication device includes a processor 302, a transceiver 304, and a memory 306. The processor 302 is coupled to the transceiver 304 and the memory 306 via a communication component 308 known in the art. The second communication device 800 also includes a communication interface 310 coupled to the transceiver 304, meaning that the second communication device 800 is used for communication within the communication system 500.

[0112] Processor 302 may be referred to as one or more general-purpose CPUs, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more programmable logic devices, one or more discrete gates, one or more transistor logic devices, one or more discrete hardware components, or one or more chipsets. Memory 306 may be read-only memory, random access memory (RAM), or non-volatile RAM (NVRAM). Transceiver 304 may be transceiver circuitry, a power controller, or an interface providing the ability to communicate with other communication modules or communication devices. Transceiver 304, memory 306, and / or processor 302 may be implemented in separate chipsets or in a common chipset. In this invention, the second communication device 800 is used to perform certain operations, which can be understood as the second communication device 800 including suitable components for performing these operations, such as processor 302 and transceiver 304.

[0113] According to an embodiment of the present invention, the second communication device 800 is used to receive an input image from the first communication device 600 via the first communication channel 510. Encoded semantic segmentation image The second communication device 800 includes the image decoding device 300 described herein. As described above, the image decoding device 300 is used to acquire an input image. Encoded semantic segmentation image The image decoding device 300 is also used for processing encoded semantically segmented images based on the lossless decoder 132. Decode to generate the input image semantic segmentation image The image decoding device 300 is also used for semantic image segmentation based on a sophisticated deep neural network 150. and semantically based distortion graphs Generate input image estimated image The image decoding device 300 is further configured to generate a reconstructed image based on the estimated image . .

[0114] As explained below, according to some embodiments, in addition to the encoded semantic segmentation image , the second communication device 800 can be configured to receive an encoded coarse image or to receive an encoded coarse image and an encoded residual image ´, such that in addition to the encoded semantic segmentation image , the image decoding device 300 can be configured to generate a reconstructed image based on the obtained encoded coarse image or based on the obtained encoded coarse image and the obtained encoded residual ´ image.

[0115] Figure 7 A communication system 500 is shown, on which the first communication device 600 can transmit at least the encoded semantic segmentation image to the second communication device 800 over a first communication channel 510. According to some embodiments as explained below, in addition to the encoded semantic segmentation image , the first communication device 600 can transmit an encoded coarse image to the second communication device 800 over the first communication channel 510 or transmit an encoded coarse image and an encoded residual image ´ to the second communication device 800 over the first communication channel 510. According to some embodiments as explained below, the first communication device 600 can further transmit a signal 620 to the second communication device 800 over a second communication channel 520. According to various embodiments, the first communication device 600 can be a mobile device or a base station device. According to various embodiments, the second communication device 800 can be a mobile device or a base station device. Thus, the encoded images can be transmitted in any direction Figure 7 between the entities, either directly between two mobile devices or between a base station device and a mobile device, or indirectly through one or more intermediate devices and / or networks.

[0116] Figure 8 A block scheme illustrating some embodiments of the image encoding device 100 is shown. As explained in connection with Figure 1 , the image encoding device 100 is configured to encode an input image Encoding is performed, and in some embodiments, the input image is associated with a video application. The input image is created based on a semantic segmentation deep neural network 120. semantic segmentation image Then, based on the lossless encoder 130, the semantic segmentation image is processed. Encode to generate encoded semantic segmentation images The encoded semantic segmentation image can be stored in the encoder memory 180 and / or transmitted to the image decoding device 300 via the first communication channel 510.

[0117] Also for the input image Encoding is performed so that the first lossy encoder 140 is based on the input image. and semantically based distortion graphs With the first coding rate Generate a coarse image after encoding And make the first lossy decoder 142 based on the encoded coarse image. Generate a rough image Semantic-based distortion maps The semantic task filter 152 can be provided to the first lossy encoder 140. According to some embodiments, the first lossy encoder 140 can use a classic compression algorithm, such as BPG or JPEG2000, for the input image. Compressed to the first coding rate Encoded input image Low-dimensional version .

[0118] Image encoding device 100 is also used for semantic image segmentation based on fine-grained deep neural network 150. Rough image and semantically based distortion graphs Generate input image Estimated image Provides the encoded semantic segmentation image from the semantic segmentation deep neural network 120 to the fine-grained deep neural network 150. A rough image from the first lossy decoder 142 and semantically based distortion graphs from semantic task filter 152. .

[0119] According to one embodiment, based on the input image Related task and semantic segmentation images In semantic task filter 152, a semantically based distortion map is determined. Then, it will be compared with the input image. Related task and semantic segmentation images The semantic task filter 152 is provided with parameters such that the semantic task filter 152 can determine a semantic-based distortion map based on these parameters . According to some embodiments, the task associated with the input image may comprise a set of labels and / or sub-images / regions of interest for which the corresponding part of the input image should be encoded at a high resolution.

[0120] According to one embodiment, the semantic-based distortion map is provided to the first lossy encoder 140. Thus, the first encoding rate of the first lossy encoder 140 can be adjusted according to the semantic-based distortion map . According to one embodiment, the semantic-based distortion map used by the first lossy encoder 140 comprises a heat map distortion of the input image . The image distortion heat map or intensity map is a kind of visual representation, e.g. composed of pixels, to show the areas in the image that are most important or relevant for a specific task, e.g. for object detection or any other task mentioned herein. Thus, each pixel of the image heat map distortion indicates the intensity of importance of the corresponding pixel in the original image.

[0121] By subtracting the input image from the estimated image in the differencing unit 154, a residual image can also be generated based on the estimated image and the input image . Thus, according to one embodiment, the residual image is a difference image between the estimated image and the input image that complies with a target distortion. The target distortion refers to a target or acceptable level of distortion that is allowed in the process of compressing an image or video. For example, in a lossy compression algorithm, the data size of the original image is reduced in compression, which results in some distortion in the encoded image, which should be below the target distortion. The differencing unit 154 is provided with the estimated image from the fine depth neural network 150 and with the input image from the input 158. The residual image is provided from the differencing unit 154 to the second lossy encoder 160, where an encoded residual image is then generated in the second lossy encoder 160. According to some embodiments, the second lossy encoder 160 generates the encoded residual image at a second encoding rate . The second encoding rate the information content and energy of the encoded residual image According to some embodiments, the semantic task filter 152 will provide the semantic based distortion map to the second lossy encoder 160. By providing the semantic based distortion map to the second lossy encoder 160, the second encoding rate of the second lossy encoder 160 can be adjusted according to the semantic based distortion map

[0122] The image encoding device 100 can also be configured to store the encoded semantic segmentation image , and in addition to the encoded semantic segmentation image , the encoded coarse image may also be stored or the encoded coarse image and the encoded residual image may be stored in the encoding memory 180.

[0123] Thus, the image encoding device 100 generates the encoded semantic segmentation image , the encoded coarse image and the encoded residual image . Then, according to various embodiments, the encoded semantic segmentation image , the encoded coarse image and the encoded residual image may be stored in a hierarchical manner, with the encoded semantic segmentation image having the highest rank. Thus, it is possible to store only the encoded semantic segmentation image , or to store the encoded semantic segmentation image with the encoded coarse image , or to store the semantic segmentation image , the encoded coarse image and the encoded residual image .

[0124] When encoding a video, the input video is processed in a frame by frame manner, i.e. each image is processed. In parallel, the input video is compressed using a classical off-the-shelf video codec 140, 142 (e.g. H265) to create a corresponding coarse / compressed video / image . The coarse / compressed video / image is then encoded using the semantic segmentation neural network 120 to create a corresponding semantic segmentation image frame-wise (i.e. image-wise) to the fine depth neural network 150. The fine depth neural network 150 creates a fine detail residual image The sequence, the images are converted to residual video / images. The residual video / images are created in a corresponding manner as described above for the image encoding device 100 and can be encoded by a selected lossy video encoder into video / images ´. Using the three layers of information for the encoded video as described above, i.e. by generating an encoded semantic segmentation video / image , an encoded coarse video / image and an encoded residual video / image ´, it is possible to encode the video / image file such that the encoding rate is reduced while providing a task-related and semantic-based encoding which is a valuable feature for many applications, e.g. Internet of Things applications.

[0125] As described above, the first communication device 600 is configured to transmit the encoded semantic segmentation image to the second communication device 800 over the first communication channel 510. According to various embodiments, the first communication device 600 is configured to transmit the encoded semantic segmentation image , the encoded coarse image and the encoded residual image in a layered manner, wherein the encoded semantic segmentation image has the highest rank. Thus, it is possible to transmit only the encoded semantic segmentation image to the second communication device 800, or to transmit the encoded coarse image in addition to the encoded semantic segmentation image , or to transmit the semantic segmentation image , the encoded coarse image and the encoded residual image .

[0126] As described above, according to one embodiment, the first lossy encoder 140 has a first encoding rate , the second lossy encoder 160 has a second encoding rate , wherein the first encoding rate and / or the second encoding rate can be adjusted according to a semantic-based distortion map . Then, the image encoding device 100 is configured to determine a sum of the first encoding rate and the second encoding rate . The sum of the first encoding rate and the second encoding rate may be based on the input image The target distortion is determined by the image target distortion. The target distortion corresponds to the optimal operating point; for example, the minimum coding rate where the resulting distortion is less than or equal to the target distortion. Therefore, if the first coding rate is determined based on this target distortion... Second coding rate The sum of these values ​​represents the optimal coding point relative to the actual distortion produced.

[0127] According to one embodiment, the sum of the first coding rate and the second coding rate can also be determined based on an estimate of the first communication channel 510. The channel estimate is an estimate or measurement of the quality of the first communication channel 510 between the first communication device 600 and the second communication device 800. If the quality of the first communication channel 510 is good, a higher coding rate can be used during transmission if the application allows. However, if the quality of the first communication channel 510 is poor, the application may not allow a higher coding rate, and therefore a lower coding rate is used during transmission.

[0128] The sum of the first coding rate and the second coding rate can also be determined based on the available channel resources of the first communication channel 510, wherein the available channel resources may be related to the number of users, such that if there are many users, each user can only be allocated a small amount of resources. For wireless communication, available resources typically include limited available wireless resources provided only to users.

[0129] The sum of the first coding rate and the second coding rate can also be based on the input image. The image requirements of the associated application settings are used to determine this. For example, if the application is used for IoT control, its image requirements may allow for a higher compression ratio corresponding to a lower code rate, while if the application is used for H2M, its image requirements may only allow for a lower compression ratio corresponding to a higher code rate. The sum of the first and second code rates can also be determined based on a combination of one or more of these parameters.

[0130] Because the first lossy encoder 140 encodes a coarse image... The first coding rate used during encoding For residual images The amount of information and energy content have an impact, therefore in the first coding rate Second coding rate There are trade-offs. For example, if using a very low first coding rate... For the encoded coarse image Encode, that is, if the encoded coarse image If it's too coarse, then it naturally assumes the residual image... This will involve more energy and will require more bits to achieve the desired distortion. On the other hand, if a high first coding rate is used... For the encoded coarse image encoding, the residual image will include less energy, and the encoded residual image after encoding in the second lossy encoder 160 will require less bits to achieve the target distortion, but at the cost of using more bits to generate the encoded coarse image .

[0131] Thus, given a certain target distortion, the second encoding rate is a function of the first encoding rate According to some embodiments, the total rate + can be optimized with respect to the first encoding rate and the fine depth neural network 150, and in turn the fine depth neural network 150 should be retrained for different values of the first encoding rate This optimization can be achieved by selecting a finite set of possible values for the first encoding rate and retraining the fine depth neural network for each first encoding rate Likewise, for each first encoding rate a large range of second encoding rates and + and corresponding pairs of distortion experience can be generated. Then, the operating points of these + and corresponding pairs of distortion are scanned and the + with a distortion not greater than a given target distortion is selected as the operating point to be used.

[0132] The image encoding device 100 is able to provide rate adaptation according to channel conditions and / or application requirements, which is an advantageous feature. In particular, for wireless applications, + the optimization of the operating points and corresponding pairs of distortion is very important, as the operating point used and thus the total rate actually used depends on the possibly varying channel conditions, resources, etc. Moreover, as the rate can also be adjusted based on application requirements, flexibility related to the application is provided. Thus, the image encoding device 100 can be task and / or application dependent, as it can adapt based on input parameters (e.g. labels / classes / regions of interest specifying the semantic-based distortion map and application requirements (e.g. rate and distortion parameters constraining the operating point).

[0133] According to some embodiments, the first communication device 600 is configured to transmit the signal 530 to the second communication device 800 over the second communication channel 520, as Figure 7As shown. Then, the transmission signal 530 can indicate the input image to the second communication device 800. The associated task, fine-grained deep neural network 150, decoder 132 corresponding to lossless encoder 130, first lossy decoder 142 corresponding to first lossy encoder 140 and / or second lossy decoder 162 corresponding to second lossy encoder 160.

[0134] Figure 9 Block schemes of some embodiments of the image decoding device 300 are shown. (See attached diagram.) Figure 4 As explained, the image encoding device 100 is used to encode the input image. Encoding is performed, and in some embodiments, the input image is associated with a video application. The image decoding device 300 is used to acquire at least the input image. Encoded semantic segmentation image The encoded semantic segmentation image It may have been stored and / or transmitted previously.

[0135] Encoded semantic segmentation image The input image is provided to the lossless decoder 132 and decoded based on the lossless decoder 132 to generate the input image. semantic segmentation image Then, the semantically segmented image... Provided to a fine-grained deep neural network 150, and based on semantic segmentation images in the fine-grained deep neural network 150. and semantically based distortion graphs Generate input image Estimated image Semantic-based distortion maps The semantic task filter 152 is provided to the fine-grained deep neural network 150 to estimate the image. The generation of [something] is task-related.

[0136] According to one embodiment, the semantic task filter 152 is based on the input image. Related task and semantic segmentation images Determine semantically based distortion maps Therefore, the task and semantic segmentation of images are combined. Provided to semantic task filter 152, semantic task filter 152 will analyze the semantically distorted graph. Provided to a fine-grained deep neural network 150.

[0137] As described above, according to some embodiments, with the input image The associated task may include the input image A set of labels and / or sub-images of interest. According to one embodiment, semantically distorted images... Including input image Heatmap distortion is used by the sophisticated deep neural network 150 to generate input images. Estimated image .

[0138] Image decoding device 300 also relies on estimated images Generate input image Reconstructed image The input image can be generated in the addition unit 156. Reconstructed image .

[0139] According to some embodiments, in addition to the encoded semantic segmentation image In addition, the image decoding device 300 can also be used to acquire the encoded coarse image. Or obtain a rough encoded image and encoded residual image ´, and in addition to the encoded semantic segmentation image In addition, it is based on the acquired encoded coarse image. or based on the acquired encoded coarse image and the acquired encoded residual image To generate reconstructed images .

[0140] More specifically, if the encoded coarse image is retrieved from memory or received via transmission... The image decoding device 300 is used to decode the coarse image based on the first lossy decoder 142. Decode to generate the input image rough image And in addition to the encoded semantic segmentation image In addition, it is also based on the input image rough image To generate estimated images Input image rough image The fine-grained deep neural network 150 is provided, and the fine-grained deep neural network 150 is also based on the input image. rough image Generate input image Estimated image Therefore, a sophisticated deep neural network 150 was subsequently used to segment images semantically. Semantic-based distortion maps and input image coarse image of the input image generating an estimated image

[0141] If, in addition to the encoded semantic segmentation image , an encoded coarse image and an encoded residual image are retrieved, e.g. from a memory or received by transmission, the image decoding device 300 is configured to decode the encoded coarse image based on the first lossy decoder 142 to generate a coarse image of the input image , to decode the encoded residual image of the input image based on the second lossy decoder 162 to generate a reconstructed residual image of the input image , and to generate a reconstructed image of the input image based on the coarse image of the input image and the reconstructed residual image of the input image in addition to the semantic segmentation image . The coarse image of the input image is provided to the fine depth neural network 150 which generates an estimated image of the input image based on the coarse image of the input image in addition to the semantic segmentation image . The reconstructed residual image of the input image is provided to the addition unit 156 which is configured to generate the reconstructed image by adding the estimated image to the reconstructed residual image .

[0142] Thus, at the image decoding device 300, the decoder has full knowledge of the semantic segmentation image / map due to the lossless encoding of the encoded semantic segmentation image in the lossless encoder 130 in the image encoding device 100. Furthermore, the semantic-based distortion map is known to both the image encoding device 100 and the image decoding device 300. The semantic-based distortion map ​No transmission or encoding is required. Image segmentation is performed by knowing the attributes and semantics of the task, using both the image encoding device 100 and the image decoding device 300. Used as input, a semantically based distortion map can be obtained from the semantic task filter 152. Semantic-based distortion maps Used as input in a fine-grained DNN 150 to generate the input image. Estimated image Due to semantically distorted images As is associated with the task, semantically based distortion maps are therefore utilized in the image decoding device 300. This makes it relevant to the task. The image decoding device 300 also grasps coarse / compressed images. All the knowledge, which means the coarse / compressed image in the image decoding device 300 With the coarse / compressed image in the image encoding device 100 The same applies because the image decoding device 300 uses the same first lossy decoder 142 as the image encoding device 100. Therefore, the decoder can synthesize the estimated missing details and generate the input image. Estimated image Combined with the residual image after lossy encoding The image decoding device 300 can ultimately generate the input image. Visually appealing reconstructed images .

[0143] Therefore, according to one embodiment, in addition to the encoded semantic segmentation image In addition, the second communication device 800 is also used to receive input images from the first communication device 100. Coarse image after encoding Then, the image decoding device 300 is used to process the encoded coarse image based on the first lossy decoder 142. Decode to generate the input image rough image And based on semantic image segmentation and input image rough image Generate input image Estimated image .

[0144] According to one embodiment, in addition to the encoded semantic segmentation image and input image Coarse image after encoding In addition, the second communication device 800 is also used to receive input images from the first communication device 100. Encoded residual image Then, the image decoding device 300 is used to decode the input image based on the second lossy decoder 162. Encoded residual image Decode to generate the input image The reconstructed residual image And based on semantic image segmentation Input image rough image and input image The reconstructed residual image Generate input image Reconstructed image Specifically, the input image The reconstructed residual image Provided to the addition unit 156, wherein the reconstructed residual image With estimated image Add to generate a reconstructed image .

[0145] According to one embodiment, the second communication device 800 is further configured to receive a signal 530 from the first communication device 600 via a second communication channel 520. This signal 530 indicates to the second communication device 800 an input image. The associated tasks include a fine-grained deep neural network 150, a lossless decoder 132, a first lossy decoder 142, and / or a second lossy decoder 162. The image decoding device 300 is used to acquire information indicated in the signal 530 and generate an input image based on that information. Reconstructed image The acquired task is used to generate semantically based distortion maps. Based on the semantic segmentation images provided by the lossless decoder 132 and the first lossy decoder 142 obtained therefrom, respectively. and rough image The refined DNN obtained therefrom is used to generate the estimated image as described above. The resulting second lossy decoder 162 generates the reconstructed residual image. and add it to the estimated image. To generate reconstructed images .

[0146] Figure 10 Signaling diagrams illustrating the interaction between a first communication device 600 and a second communication device 800 according to various embodiments of the present invention are shown.

[0147] exist Figure 10 In step I, the image encoding device 100 of the first communication device 600 generates the input image. the semantic segmentation image and encodes the semantic segmentation image to generate an encoded semantic segmentation image as described above.

[0148] In step II, the image encoding device 100 of the first communication device 600 generates an encoded coarse image as described above.

[0149] In step III of the method Figure 10 , the image encoding device 100 of the first communication device 600 generates a residual image and encodes the residual image to generate an encoded residual image ´ as described above.

[0150] The first communication device 600 transmits at least the encoded semantic segmentation image to the second communication device 800 over the first communication channel 510.

[0151] According to some embodiments, the first communication device 600 transmits, in addition to the encoded semantic segmentation image , the encoded coarse image to the second communication device 800 over the first communication channel 510.

[0152] According to some embodiments, the first communication device 600 transmits, in addition to the encoded semantic segmentation image , the encoded coarse image and the encoded residual image to the second communication device 800 over the first communication channel 510.

[0153] According to some embodiments, the image encoding device 100 and / or the first communication device 600 determines, in step IV of the method Figure 10 , a signal 530 indicative of a task associated with the input image , the fine depth neural network 150, the decoder 132 corresponding to the lossless encoder 130, the first lossy decoder 142 corresponding to the first lossy encoder 140 and / or the second lossy decoder 162 corresponding to the second lossy encoder 160. Then, the first communication device 600 transmits the signal 530 to the second communication device 800 over the second communication channel 520.

[0154] In step V, the second communication device 800 receives, from the first communication device 600 over the first communication channel 510, the encoded semantic segmentation image of the input image and the image decoding device 300 generates a reconstructed image of the input image based on the received encoded semantic segmentation image . as described above.

[0155] According to one embodiment, in step V, the second communication device 800 receives, in addition to the encoded semantic segmentation image , an encoded coarse image . Then, the image decoding device generates a reconstructed image of the input image based on the received encoded semantic segmentation image and the received encoded coarse image . as described above.

[0156] According to one embodiment, in step V, the second communication device 800 receives, in addition to the encoded semantic segmentation image , an encoded coarse image and an encoded residual image ´. Then, the image decoding device generates a reconstructed image of the input image based on the received encoded semantic segmentation image , the received encoded coarse image and the received encoded residual image ´. as described above.

[0157] According to some embodiments, in step V, the second communication device 800 receives, in addition to the encoded semantic segmentation image , a signal 530 indicating a task associated with the input image , the fine depth neural network 150, a decoder 132 corresponding to the lossless encoder 130, a first lossy decoder 142 corresponding to the first lossy encoder 140 and / or a second lossy decoder 162 corresponding to the second lossy encoder 160. Then, the image decoding device generates a reconstructed image of the input image based on the information provided by the signal 530.

[0158] The image encoding device 100 and the image decoding device 300 proposed herein aim at defining an architecture of efficient and adaptive semantic-based and task-dependent source coding and decoding schemes as part of the first communication device 600 and the second communication device 800 of the communication system 500 in order to reduce the communication payload of the communication system 500. In particular, the source coding of the image encoding device 100 and the image decoding device 300 comprises a combination of different artificial intelligence-based sub-blocks (e.g. various forms of neural networks) and classical compression / decompression and lossless / lossy coding / decoding sub-blocks to implement semantic-based and task-dependent compression / coding and reconstruction / decoding schemes for images and videos.

[0159] The image encoding device 100 and the image decoding device 300 proposed herein can basically be used in all potential products comprising image / video source coding blocks. For example, the first communication device 600 can be a user terminal, a vehicle, a robot, a general machine or even a base station, wherein the information source to be encoded is an image or a video produced by a camera or stored in a server. For example, the second communication device 800 comprising the image decoding device 300 can be a user terminal, a vehicle, a robot, a general machine or even a base station, wherein the image decoding device 300 decodes compressed images / videos and pictures so that the original images / videos can be synthesized or only specific operations can be performed.

[0160] Therefore, the image encoding device 100 and the image decoding device 300 proposed herein can be placed either at the terminal side or at the base station side. However, in order to be able to reconstruct the original images / videos, the image encoding device 100 and the image decoding device 300 need to share some common knowledge, e.g. the task and the fine DNN associated with the input image By sharing this common knowledge, the image encoding device 100 and the image decoding device 300 are fully compatible. The semantic-based nature of the coding scheme leads to a certain level of unique representation of the original images / videos in terms of performance / complexity as well as visual appearance.

[0161] Image encoding device 100 and image decoding device 300 provide the possibility of incorporating semantic-based distortion metrics into source coding, particularly integrating heatmap distortion into the encoding and decoding processes. This allows the encoding and decoding processes to focus on specific regions of interest in the image, such as those captured by surveillance cameras, traffic cameras, or augmented / virtual reality (AR / VR) applications. Image encoding device 100 and image decoding device 300 also enable the integration of standard lossy image and video coding into source coding and decoding, making these schemes compatible with any standard player application. Image encoding device 100 and image decoding device 300 further optimize the coarse image after encoding. Compression is achieved by adapting the source coding through a wide range of rate and distortion pairs.

[0162] As non-limiting examples, many possible applications of the image encoding device 100 and the image decoding device 300 are mentioned below. One example is monitoring the behavior of AI-based autonomous vehicles. For autonomous vehicle applications, it is important not to distort key pixels associated with labels, such as cars, traffic lights, traffic signs, pedestrians, etc., while other parts of the scene, such as trees, sky, buildings, etc., can be synthesized by the neural network portion. The image encoding device 100 will use an appropriate semantically distorted image generated based on a specific task. This masks anything other than these key labels, and then the encoded residual image is used at a relatively high rate. The regions containing these key labels are encoded. Other parts of the scene, i.e., less important labels, are encoded using a neural network and a compressed version of the image (i.e., a coarsely encoded image). Therefore, the image encoding device 100 and the image decoding device 300 significantly reduce the rate of synthesis.

[0163] Another possible application of the image encoding device 100 and the image decoding device 300 is security video surveillance in public places. Semantic-based distortion images generated based on specific tasks can be defined. A set of relevant attention tags, such as people, animals, bags, etc., are then used. The image encoding device 100 should then transmit the pixels related to the attention tags to a security center at a high rate, while other parts of the scene can be synthesized by a neural network. In this case, an readily available semantic segmentation algorithm using the FLIF algorithm can be used to obtain the encoded semantic segmentation image. A sequence. A compressed version of the video, i.e., a coarsely encoded image, can be obtained using the H.265 algorithm. The sequence, and the encoding of the residual video, i.e., the encoded residual image. sequence.

[0164] Another possible application for the image encoding device 100 and the image decoding device 300 is augmented / virtual reality or meta-applications. In such applications, it is important not to distort the semantically segmented images. Related pixels, and semantically distorted images generated based on a specific task. The relevant pixels focus on certain specific areas, i.e., the user's field of view. The image encoding device 100 then transmits the pixels associated with the label and region of interest at a high rate by encoding them at high resolution, while other parts of the scene can be synthesized by a neural network. Using the image encoding device 100 and image decoding device 300 can significantly reduce the rate, as explained above with reference to other embodiments.

[0165] Therefore, image encoding device 100 and image decoding device 300 refer to the basic components of a transceiver communication system that uses semantic image / video source coding. Image encoding device 100 and image decoding device 300 can generally be applied to any image / video transmission-related application in any wireless or wired communication system / network where semantic fidelity is the primary concern, such as the aforementioned autonomous vehicles, surveillance, or AR / VR applications. Image encoding device 100 and image decoding device 300 can then reduce the communication payload according to specific objectives. Image encoding device 100 provides object-based or region-of-interest-based coding reconstructed by image decoding device 300, enabling operation at the second communication device 800.

[0166] Semantic coding has been proposed as a potential technology for 6G networks. Furthermore, Wi-Fi or other fixed networks can utilize semantic coding, as well as local networks within vehicles or factories, for local decision-making. Finally, this image / video source coding scheme can be used as a semantic compression algorithm to store compressed versions of images / videos focused on specific classes of interest, while semantic viewers / players can reconstruct the images / videos using the original regions of interest and the synthesized portions.

[0167] Furthermore, any of the methods according to embodiments of the application can be implemented in a computer program, having code means, which is executed by a processing means and which causes the processing means to perform the method steps. The computer program is included in a computer readable medium, for instance a floppy disk, CD ROM or an internet distribution. The computer readable medium is essentially a tangible memory, for instance the above mentioned ROM, programmable ROM (PROM), erasable programmable ROM (EPROM), flash memory, electrically erasable programmable ROM (EEPROM), a hard disk, etc.

[0168] Furthermore, it should be appreciated that the image encoding device 100 and the image decoding device 300 comprise necessary communication capabilities in the form of functions, means, units, elements, etc. for performing or realizing embodiments of the application. Other such embodiments of means, units, elements and functions are: processors, memories, buffers, control logic, encoders, decoders, rate matchers, de-rate matchers, mapping units, multipliers, decision units, selection units, switches, interleavers, de-interleavers, modulators, modems, inputs, outputs, antennas, amplifiers, receiving units, transmitting units, DSPs, MSDs, TCM encoders, TCM decoders, power supply units, power feeders, communication interfaces, communication protocols, etc. which are suitably arranged together to perform the technical solutions.

[0169] Hence, the one or more processors of the image encoding device 100 and / or the image decoding device 300 can comprise, e.g., one or more instances of a central processing unit (CPU), a processing unit, a processing circuit, a processor, an application-specific integrated circuit (ASIC), a microprocessor, or other processing logic that can interpret and execute instructions. Hence, the expression "processor" can mean any processing circuitry comprising a plurality of functional entities such as any of the above, in whole or in part. The processing circuitry can further perform data processing functions for inputting, outputting, and processing of data comprising data buffering and device control functions, such as call processing control, user interface control, or the like.

[0170] Finally, it should be understood that the application is not limited to the embodiments described above, but also relates to all embodiments within the scope of the appended claims.

Claims

1. An image encoding apparatus (100), characterized by, for: generating a semantic segmentation image of the input image based on a semantic segmentation deep neural network (SSDNN) (120) ;​ based on a lossless encoder (130) encoding the semantic segmentation image to generate an encoded semantic segmentation image ; based on a first lossy encoder (140), a corresponding first lossy decoder (142), and a semantic-based distortion map encoding the input image to generate an encoded coarse image and a coarse image ; based on the semantic segmentation image in a refinement deep neural network (refinement, DNN) (150) , the coarse image and the semantic-based distortion map generating an estimated image of the input image ; based on the estimated image and the input image generating a residual image ; based on a second lossy encoder (160) encoding said residual image to generate an encoded residual image ´.

2. The image coding apparatus (100) according to claim 1, characterized by for: storing the encoded semantic segmentation image .

3. The image coding apparatus (100) according to claim 2, characterized by for: storing the encoded coarse image .

4. The image coding (100) of claim 3, characterized by for: storing the encoded residual image .

5. The image coding apparatus (100) according to any one of the above claims, characterized by based on a task associated with the input image and the semantic segmentation image determining the semantic-based distortion map .

6. The image coding apparatus (100) according to claim 5, characterized by The task includes a set of labels and / or sub-images of interest of the input image ​ 7. The image coding apparatus (100) according to claim 5 or 6, characterized by the semantic-based distortion map including the input image heat map distortion.

8. The image coding apparatus (100) according to any one of the above claims, characterized by The residual image is a difference image between the estimated image and the input image .

9. The image coding apparatus (100) according to any one of the above claims, characterized by The input image For video applications.

10. A first communication device (600) for a communication system (500), characterized by The first communication device (600) comprises the image encoding device (100) according to any of the preceding claims, the first communication device (600) being configured to: The encoded semantic segmentation image is transmitted through the first communication channel (510). Transmitted to the second communication device (800).

11. The first communication device (600) according to claim 10, characterized by for: transmitting the encoded coarse image to the second communication device (800).

12. The first communication device (600) according to claim 11, characterized by for: transmitting the encoded residual image to the second communication device (800).

13. The first communication device (600) according to any one of claims 10 to 12, characterized by The first lossy encoder (140) has a first encoding rate The second lossy encoder (160) has a second encoding rate The image encoding device (100) is configured to: determining a sum of the first code rate and the second code rate ​ 14. The first communication device (600) according to claim 13, characterized by the first encoding rate and the second encoding rate is determined based on one or more of an image target distortion of the input image an estimate of the first communication channel (510), available channel resources of the first communication channel (510), and image requirements set by an application associated with the input image ​ 15. The first communication device (600) according to any one of claims 10 to 14, characterized by for: transmitting a signal (530) to the second communication device (800) over a second communication channel (520), the signal (530) indicating one or more of: the task associated with the input image the fine DNN (150), a decoder (132) corresponding to the lossless encoder (130), the first lossy decoder (142) corresponding to the first lossy encoder (140), a second lossy decoder (162) corresponding to the second lossy encoder (160).

16. An image decoding apparatus (300) characterized by comprising: for: acquiring an input image an encoded semantic segmentation image of the input image ; based on a lossless decoder (132) on the encoded semantic segmentation image decoded to generate the input image of the semantic segmentation image ; based on the semantic segmentation image and semantic-based distortion map in fine DNN (150) generating the input image estimated image ; based on the estimated image generating the input image reconstructed image .

17. The picture decoding apparatus (300) according to claim 16, characterized by for: acquiring the input image of the encoded coarse image ; based on a first lossy decoder (142) decoding the encoded coarse image to generate the input image of the input image ​ further based on the input image the coarse image the input image the estimated image is generated 18. The picture decoding apparatus (300) according to claim 17, characterized by for: acquiring the input image of the encoded residual image ; based on a second lossy decoder (162) decoding the encoded residual image of the input image to generate a reconstructed residual image of the input image ​ further based on the input image the reconstructed residual image generating the reconstructed image of the input image .

19. The picture decoding apparatus (300) according to claim 18, characterized by the reconstructed image is an addition of the estimated image and the reconstructed residual image .

20. The picture decoding apparatus (300) according to any of claims 16 to 19, characterized by based on a task associated with the input image and the semantic segmentation image determining the semantic-based distortion map .

21. The picture decoding apparatus (300) according to any of claims 16 to 20, characterized by The task includes a set of labels and / or sub-images of interest of the input image ​ 22. The picture decoding apparatus (300) according to claim 20 or 21, characterized by the semantic-based distortion map including the input image heat map distortion.

23. The picture decoding apparatus (300) according to any of claims 16 to 22, characterized by The input image For video applications.

24. A second communication device (800) for a communication system (500), characterized by The second communication device (800) comprises the image decoding device (300) according to any of claims 16 to 23, the second communication device (800) being configured to: receiving the input image from a first communication device (600) over a first communication channel (510) the encoded semantic segmentation image .

25. The second communication device (800) according to claim 24, characterized by for: receiving the input image from the first communication device (100) the encoded coarse image .

26. The second communication device (800) according to claim 25, characterized by for: receiving the input image from the first communication device (100) an encoded residual image of the input image .

27. The second communication device (300) according to any one of claims 24 - 26, characterized by for: receiving a signal (530) from the first communication device (100) over a second communication channel (520), the signal (530) indicating one or more of: the task associated with the input image the fine DNN (150), the lossless decoder (132), the first lossy decoder (142), and the second lossy decoder (162).

28. A method (200) for an image encoding device (100), characterized by, The method (200) comprises: generating (202) a semantic segmentation image of the input image based on a semantic segmentation deep neural network (SSDNN) (120) ;​ based on a lossless encoder (130) on the semantic segmentation image encoding (204) to generate an encoded semantic segmentation image ; based on a first lossy encoder (140), a corresponding first lossy decoder (142), and a semantic-based distortion map encoding (204) the input image to generate an encoded coarse image and a coarse image ; The semantic image segmentation based on the refinement deep neural network (refinement DNN) (150) The rough image and the semantic-based distortion map Generate (206) the input image Estimated image ; based on the estimated image and the input image generating (208) a residual image encoding (210) the residual image based on a second lossy encoder (160) to generate an encoded residual image ´.

29. A method (400) for an image decoding device (300), characterized by, The method (400) comprises obtaining (402) an input image of an encoded semantic segmentation image ; decoding (404) the encoded semantic segmentation image based on a lossless decoder (132) to generate a semantic segmentation image of the input image ; based on the semantic segmentation image and semantic-based distortion map in fine DNN (150) generating (406) an estimated image of the input image ; based on the estimated image generating (408) a reconstructed image of the input image of the input image .

30. A computer program having a program code for performing a method according to one of the preceding claims 27 to 29 when the program code is executed on a computer. The program code is for performing the method according to any of claims 28 to 29 when the computer program is run on a computer.