Method and device for encoding and decoding images

Abstract

The invention relates to a method and device for encoding and decoding an image (Id) comprising a plurality of regions (Rdj). The decoding method comprises the following steps: - obtaining (F22) latent values ​​called primary values ​​( ), representative of said image to be decoded (Id), - obtaining (F23) a plurality of neural networks ( ), called synthetic neural networks, - For at least two regions (Rd1, Rd2) of said image to be decoded: - processing (F24) of said primary latent values ​​identified ( ) by at least two synthetic neural networks distinct from the plurality ( ), to produce at least two sets of respective secondary latent values ​​(FM2j, FM21, FM22), - combining, according to information enabling the identification of at least one of said regions, said at least two sets of secondary latent values ​​to obtain said at least two decoded regions. Figure for the abstract: Fig. 3

Description

Title of the invention: Method and device for encoding and decoding images. Prior art.

[0001] The invention relates to the general field of coding one-dimensional or multidimensional signals. It relates more particularly to the compression of digital images or videos.

[0002] Digital videos are generally subject to source coding aimed at compressing them in order to limit the resources required for their transmission and / or storage. Numerous coding standards exist, such as the ITU / MPEG standards (H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.) and their extensions (MVC, SVC, 3D-HEVC, etc.). In these approaches, image encoding is generally performed by predicting pixels using previously encoded and then decoded pixels present in the image being encoded, in which case it is called "Intra prediction," or previously encoded images, in which case it is called "Inter prediction."

[0003] In addition to these classic approaches, approaches based on artificial intelligence, and in particular neural networks, tend to develop.

[0004] Some approaches use the concept of an "autoencoder." Autoencoders are learning algorithms based on artificial neural networks that allow the construction of a new representation of a dataset. The architecture of an autoencoder consists of two parts: the encoder and the decoder. The encoder is made up of a set of neural network layers that process the data to construct new representations called "encoded" representations, also known as "latent representations." In turn, the decoder's neural network layers receive these representations and filter them to try to reconstruct the original data. The differences between the reconstructed data and the initial data allow the error made by the autoencoder to be measured. Training consists of modifying the autoencoder's parameters to reduce the reconstruction error measured on the different samples of the dataset.The performance of such autoencoder-based systems comes at the cost of a considerable increase in memory footprint and complexity compared to conventional approaches such as those offered by compression standards. They can have millions of parameters and may require on the order of a million MACs (multiplication-accumulation) to decode a single pixel. This makes such decoders far more complex than conventional decoders. which could hinder the adoption of machine learning-based compression. Furthermore, the neural networks used in the encoder and decoder are different. US patent 2022 / 0385907 describes such an autoencoder system.

[0005] More recently, a simple image encoding technique based on a neural network was described in the article "COOL-CHIC: Coordinate-based Low Complexity Hierarchical Image Codée" by T. Ladune et al. (https: / / arxiv.org / abs / 2212.05458). The proposed encoding technique involves fitting a neural network to an image, quantifying the network weights, and transmitting them along with a latent representation of the signal. At decoding time, the neural network is decoded and applied to the decoded latents for each pixel position to reconstruct the image. However, such a technique remains relatively inefficient in terms of compression.

[0006] One drawback of the classical and neural approaches described above is that they do not allow, in addition, for the coding and decoding of spatial areas independently of each other, which limits the quality of the different regions and the overall quality of the decoded image.

[0007] There is therefore a need for a solution that improves upon the classical and neural approaches described above. Summary of the invention

[0008] The invention relates to a method of encoding an image comprising a plurality of regions, called regions to be coded, comprising the following steps: - generation of latent values ​​called primary values; - obtaining a plurality of neural networks, called synthetic neural networks, defined by a set of synthetic neural network parameters; - For at least two regions of said image: - processing of said primary latent values ​​identified by at least two distinct neural networks of the plurality, to produce as output at least two sets of respective secondary latent values; - combination, based on information enabling the identification of at least one of said regions, of said at least two sets of secondary latent values ​​to obtain said at least two decoded regions; - updating at least part of said primary latents and / or said synthetic neural network parameters, based on a coding performance measure; - coding at least part of said primary latents and / or said synthetic neural network parameters, as a function of a coding performance measure.

[0009] The invention also relates to a method for decoding an image comprising a plurality of regions, called regions to be decoded, said decoding method comprising the following steps: - obtaining latent values ​​called primary values, representative of the said image to be decoded; - obtaining a plurality of neural networks, known as synthetic neural networks; - For at least two regions of said image: - processing of said primary latent values ​​by at least two distinct synthetic neural networks of the plurality, to produce as output at least two sets of respective secondary latent values; - combination, based on information enabling identification of at least one of said regions, of said at least two sets of secondary latent values ​​to obtain said at least two decoded regions.

[0010] For the purposes of this invention, encoding, or "coding," means the operation of representing a set of samples, or pixels, in a compact form carried, for example, by a digital binary stream. "Decoding" means the operation of processing a digital binary stream to produce decoded samples.

[0011] By "image" we mean a signal of at least two dimensions, consisting of a set of samples (two-dimensional image, stereoscopic image, multiscopic image, image associated with a depth map, video, etc.) In the case of a two-dimensional image, the position of the sample, or pixel, is located by its coordinates in abscissa (x) and ordinate (y).

[0012] By "latent values" is meant an abstract representation of the signal comprising a plurality of variable scalar data, for example, real numbers or integers, signed or unsigned. The abstract representation of an image or sequence may, for example, include one or more feature maps of latent values. The primary latent values ​​are those applied to the input of the first synthesis neural network. The secondary latent values ​​are those corresponding to the output of the last synthesis neural network and are applied to the input of the combination module.

[0013] By "combination of said at least two sets of latent values", we mean any operation aimed at producing a final set of latent values ​​from at least two sets of secondary latent values ​​as input. The combination can, for example, be performed value by value (addition, multiplication, neural network, etc.), between two latent values ​​coming respectively from the two sets of latent values. According to another example, it can be done by blocks of latent values.

[0014] The term "neural network" refers to a neural network such as a convolutional neural network, a multilayer perceptron, an LSTM (Long Short Tertiary Memory), etc. The neural network is defined by at least one layer of artificial neurons and at least one activation, weighting, or addition function. For example, a layer might compute a function of the form y = f(Ax+b), where y and b are N-dimensional vectors, x is an M-dimensional vector, A is an MxN matrix, and f is the activation function.

[0015] By "neural network parameter" is meant one of the values ​​that characterizes the neural network, for example a weight associated with one of the neurons (filter coefficient, weighting, bias, value affecting the functioning of non-linearity, etc.)

[0016] By "processing by a synthetic neural network" is meant the application of a function expressed by a synthetic neural network to input latent values ​​to produce output latent values ​​representative of a set of pixels to be encoded (resp. decoded). In general, the output latent values ​​constitute a plurality of latents. In a particular case, the output latent values ​​are values ​​(for example (R,G,B) or (Y,U,V)) of the pixel to be reconstructed, before the combination step.

[0017] By "information enabling the identification of at least one of said regions," we mean any type of information enabling the identification of the position of a region (contour, number of pixels, position of the first pixel, etc.). Of course, if the image has two regions, identifying the first allows the second to be deduced by subtracting the first region from the image. It is therefore not necessary to have the segmentation of both regions; one is sufficient.

[0018] By "performance measurement," we mean a measurement between at least one value of a sample to be encoded and a decoded value of said sample. The measurement may, for example, assess distortion or perceptual error. It may be performed on one or more samples (for example, a current sample, or the current image, etc.). The measurement may also include a measurement of throughput, particularly associated with the encoding of the synthetic neural network and / or the encoding of the feature maps of the first group. The measurement may be a joint measurement of throughput and distortion through their weighting. As is well known in the prior art, the value of this measurement is generally minimized until a target value is reached.

[0019] Generally speaking, the steps of an encoding or decoding process should not be interpreted as being linked to a notion of temporal succession. In other words, the steps can be carried out in a different order than that indicated in the independent claim of encoding or decoding, or even in parallel.

[0020] The coding method according to the invention thus encodes a segmented image into several regions from a representation of this image in the form of latent values. The different regions of the image are processed by a plurality of distinct synthetic neural networks. In this way, it is possible to obtain a representation of the image that respects the characteristics of the different regions. The decoding method according to the invention decodes the different regions of the image from the representation of the image in the form of latent values ​​and the identification of the different regions, each region advantageously being processed by one or more distinct neural networks from the plurality before combining the different regions.

[0021] The decoding method (and symmetrically the encoding method) may further include one or more of the following optional features, taken individually or in any technically possible combination:

[0022] - said information enabling the identification of at least one of said regions includes Segmentation information is associated with the image to be decoded. Advantageously, in this method, the segmentation information allows the identification of the different regions that must be specifically synthesized for decoding (or encoding) by one or more synthetic neural networks. For example, the segmentation information indicates a division of the image into a plurality of regular areas of identical size and shape: the image can be divided into rectangular areas with horizontal and vertical boundaries, or divided into areas corresponding to samples (or blocks) processed in a lexicographical order. In this case, the segmentation information can indicate the beginning and / or end of the set. As another example, a region can be of arbitrary size and shape. In this case, the segmentation information can indicate the shape and size of the region.

[0023] - said information enabling the identification of at least one of said regions includes Combination information associated with the image to be decoded, allowing the combination of said at least two regions; such combination information advantageously includes one or more data indicating how to use the two sets of secondary latent values, this method potentially including, but not limited to: averaging the two input sets, masking, or weighting the respective importance of the input sets, allowing for weighted blending between the different input sets. The combination information may thus include data indicating a weighting factor, a masking factor, etc., to improve the image by taking into account its content, for example, occlusal zones. Static areas, transparent areas, or areas of high motion or texture in the image require specific processing. This improves image processing.

[0024] - said combination information and said segmentation information are grouped into a combination map; such a map is hereafter referred to as a "combination map" or "combination latent map"; advantageously, in this mode, a combination map allows the identification, pixel by pixel, of the different regions of arbitrary shape and size that must be combined from the output of the synthesis neural networks, as well as how they should be combined. For example, a map includes a value mask that can indicate both the pixels belonging to the different regions to be combined and the weighting factor to be used on each region: if the mask value is 0, the pixel and its associated latent values ​​do not belong to the region; otherwise, the value contains the weighting factor to be used. The regions can be of random shape and size. A map can be of the same resolution as the image, or of lower resolution.For example, a map can define two regions: a background processed by a first network and a foreground processed by a second network, along with a weighting factor to be applied to each region during combination. In another example, a map can define a single region and a weighting factor to be applied to that region during combination. The weighting factor (combination information) can be included with the map or embedded within it. It's important to note that there are no restrictions related to the type of segmentation algorithm used. Latent values ​​are processed in parallel by the first and second synthesis networks. Then, the regions are reconstructed by the decoder by performing the appropriate combination of latent values.In a first example, there is a single two-dimensional map indicating which values ​​belong to the first region (for example, the value 1) and which values ​​belong to the second region (for example, the value 0). The values ​​can then be mutually exclusive. In another embodiment, there are two maps: the first map indicating, for example, which values ​​to keep from the output of network 1 (which processes the background), while the second map indicates which values ​​to keep from the output of network 2 (which processes the foreground).

[0025] - the combination information is a blending parameter (combination Advantageously, according to this method, a combination map can indicate the weighting to be applied to the sets of secondary latent values. For example, according to the previous example, a combination map can indicate weighting information (al) to be applied to the output of the first synthesis network, this weighting being able to take values ​​between 0 and 1, pixel by pixel. We will note If the values ​​are 0 or 1, this allows unwanted areas to be eliminated by multiplication or, conversely, to be selected.

[0026] - the combination is performed by a combination neural network (CNR); Advantageously, combination functions can be more complex than the previously mentioned operations of averaging, exclusion, selection, blending... In this case, one or more neural networks, which may or may not be transmitted, can be responsible for the combination.

[0027] - the processing step is preceded by a latent value identification step primary latents are associated with the regions to be decoded; advantageously, in this mode, synthetic neural networks can be applied only to the relevant primary latent values; thus, it is not necessary to apply the network to a region not targeted by that neural network. This avoids unnecessary operations on the primary latent values.

[0028] - combination information and segmentation information are obtained by decoding latent combination and segmentation values ​​and processing said values ​​by a neural network for decoding combination and segmentation information; advantageously, according to this method, a neural network for decoding combination and segmentation information can be used. This has the effect of offering the possibility of obtaining more precise combination and segmentation information.

[0029] - combination information and segmentation information are obtained by primary latent analysis. Advantageously, with this method, it is not necessary to transmit the combination and segmentation information, which can be reconstructed by the decoder through primary latent analysis. This analysis may include, for example, a primary latent segmentation step.

[0030] - at least some of the parameters of a synthetic neural network and / or The combination is decoded; advantageously, in this mode, the neural network parameters are transmitted in the stream. The module can be transmitted in a quantized form and encoded in a compact form using any quantizer and entropy coder accessible to a person skilled in the art. A format analogous to that of the MPEG-7 NNR standard can be used.

[0031] - at least some of the parameters of a synthetic neural network and / or combination is obtained from predetermined values; advantageously according to this mode, the parameters of the neural network are obtained, for example from a network, or from a storage memory.

[0032] - at least some of the parameters of a synthetic neural network and / or The combination is decoded according to reference parameters. Advantageously, according to this mode, the parameters of the neural network are decoded by taking into account Reference parameters can be stored in accessible memory of the decoder or obtained from a network. This allows for efficient reduction of storage space or network transmission bandwidth: some parameters can be coded separately, while others are not coded at all because they are available in the reference parameters. According to non-limiting embodiments, which can be combined: • The reference parameters of a synthesis or combination neural network can constitute a set of network variants: convolution type, attention modules, etc. Thus, only these variants need to be known by the decoder. The parameters associated with these variants can, for example, be encoded, inserted into the stream, and decoded by the decoder, which can then reconstruct the network from the reference parameters. • The parameters of a network are coded in a way that complements the parameters of a reference network. Thus, the decoder only needs to decode these residual parameters and then correct them by adding, multiplying (or combining them in any other known way) to those of the reference neural network. • Part of the parameters of a reference network is reused for a synthesis or combination network. For example, a complete layer of an oversampling network can be copied from the reference network; the parameters of this layer are therefore neither encoded, nor transmitted, nor decoded.

[0033] - at least part of the parameters of a synthesis and / or combination network is obtained based on a network indicator transmitted in the stream; it is thus possible to indicate a network to use for decoding from among a plurality of possible networks known to the decoder.

[0034] - a distinct combination and segmentation information is associated with each synthetic neural network;

[0035] - the plurality of synthetic neural networks includes a first network of pretreatment of all primary latents.

[0036] - the combination network is followed by a post-processing network.

[0037] Correspondingly, the invention also relates to an coding device and a decoding device.

[0038] The characteristics and advantages of the coding or decoding process apply in the same way to the coding or decoding device according to the invention and vice versa.

[0039] The invention also relates to a computer program on a recording medium, this program being capable of being implemented in a computer or an encoding or decoding device according to the invention. The program includes instructions adapted to the implementation of the corresponding process. This program can use any programming language and be in the form of source code, object code, or code intermediate between source and object code, such as in a partially compiled form, or in any other desirable form.

[0040] The invention also relates to a computer-readable information or recording medium comprising the computer program instructions mentioned above. The information or recording medium may be any entity or device capable of storing programs. For example, the medium may include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a floppy disk or a hard drive, a DNA sequence, or flash memory. Furthermore, the information or recording medium may be a transmissible medium such as an electrical or optical signal, which may be transmitted via an electrical or optical cable, by radio link, by wireless optical link, or by other means. The program according to the invention may, in particular, be uploaded to a network such as the Internet.Alternatively, each information or recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of a process according to the invention. Brief description of the figures

[0041] The invention will be better understood with the aid of the following description, given solely by way of example and made with reference to the accompanying drawings in which: - Figure [1] schematically represents a decoding device used in the context of the invention, - Figure [Fig. 2] schematically represents a coding device used in the context of the invention, - Figure [Fig. 3] schematically represents a decoding device according to a first embodiment of the invention, - Figure 4 schematically represents a decoding device according to a second embodiment of the invention, - Figure 5 schematically represents a module for decoding information, combining and segmenting it according to one embodiment of the invention. - Fig. 6 illustrates an example of a network for decoding information by combining and segmenting. - Fig. 7 illustrates an example of a network for decoding information by combining and segmenting. - [Fig.8] is a flowchart representing an example of a decoding process that can be implemented by the decoding device of [Fig.1], - Figure 9 is a flowchart representing an example of a coding method that can be implemented by the coding device in Figure 2. Detailed description of the invention

[0042] Figure 1 schematically represents a decoding device used within the framework of the invention,

[0043] The decoding device DEC of [Fig.1] decodes the coded information from a binary stream BS to generate a decoded signal Id, which can be, for example, an image from a sequence of digital images.

[0044] The DEC decoder receives as input the data necessary to decode the Id signal:

[0045] - FMci encoded data corresponding to one or more characteristic maps encoded, for example a plurality of I multiresolution feature maps.

[0046] - the Wck-coded parameters of at least one processing neural network, on the figure includes a plurality of L synthesis neural networks, denoted RSdi, a combination neural network, denoted RdCO, and optionally a final network denoted RdF. Other neural networks can be used, such as neural networks:

[0047] - for predictive decoding of latents;

[0048] - of oversampling and / or processing of latents;

[0049] - etc.

[0050] The coded information is extracted from the BS stream which can be received on a communication network, obtained from an accessible memory of the decoder, etc.

[0051] The DEC decoding module includes an FMD module for decoding feature maps, an NND module for decoding neural networks, and a SCOD module for decoding combination and segmentation information.

[0052] The SCOD decoding module decodes segmentation information and combination information Ce and outputs the decoded combination and segmentation information Cd. The segmentation information and combination information are more compactly referred to as "combination and segmentation information". This decoding can take the form of a lossless decoder (for example, using the JPEG-LS algorithm defined by the international standard ISO / IEC 14495-1). It can also take the form of a synthetic neural network. The decoded segmentation information makes it possible to identify at least one region to be decoded, which can be in the form of a rectangle or a juxtaposition of A segment can be a rectangle of contiguous pixels, or even a region of arbitrary shape and size. For example, the segmentation operation can identify two different areas within the image signal. In one example, one corresponds to the background of the image and the other to the foreground. Combining information is associated with the segmentation information. This could be, for example, a multiplication factor to be applied to a region. Combining and segmenting information is represented, for example, as a set of maps, or masks, each identifying an area. Each mask is associated with a value that can be different from the values ​​associated with the other masks, or identical (for example, 0.5 and 0.5 in the case of two masks). This value can be a combining value, such as a multiplication factor for the area.

[0053] The SCOD module is optional. Indeed, the combination and segmentation information can be deduced from the latent maps, or known to the decoder: for example the segmentation can be fixed, and a default combination information associated with each zone.

[0054] The FMD decoding module decodes so-called primary latents, coded FMci. In one embodiment, it is a conventional decoder, for example, an entropic decoder of the JPEG or MPEG type, etc., which outputs a set of at least one decoded latent map, denoted FMdi. In another embodiment, the FMD module may include a latent synthesis neural network for this purpose. The latents to be decoded are, for example, entropically decoded. In another embodiment, the primary latents decoded by the FMD module, numbering I, are denoted FMdi and indexed by i. They can be in the form of 2D vectors or matrices with sizes related to that of the decoded or undecoded image; in this case, they are called latent value maps. They can be organized in a pyramidal or hierarchical manner and have different resolutions.

[0055] The neural network decoding module (NND) decodes the encoded parameters of the synthesis, combination, and optionally final neural networks, noted after Wdk decoding. This module is optional. Indeed, the networks (synthesis and / or combination) may be known to the decoder (stored in its memory, downloaded, etc.). For example, a simple indicator is provided to choose a network from among several reference networks.

[0056] The RS synthesis module, comprising the RSD1 synthesis neural networks, uses as input the decoded FMDi maps, referred to as primary maps, and produces as output a set of maps referred to as secondary maps, denoted FM2i. The RS module will be detailed later in the description. It comprises at least two networks including one or more synthesis layers. These networks are preferably of the convolutional neural network (CNN) or fully connected neural network (MLP) type. The convolutional layers may It may or may not include a residual structure and an attention module. It may also include oversampling, undersampling, etc. layers.

[0057] The RdCO module, comprising the combination neural network, takes as input the secondary feature maps FM2i, for example in the figure a plurality of sets denoted FM2m, and produces as output a set of composite latents denoted FMco. RdCO can be in the form of a neural network and include weighting and nonlinearity elements. It can include a multi-layer programming (MLP). It can also include convolutional layers, including convolutional elements, with or without a residual structure and with or without an attention module.

[0058] The optional RdF module corresponds to a final neural network, responsible for reconstructing the image Id from latents generated by the combination module. According to the example in [Fig. 1], the image is obtained in the form of three decoded components, for example R, G, B or Y, U, V. It can be positional (the samples are reconstructed pixel by pixel, as in the aforementioned prior art of COOL-CHIC) or convolutional, etc.

[0059] The DEC decoding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be realized by the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to perform the functionalities of the module concerned, in particular as described below, when these instructions are executed by the processor.

[0060] Fig. 2 schematically represents a coding device used in the context of the invention.

[0061] The ENC coding device of [Fig. 2] receives as input the image I, for example, one or more two-dimensional images from a video sequence, and produces as output coded parameters. These parameters include FMc encoded data corresponding to one or more encoded latents, for example, a plurality of multiresolution maps, optionally the Wck coded parameters of at least one synthesis neural network and / or a combination neural network and / or a combination and segmentation information decoding neural network, referred to as combination and / or synthesis and / or combination information decoding neural network parameters, and optionally the coded segmentation information, SCOC. As mentioned in the description of [Fig. 1], other neural networks can be used.

[0062] It should be noted that the generation of segmentation information can be carried out by any technique within the reach of a person skilled in the art: a manual definition can be Performed by an operator, segmentation can also be automatic or semi-automatic depending on the segmentation algorithm used. There are no restrictions related to the type of segmentation algorithm used.

[0063] The ENC coding module includes, for a current image, an NNC module for coding neural network(s) parameters, an FMC module for coding latents, a SCOC module for coding combination and segmentation information, an RD-OPT evaluation module, an INIT / MAJ initialization and update module.

[0064] Furthermore, in a conventional manner, the encoding module includes a decoder similar to the DEC decoder previously described, in order to reconstruct an encoded and then decoded signal 1'. This decoder is denoted DEC'.

[0065] During the coding training or construction process—that is, as long as the performance evaluation stage is not satisfactory—the coding modules perform a coding simulation, followed by decoding, for the RD-OPT evaluation module. Subsequently, they perform the actual coding of the data. As is known, the coding simulation can be identical to the actual coding, or it can provide an approximation of it.

[0066] The INIT / MAJ module is responsible for initializing and updating the latent values, neural network parameters, and combination and segmentation information. It updates the image values ​​to be encoded based on the results of a performance function. Once the latent values, combination and segmentation information, and neural network values ​​are established, the latent values, combination and segmentation information, and neural networks can be encoded. The INIT / MAJ module can update a subset of the latent values ​​and neural network parameters. The unupdated elements may, if necessary, retain values ​​known to the decoder.

[0067] The RD-OPT module performs an evaluation and minimization of coding performance. The evaluation function is, for example, of the rate-distortion type. The distortion can be evaluated between the original signal I and the signal I' resulting from coding or a coding-then-decoding simulation. The rate can include the sum of the rates related to the contributions of Ce, Wck, and FMci, or only a subset. Minimization can be performed by gradient descent and updating the parameters of the neural network(s), or any other optimization method within the grasp of a person skilled in the art. When the coding performance is satisfactory, the latent values ​​and neural network values ​​are established.

[0068] The ENC coding device can be implemented by means of an electronic device comprising a processor and a memory, not shown; each of the modules mentioned above can then be implemented through the cooperation of the processor and computer program instructions stored in the aforementioned memory and designed to perform the functions of the module concerned, including as described below, when these instructions are executed by the processor.

[0069] Fig. 3 schematically represents a more detailed decoding device than that of Fig. 1, according to a first embodiment of the invention.

[0070] The modules are similar to those in [Fig. 1], but here they are assumed to be applied to the processing of two regions, illustrated on the CDk segmentation maps (CD1, CD2) resulting from the decoding of the combination and segmentation information Ce. These are presented in the figure, exemplarily, as two masks, one representing a first region, for example a foreground (black ellipse on a white background) and the other a second region (black background). Naturally, the segmentation information can take any other form known to those skilled in the art, such as: - a rectangle or a juxtaposition of contiguous pixel rectangles. - Block partitioning in accordance with the HEVC standard in the form of a quadtree or any other enhanced form - a concatenation of rectangles or geometric partitions as described in the VVC standard. - a division of the image into regular areas, for example of identical size and shape (except possibly at the edges of the image). For example, the image can be divided into rectangular areas with horizontal and vertical borders. Such a division corresponds to the concept of "tiles" implemented by encoding standards such as H.264 / AVC, H.265 / HEVC and H.266 / VVC. - samples (or blocks) traversed in a lexicographical order. Such a division corresponds to the concept of "slices" implemented by coding standards such as H.264 / AVC, H.265 / HEVC and H.266 / VVC. - semantic zones, for example a background and a foreground.

[0071] To a region to be decoded (for example that corresponding to the Cdl map), a synthetic neural network is associated with the input of which the decoded latents, called primary latents, are applied.

[0072] For example, the latents to be decoded are entropically decoded by mirroring the entropy encoder possibly used by the encoder, and then undergo a dequantization and / or dynamic range adjustment step by multiplication. The resulting decoded latents (FMdi) are in the form of 2D vectors or matrices whose sizes are related to those of the decoded or undecoded image. Preferably, there are 7 decoded latents; advantageously, they do not all have the same number of values. They can be arranged in a pyramidal structure, meaning that each latent has a number of values ​​less than that of the latent value preceding (or following) it in the pyramid.

[0073] According to the embodiment shown in [Fig. 3], the FMdi latents are processed by a first RDdO neural network, or preprocessing network, of the RS module, which may be of the convolutional neural network (CNN) or fully connected neural network (MLP) type. Preferably, this network produces a number of latents greater than or equal to the number of FMdi latents, for example, 16 or 40 latents. The latents obtained may or may not have the same resolution. In the first case, the RDdO network may include an oversampling module, conforming, for example, to that used in the aforementioned prior art "Cool-Chic" or of the bicubic type, etc.

[0074] The latents obtained are processed by the RSdi and RSd2 networks. These networks are preferably of the convolutional type (CNN) or of the fully connected type (MLP).

[0075] According to this embodiment, the set of latent values ​​is processed by both networks, independently of the segmentation; that is, all the values ​​of a latent value are processed by each of the neural networks. In other words, the RS synthesis module is unaware of the segmentation.

[0076] The respective RSdi and RSd2 networks produce two sets of latents, called secondary latents, FM2i and FM22. These sets of latents preferably consist of latents organized identically; that is, in this case, they are identical in number and homogeneous in resolution. In a preferred mode, there are three of them, with a constant resolution equal to the dimensions (height, width) of the image. In another mode, there are three of them: one (Y) is the same size as the image, and the other two (U and V) have half the resolution in each dimension.

[0077] These two sets of latent values ​​are processed by a combination neural network RdCO. It receives as input the latent values ​​FM2i and FM22 and combination and segmentation information Cdk in the form of two maps Cdi and Cd2 corresponding to segmentation information and combination information, respectively. For example, these two maps contain a set of values ​​positioned at a value ai (in CdJ) or a2 (in Cd2), respectively. When the pixels at position (x,y) are part of their respective regions, the value of aieta2 is non-zero. If the pixels are outside the region designated by their respective map, they can be set, for example, to a value of zero. In a simplified mode, the combination network takes the form of two multipliers and one adder.The combination and segmentation information takes the form of the multiplier values ​​ai, a2 obtained from the Cdk map (Cdi or Cd2) whose resolution is identical to that of the FM2i and FM22 latents. The multipliers are applied point by point to each . The value of each latent value is then multiplied, and the resulting latent values ​​are added together to provide M latent values ​​FMd, that is, for each value of latent value FMd at position (x,y), we have:

[0078] FMd (x,y) = cq(x,y)* FM2j (x,y) +a2 (x,y)*FM22(x,y)

[0079] In one variant, these multipliers can advantageously be derived from one another. For example, "2 ~ 1" f / i, so the preceding calculation can be written in the form:

[0080] FMd(x,y) = ai(x,y) * (FM2i(x,y)- FM22(x,y)) + FM22 (x,y)

[0081] This expression saves a multiplication by latent value.

[0082] The multipliers can take a binary form (“1 / 2 = 0 or “1 / 2 = 1”, in which case the combination takes the form of a combination, which allows certain areas of the latents to be treated by a single network.

[0083] Advantageously, the combination network can be more complex and take the form of a neural network, incorporating convolutional processing including nonlinearities. The combination and segmentation information is then applied by modifying biases, gains, or convolutional filters according to the latent region described by the segmentation and combination information. This network can be decoded by the NND module, in part or in full, or obtained from the decoder by other means.

[0084] These latents obtained are optionally processed by a final network (FN), for example a post-filtering network of the loop filtering type, or enhancement network to provide a decoded image in the desired format (YUV, RGB,...).

[0085] If they are not processed by the RdF network, the latent FMd at the output of the combination network represent the decoded image Id in the desired domain, for example YUV or RGB.

[0086] The combination and segmentation information can take the form of an image, for example monochrome, coded by a well-known encoder (JPEG, HEIF, PNG,...) where each pixel of the image is associated with an intensity proportional to the multiplier af / 2 described above.

[0087] According to a variant, not shown, only one card is offered to the combination module, containing, by region, the values ​​of the multipliers of that region.

[0088] Figure 4 schematically represents a decoding device according to a second method of embodiment of the invention,

[0089] According to this embodiment, a subset of the latents can be processed based on the segmentation information. For example, if the segmentation indicates that the processing for a part of the image does not consider network 1, then only network 2 will process the considered portion of the latents. In this case, it is necessary that the RS synthesis module has the segmentation information (which corresponds to the dashed arrow in the figure).

[0090] In this case, the segmentation and combination information from the SCOD module is applied to the latents in the RS module. These latents correspond to the decoded FMd latents or those processed by a set of initial synthesis layers (RSdO). The Cdi and Cd2 maps, corresponding to segmentation and combination information, are applied respectively to the input latents of RSD1 and RSD2. When the values ​​of the Cdl or Cd2 maps are zero, the corresponding latents can be omitted from the RSdl or RSd2 synthesis processing to limit the number of operations. Non-zero values ​​are applied to these latents or to the RdCO combination network.

[0091] The RdCO network can be limited to a simple addition in the case where the maps are fully applied in the synthesis.

[0092] Fig. 5 schematically represents a combination and segmentation information decoding module according to one embodiment of the invention.

[0093] Combination and segmentation information Ce is presented in coded form to the combination and segmentation information decoding module SCOD. It is decoded by a submodule denoted DECe, for example, a well-known state-of-the-art decoder (JPEG, HEIF, PNG, etc.) or any image decoder, for example, entropic (JPEG LS, etc.). This yields a plurality of decoded intermediate combination and segmentation latents LCSi, which are processed by a combination information decoding neural network RDIC. This network outputs combination and segmentation information, also called combination latents, or combination maps, denoted Cdi and Cd2, which are applied by the RDCo module.Preferably, these combination latents have the same resolution as the FM2 latents; If the resolution of the intermediate combination and segmentation LCS latents is different from that of the FM2 latents, then resampling is applied to obtain the expected WxH resolution of the combination maps, Cdi and Cd2, as described in support of Figures 6 and 7.

[0094] Figure 6 illustrates an example of a combination and segmentation information decoding network according to one embodiment.

[0095] In this mode, an intermediate LCSi combination and segmentation latent composed of integer values ​​is decoded by the DECe decoder. It contains (w / n) x (h / n) values, where w and h correspond to the width and height of the latent to which the combination information ai, a2 is applied. In [Fig. 6], n=2. This latent is processed by an oversampler, for example, a bilinear, bicubic, nonlinear, or any other type of interpolator, to obtain, as an oversampling output, a latent, or combination map Cdi / 2, at the expected resolution WxH. This map has for each position a value aioua2 The value a2 can be decoded, or deduced by a2= 1- ai.

[0096] Figure 7 illustrates an example of a combination and segmentation information decoding network according to another embodiment.

[0097] In this embodiment, several intermediate combination and segmentation latents (LCSi, LCS2...) are decoded by the DECe decoder. They contain (w / n) x (h / n) and (w / n") x (h / n") values, respectively. Subsequently, the intermediate latents are resampled by two UP modules to obtain a corresponding number of LCS'i and LCS'2 latents with the same resolution (w / n") x (h / n"). These latents are processed by an NN neural network from the RDIC module, for example, of the transposed convolution type, having one or more linear or non-linear layers, as many inputs as there are intermediate latents, and as many outputs as desired: in the figure, three outputs are shown, corresponding to the maps / latents CdiCd2, Cd3, with respective combination parameters ai, a2, and a3. The network is configured to provide these latents at the resolution wxh.For example, a stride of size n” will be used on a transposed convolution linear layer with 2 inputs and 3 outputs as shown in the figure. At the output of this convolution network, an optional nonlinearity, for example of the softmax type, allows the value of the combination weights to be adjusted and restricted within a given dynamic range.

[0098] Fig. 8 is a flowchart representing an example of a decoding process that can be implemented by the decoding device of Fig. 1.

[0099] The decoding described relates to an image to be decoded, comprising at least two regions that one wishes to decode differently.

[0100] During a step F20, the sub-streams are extracted from the encoded stream. They contain respectively, for the image, the encoded representations of the current latents FMc; and optionally parameters of the synthesis neural networks RSDi, the combination and segmentation information decoding network RDIC and the final network RdF, and segmentation and combination parameters Ce.

[0101] In one embodiment, the portion of the BS stream containing the neural networks is received and decoded. In another embodiment, a portion of the neural network parameters is stored in memory, for example in ROM or RAM. This storage space can contain a plurality of parameters, allowing for the creation of neural network variants. An indicator can then be read from the stream to inform the decoder of which network to use or which to construct from a combination of parameters. In one embodiment, the Wck parameters of a network can be decoded in relation to a reference network stored in memory. For example, the received parameters are relative correction values. to the reference network, which can be decoded using an entropic code and combined with the parameters of the reference network.

[0102] The two embodiments mentioned above can be combined: a reference indicator can then be read from the stream to inform the decoder of the reference network to be used, and the parameters received are used to reconstruct at least one layer of a network from the reference network.

[0103] Combination and segmentation information can likewise be received in the stream and decoded, or stored in whole or in part in memory.

[0104] During step F21, the segmentation information and the combination information are decoded. For this decoding, an entropy decoder, or a JPEG decoder, etc., corresponding to the encoding that was performed in the encoder, is used for example.

[0105] During step F22, the latent values ​​FMd are generated by decoding the FMCI values. For this decoding, an entropy decoder and an inverse quantizer are used, for example. A neural network for predicting latent values ​​can also be used.

[0106] In some embodiments, the latents take the form of FMd maps; at a resolution less than or equal to that of the image Id to be reconstructed, or at the same resolution but less than the resolution of the image. An FMd latent comprises, for each sample, a vector of scalar values ​​to be decoded. Even if the data are organized differently, it is assumed hereafter that there are a plurality of latent maps to be decoded.

[0107] During step F23, the neural networks are decoded. The Wdk parameters of the synthesis, combination, and final neural networks are generated by decoding the Wck values ​​of the BS stream. Any known decoding technique corresponding to the encoding technique used by the encoder can be used for this purpose, for example, a format analogous to that of the MPEG-7 NNR standard. The decoding neural networks are similar to their counterparts used by the encoder; that is, they have the same structure and the same parameters, except for the encoding, which can be lossy or lossless.

[0108] During step F24, the synthesis neural networks are implemented. The M decoded FMd maps are processed in parallel or sequentially by the RSDi synthesis neural networks to generate secondary FM2 latents as output, the number of which may differ from M. There are as many synthesis neural networks as there are regions to be processed in the image. According to one embodiment, corresponding to [Fig. 3], a synthesis neural network processes all the latent values ​​it receives as input. According to another embodiment, corresponding to [Fig. 4], it processes only the values ​​that correspond to the region it is to process.

[0109] During step F25, the combination neural network is implemented. Secondary latents are applied as input to the network, which also receives combination and segmentation information. The module takes this information into account to combine the secondary latents, as explained previously.

[0110] In an optional step F26, the final neural network RdF is implemented to generate a synthesis of the decoded samples from the latent maps resulting from the combination. For example, an MLP or convolutional neural network, or a combination of the two, transforms an input vector into an output vector representative of the sample being processed, for example, a pixel. For example, extraction is performed in the oversampled maps based on the coordinates (xn, yn) of the sample Pn of the input signal. The pixels to be decoded are, for example, processed sequentially, from n=l to n=N. When all the pixels of the image have been processed, the decoded signal corresponding, for example, to image I (Pdn) is available.

[0111] [Fig.9] is a logic diagram representing an example of a coding process that can be implemented by the coding device of [Fig.2]

[0112] The encoding takes place in two main phases:

[0113] In a first phase, called the construction phase, a learning process is performed to determine, for image I, the values ​​of the FMo maps and optionally the combination and segmentation information Co and the parameters Wo of the neural network(s) to be transmitted to the decoder, in order to optimize an overall cost function. The learning is, for example, performed by gradient descent, followed by an update of this data and these parameters. As is known in the prior art, the cost function can be of the rate-distortion type, or rate, or distortion; the distortion can include a perceptual metric. To measure the rate R, it is necessary to simulate the encoding of the FMo latents, and then to measure the associated encoding rate (the size of the flow). According to one embodiment, the encoding of the parameters Wo and / or the information Co is not simulated because their influence is less significant than that of the latents.In one embodiment, the encoding of parameters Wo and / or information Co is also simulated, and the associated data rate (the size of the stream) is measured. To measure the distortion D, it is necessary to simulate the encoding and then the decoding of at least a part of the image I, to obtain at least one pixel resulting from a simulation of encoding and then decoding, and then to measure the difference between this part of the input image and a corresponding part of the encoded and then decoded image F.

[0114] Then, during a second phase, called the coding phase, the FMO latents and optionally the Wo parameters and Co information are encoded before transmission or storage. They constitute the compressed representation of the input signal I.

[0115] We will now describe the steps of a process according to an embodiment of the invention.

[0116] During a step E20, an image I to be coded, comprising a plurality of WxH samples, or pixels, is provided as input to the process.

[0117] During step E21, the latent FMo, the neural network parameters Wo, and the segmentation and combination information Co are initialized. Subsequently, they must be optimized during the construction phase.

[0118] According to one embodiment, the FMo latents have a resolution less than or equal to that of the input signal I and therefore, for at least one of them, have a number of values ​​to be encoded less than WxH. According to another variant, the first FMo latent has the resolution of the images and each subsequent card has half the resolution of the previous one. According to another embodiment, several FMo latents have the same resolution, lower than that of the input signal I.

[0119] According to one embodiment, FMo, Co, and Wo are initialized by the INIT module to predefined constant values. According to another embodiment, they are initialized by the INIT module to a set of random real values.

[0120] The values ​​of FMo, Co and Wo are subsequently updated, or refined, during an E22 step, by the encoder update module during its learning.

[0121] During step E23, the FMo values, and optionally Co and Wo, are encoded by their respective FMC, SCOC, and NNC modules of the encoder. During the construction phase, this operation is a coding simulation. During the encoding phase, this operation is the actual encoding. The simulation may be identical to the actual encoding, but it may also be different (for example, simplified). For this encoding, for example, entropic coding of the quantized latent values ​​is used, and the neural network coding standard proposed by the MPEG-7 Part 17 standard, also called NNR (Neural Network Representation), is used.

[0122] During a step E24, a decoding similar to that described in support of the previous figure is performed using the FMo, Co and Wo values ​​from the coding simulation. A decoded image 1' is obtained.

[0123] During step E25, a performance measure is evaluated. For this purpose, the coding simulation rates associated with FMo, and optionally Co and Wo, are measured. In one embodiment, the cost function is of the rate-distortion type, denoted (D+L*R), where D, for example, is the root mean square error measured between the input image and the decoded image F (or the error measured on a subset of image samples). In another example, D is calculated from a perceptual function such as SSIM (for Structural SIMilariously) or MSSSIM (for Multi-scale Structural SIMilariously). SlMilarity). As long as the cost function has not reached its minimum, the performance measurement is not satisfactory, and the process is repeated from step E22. This minimization can be performed by a mechanism known as gradient descent with parameter updates during step E22 for the values ​​FMo, and optionally Co and Wo.

[0124] During step E26, if the cost function has reached its minimum, the training stops. This is followed by step E27 for generating the coded data. If a coded version corresponding to the last simulation of FMo, and optionally Co and Wo, is available, the BS stream can be generated. According to another embodiment, the actual encoding of these values ​​is performed at this step to produce the encoded parameters FMc, and optionally Ce and Wck.

[0125] In the detailed presentation of the invention given above, the terms used shall not be interpreted as limiting the invention to the embodiments set forth in this description, but shall be interpreted as including all equivalents which can be foreseen by a person skilled in the art by applying their general knowledge to the implementation of the teaching which has just been disclosed to them.

Claims

[Claim 1] Demands Encoding device (ENC) of an image (I) comprising a plurality of regions (Rdj), called regions to be coded, said device being configured to implement the following steps: - generation (INIT / MAJ) of so-called primary latent values ​​(FMo); - obtaining a plurality of neural networks (RSd;), called synthetic neural networks, defined by a set of synthetic neural network parameters (Wo); - For at least two regions (RB R2) of said image: - processing of said primary latent values ​​identified (FMd;) by at least two distinct neural networks of the plurality (RSdz, RSdl, RSd2), to produce as output at least two sets of secondary latent values ​​(FM2j, FM2i FM22) respectively, - combination, according to information allowing to identify at least one of said regions, of said at least two sets of secondary latent values ​​to obtain said at least two decoded regions; - updating (INIT / MAJ) at least part of said primary latents and / or said synthetic neural network parameters, based on a coding performance measure; - coding (FMC, NNC) of at least part of said primary latents and / or said synthetic neural network parameters, as a function of a coding performance measure.

Images