Encoding method, decoding method, training method, and related devices

Abstract

Embodiments of the present application provide an encoding method, a decoding method, a training method, and related devices. The encoding method may include: obtaining a latent representation of a target signal based on reference signals by using an encoding model; performing a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation; and performing an entropy encoding operation on the scaled latent representation to obtain a compressed result. According to the above technical solution, the quality and efficiency of encoding and decoding may be enhanced.

Description

ENCODING METHOD, DECODING METHOD, TRAINING METHOD, ANDRELATED DEVICESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application is related to, and claims priority to, PCT patent application Serial No.PCT / RU2024 / 000078, entitled "TRAINING METHOD AND RELATED DEVICES", filed on March 06, 2024.]0002] The disclosure of the aforementioned application is hereby incorporated by reference in its entirety.TECHNICAL FIELD]0003] Embodiments of the present application relate to the field of signal processing technologies, and more specifically, related to an encoding method, a decoding method, a training method, and related devices.BACKGROUND

[0004] Artificial intelligence (Al) is a rapidly growing field of technology that involves simulating human intelligence processes through machine learning, deep learning, natural language processing, computer vision, and other advanced techniques.

[0005] Al has been widely used in many fields, such as image signal processing, video signal processing, and audio signal processing. In the field of coding and decoding, Al is used for data compression such as image compression, video compression, audio compression, 3D data (like point-cloud data) compression, and so on. Video compression is taken as an example. There are a lot of approaches of applying neural networks to enhance or even replace some modules in the traditional video codec architecture. Al is expected to be used to enhance the efficiency and quality of video compression.,0006] However, the introduction of Al may not necessarily guarantee good encoding and decoding performance. In some scenarios, Al-based encoding and decoding performance may not even be as good as traditional encoding and decoding methods. There are still some issues with the application of Al in encoding and decoding.SUMMARY

[0007] Embodiments of the present application provide an encoding method, a decoding method, a training method.and related devices. The proposed technical solution may enhance encoding and decoding efficiency and quality.

[0008] According to a first aspect, an embodiment of the present application provides an encoding method. The encoding method may be performed by an encoding device.

[0009] The encoding method may include obtaining a latent representation of a target signal based on reference signals by using an encoding model; performing a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation; and performing an entropy encoding operation on the scaled latent representation to obtain a compressed result.

[0010] According to the aforementioned technical solution, the latent representation of the target signal is scaled according to the information about the hierarchy level of the target signal before being compressed. The scaling of the latent representation influences both the bitrate and the reconstruction quality. Scaling the latent representation according to the hierarchy level of the target signal may be beneficial for reducing bitrate and improving reconstruction quality, thereby enhancing encoding and decoding efficiency and quality.

[0011] The method may be applied to data encoding using the hierarchical coding scheme.,0012] The reference signals refer to reference signals of the target signal. The reference signals of the target signal may be understood as the signals dependent on for encoding the target signal.

[0013] Optionally, the method may be applied to video encoding. The target signal may be a video frame. The reference signals may be reference frames of the target frames. The reference frames may be reconstructed frames. For example, the reference frames may be stored in a buffer. In this case, the reference frames may be obtained from the buffer.

[0014] In a possible implementation of the first aspect, the performing a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation, includes: performing a multiplying operation on the latent representation by a first scaling value corresponding to the hierarchy level of the target signal to obtain the scaled latent representation.

[0015] Optionally, the first scaling value may be taken from a look up table according to the hierarchy level of the target signal.

[0016] In a possible implementation of the first aspect, the first scaling value is a scalar.

[0017] According to the aforementioned technical solution, scaling value using scalar is simpler and helps reduce storage and computing resources.

[0018] In a possible implementation of the first aspect, the first scaling value is a trainable parameter.

[0019] According to the aforementioned technical solution, scaling values determined through training may be more appropriate for encoding and decoding, which is beneficial for reducing bitrate and improving reconstruction quality, therebyimproving the efficiency and quality of encoding.

[0020] In a possible implementation of the first aspect, the method further includes: transmitting a bitstream carrying the compressed result.

[0021] According to a second aspect, an embodiment of the present application provides a decoding method.(0022] The decoding method may be performed by a decoding device.

[0023] The decoding method may include: obtaining a bitstream corresponding to a target signal; performing an entropy decoding operation on the bitstream to obtain a reconstructed latent representation of the target signal; performing a scaling operation on the reconstructed latent representation according to a hierarchy level of the target signal to obtain a scaled reconstructed latent representation; and obtaining a reconstructed signal based on the scaled reconstructed latent representation and reference signals by using a reconstruction model.

[0024] According to the aforementioned technical solution, the reconstructed latent representation of the target signal is scaled according to the information about the hierarchy level of the target signal after being decompressed by the entropy model. The scaling influences both the bitrate and the reconstruction quality. Scaling the reconstructed latent representation according to the hierarchy level of the target signal may be beneficial for reducing bitrate and improving reconstruction quality, thereby enhancing encoding and decoding efficiency and quality.

[0025] The reference signals refer to reference signals of the target signal. The reference signals of the target signal may be understood as the signals dependent on for decoding the encoded target signal.[0026| Optionally, the method may be applied to video decoding. The target signal may be a video frame. The reference signals may be reference frames of the target frames. The reference frames may be reconstructed frames. For example, the reference frames may be stored in a buffer. In this case, the reference frames may be obtained from the buffer.(0027] In a possible implementation of the second aspect, the performing a scaling operation on reconstructed latent representation according to a hierarchy level of the target signal to obtain a scaled reconstructed latent representation, includes: performing a multiplying operation on the reconstructed latent representation by a second scaling value corresponding to the hierarchy level of the target signal to obtain the scaled reconstructed latent representation.

[0028] In a possible implementation of the second aspect, the second scaling value is a trainable parameter.,0029] According to a third aspect, an embodiment of the present application provides a training method.

[0030] The training method may be performed by a training device.

[0031] The training method may include: obtaining multiple training datasets including a first training dataset and a second training dataset, where the first training dataset includes one or more signal sequences, number of signals in each signal sequence in the first training dataset is a first sequence size, the second training dataset includes one or more signal sequences,number of signals in each signal sequence in the second training dataset is a second sequence size, and the first sequence size is different from the second sequence size; and training a codec model with the multiple training datasets, where the training process includes multiple training stages including a first training stage and a second training stage, parameters of the codec model are adjusted based on the first training dataset during the first training stage, and parameters of the codec model are adjusted based on the second training dataset during the second training stage.

[0032] According to the aforementioned technical solution, different training datasets including signal sequences with different sizes are used to adjust the parameters of the codec model during different training stages, which is conductive to leading to better generalization, making the codec model suitable for inference of multiple sequence lengths. In this way, the codec may be able to work on large sequence sizes.

[0033] The codec may be applied to data encoding and decoding using the hierarchical coding scheme.,0034] Optionally, the codec may be applied to video compression. The signal sequences may be video frame sequences.A signal sequence may be referred to as a group of frames or a group of pictures.]0035] Further, optionally, the signal sequence may be consecutive frames from a video sequence.

[0036] In a possible implementation of the third aspect, the first sequence size is smaller than the second sequence size, and the first training stage is earlier than the second training stage.

[0037] According to the aforementioned technical solution, the size of signal sequence is increased during the training process. Larger size of signal sequence during the training process allows bigger distances between target signal and reference signals while using hierarchical coding scheme, which leads to better generalization.

[0038] In a possible implementation of the third aspect, where the codec model is applied to video compression, during the first training stage, parameters of the codec model are adjusted based on a first loss, the first loss is determined based on a loss corresponding to a signal sequence in a subset of the first training dataset, the loss corresponding to the signal sequence is determined based on one or more losses corresponding to one or more B-frames in the signal sequence, and the one or moreB-frames are part of B-frames in the signal sequence.[0039| According to the aforementioned technical solution, part of B-frames may be extracted from the signal sequence for calculating loss to update the codec parameters. Compared to processing the entire signal sequence during the training process, this approach is beneficial in reducing memory consumption and saving training time.

[0040] In a possible implementation of the third aspect, the one or more B-frames in the signal sequence includes one or more non-reference frames in the signal sequence.

[0041] In a possible implementation of the third aspect, the one or more losses corresponding to the one or more B- frames are determined according to one or more hierarchy levels of the one or more B-frames.

[0042] According to the aforementioned technical solution, hierarchy level(s) of the part of B-frames may be introduced to the calculation of the one or more losses corresponding to the one or more B-frames. In this way, the influence of reference distance on frame reconstruction may also be considered in the training process, which is beneficial for improving reconstruction quality.

[0043] In a possible implementation of the third aspect, the loss corresponding to the signal sequence is obtained according to:(0044] RIis a rate of a I-frame, DIis a distortion of the I-frame, λ is a coefficient, Rt8is a rate of a B-frame t,D8is a distortion of the B-frame t, path includes the one or more B-frames, and level(t) is the hierarchy level of the B-frame t.

[0045] In a possible implementation of the third aspect, the one or more losses corresponding to the one or more B- frames are determined according to one or more distortions of the one or more B-frames, and the one or more distortions of the one or more B-frames are related to the one or more hierarchy levels of the one or more B-frames.(0046] In a possible implementation of the third aspect, the one or more distortions of the one or more B-frames are determined based on distortions of channel Y, channel U and channel V of the one or more B-frames, and distortion weights corresponding to the distortions of the channel Y, the channel U and the channel V are related to the one or more hierarchy levels of the one or more B-frames.

[0047] According to the aforementioned technical solution, the distortion weights corresponding to the distortion of each channel may be used to control the trade-off between the distortion of each channel at different hierarchy level, which is conductive to balancing the reconstruction quality of each channel among hierarchy levels.

[0048] According to a fourth aspect, an embodiment of the present application provides an electronic device, and the electronic device has a function of implementing the method in the first aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware of the software includes one or more units corresponding to the function.

[0049] According to a fifth aspect, an embodiment of the present application provides an electronic device, and the electronic device has a function of implementing the method in the second aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware of the software includes one or more units corresponding to the function.(0050] According to a sixth aspect, an embodiment of the present application provides an electronic device, and theelectronic device has a function of implementing the method in the third aspect. The function may be implemented by hardware, or may be implemented by hardware executing corresponding software. The hardware of the software includes one or more units corresponding to the function.

[0051] According to a seventh aspect, an embodiment of the present application provides a computer readable storage medium including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect.

[0052] According to an eighth aspect, an embodiment of the present application provides a computer readable storage medium including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the method in the second aspect or any possible implementation of the second aspect.

[0053] According to a ninth aspect, an embodiment of the present application provides a computer readable storage medium including instructions. When the instructions run on an electronic device, the electronic device is enabled to perform the method in the third aspect or any possible implementation of the third aspect.

[0054] According to a tenth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the first aspect or any possible implementation of the first aspect.,0055] According to an eleventh aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the second aspect or any possible implementation of the second aspect.

[0056] According to a twelfth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The processor is connected to the memory. The memory is configured to store instructions, and the processor is configured to execute the instructions. When the processor executes the instructions stored in the memory, the processor is enabled to perform the method in the third aspect or any possible implementation of the third aspect.

[0057] According to a thirteenth aspect, an embodiment of the present application provides a chip system, where the chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the method in the first aspect or any possible implementation of the first aspect.

[0058] According to a fourteenth aspect, an embodiment of the present application provides a chip system, where the chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processoris configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the method in the second aspect or any possible implementation of the second aspect.]0059] According to a fifteenth aspect, an embodiment of the present application provides a chip system, where the chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the method in the third aspect or any possible implementation of the third aspect.

[0060] According to a sixteenth aspect, an embodiment of the present application provides a computer program product, where when the computer program product runs on an electronic device, the electronic device is enabled to perform the method in the first aspect or any possible implementation of the first aspect.

[0061] According to a seventeenth aspect, an embodiment of the present application provides a computer program product, where when the computer program product runs on an electronic device, the electronic device is enabled to perform the method in the second aspect or any possible implementation of the second aspect.(0062] According to an eighteenth aspect, an embodiment of the present application provides a computer program product, where when the computer program product runs on an electronic device, the electronic device is enabled to perform the method in the third aspect or any possible implementation of the third aspect.(0063] According to a nineteenth aspect, an embodiment of the present application provides computer readable storage medium, where the computer readable storage medium stores a bitstream, the bitstream is obtained by using the method according to the first aspect or any possible implementation of the first aspect.DESCRIPTION OF DRAWINGS[0064| FIG. 1 is a schematic block diagram illustrating a coding system according to some embodiments of the present application.

[0065] FIG. 2 illustrates an example of a GoP structure for the hierarchical coding.

[0066] FIG. 3 illustrates an encoding method according to some embodiments of the present application.

[0067] FIG. 4 is a schematic block diagram of codec model architecture according to some embodiments of the present application.

[0068] FIG. 5 is a schematic block diagram of an HGU module according to some embodiments of the present application.

[0069] FIG. 6 illustrates a decoding method according to some embodiments of the present application.

[0070] FIG. 7 illustrates a training method according to some embodiments of the present application.(0071] FIG. 8 illustrates an example of a choose frame according to some embodiments of the present application.

[0072] FIG. 9 is a schematic block diagram of an electronic device according to some embodiments of the present application.

[0073] FIG. 10 is a schematic block diagram of an electronic device according to some embodiments of the present application.

[0074] FIG. 11 is a schematic block diagram of an electronic device according to some embodiments of the present application.

[0075] FIG. 12 is a schematic block diagram of an electronic device according to some embodiments of the present application.

[0076] FIG. 13 is a schematic block diagram of a system architecture according to some embodiments of the present application.DESCRIPTION OF EMBODIMENTS

[0077] The following describes the technical solutions in the present application with reference to the accompanying drawings.

[0078] FIG. 1 is a schematic block diagram illustrating a coding system according to some embodiments of the present application. Referring to FIG. 1, the coding system 100 includes a source device 110 configured to provide encoded picture data to a destination device 120 for decoding the encoded picture data. For convenience, it is assumed that the coding system100 is a picture coding system. As will be apparent for the skilled person, the picture coding system is just example embodiments of the present application and embodiments of the present application are not limited thereto.

[0079] The source device 110 may include an encoding unit 111. Optionally, the source device 110 may further include a picture source unit 112, a pre-processing unit 113, and a communication unit 114.

[0080] The picture source unit 112 may include or be any kind of picture capturing device, for example for capturing a rea-word picture, and / or any kind of a picture generating device, for example a computer- graphics processor for generating a computer animated picture, or any kind of device for obtaining and / or providing a real-word picture, a computer animated picture (e.g., a screen content, a virtual reality (VR) picture) and / or any combination thereof (e.g., an augmented reality (AR) picture). In the following, all these kinds of pictures and any other kind of picture will be referred to as “picture".

[0081] A (digital) picture is or can be regarded as a two-dimensional array or matrix of samples with intensity values.A sample in the array may also be referred to as pixel (short form of picture element) or a pel. The number of samples in horizontal and vertical direction (or axis) of the array or picture define the size and / or resolution of the picture. For representation of color, typically three-color components are employed, i.e. the picture may be represented or include three sample arrays. In RBG format or color space a picture comprises a corresponding red, green and blue sample array. However, in video coding each pixel is typically represented in a luminance / chrominance format or color space, e.g. YCbCr, which comprises a luminance component indicated by Y (sometimes also L is used instead) and two chrominance components indicated by Cb and Cr. The luminance (or short luma) component Y represents the brightness or grey level intensity (e.g. like in a grey-scale picture), while the two chrominance (or short chroma) components Cb and Cr represent the chromaticity or color information components. Accordingly, a picture in YCbCr format comprises a luminance sample array of luminance sample values (Y), and two chrominance sample arrays of chrominance values (Cb and Cr). Pictures in RGB format may be converted or transformed into YCbCr format and vice versa, the process is also known as color transformation or conversion.If a picture is monochrome, the picture may comprise only a luminance sample array.

[0082] The picture source unit 112 may be, for example a camera for capturing a picture, a memory, e.g. a picture memory, comprising or storing a previously captured or generated picture, and / or any kind of interface (internal or external) to obtain or receive a picture. The camera may be, for example, a local or integrated camera integrated in the source device, the memory may be a local or integrated memory, e.g. integrated in the source device. The interface may be, for example, an external interface to receive a picture from an external video source, for example an external picture capturing device like a camera, an external memory, or an external picture generating device, for example an external computer- graphics processor, computer or server. The interface can be any kind of interface, e.g. a wired or wireless interface, an optical interface, according to any proprietary or standardized interface protocol. The interface for obtaining the picture data 312 may be the same interface as or a part of the Communication unit 114.

[0083] In distinction to the pre-processing unit 113 and the processing performed by the pre-processing unit 113, the picture or picture data 131 may also be referred to as raw picture or raw picture data 131.

[0084] The pre-processing unit 113 is configured to receive the (raw) picture data 131 and to perform pre-processing on the picture data 131 to obtain a pre-processed picture 132 or pre-processed picture data.[0085| The pre-processing performed by the pre-processing unit 113 may, e.g., comprise trimming, color format conversion (e.g. from RGB to YCbCr), color correction, or de-noising. The encoding unit 111 is configured to receive the pre- processed picture data 132 and provide encoded picture data 133.

[0086] The communication unit 114 of the source device 110 may be configured to receive the encoded picture data133 and to directly transmit it to another device, e.g. the destination device 120 or any other device, for storage or direct reconstruction, or to process the encoded picture data 133 for respectively before storing the encoded picture data 133 and / or transmitting the encoded picture data 133 to another device, e.g. the destination device 120 or any other device for decoding or storing.

[0087] The destination device 120 comprises a decoding unit 121, and may additionally, i.e. optionally, comprise a communication unit 124, a post-processing unit 123 and a display unit 122.

[0088] The communication unit 124 of the destination device 120 is configured receive the encoded picture data 133, e.g. directly from the source device 110 or from any other source, e.g. a memory, e.g. an encoded picture data memory.]0089] The communication unit 114 and the communication unit 124 may be configured to transmit respectively receive the encoded picture data 133 via a direct communication link between the source device 110 and the destination device 120, e.g. a direct wired or wireless connection, or via any kind of network, e.g. a wired or wireless network or any combination thereof, or any kind of private and public network, or any kind of combination thereof.

[0090] The communication unit 114 may be, e.g., configured to package the encoded picture data 133 into an appropriate format, e.g. packets, for transmission over a communication link or communication network, and may further comprise data loss protection and data loss recovery.,0091] The communication unit 124, forming the counterpart of the communication unit 114, may be, e.g., configured to de-package the packets to obtain the encoded picture data 133 and may further be configured to perform data loss protection and data loss recovery, e.g. comprising error concealment.(0092] The communication unit 124, forming the counterpart of the communication unit 114, may further be configured to perform communication without or with limited data loss protection, and without re -transmission of lost of corrupted data to minimize communication delay and end-to-end latency between source and destination device. In such configuration the encoded picture data 133 may contain errors after receiving by communication unit 124. For binary represented signals the communication errors will lead to inverting 0 to 1 and vice versa.

[0093] Both, the communication unit 114 and the communication unit 124 may be configured as unidirectional communication interfaces as indicated by the arrow for the encoded picture data 133 in FIG. 1 pointing from the source device110 to the destination device 120, or bi-directional

[0094] The communication units, and may be configured, e.g. to send and receive messages, e.g. to set up a connection, to acknowledge and / or re-send lost or delayed data including picture data, and exchange any other information related to the communication link and / or data transmission, e.g. encoded picture data transmission.,0095] The decoding unit 121 is configured to receive the encoded picture data 133 and provide decoded picture data134.

[0096] The post-processing unit 123 of destination device 120 is configured to post -process the decoded picture data134 to obtain post-processed picture data 135. The post-processing performed by the post-processing unit 123 may comprise, e.g. color format conversion (e.g. from YCbCr to RGB), color correction, trimming, or re-sampling, or any other processing, e.g. for preparing the decoded picture data 134 for display, e.g. by display unit 122.

[0097] The display unit 122 of the destination device 120 is configured to receive the post -processed picture data 135 for displaying the picture, e.g. to a user or viewer. The display unit 122 may be or comprise any kind of display for representing the reconstructed picture, e.g. an integrated or external display or monitor. The displays may, e.g. comprise cathode ray tubes(CRT), liquid crystal displays (LCD), plasma displays, organic light emitting diodes (OLED) displays or any kind of other display, such as beamer, hologram (3D), or the like.

[0098] Although FIG. 1 depicts the source device 110 and the destination device 120 as separate devices, embodiments of devices may also comprise both or both functionalities, the source device 110 or corresponding functionality and the destination device 120 or corresponding functionality. In such embodiments the source device 110 or corresponding functionality and the destination device 120 or corresponding functionality may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0099] As will be apparent for the skilled person based on the description, the existence and (exact) split of functionalities of the different units or functionalities within the source device 110 and / or destination device 120 as shown inFIG. 1 may vary depending on the actual device and application.

[0100] Therefore, the source device 110 and the destination device 120 as shown in FIG. 1 are just example embodiments of the present application and embodiments of the present application are not limited to those shown in FIG. 1.

[0101] The source device 110 and the destination device 120 may comprise any of a wide range of devices, including any kind of handheld or stationary devices, e.g. notebook or laptop computers, mobile phones, smart phones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices, broadcast receiver device, or the like and may use no or any kind of operating system.

[0102] The embodiments of this application relate to application of a large quantity of neural networks. Therefore, for ease of understanding, related terms and related concepts such as the neural network in the embodiments of this application are first described below.

[0103] (1) Neural network (NN)

[0104] Any neural network can be formalized and fully defined as a directed acyclic graph with set of nodes Z. Each nod represents a tensor (multi-dimensional array) and associated with an operation (neural network laye ona set of its parent nodes Only exception is input node x which doesn’t have input nodes and associated operations.Computations at node k:( j Set of operations 0 includes unary operations (convolutions, pooling, activations, batchnorms, etc.) and multivariate operations (concatenation, addition, etc.). Any representation that specifies a set of parents and an operation of each node completely defines a neural network.[0

[0005] Alternatively, neural network may also be understood in the following way.[0[06[ The neural network may include neurons. The neuron may be an operation unit that uses xsand an intercept 1 as inputs, and an output of the operation unit may be as follows:

[0107] (0108] Herein, s=l, 2, . . . , or n, n is a natural number greater than 1, Wsis a weight of xs, and b is bias of the neuron. f is an activation function of the neuron, and the activation function is used to introduce a non-linear feature into the neural network, to convert an input signal in the neuron into an output signal. The output signal of the activation function may be used as an input of a next convolutional layer. The activation function may be a sigmoid function. The neural network is a network formed by connecting many single neurons together. To be specific, an output of a neuron may be an input of another neuron.An input of each neuron may be connected to a local receptive field of a previous layer to extract a feature of the local receptive field. The local receptive field may be a region including several neurons.

[0109] (2) Deep neural network (DNN)[01

[0000] The deep neural network, also referred to as a multi-layer neural network, may be understood as a neural network having many hidden layers. The “many” herein does not have a special measurement standard. The DNN is divided based on locations of different layers, and a neural network in the DNN may be divided into three types: an input layer, a hidden layer, and an output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layer is the hidden layer. Layers are fully connected. To be specific, any neuron at the ith layer is certainly connected to any neuron at the(i+l)th layer. Although the DNN looks to be complex, the DNN is actually not complex in terms of work at each layer, and is simply expressed as the following linear relationship expression: where is an input vector, is anoutput vector, is a bias vector, W is a weight matrix (also referred to as a coefficient), and a( ) is an activation function. At each layer, the output vector is obtained by performing such a simple operation on the input vector . Because there aremany layers in the DNN, there are also many coefficients W and bias vectors Definitions of these parameters in the DNNare as follows: The coefficient W is used as an example. It is assumed that in a DNN having three layers, a linear coefficient from the fourth neuron at the second layer to the second neuron at the third layer is defined as . The superscript 3 representsa layer at which the coefficient W is located, and the subscript corresponds to an output third-layer index 2 and an input second-layer index 4. In conclusion, a coefficient from the kthneuron at the (L-1 )thlayer to the jthneuron at the Lthlayer is defined as It should be noted that there is no parameter W at the input layer. In the deep neural network, more hidden layers makethe network more capable of describing a complex case in the real world.

[0111] Theoretically, a model with a larger quantity of parameters indicates higher complexity and a larger “capacity”, and indicates that the model can complete a more complex learning task. Training the deep neural network is a process of learning a weight matrix, and a final objective of the training is to obtain a weight matrix of all layers of the trained deep neural network (a weight matrix including vectors W at many layers).

[0112] (3) Convolutional neural network (CNN)

[0113] The convolutional neural network is a deep neural network having a convolutional structure. The convolutional neural network includes a feature extractor including a convolutional layer and an optional sub sampling layer. The feature extractor may be considered as a filter. A convolution process may be considered as using a trainable filter to perform convolution on an input image or a convolutional feature plane (feature map). The convolutional layer is a neuron layer that is in the convolutional neural network and at which convolution processing is performed on an input signal. At the convolutional layer of the convolutional neural network, one neuron may be connected only to some adjacent-layer neurons. A convolutional layer usually includes a plurality of feature planes, and each feature plane may include some neurons arranged in a rectangular form. Neurons in a same feature plane share a weight. The shared weight herein is a convolution kernel. Weight sharing may be understood as that an image information extraction manner is irrelevant to a location. A principle implied herein is that statistical information of a part of an image is the same as that of another part. This means that image information learned in a part can also be used in another part. Therefore, image information obtained through same learning can be used for all locations in the image. At a same convolutional layer, a plurality of convolution kernels may be used to extract different image information. Usually, a larger quantity of convolution kernels indicates richer image information reflected by a convolution operation.[0114| The convolution kernel may be initialized in a form of a random-size matrix. In a process of training the convolutional neural network, the convolution kernel may obtain an appropriate weight through learning. In addition, a direct benefit brought by weight sharing is that connections between layers of the convolutional neural network are reduced and an overfitting risk is lowered.

[0115] (4) Loss function[0116| Training is the adaptation of the network to better handle a task by considering sample observations. Training involves adjusting the weights and other parameters of the network to improve the accuracy of the result. This is done by minimizing the observed errors. After finish of the training neural network with adapted weights called trained neural network.

[0117] In a process of training the deep neural network, because it is expected that an output of the deep neural network is as much as possible close to a predicted value that is actually expected, a predicted value of a current network and a target value that is actually expected may be compared, and then a weight vector of each layer of the neural network is updated based on a difference between the predicted value and the target value (certainly, there is usually an initialization process before the first update, to be specific, parameters are preconfigured for all layers of the deep neural network). For example, if the predicted value of the network is large, the weight vector is adjusted to decrease the predicted value, and adjustment is continuously performed until the deep neural network can predict the target value that is actually expected or a value that is very close to the target value that is actually expected. Therefore, “how to obtain, through comparison, a difference between a predicted value and a target value” needs to be predefined. This is the loss function or an objective function. The loss function and the objective function are important equations used to measure the difference between the predicted value and the target value.

[0118] The loss function is used as an example. Loss function is a function that maps values of one or more variables onto a real number intuitively representing some "cost" associated with the observed variables. For example, if we consider values of errors on multi-dimensional array (for example image), then a loss function could be mean square error(MSE) - average of the squares of the errors. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training of the deep neural network is a process of minimizing the loss as much as possible.

[0119] Backpropagation is a method to adjust the weights to compensate for each error found during learning.Technically, backpropagation calculates the gradient (the derivative) of the cost function associated with a given state with respect to the weights. The weight updates can be done via stochastic gradient descent (SGD) or other methods.

[0120] (5) Frame]0121] Frame is one image from video.

[0122] Group of pictures (GoP) is a sequence of frames.

[0123] Intra-frame (I-frame) is a frame, compressed independently of any other frame in GoP.

[0124] Bidirectional predicted frame (B-frame) is a frame, compressed via utilizing information from already decoded frames: from past and future, judging by order in GoP.

[0125] Non-reference frame is frame on which no one is dependent or, from oriented graph perspective, sink.

[0126] (6) Hierarchy

[0127] Hierarchy is an ordered set of triplets with frame indexes, which defines coding procedure and dependency between frames. Usually represented as connected oriented graph with I-frames having only outcoming edges.

[0128] (7) Deep learning based intra prediction

[0129] In the traditional intra prediction, the neighboring reconstructed samples of a coding block are used to get theprediction of the samples inside the coding block along a specific straight direction (or some fixed pattern) which is indicated by an intra prediction mode. With the deep learning with the reference samples, the generated prediction sample value could be more flexible, and could be more similar with the samples inside the current coding block.

[0130] (8) Deep learning based inter prediction

[0131] In the traditional inter prediction, the reference block in a reference picture are used to get the prediction of the samples inside the current coding block, by using a simple weighting method. By using deep learning with the reference blocks, more flexible predictions can be obtained, which could be more similar with the samples inside the current coding block.

[0132] (9) Deep learning based entropy coding

[0133] In the traditional entropy coding, some neighboring information or priori knowledge are used as context, which will be used to estimate the probability of a syntax value for arithmetic coding. By using deep learning with context, more accurate probability could be estimated.

[0134] (10) Video compression

[0135] Video compression has been a critical aspect of digital media for a long time, enabling the efficient storage and transmission of video content. It plays crucial role in various applications, from video streaming to video conferencing, demanding efficient compression techniques to transmit data effectively over limited bandwidth networks. Among the diverse set of video coding methodologies, two prominent approaches stand out: low-delay P-frame coding and random access (RA) coding.

[0136] Low-delay P-frame coding prioritizes minimizing the time delay between encoding and decoding to facilitate real-time applications such as video conferencing and live streaming. In this approach, frames are typically encoded in a predictive manner, where each frame is predicted based on previous frame. This simplifies the encoding process, making it computationally less demanding compared to other coding schemes. Since P -frames rely heavily on temporal prediction, any transmission errors can propagate and affect subsequent frames, leading to noticeable artifacts and degradation in video quality.Accessing specific frames for editing or seeking purposes can be challenging due to the interdependency between frames, impacting the flexibility of the video stream.

[0137] Random access coding emphasizes the ability to access any frame independently, without the need to decode preceding or subsequent frames fully. This approach is essential for applications requiring random frame access, such as video editing and content-based retrieval systems. Key component of the RA coding scheme is a bidirectional frame (B -frame). B- frames utilize both past and future reference frames for prediction, resulting in higher compression efficiency compared to P- frames. By incorporating information from future frames, B-frames can better conceal errors and maintain video quality in the presence of transmission errors.

[0138] Video sequence consists of ordered frames and during encoding / decoding they are split into GoP. The most common approach in RA scenario is the hierarchical coding.

[0139] FIG. 2 illustrates an example of GoP structure for the hierarchical coding. First frame of every GoP may be coded as an intra frame (I-frame). All other frames are coded as bi-predictive frames (B-frames), using two already decoded frames as references. For example, as shown in FIG. 2, the GoP size equals 8. Arrows are connecting frames with their child(those that use them as a reference) frames.]0140] The emergence of neural network based hybrid and end-to-end (E2E) codecs represents a significant advancement in encoding and decoding field. However, the introduction of Al may not necessarily guarantee good encoding and decoding performance. In some scenarios, Al-based encoding and decoding performance may not even be as good as traditional encoding and decoding methods.

[0141] In view of this, the present application provides an encoding method and a decoding method, which may be conducive to improving the quality and efficiency of encoding and decoding. The present application also provides a training method for a codec model, which may be conducive to improving the quality and efficiency of encoding and decoding.

[0142] For example, the technical solutions provided in the present application may be applied to video compression using hierarchical coding scheme, which is beneficial for enhancing the efficiency and quality of video compression.(0143] Further, the technical solutions provided in the present application may be applied to video compression in random access scenario.

[0144] The present application provides an encoding method and a decoding method. According to the encoding method, the encoding device may obtain a compressed result of a target signal according to the hierarchy level of the target signal and reference signals. Correspondingly, the decoding device may obtain a reconstructed signal according to the hierarchy level of the target signal and reference signals.

[0145] The reference signals may include two or more reference signals. In some other embodiments, part of the reference signal(s) may be previous signal(s) of the target signal, while other signal(s) may be further signal(s) of the target signal. In other words, the frame numbers) of part of the reference signal(s) may be smaller than the frame number of the target signal, while the frame number(s) of other signal(s) may be greater than the frame number of the target signal.

[0146] The embodiments provided by the present application may be used to encode a video frame, an audio frame, or the like, and the present application is not limited thereto. For example, when the present application is used to encoding the video frame, the signal may be a video frame. When the present application is used to encoding the audio frame, the signal may be an audio frame.(0147] For convenience, in the following embodiments, it is assumed that the signal is a video frame. In someembodiments, the video frame may also be referred to “frame”, while the reference signal may be referred to “reference frame”.[0148J FIG. 3 illustrates an encoding method provided by some embodiments of the present application. For convenience, it is assumed that the operation illustrated in FIG. 3 is performed by an encoding device. The encoding device may also be referred to as an encoder side. In some embodiments, the encoding device may be a terminal device (e.g., a personal computer, a laptop, a mobile phone, a tablet computer, an augmented reality (AR) / virtual reality (VR) terminal, or the like), a server, a network device or the like. In some other embodiments, the encoding device may be a component in the terminal, the server, the network device or the like. For example, the encoding device may be a chip, a system on chip (SoC), a circuit, and so on.

[0149] The method 300 in FIG. 3 may be applied to all encoding methods utilizing a hierarchical coding scheme. For example, the method 300 may be applied to all video compression methods utilizing a hierarchical coding scheme.

[0150] As shown in FIG. 3, method 300 may include the following steps.

[0151] 301, obtain a latent representation of a target signal based on reference signals by using an encoding model.(0152] 302, perform a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation.

[0153] 303, perform an entropy encoding operation on the scaled latent representation to obtain a compressed result.

[0154] The reference signals in step 301 refer to reference signals of the target signal. The reference signals of the target signal may be understood as the signals dependent on for encoding the target signal.

[0155] As mentioned before, method 300 may be used to compress a video frame. For example, every B-frame may be compressed through the method 300. The target signal may be a video frame, such as a B-frame. The reference signals may be reference frames of the target frame. A B-frame utilizes both previous and future reference frames for prediction. The reference signals may be obtained from the previous reference frames of the target frame and the future reference frames of the target frame. For example, the target signal may be a B-frame. The reference signals may include two reference frames, one of which is a previous reference frame and the other is a future reference frame.

[0156] The following describes the reference frames.[0157| Optionally, the method 300 may also include: obtaining the reference frames.

[0158] The reference frames may be stored in a buffer. In this case, obtaining the reference frames may include taking reference frames from the buffer.

[0159] For example, the reference frames may be reconstructed frames. The reconstructed frame may also be referred to as a decoded frame. The reconstructed frames may be stored in a buffer, such as a decoded picture buffer. Obtaining the reference frames may include taking reconstructed frames from the decoded picture buffer.

[0160] FIG. 2 is taken as an example. For example, the target frame may be frame 3. The reference frames of frame 3 include reconstructed frame 2 and reconstructed frame 4. For example, the target frame may be frame 4. The reference frames of frame 4 include reconstructed frame 0 and reconstructed frame 8.

[0161] The reconstruction method of a frame may refer to the method 600, and will not be described here.

[0162] The following describes the latent representation of the target frame.

[0163] In step 301, the latent representation of the target frame may also be referred to as a latent or an encoded result of the target frame.

[0164] The encoding model may be regarded as an encoding module or an encoding network on the encoder side.[0165| The encoding model may be achieved through multiple model structures.

[0166] As an example, the encoding model may include sequence of convolutional layers and activation functions, such as rectified linear unit (ReLU). For example, the encoding model may include the first layer of a residual network (ResNet) orVisual Geometry Group (VGG).

[0167] The above is only an example. The encoding model may also include models with other structures, as long as the latent representation of the target frame can be obtained.

[0168] For a target frame, there may be one or more latents obtained by using one or more encoding models, respectively.[0169[ For example, the latent of the target frame may include a motion latent of the target frame and the encoding model may include a motion encoding model. For another example, the latent of the target frame may include a residual confidence (RC) latent of the target frame and the encoding model may include a RC encoding model.

[0170] For convince, one latent is taken as an example for illustration. Other possible latent(s) of the target signal may be processed in the same way.

[0171] The following describes the hierarchy level.[0172[ In step 302, the hierarchy level of the target frame is determined according to the distance between the reference frames and the target frame. The distance between the reference frame and the target frame may be called a reference distance.

[0173] For example, hierarchy level of the target frame may be determined according to the frame numbers of the reference frames and the target frame.[0174| The target frame is a B-frame as an example. For example, the target frame may be located in the middle of two reference frames. In this case, the distance between one reference frame and the target frame is the same as the distance between the other reference frame and the target frame. The reference distance is half of the distance between the two reference frames.For example, the reference distance may be the difference between the frame number of the reference frame and the frame number of the target frame, which is half of the difference between the frame numbers of the two reference frames.

[0175] The hierarchy level of frame t may meet the following formula:

[0176] level(t) represents the hierarchy level of the frame 1. 1 is a non-negative integer. GoP represents the size of the GoP to which the frame t belongs. G s a non-negative integer. AZ represents the reference distance. AZis a positive integer less than GoP .[0177| FIG. 2 shows an example of the hierarchical coding scheme when GoP size equals 8. The end pointed by an arrow may be regarded as the target frame, and the other end may be the reference frame for that target frame. For example, t=4. The reconstructed frame 0 and the reconstructed frame 8 may be the reference frames for the frame 4. In this case, Correspondingly, The hierarchy level of frame 4 is 1.]0178] The step 302 may be performed by using a hierarchical gain unit (HGU) module. Information about different hierarchy levels, such as the scaling values corresponding to different hierarchy levels, allows the HGU module processes the features, such as the latent representation, before compression by the entropy model.

[0179] FIG. 4 is a schematic block diagram of architecture of codec model according to some embodiments of the present application. The codec model includes an encoder and a decoder. The codec model may also be referred to as a codec in the embodiments of the present application.[0180[ The compressed result of the target frame is obtained according to the encoder, the target frame, and the reference frames.

[0181] As an example, the target frame and the reference frames may be used as the input of the encoder. In this case, the encoder may output the compressed result of the target frame by processing the target frame and the reference frames.

[0182] The reconstructed frame of the target frame is obtained according to the decoder, the compressed result and the reference frames.[0183| As an example, the compressed result and the reference frames may be used as the input of the decoder. In this case, the decoder may output the reconstructed frame of the target frame by processing the compressed result and the reference frames.

[0184] As shown in FIG. 4, the encoder may include an encoding module, an HGU module and an entropy module.

[0185] The encoding module may be used to obtain the latent of the target frame based on reference frames.

[0186] The HGU module on the encoder side may be used to scale the latent according to the hierarchy level of thetarget frame. The scaling method in the HGU module in FIG. 4 is only an example and does not constitute a limitation on theHGU module in the embodiments of the present application.[0187| The entropy module may be used to compress the scaled latent from the HGU module to obtain a compressed result of the target frame. In other words, the entropy module may be used to perform an entropy encoding operation on the latent the scaled latent from the HGU module to obtain a compressed result of the target frame.

[0188] The codec model in FIG. 4 may be used as an E2E Al codec for random access scenario.

[0189] For example, the encoder in FIG. 4 may be deployed in the encoding device. The decoder in FIG. 4 may be deployed in the decoding device.

[0190] FIG. 5 is a schematic block diagram of the HGU module according to some embodiments of the present application. The HGU module be used on the encoder side of the codec in FIG. 4.

[0191] Step 302 may be understood as a hierarchy level-dependent scaling of the latent. The hierarchy level-dependent scaling may also be referred to as hierarchy level-adaptive scaling.

[0192] The following describes details about how to scale the latent.

[0193] In step 302, the latent may be scaled by a scaling value corresponding to the hierarchy level of the target frame.

[0194] There is a corresponding relationship between the hierarchy levels of the frames and the scaling values.[0195| In some embodiments, the step 302 may include: performing a multiplying operation on the latent representation by a first scaling value corresponding to the hierarchy level of the target frame.

[0196] The “first” in the “first scaling value” is only used to indicate that the scaling value is used to scale the latent representation on the encoder side, and does not have any other limiting effect.

[0197] For example, as shown in FIG. 4 or FIG. 5, multiply the latent representation by the first scaling value corresponding to the hierarchy level of the target frame.

[0198] Optionally, the scaling values corresponding to hierarchy levels may be stored in a look up table.

[0199] For example, the scaling value used to scale the latent may be taken from the look up table by the index of the hierarchy level of the target frame, as shown in FIG. 5.[0200| Optionally, the scaled latent may meet the following formula:[0201| represents the latent of the frame t / represents the scaled latent of the frame represents thescaling value corresponding to the level level (t) on the encoder side (i.e. the first scaling value). The latent may be a tensor, or a latent tensor, c, h and w represent channel index, height index and width index of the latent tensor, respectively.

[0202] In some embodiments, the first scaling value may be represented as a vector.(0203] For example, as shown in the formula mentioned above, the first scaling value may be a c-dimensional vector for channel-wise scaling of the latent with c channels.

[0204] In some other embodiments, the first scaling value may be represented as a tensor.

[0205] For example, the first scaling value may be used to scale the whole latent tensor.[0206[ In some other embodiments, the first scaling value may be represented as a scalar.

[0207] For example, the scalar may be used to scale the whole latent tensor.

[0208] For another example, the scalar may be used to scale some dimensions in the latent tensor.

[0209] As an example, the scalar may be used for channel-wise scaling of the latent.

[0210] Scaling value using scalar is simpler and helps reduce storage and computing resources.

[0211] The first scaling value may be of arbitrary dimensionality.

[0212] For target frames with different distances from the reference frames, the degree of difference between them also varies. The required bits to ensure the reconstruction quality of the target frames are also different.

[0213] The scaling leads to increasing or decreasing of the number of unique values in the latent (which is quantized), which influences both the bitrate and the reconstruction quality. The hierarchy level of the target frame may indicate the reference distance. Scaling the latent representation according to the hierarchy level of the target frame may be able to scale the latent representation appropriately, which may be beneficial for reducing bitrate and improving reconstruction quality.

[0214] The scaling value may also be referred to as the value of the scaling parameter.

[0215] In some embodiments, the first scaling value may be a learnable parameter, also referred to as a trainable parameter.

[0216] The first scaling value may be obtained through the training process of the codec. In other words, during the training process of the codec, not only are the weight parameters of the codec adjusted, but the scaling parameter may also be adjusted. After the training is completed, the first scaling values may be stored.

[0217] Scaling values determined through training may be more appropriate for encoding and decoding, which is beneficial for beneficial for reducing bitrate and improving reconstruction quality, thereby improving the efficiency and quality of encoding.

[0218] In case there are more hierarchy levels during inference than during training, extrapolation between learned scaling values may be applied.

[0219] Optionally, the scaling value(s) absent during training may be approximated on inference by exponential function. For example, there are 5 hierarchy levels during inference (i.e. levels 0-4) and 4 hierarchy levels during training (i.e.levels 0-3). The scaling values corresponding to levels 0-3 may be learned through training. The scaling values corresponding to level 4 may be approximated by exponential function according to the learned scaling values.

[0220] As mentioned before, for a target frame, there may be one or more latents. Correspondingly, there may be one or more scaled latents. The scaling values for different latents of the target frame may be independent of each other. The one or more scaled latents may be obtained by using one or more HGU modules, respectively. For example, the scaling values for the motion latent and the RC latent of the target frame may be independent. There may be a motion HGU module used to scale the motion latent and a RC HGU module used to scale RC latent.

[0221] In this way, the scaling values for different latents on each hierarchy level may be set independently. For example, the scaling values for the motion latent and the RC latent on each hierarchy level may be learned independently. This is beneficial for obtaining more appropriate scaling values for different types of latents, thereby improving the efficiency and quality of encoding.

[0222] The step 303 may be performed by an entropy module, which is used for entropy encoding.

[0223] The entropy module on the encoder side may also be referred to as a compression module, a compression model or an entropy encoding model.

[0224] In other words, the step 303 may be understood as compressing the scaled latent representation by using the entropy module to obtain a compressed result.

[0225] In some embodiment, the entropy coding may be an arithmetic coding, a context -based adaptive variable length coding, a context-adaptive binary arithmetic coding, or the like.

[0226] Alternately, the step 303 may also be understood as obtaining a bitstream corresponding to the target signal by using the entropy module. The compressed result is represented in form of bitstream.(0227] The bitstream may be stored or transmitted.

[0228] Further, optionally, the method 300 may also include step 304 (not shown in FIG. 3).

[0229] 304, transmit a bitstream carrying the compressed result corresponding to the target signal.

[0230] The encoding device may transmit the bitstream to the decoding device.

[0231] The decoding device may also obtain the bitstream through other ways.

[0232] For example, the encoding device and the decoding device may be two components in the same device, such as a terminal device. The bitstream may be stored in the terminal device. The decoding device may read the bitstream.

[0233] As mentioned before, for a target frame, there may be one or more scaled latents. Correspondingly, the bitstream may carry one or more compressed results. In case of multiple scaled latents, the multiple scaled latents may be compressed by using one entropy module. Alternatively, the multiple scaled latents may be compressed by using multiple entropy modules,respectively.(0234] FIG. 6 illustrates a decoding method provided by some embodiments of the present application. For convenience, it is assumed that the operation illustrated in FIG. 6 is performed by a decoding device. The decoding device may also be referred to as a decoder side. In some embodiments, the decoding device may be a terminal device (e.g., a personal computer, a laptop, a mobile phone, a tablet computer, an augmented reality (AR) / virtual reality (VR) terminal, or the like), a server, a network device or the like. In some other embodiments, the decoding device may be a component in the terminal, the server, the network device or the like. For example, the decoding device may be a chip, a system on chip (SoC), a circuit, and so on.

[0235] Exemplarily, the encoding device in the method 300 and the decoding device in the method 600 may be two components in the same device.

[0236] Exemplarily, the encoding device in the method 300 and the decoding device in the method 600 may be two components from different devices.

[0237] Exemplarily, the encoding device in the method 300 may be a terminal device, a server, a network device or the like. The decoding device in the method 600 may be a terminal device, server, network device, or the like, that is different from the encoding device.

[0238] The method 600 may be applied to all decoding methods utilizing a hierarchical coding scheme. For example, the method 600 may be applied to all learned video compression approaches utilizing a hierarchical coding scheme.

[0239] The method 600 may be considered as the inverse process of the method 300. The relevant description can refer to the method 300. To avoid repetition, it is appropriate to omit some descriptions when describing the method 600.10240] As shown in FIG. 6, method 600 may include the following steps.

[0241] 601, obtain a bitstream corresponding to a target signal.

[0242] 602, perform an entropy decoding operation on the bitstream to obtain a reconstructed latent representation of the target signal.]0243| 603, perform a scaling operation on the reconstructed latent representation according to a hierarchy level of the target signal to obtain a scaled reconstructed latent representation.

[0244] 604, obtain a reconstructed signal based on the scaled reconstructed latent representation and reference signals by using a reconstruction model.

[0245] As mentioned before, method 600 may be used to decode the bitstream to obtain a video frame. For example, every B-frame may be reconstructed through the method 600. The target signal may be a video frame, such as a B-frame. The reconstructed signal may be a reconstructed frame of the target frame, such as a reconstructed B-frame. The reference signals may be reference frames of the target frame. The relevant description about the reference frames may refer to the method 300,and will not be repeated here.

[0246] Optionally, in step 601, the decoding device may receive the bitstream.

[0247] For example, the decoding device may receive the bitstream from the encoding device.

[0248] The decoding device may also obtain the bitstream through other ways.

[0249] For example, the encoding device and the decoding device may be two components in the same device, such as a terminal device. The bitstream may be stored in the terminal device. The decoding device may read the bitstream.

[0250] The step 602 may be performed by an entropy module, which is used for entropy decoding.

[0251] The entropy module on the decoder side may also be referred to as a decompression module, a decompression model, or an entropy decoding model.

[0252] In other words, the step 602 may be understood as decompressing the bitstream to obtain a reconstructed latent representation of the target signal by using an entropy module.[0253[ The reconstructed latent representation of the target frame may also be referred to as a decompressed result of the bitstream. The decompression may be understood as used for reconstructing the latent of the target frame.

[0254] As mentioned in method 300, the bitstream may carry one or more compressed results. In step 602, there may be one or more decompressed results obtained from the bitstream. For example, the reconstructed latent of the target frame may include a reconstructed motion latent of the target frame. For another example, reconstructed latent of the target frame may include a reconstructed RC latent of the target frame.

[0255] For convince, one decompressed result is taken as an example for illustration. Other possible decompressed result(s) of the bitstream may be processed in the same way.[0256| The step 603 may be performed by using an HGU module. In order to distinguish it from the HGU module on the encoder side, the HGU module on the decoder side may also be referred to as the inverse HGU (Inv HGU) module.[0257| As shown in FIG. 4, the decoder may include an entropy module, an Inv HGU module and a reconstruction model.[0258| The entropy module may be used to decompress the bitstream to obtain the reconstructed latent representation of the target frame. In other words, the entropy module may be used to perform an entropy decoding operation on the bitstream to obtain the reconstructed latent representation of the target frame.

[0259] The Inv HGU module on the decoder side may be used to scale the reconstructed latent according to the hierarchy level of the target frame. The scaling method in the Inv HGU module in FIG. 4 is only an example and does not constitute a limitation on the Inv HGU module in the embodiments of the present application.

[0260] The reconstruction module may be used to obtain the reconstructed frame of the target frame based on the scaledreconstructed latent representation from the Inv HGU module and reference frames.

[0261] The relevant description about the hierarchy level of the target frame may refer to the step 302 in the method300, and will not be repeated here.

[0262] In step 603, the reconstructed latent may be scaled by a scaling value corresponding to the hierarchy level of the target frame.

[0263] There is a corresponding relationship between the hierarchy levels of the frames and the scaling values.

[0264] In some embodiments, the step 603 may include: performing a multiplying operation on the reconstructed latent representation by a second scaling value corresponding to the hierarchy level of the target frame.[0265| The “second” in the “second scaling value” is only used to indicate that the scaling value is used to scale the reconstructed latent representation on the decoder side, and does not have any other limiting effect. The first scaling value and the second scaling value may be the same or different.]0266| For example, as shown in FIG. 4, multiply the reconstructed latent representation by the second scaling value corresponding to the hierarchy level of the target frame.(0267] Optionally, the scaling values corresponding to hierarchy levels may be stored in a look up table.

[0268] For example, the scaling value used to scale the reconstructed latent may be taken from the look up table by the index of the hierarchy level of the target frame.

[0269] Optionally, the scaled reconstructed latent may meet the following formula:

[0270] , represents the reconstructed latent of the frame represents the scaled reconstructed latent of theframe represents the scaling value corresponding to the level on the decoder side (i.e. the second scalingvalue). The reconstructed latent may be a tensor, or a latent tensor, c, h and w represent channel index, height index and width index of the latent tensor, respectively.

[0271] In some embodiments, the second scaling value may be represented as a vector.(0272] For example, as shown in the formula mentioned above, the second scaling value may be a c-dimensional vector for channel-wise scaling of the reconstructed latent with c channels.

[0273] In some other embodiments, the second scaling value may be represented as a tensor.

[0274] In some other embodiments, the second scaling value may be represented as a scalar.

[0275] For example, the scalar may be used to scale the whole latent tensor.

[0276] For another example, the scalar may be used to scale some dimensions in the latent tensor.

[0277] As an example, the scalar may be used for channel-wise scaling of the reconstructed latent.

[0278] Scaling value using scalar is simpler and helps reduce storage and computing resources.(0279] The second scaling value may be of arbitrary dimensionality.,0280] The scaling value may also be referred to as the value of the scaling parameter.

[0281] In some embodiments, the second scaling value may be a learnable parameter, also referred to as a trainable parameter.

[0282] The second scaling value may be obtained through the training process of the codec.[0283| In case there are more hierarchy levels during inference than during training, extrapolation between learned scaling values may be applied.

[0284] As mentioned before, there may be one or more reconstructed latents. Correspondingly, there may be one or more scaled reconstructed latents. The scaling values for different reconstructed latents of the target frame may be independent of each other. The one or more scaled reconstructed latents may be obtained by using one or more Inv HGU modules, respectively. For example, the scaling values for the reconstructed motion latent and the reconstructed RC latent of the target frame may be independent. There may be a motion Inv HGU module used to scale the reconstructed motion latent and a RCInv HGU module used to scale reconstructed RC latent.

[0285] In this way, the scaling values for different reconstructed latents on each hierarchy level may be set independently. For example, the scaling values for the reconstructed motion latent and the reconstructed RC latent on each hierarchy level may be learned independently. This is beneficial for obtaining more appropriate scaling values for different reconstructed latents, thereby improving the efficiency and quality of encoding.[0286| The details about the scaling values on the decoder side may refer to the scaling values on the encoder side, and will not repeated here.

[0287] In step 604, the reconstructed frame may be obtained according to the reconstruction model, the scaled reconstructed latent and reference frames.

[0288] For example, the scaled reconstructed latent and reference frames may be used as the input to the reconstruction model.

[0289] The reconstruction model may be regarded as a reconstruction module or a reconstruction network on the decoder side.[0290| The reconstruction model may be achieved through multiple model structures.

[0291] As an example, the reconstruction model may include sequence of transposed convolution layers and activation functions, such as ReLU. The reconstruction model may also include some refinement block, with few convolutional layers oreven UNet, which may increase final quality.

[0292] The above is only an example. The reconstruction model may also include models with other structures, as long as the target frame can be reconstructed.

[0293] Further, optionally, the method 600 may also include step 605 (not shown in FIG. 6).

[0294] 605, store the reconstructed frame.

[0295] For example, add the reconstructed frame to the buffer, such as the decoded picture buffer mentioned in method300.

[0296] The present application provides a training method for a codec model. According to the training method, the training device may train a codec model with multiple training datasets. A training dataset in the multiple datasets includes one or more signal sequences.(0297] The embodiments provided by the present application may be used to train a codec model used for encoding and decoding a video frame, an audio frame, or the like, and the present application is not limited thereto. For example, when the present application is used to train a codec model used for encoding and decoding the video frame, the signal may be a video frame. When the present application is used to train a codec model used for encoding and decoding the audio frame, the signal may be an audio frame.(0298] For convenience, in the following embodiments, it is assumed that the signal is a video frame. In some embodiments, the video frame may also be referred to “frame”.]0299] The present application also provides a training method for an encoder and a decoder. In other words, the present application also provides a training method for a codec model. The codec model may also be called a codec in the embodiments of the present application.

[0300] FIG. 7 illustrates a training method provided by some embodiments of the present application. For convenience, it is assumed that the operation illustrated in FIG. 7 is performed by a training device. In some embodiments, the training device may be a terminal device (e.g., a personal computer, a laptop, a mobile phone, a tablet computer, an augmented reality(AR) / virtual reality (VR) terminal, or the like), a server, a network device or the like. In some other embodiments, the training device may be a component in the terminal, the server, the network device or the like. For example, the training device may be a chip, a system on chip (SoC), a circuit, and so on.

[0301] The method 700 in FIG. 7 may be used for any codec which utilize hierarchical coding scheme. For example, the method 700 may be applied to any codec used for video compression based on the hierarchical coding scheme. In other words, the method 700 may be applied to any learnable video compression methods utilizing the hierarchical coding scheme.Further, the method 700 may be applied for training E2E Al codec for random access scenario.

[0302] For example, the method 700 may be applied to obtain the codec in FIG. 4.

[0303] For example, the method 700 may be applied to obtain the encoder in the method 300. In other words, the encoder in the codec trained by method 700 can be used as the encoder in method 300.

[0304] For example, the method 700 may be applied to obtain the decoder in the method 600. In other words, the decoder in the codec trained by method 700 can be used as the decoder in method 600.

[0305] As shown in FIG. 7, method 700 may include the following steps.

[0306] 701, obtain multiple training datasets including a first training dataset and a second training dataset. The first training dataset may include one or more signal sequences. The number of signals in each signal sequence in the first training dataset is a first sequence size. The second training dataset may include one or more signal sequences. The number of signals in each signal sequence in the second training dataset is a second sequence size. The first sequence size is different from the second sequence size.

[0307] 702, train a codec with the multiple training datasets. The training process includes multiple training stages. The multiple training stages includes a first training stage and a second training stage. The parameters of the codec model are adjusted based on the first training dataset during the first training stage, and the parameters of the codec model are adjusted based on the second training dataset during the second training stage.

[0308] A signal sequence may be a frame sequence including multiple frames. The frame sequence may also be replaced with a group of frames or a GoP. The number of the signals in the signal sequence may also be referred to as the size of the signal sequence, the length of the signal sequence, the number of frames in the group for frames, or the GoP size.

[0309] The "first" in the “first training dataset” and the "second" in the “second training dataset” are only used to distinguish between two different training datasets in the multiple training datasets and do not have any other limiting effect.]0310] The "first" in the “first sequence size” is only used to indicate that the sequence size corresponds to the first training dataset and does not have any other limiting effect.

[0311] The "second" in the “second sequence size” is only used to indicate that the sequence size corresponds to the second training dataset and does not have any other limiting effect.

[0312] In some embodiments, each training dataset in the multiple training datasets may correspond to a sequence size.Different training datasets may correspond to different sequence sizes. Each training dataset may include one or more groups of frames. The number of frames in a group in each training dataset may be the corresponding sequence size for the training dataset.

[0313] The multiple training datasets may be obtained at once or multiple times.

[0314] For example, the multiple training datasets may be obtained before the training begins.(0315| For another example, the training dataset may be obtained before the stage begins corresponding to the training dataset.

[0316] The present embodiment does not limit the timing of obtaining multiple training datasets.

[0317] The frames in one group in the training dataset may be consecutive frames from a video sequence.

[0318] There may be multiple video sequences in a dataset. The multiple training datasets may be obtained by sampling frames from video sequences in the dataset.

[0319] One training dataset in the multiple training dataset is taken as an example. There may be one or more groups of frames in the training dataset. Each group may include frames from a video sequence in the dataset. In the case of multiple groups, the frames in different groups may be from the same video sequence or different video sequences. The number of frames in a group is equal to the sequence size corresponding to the training dataset. The sequence size may also refer to as theGoP size.

[0320] For example, it is assumed that there is a training dataset #1 in the multiple training datasets, and the GoP size corresponding to the training dataset #1 is 8. Take consecutive 8 frames from every video sequence in the dataset to obtain multiple groups of frames in the training dataset #1.

[0321] The above is only one method for obtaining training datasets and does not constitute a limitation on the embodiments of the present application. For example, training datasets with different GoP sizes may also be from public datasets with corresponding GoP sizes.

[0322] The GoP size may be set as needed.

[0323] The "first" in the “first training stage” and the "second" in the “second training stage” are only used to distinguish between two different training stages in the multiple training stages and do not have any other limiting effect.

[0324] In some embodiments, the step 702 may also be understood as the following.

[0325] During the multiple training stages, the parameters of the codec are adjusted based on the multiple training datasets, respectively.

[0326] The training stage in the multiple training stages may include one or more epochs.

[0327] In step 702, the training process is split to several training stages. Different training stages may correspond to different training datasets. Correspondingly, different training stages may correspond to different GoP sizes.

[0328] In some embodiments, the number of the training stages may be the same as the number of the training datasets.[0329| During different training stages, different training datasets with different GoP sizes are used to adjust the parameters of the codec.

[0330] When the GOP size is large, the distance between the reference frame and the target frame may be large, resultingin significant differences between the reference frame and the target frame, which may affect the reconstruction quality.

[0331] According to the technical solution in method 700, different training datasets including signal sequences with different size are used to adjust the parameters of the codec during different training stages, which is conductive to leading to better generalization. In this way, the codec may be able to work on large GoP sizes, which is conductive to ensure the reconstruction quality of the frame sequence with large GoP size.

[0332] In some embodiments, the method 700 may adopt a progressive training strategy.[0333| The first sequence size is smaller than the second sequence size, and the first training stage is earlier than the second training stage.

[0334] Optionally, the sequence size corresponding to the training dataset in the previous training stage is smaller than the sequence size corresponding to the training dataset in the later training stage.

[0335] During any two training stages of the multiple training stages, the sequence size corresponding to the training dataset in the training stage that occurs first may be smaller than the sequence size corresponding to the training dataset in the training stage that occurs later.

[0336] In other words, stages organized in a way to increase GoP size from stage to stage and up the biggest GoP size.The biggest GoP size may be the number expected on test.

[0337] According to the technical solution mentioned above, the size of signal sequence is increased during the training process. Larger size of signal sequence during the training process allows bigger distances between target signal and reference signals while using hierarchical coding scheme, which leads to better generalization. For example, the larger GoP size during the training process may be beneficial for better codec performance on videos with more GoP sizes, which means that the codec may behave well on the video with the short GoP size and the video with a long GoP size. The codec performance may be indicated by the Bjontegaard-Delta rate (BD-rate). The BD-rate may include peak signal to noise ratio (PSNR)-BD-rate and / or visual multimethod assessment fusion (VMAF)-BD-rate.

[0338] In some embodiments, the codec may include an HGU module in the encoder and an HGU module in the decoder.As mentioned in the method 300 and method 600, the scaling parameters which may be regarded as the HGU weights may be trainable parameters. In this case, the HGU weights may be updated as well as the weights of the codec.

[0339] In addition, as mentioned in the method 300 and method 600, the scaling parameters absent in training are approximated on inference by exponential function. The training method of the present embodiment may effectively utilize the large GoP size which is conductive to reducing uncertainty in scaling parameter, leading to more efficient bit allocation.

[0340] What’s more, the scaling parameters may be scalars, which may be more robust to train.

[0341] Step 702 includes: updating the parameters in the codec based on loss calculated through loss function.

[0342] For convince, the parameters update process during the first training stage is taken as an example, and parameters update process during other training stages may refer to the first training stage.

[0343] In some embodiments, during the first training stage, the parameters of the codec may be updated based on a first loss. The first loss is determined based on a loss corresponding to a group in a subset of the first training dataset. The loss corresponding to the group is determined based on the loss corresponding to one or more frames in the group.(0344] The "first" in the “first loss” is only used to indicate the loss corresponding to the subset of the first training dataset and does not have any other limiting effect.(0345] The subset of the first training dataset may be regarded as a batch in the first training dataset.[0346| There may be one or more groups in the subset of the first training dataset.(0347] In the case where the subset includes one group, the first loss may be determined based on the loss corresponding to the group.

[0348] For example, the first loss may be the loss corresponding to the group in the subset of the first training dataset.

[0349] In the case where the subset includes multiple groups, the first loss may be determined based on the losses corresponding to the multiple groups.

[0350] For example, the first loss may be the sum of the losses corresponding to the multiple groups in the subset of the first training dataset.

[0351] For another example, the first loss may be the average of the losses corresponding to multiple groups in the subset of the first training dataset.

[0352] The following describes the loss corresponding to the group in the subset. For convince, the loss corresponding to one group in the subset is taken as an example. The losses corresponding to multiple groups in the subset may refer to the loss corresponding to the group.(0353] The loss corresponding to the group may also referred to as the loss for a GoP.

[0354] In some embodiments, the loss corresponding to the group may be determined based on the losses corresponding to all frames in the group.,0355] For example, the loss corresponding to the group may be the sum of the losses corresponding to all frames in the group.

[0356] For another example, the loss corresponding to the group may be the average of the losses corresponding to all frames in the group.

[0357] All frames in a group may be regarded as a full video sequence. There may be some limits arising from full video sequence processing. For example, limited memory may not support the processing of full video sequences, especiallywhen the GoP size is large. For another example, processing of full video sequences may require a significant amount of time, especially when the GoP size is large.

[0358] In some other embodiments, the loss corresponding to the group may be determined based on the loss(s) corresponding to part of frames in the group.

[0359] As a possible implementation, the first loss based on which the codec parameters are updated may be determined based on the loss(s) corresponding to part of frames in each group in the subset.,0360] The part of the frames in the group may include one or more frames.

[0361] In the case where the part of the frames in the group includes multiple frames, the loss corresponding to the group may be determined based on the losses of the multiple frames.

[0362] For example, the loss corresponding to the group may be the sum of the losses corresponding to the multiple frames.

[0363] For another example, the loss corresponding to the group may be the average of the losses corresponding to the multiple frames.

[0364] Further, optionally, the part of the frames in the group may include part of B-frames in the group.

[0365] In other words, the loss corresponding to the group is determined based on one or more losses corresponding to one or more B-frames in the group, and the one or more B-frames are part of B-frames in the group.

[0366] The part of the B-frames in the group referred to the B-frame(s) in which the loss(s) needs to be calculated.When calculating the loss corresponding to the group, for B frames in the group, only a portion of B frames are considered for their loss, rather than considering the losses of all B frames in the group.,0367] Further, optionally, the part of the B-frames in the group may include one or more non-reference frames in the group.

[0368] Further, optionally, the part of the B-frames may include one or more non-reference frames in the group and all frames that are necessary to reconstruct the one or more non-reference frames.,0369] For example, for each group in the subset, choose one non-reference frame and all frames sufficient for its reconstruction as defined by hierarchical coding scheme. As shown in FIG. 8, L is the hierarchy level. For example, frames with the hierarchy level of 0 (i.e. L=0) include frame 0 and frame 8. The non-reference frames may include the frames with the highest value at the hierarchical level. As shown in (a) of FIG. 8, non-reference frames include frames 1, 3, 5, and 7. As shown in (b) of FIG. 8, only frame 3 is chosen. Frame 3 is the chosen frame. The frames sufficient for the reconstruction of the frame3 include frames 0, 8, 4 and 2. The part of the B-frames may include frames 3, 0, 8, 4 and 2.

[0370] In some cases, the first training dataset can also be considered as a subset of itself.[0371 ) As a possible implementation, the loss for each update step may be computed as sum of losses of every decoded frame in the decoded picture buffer. For example, the decoded frames in the decoded picture buffer may be the decoded frames of the part of the frames in each group in the subset.(0372) According to the technical solutions provided by the present application, part of frames may be extracted from the signal sequence for calculating loss to update the codec parameters. Compared to processing the entire signal sequence during the training process, this approach is beneficial in reducing memory consumption and saving training time.[0373[ In some embodiments, the loss corresponding to a B-frame in the group may be determined according to the hierarchy level of the B frame.

[0374] Correspondingly, the one or more losses corresponding to the one or more B-frames in the group may be determined according to one or more hierarchy levels of the one or more B-frames in the group.

[0375] In this way, the first loss may be regarded as a hierarchical level-dependent loss.[0376| As an example, the codec may be trained with rate-distortion loss (RD-loss). The RD-loss for one GoP may be defined as:[0377| is a rate of a I-frame,is a distortion of the I-frame, is a coefficient, is a rate of a B-frame t,D® is a distortion of the B-frame t, path includes the part of the B-frames in the GoP, and level(t) is the hierarchy level of the B-frame t. is the loss corresponding to the B-frame t. is the loss corresponding to the I-frame.[0378| Further, the RD-loss for one GoP may be defined as:[0379| is a rate of I-frame,1is a distortion of I-frame. is a coefficient, which is used to control trade-offbetween the rate and the distortion. X is a constant. is a rate of B-frame t. is the rate of the motion of theB-frame t ” is the rate of RC of the B-frame t. is a distortion of B-frame t. path includes the chosen frameintroduced specifically to account for data sampling procedure.[0380[ As shown in the formula, the loss function may be used to weight losses of individual frames according to such sampling procedure and hierarchical coding scheme. The function may reflect frames distribution in an update step.[0381| The distortion may be mean squared error (MSE), multi-scale structural similarity index, or the like.[0382| In some embodiments, the distortion of the B-frame is related to the hierarchy level of the B-frame.[0383| Correspondingly, the distortion(s) of the part of the B-frames is related to the hierarchy level of the part of theB-frames.

[0384] In other words, the one or more losses corresponding to the one or more B-frames are determined according to one or more distortions of the one or more B-frames, and the one or more distortions of the one or more B-frames are related to the one or more hierarchy levels of the one or more B-frames.

[0385] In this way, the distortions may be rebalanced between different hierarchy levels.

[0386] further, optionally, the distortion of the B-frame is determined based on the distortion of channel Y, channel U and channel V, and the distortion weights corresponding to the distortion of each channel are related to the hierarchy level of the B-frame.

[0387] Correspondingly, the distortion(s) of the part of the B-frames is determined based on distortions of channel Y, channel U and channel V of the part of the B-frames, and distortion weights corresponding to the distortions of the channel Y, the channel U and the channel V are related to the hierarchy level of the part of the B-frames.

[0388] In other words, the one or more distortions of the one or more B-frames are determined based on distortions of channel Y, channel U and channel V of the one or more B-frames, and distortion weights corresponding to the distortions of the channel Y, the channel U and the channel V are related to the one or more hierarchy levels of the one or more B-frames.

[0389] The distortion weights corresponding to the distortion of each channel may be used to control the trade-off between the distortion of each channel at different hierarchy level, which is conductive to balancing the reconstruction quality of each channel among hierarchy levels.

[0390] Bor example, the distortion of B-frame may be defined as:

[0391] andare distortion coefficients for each hierarchy. Distortion coefficients may also be referred to asdistortion weights. represent MSB corresponding to the channel Y, channel U and channelV, respectively.

[0392] The distortion function may be conductive to balancing Y / UV reconstruction quality among hierarchy levels by incorporating distortion coefficients which may be used to control the trade-offbetween MSE of the luma and chroma channels.In other words, the YUV-distortion mentioned above is conductive to effectively balancing quality between luma and chroma channels.

[0393] The distortion coefficients may be learnable parameters. In this case, the distortion coefficients may be updatedas well as other weights of the codec during the training process.

[0394] The following is an example of the method 700 written in pseudocode.

[0395] The architecture and weights W of the codec may be initialized randomly or already pretrained. As shown in the pseudocode, the method 700 may be applied in subsequent training. It returns updated weights with two additional sets of tensors corresponding to HGU modules (i.e. qenc, qdec), which are uniformly initialized by 1 and updated as well as weights of the codec.

[0396] As can be seen, training procedure adjusts weights following update rule based on loss. Function sample_gop takes consecutive gop_size frames from every video sequence. In other words, function sample_gop may be used to obtain the multiple training datasets. Function sample_path samples only part of all frames from the GoP. In other words, function sample_path is used to determine the frames that are used to calculate the loss. For example, function sample_path is used to determine the part of the B-frames in each group in the subset in the first training dataset. Function forward— pass outputs frames reconstructions and bitrates based on model’s weights, HGU tensors and model’s architecture. Function loss function computes the loss in way mentioned before. For example, function loss_function may be used to calculate the first loss. As HGU follows training procedure and perform updates together with model, it complements and improves final model.

[0397] The comparison results of the technical solution provided by the present application with the baseline model in terms of BD-Rate metrics are shown in Table 1.Table 1

[0398] As shown in table 1, the training method for a codec with HGU modules provided by the present application outperforms baseline by BD-Rate metrics on JVET common test conditions (CTC). The models are tested in the JVET CTC setup.

[0399] FIG. 9 is a schematic block diagram of an electronic device 900 according to some embodiments of the present application. The electronic device 900 may be the aforementioned encoding device. Referring to FIG. 9, the electronic device900 includes an obtaining unit 901, a scaling unit 902, and a compression unit 903.

[0400] The obtaining unit 901 may be configured to obtain a latent representation of a target signal based on reference signals by using an encoding model.

[0401] The scaling unit 902 may be configured to perform a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation.

[0402] The compression unit 903 may be configured to perform an entropy encoding operation on the scaled latent representation to obtain a compressed result.,0403] Optionally, the scaling unit 902 may configured to perform a multiplying operation on the latent representation by a first scaling value corresponding to the hierarchy level of the target signal to obtain the scaled latent representation.

[0404] Optionally, the electronic device 900 may further includes a transmitting unit, configured to transmit a bitstream carrying the compressed result.

[0405] The scaling unit 902 and the compression unit 903 may be implemented by a processor. In some embodiments, the scaling unit 902 and the compression unit 903 may be implemented by the one processor. In some other embodiments, the scaling unit 902 and the compression unit 903 may be implemented by different processors. In some embodiments, the obtaining unit 901 may be implemented by a receiver or a receiving circuit of the processor. In some embodiments, transmitting unit may be implemented by transmitter or a transmitting circuit of the processor.

[0406] Details on how to encode signal may refer to the above-mentioned embodiments and will not be described here.]0407] FIG. 10 is a schematic block diagram of an electronic device 1000 according to some embodiments of the present application. The electronic device 1000 may be the aforementioned decoding device. Referring to FIG. 10, the electronic device1000 includes an obtaining unit 1001, a decompression unit 1002, a scaling unit 1003, and a reconstruction unit 1004.,0408] The obtaining unit 1001 may be configured to obtain a bitstream corresponding to a target signal.

[0409] The decompression unit 1002 may be configured to perform an entropy decoding operation on the bitstream to obtain a reconstructed latent representation of the target signal.

[0410] The scaling unit 1003 may be configured to perform a scaling operation on the reconstructed latent representation according to a hierarchy level of the target signal to obtain a scaled reconstructed latent representation.

[0411] The reconstruction unit 1004 may be configured to obtain a reconstructed signal based on the scaled reconstructed latent representation and reference signals by using a reconstruction model.(0412] Optionally, the scaling unit 1003 may be configured to perform a multiplying operation on the reconstructed latent representation by a second scaling value corresponding to the hierarchy level of the target signal to obtain the scaled reconstructed latent representation.(0413] The decompression unit 1002, the scaling unit 1003 and the reconstruction unit 1004 may be implemented by a processor. In some embodiments, the decompression unit 1002, the scaling unit 1003 and the reconstruction unit 1004 may be implemented by the one processor. In some other embodiments, the decompression unit 1002, the scaling unit 1003 and the reconstruction unit 1004 may be implemented by two or more processors. The obtaining unit 1001 may be implemented by a receiver or a receiving circuit of the processor.

[0414] Details on how to decode the encoded signal may refer to the above-mentioned embodiments and will not be described here.

[0415] FIG. 11 is a schematic block diagram of an electronic device 1100 according to some embodiments of the present application. The electronic device 1100 may be the aforementioned training device. Referring to FIG. 11, the electronic device1100 includes an obtaining unit 1101 and a training unit 1102.

[0416] The obtaining unit 1101 may be configured to obtain multiple training datasets including a first training dataset and a second training dataset. The first training dataset includes one or more signal sequences. The number of signals in each signal sequence in the first training dataset is a first sequence size. The second training dataset includes one or more signal sequences. The number of signals in each signal sequence in the second training dataset is a second sequence size. The first sequence size is different from the second sequence size.

[0417] The training unit 1102 may be configured to train a codec model with the multiple training datasets. The training process includes multiple training stages including a first training stage and a second training stage. The parameters of the codec model are adjusted based on the first training dataset during the first training stage, and the parameters of the codec model are adjusted based on the second training dataset during the second training stage.

[0418] The training unit 1102 may be implemented by a processor. In some embodiments, training unit 1102 and the obtaining unit 1101 may be implemented by the one processor. In some other embodiments, training unit 1102 and the obtaining unit 1101 may be implemented by two or more processors. In some other embodiments, the obtaining unit 1101 may be implemented by a receiver or a receiving circuit of the processor.

[0419] Details on how to training the codec model may refer to the above-mentioned embodiments and will not be described here.

[0420] As shown in FIG. 12, an electronic device 1200 may include a receiver 1201, a processor 1202, a memory 1203, and a transmitter 1204. The memory 1203 may be configured to store code, instructions, and the like executed by the processor1202.

[0421] It should be understood that the processor 1202 may be an integrated circuit chip and has a signal processing capability. In an implementation process, steps of the foregoing method embodiments may be completed by using a hardware integrated logic circuit in the processor, or by using instructions in a form of software. The processor may be a general purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a system on chip(SoC) or another programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor may implement or perform the methods, the steps, and the logical block diagrams that are disclosed in the embodiments of the present application. The general purpose processor may be a microprocessor, or the processor may be any conventional processor or the like. The steps of the methods disclosed with reference to the embodiments of the present application may be directly performed and completed by the processor, or may be performed and completed by using a combination of hardware in the processor and a software module. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory, and the processor reads information in the memory and completes the steps of the foregoing methods performed by the encoding device, the decoding device or the training device in combination with hardware in the processor.

[0422] It may be understood that the memory 1203 in the embodiments of the present application may be a volatile memory or a nonvolatile memory, or may include both a volatile memory and a nonvolatile memory. The nonvolatile memory may be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), or a flash memory. The volatile memory may be a random access memory (RandomAccess Memory, RAM) and is used as an external cache. By way of example rather than limitation, many forms of RAMs may be used, and are, for example, a static random access memory (Static RAM, SRAM), a dynamic random access memory(Dynamic RAM, DRAM), a synchronous dynamic random access memory (Synchronous DRAM, SDRAM), a double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), an enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), a synchronous link dynamic random access memory(Synchronous link DRAM, SLDRAM), and a direct rambus random access memory (Direct Rambus RAM, DR RAM).

[0423] It should be noted that the memory in the electronic device and the methods described in this specification includes but is not limited to these memories and a memory of any other appropriate type.

[0424] Referring to FIG. 13, an embodiment of the present application provides a system architecture 1300. As shown in the system architecture 1300, a data collection device 1360 is configured to collect training data. The training data may be stored into a database 1330. A training device 1320 may obtain at least one target model / rule 1301 through training based on the training data maintained in the database 1330. Each of the at least one target model / rule 1301 may be corresponding to one of the aforementioned models. The at least one target model / rule 1301 in this embodiment of this application may specifically be a neural network, a convolution network or the like to obtained through training. In this embodiment provided in this application, the network is obtained by training an initialized network. It should be noted that, in actual application, the training data maintained in the database 1330 is not necessarily all collected by the data collection device 1360, and may be received from another device. In addition, it should be noted that the training device 1320 does not necessarily perform training completely based on the training data maintained in the database 1330 to obtain the at least one target model / rule 1301, and may obtain training data from a cloud or another place to perform model training. The foregoing description shall not be construed as a limitation on this embodiment of this application.

[0425] The at least one target model / rule 1301 obtained by the training device 1320 through training may be applied to different systems or devices, for example, applied to an execution device 1310 shown in FIG. 13. The execution device 1310 may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) device, a virtual reality (VR) device, or a vehicle-mounted terminal, or may be a server or the like. In FIG. 13, an I / O interface 1312 is configured on the execution device 1310 and is configured to exchange data with an external device. A user may input data into the I / O interface 1312 by using a customer device 1340. The input data may be data collected by the execution device 1310 by using the data collection device 1360, may be data in the database 1330, or may be data from the customer device 1340.The input data may be an image, a video frame, an audio data, a text data or the like, and the present application is not limited thereto.(0426] A preprocessing module 1313 is configured to perform preprocessing based on the input data (for example, the input image) received by the I / O interface 1312. For example, the preprocessing module 1313 may be configured to implement one or more of the following operations: flipping, rotation, scaling, grayscale conversion, demozaicking, color correction, lensshading correction, auto white balance and the like; and is further configured to implement another preprocessing operation.This is not limited in this application.

[0427] In a related processing procedure in which the execution device 1310 preprocesses the input data or a calculation module 1311 of the execution device 1310 performs calculation, the execution device 1310 may invoke data, code, and the like in a data storage system 1350 to implement corresponding processing, and may also store, into the data storage system 1350, data, an instruction, and the like obtained through corresponding processing.

[0428] Finally, the I / O interface 1312 returns a processing result, for example, the foregoing obtained image processing result, to the customer device 1340, to provide the processing result for the user.

[0429] As previously mentioned, in some embodiments, the training device 1320 may obtain, through training based on the same training data, a part of or all of the at least one target model / rule 1301 together. The corresponding target models / rules 1301 may be used to implement the foregoing targets or complete the foregoing tasks, to provide a required result for the user.

[0430] In a case shown in FIG. 13, the user may manually provide the input data. The manually providing may be performed by using a screen provided on the I / O interface 1312. In another case, the customer device 1340 may automatically send the input data to the I / O interface 1312. If it is required that the customer device 1340 needs to obtain authorization from the user to automatically send the input data, the user may set corresponding permission on the customer device 1340. The user may view, on the customer device 1340, a result output by the execution device 1310. Specifically, the result may be displayed or may be presented in a form of sound, an action, or the like. The customer device 1340 may also be used as a data collection end to collect the input data that is input into the I / O interface 1312 and an output result that is output from the I / O interface1312, as shown in the figure, use the input data and the output result as new sample data, and store the new sample data into the database 1330. Certainly, alternatively, the customer device 1340 may not perform collection, and the I / O interface 1312 directly stores, into the database 1330 as new sample data, the input data that is input into the I / O interface 1312 and an output result that is output from the I / O interface 1312, as shown in the figure.

[0431] It should be noted that FIG. 13 is merely a schematic diagram of a system architecture provided in an embodiment of the present application. A location relationship between a device, a component, a module, and the like shown in the figure constitutes no limitation. For example, in FIG. 13, the data storage system 1350 is an external memory relative to the execution device 1310. In another case, the data storage system 1350 may be alternatively disposed in the execution device1310. In this application, the target model / rule 1301 obtained through training based on the training data may be a neural network used for performing the aforementioned encoding / decoding operation.

[0432] The present application provides a computer readable storage medium including instructions. When theinstructions run on an electronic device, the electronic device is enabled to perform the aforementioned method.

[0433] The present application provides a computer readable storage medium. The computer readable storage medium stores the bitstream obtained by the aforementioned method.

[0434] The present application provides a chip system. The chip system includes a memory and a processor, and the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the aforementioned method.

[0435] The present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is enabled to perform the aforementioned method.

[0436] In the embodiments of the present application, “at least one” means one or more, and “a plurality of’ means two or more. The term “and / or” describes an association relationship between associated objects and represents that three relationships may exist. For example, A and / or B may represent the following three cases: only A exists, both A and B exist, and only B exists, where A and B may be singular or plural. The character “ / ” generally indicates an “or” relationship between the associated objects. “At least one of the following” and a similar expression thereof refer to any combination of these items, including any combination of one item or a plurality of items. For example, at least one of a, b, and c may indicate: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c may be singular or plural.

[0437] A person of ordinary skill in the art may be aware that, in combination with the examples described in the embodiments disclosed in this specification, units and algorithm steps can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed by hardware or software depends on particular applications and design constraints of the technical solutions. A person skilled in the art may use different methods to implement the described functions for each particular application, but it should not be considered that the implementation goes beyond the scope of this application.

[0438] It may be clearly understood by a person skilled in the art that, for the purpose of convenient and brief description, for a detailed working process of the foregoing system, apparatus, and unit, refer to a corresponding process in the foregoing method embodiment. Details are not described herein again.

[0439] In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the described apparatus embodiment is merely an example. For example, the unit division is merely logical function division and may be other division in actual implementation.For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In addition, the displayed or discussed mutual couplings or direct couplings or communicationconnections may be implemented through some interfaces. The indirect couplings or communication connections between the apparatuses or units may be implemented in electronic, mechanical, or other forms.[0440) The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0441] In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit.

[0442] When the functions are implemented in a form of a software functional unit and sold or used as an independent product, the functions may be stored in a computer readable storage medium. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a storage medium, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform all or some of the steps of the methods described in the embodiments of this application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk, or an optical disc.

[0443] The foregoing descriptions are merely specific implementations of this application, but are not intended to limit the protection scope of this application. Any variation or replacement readily figured out by a person skilled in the art within the technical scope disclosed in this application shall fall within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

CLAIMSWhat is claimed is:

1. An encoding method, comprising: obtaining a latent representation of a target signal based on reference signals by using an encoding model; performing a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation; and performing an entropy encoding operation on the scaled latent representation to obtain a compressed result.

2. The method according to claim 1, wherein the performing a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation, comprises: performing a multiplying operation on the latent representation by a first scaling value corresponding to the hierarchy level of the target signal to obtain the scaled latent representation.

3. The method according to claim 2, wherein the first scaling value is a trainable parameter.

4. The method according to any one of claims 1 to 3, further comprising: transmitting a bitstream carrying the compressed result.

5. A decoding method, comprising: obtaining a bitstream corresponding to a target signal; performing an entropy decoding operation on the bitstream to obtain a reconstructed latent representation of the target signal; performing a scaling operation on the reconstructed latent representation according to a hierarchy level of the target signal to obtain a scaled reconstructed latent representation; and obtaining a reconstructed signal based on the scaled reconstructed latent representation and reference signals by using a reconstruction model.

6. The method according to claim 5, wherein the performing a scaling operation on reconstructed latent representation according to a hierarchy level of the target signal to obtain a scaled reconstructed latent representation, comprises: performing a multiplying operation on the reconstructed latent representation by a second scaling value corresponding to the hierarchy level of the target signal to obtain the scaled reconstructed latent representation.

7. The method according to claim 6, wherein the second scaling value is a trainable parameter.

8. A training method, comprising:obtaining multiple training datasets comprising a first training dataset and a second training dataset, wherein the first training dataset comprises one or more signal sequences, number of signals in each signal sequence in the first training dataset is a first sequence size, the second training dataset comprises one or more signal sequences, number of signals in each signal sequence in the second training dataset is a second sequence size, and the first sequence size is different from the second sequence size; and training a codec model with the multiple training datasets, wherein the training process comprises multiple training stages comprising a first training stage and a second training stage, parameters of the codec model are adjusted based on the first training dataset during the first training stage, and parameters of the codec model are adjusted based on the second training dataset during the second training stage.

9. The method according to claim 8, wherein the first sequence size is smaller than the second sequence size, and the first training stage is earlier than the second training stage.

10. The method according to claim 8 or 9, wherein the codec model is applied to video compression, during the first training stage, parameters of the codec model are adjusted based on a first loss, the first loss is determined based on a loss corresponding to a signal sequence in a subset of the first training dataset, the loss corresponding to the signal sequence is determined based on one or more losses corresponding to one or more B-frames in the signal sequence, and the one or moreB-frames are part of B-frames in the signal sequence.

11. The method according to claim 10, wherein the one or more B-frames in the signal sequence comprise one or more non-reference frames in the signal sequence.

12. The method according to claim 10 or 11, wherein the one or more losses corresponding to the one or more B-frames are determined according to one or more hierarchy levels of the one or more B-frames.

13. The method according to claim 12, wherein the loss corresponding to the signal sequence is obtained according to:R1is a rate of a I-frame,is a distortion of the I-frame,is a coefficient is a rate of a B-frame " is adistortion of the B-frame t, path comprises the one or more B-frames, and / eve / (t) is a hierarchy level of the B-frame t.

14. The method according to claim 12 or 13, wherein the one or more losses corresponding to the one or more B-frames are determined according to one or more distortions of the one or more B-frames, and the one or more distortions of the one or more B-frames are related to the one or more hierarchy levels of the one or more B-frames.

15. The method according to claim 14, wherein the one or more distortions of the one or more B-frames are determined based on distortions of channel Y, channel U and channel V of the one or more B-frames, and distortion weights correspondingto the distortions of the channel Y, the channel U and the channel V are related to the one or more hierarchy levels of the one or more B-frames.

16. An electronic device, comprising: an obtaining unit, configured to obtain a latent representation of a target signal based on reference signals by using an encoding model; a scaling unit, configured to perform a scaling operation on the latent representation according to a hierarchy level of the target signal to obtain a scaled latent representation; and a compression unit, configured to perform an entropy encoding operation on the scaled latent representation to obtain a compressed result.

17. The electronic device according to claim 16, wherein the scaling unit is configured to: perform a multiplying operation on the latent representation by a first scaling value corresponding to the hierarchy level of the target signal to obtain the scaled latent representation.

18. The electronic device according to claim 17, wherein the first scaling value is a trainable parameter.

19. The electronic device according to any one of claims 16 to 18, further comprising: a transmitting unit, configured to transmit a bitstream carrying the compressed result.

20. An electronic device, comprising: an obtaining unit, configured to obtain a bitstream corresponding to a target signal; a decompression unit, configured to perform an entropy decoding operation on the bitstream to obtain a reconstructed latent representation of the target signal; a scaling unit, configured to perform a scaling operation on the reconstructed latent representation according to a hierarchy level of the target signal to obtain a scaled reconstructed latent representation; and a reconstruction unit, configured to obtain a reconstructed signal based on the scaled reconstructed latent representation and reference signals by using a reconstruction model.

21. The electronic device according to claim 20, wherein the scaling unit is configured to: perform a multiplying operation on the reconstructed latent representation by a second scaling value corresponding to the hierarchy level of the target signal to obtain the scaled reconstructed latent representation.

22. The electronic device according to claim 21, wherein the second scaling value is a trainable parameter.

23. An electronic device, comprising: an obtaining unit, configured to obtain multiple training datasets comprising a first training dataset and a second training dataset, wherein the first training dataset comprises one or more signal sequences, number of signals in each signal sequencein the first training dataset is a first sequence size, the second training dataset comprises one or more signal sequences, number of signals in each signal sequence in the second training dataset is a second sequence size, and the first sequence size is different from the second sequence size; and a training unit, configured to train a codec model with the multiple training datasets, wherein the training process comprises multiple training stages comprising a first training stage and a second training stage, parameters of the codec model are adjusted based on the first training dataset during the first training stage, and parameters of the codec model are adjusted based on the second training dataset during the second training stage.

24. The electronic device according to claim 23, wherein the first sequence size is smaller than the second sequence size, and the first training stage is earlier than the second training stage.

25. The electronic device according to claim 23 or 24, wherein the codec model is applied to video compression, during the first training stage, parameters of the codec model are adjusted based on a first loss, the first loss is determined based on a loss corresponding to a signal sequence in a subset of the first training dataset, the loss corresponding to the signal sequence is determined based on one or more losses corresponding to one or more B-frames in the signal sequence, and the one or moreB-frames are part of B-frames in the signal sequence.

26. The electronic device according to claim 25, wherein the one or more B-frames in the signal sequence comprise one or more non-reference frames in the signal sequence.

27. The electronic device according to claim 25 or 26, wherein the one or more losses corresponding to the one or moreB-frames are determined according to one or more hierarchy levels of the one or more B-frames.

28. The electronic device according to claim 27, wherein the loss corresponding to the signal sequence is obtained according to:is a rate of a I- frame,is a distortion of the I-frame is a coefficient, is a rate of a B-frame is adistortion of the B-frame t, path comprises the one or more B-frames, and level(t) is a hierarchy level of the B-frame t.

29. The electronic device according to claim 27 or 28, wherein the one or more losses corresponding to the one or moreB-frames are determined according to one or more distortions of the one or more B-frames, and the one or more distortions of the one or more B-frames are related to the one or more hierarchy levels of the one or more B-frames.

30. The electronic device according to claim 29, wherein the one or more distortions of the one or more B-frames are determined based on distortions of channel Y, channel U and channel V of the one or more B-frames, and distortion weights corresponding to the distortions of the channel Y, the channel U and the channel V are related to the one or more hierarchylevels of the one or more B-frames.

31. A computer readable storage medium, wherein the computer readable storage medium stores instructions, and when the instructions run on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 4, the method according to any one of claims 5 to 7, or the method according to any one of claims 8 to 15.

32. A computer readable storage medium, wherein the computer readable storage medium stores a bitstream, wherein the bitstream is obtained by using the method according to any one of claims 1 to 4.

33. An electronic device, comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that the electronic device performs the method according to any one of claims 1 to 4, the method according to any one of claims5 to 7, or the method according to any one of claims 8 to 15.

34. A chip system, comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program, so that an electronic device on which the chip system is disposed performs the method according to any one of claims 1 to 4, the method according to any one of claims 5 to 7, or the method according to any one of claims 8 to 15.

35. A computer program product, wherein when the computer program product runs on an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 4, the method according to any one of claims 5 to7, or the method according to any one of claims 8 to 15.

Images