Encoding method, decoding method, and encoding apparatus, decoding apparatus

Abstract

The application provides an encoding method, a decoding method, and an encoding device and a decoding device, comprising: encoding a to-be-encoded image by using a neural network-based encoding technology to obtain first encoding data; compressively encoding a model parameter set of a neural network model included in the neural network-based encoding technology by using a differential prediction mode to obtain second encoding data; and transmitting the first encoding data and the second encoding data. The scheme provided by the application can save transmission bandwidth.

Description

[0001] Copyright Statement

[0002] This patent document discloses material protected by copyright. The copyright belongs to the copyright holder. The copyright holder does not object to anyone copying this patent document or the patent disclosure as it exists in the official records and archives of the Patent and Trademark Office. Technical Field

[0003] This application relates to the field of image processing, and more specifically, to an encoding method, a decoding method, an encoding apparatus, and a decoding apparatus. Background Technology

[0004] Currently, one approach to managing and transmitting model parameters of neural network models in neural network-based encoding techniques is to fix the model parameters of the neural network model in a common library file that is available to both the encoder and decoder. The encoder and decoder can obtain the model parameters of the neural network model from this common library file when performing encoding or decoding. However, this approach will be less effective if some encoding tools of the encoder change.

[0005] Another approach is to transmit the model parameters of the neural network model to the decoder via a bitstream. This allows for flexible adjustment of the model parameters according to the encoder's requirements. However, for deep learning neural network models with greater depth, the number of parameters is generally larger, and directly writing them into the bitstream for transmission increases the transmission bandwidth. Summary of the Invention

[0006] This application provides an encoding method, a decoding method, and an encoding and decoding apparatus that can save transmission bandwidth.

[0007] In a first aspect, an encoding method is provided, comprising: encoding an image to be encoded using a neural network-based encoding technique to obtain first encoded data; compressing and encoding the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction to obtain second encoded data; and transmitting the first encoded data and the second encoded data.

[0008] Secondly, a decoding method is provided, comprising: receiving first encoded data and second encoded data; decoding the second encoded data to obtain model parameters in a model parameter set of a neural network model; and using the model parameters in the model parameter set of the neural network model to decode the first encoded data to obtain a decoded image.

[0009] Thirdly, an encoding apparatus is provided, comprising: a processor, configured to encode an image to be encoded using a neural network-based encoding technique to obtain first encoded data; to compress and encode the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction to obtain second encoded data; and to transmit the first encoded data and the second encoded data.

[0010] Fourthly, a decoding apparatus is provided, comprising: a processor, configured to receive first encoded data and second encoded data; decode the second encoded data to obtain model parameters in a model parameter set of a neural network model; and decode the first encoded data using the model parameters in the model parameter set of the neural network model to obtain a decoded image.

[0011] Fifthly, an encoding apparatus is provided, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to perform the methods described in the first aspect or its various implementations.

[0012] Sixthly, a decoding apparatus is provided, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to perform the methods described in the second aspect or its various implementations.

[0013] In a seventh aspect, a chip is provided for implementing the methods of the first aspect or its various implementations described above.

[0014] Specifically, the chip includes a processor for calling and running a computer program from memory, causing a device on which the chip is mounted to perform the methods described in the first aspect or its various implementations above.

[0015] Eighthly, a chip is provided for implementing the methods of the second aspect or its various implementations described above.

[0016] Specifically, the chip includes a processor for calling and running a computer program from memory, causing a device on which the chip is installed to perform the methods described in the second aspect above or in various implementations thereof.

[0017] A ninth aspect provides a computer-readable storage medium for storing a computer program including instructions for performing the methods of the first aspect or any possible implementation thereof.

[0018] In a tenth aspect, a computer-readable storage medium is provided for storing a computer program including instructions for performing the methods of the second aspect or any possible implementation thereof.

[0019] Eleventhly, a computer program product is provided, including computer program instructions that cause a computer to perform the methods in the first aspect or various implementations thereof.

[0020] In a twelfth aspect, a computer program product is provided, including computer program instructions that cause a computer to perform the methods of the second aspect or various implementations thereof.

[0021] The solution provided in this application uses differential prediction to compress and encode the model parameter set of the neural network model at the encoding end, and then transmits the second encoded data obtained after compression and encoding, which can save transmission bandwidth. Attached Figure Description

[0022] Figure 1 This is an architecture diagram of the technical solution applied in the embodiments of this application; Figure 2 This is a schematic diagram of a video coding framework 2 according to an embodiment of this application; Figure 3 This is a schematic flowchart of an embodiment of the encoding method provided in this application; Figure 4 This is a schematic flowchart of a decoding method provided in an embodiment of this application; Figure 5 This is a schematic diagram of an encoding framework 2 provided in an embodiment of this application; Figure 6 This is a schematic diagram of image transmission using the intelligent encoding technology provided in the embodiments of this application; Figure 7 This is a schematic diagram of a process for training a neural network model provided in an embodiment of this application; Figure 8a This is a flowchart illustrating the application of intelligent encoding technology in a video encoder according to an embodiment of this application; Figure 8b This is a flowchart illustrating the application of intelligent encoding technology in a video decoder according to an embodiment of this application; Figure 9 This is a schematic diagram of an encoding device provided in an embodiment of this application; Figure 10 This is a schematic diagram of a decoding device provided in an embodiment of this application; Figure 11 This is a schematic diagram of an encoding device provided in another embodiment of this application; Figure 12 This is a schematic diagram of a decoding device provided in another embodiment of this application; Figure 13 This is a schematic structural diagram of the chip provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions in the embodiments of this application are described below.

[0024] Unless otherwise stated, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application.

[0025] Figure 1 This is an architecture diagram of the technical solution applied in the embodiments of this application.

[0026] like Figure 1 As shown, system 100 can receive data to be processed 102, process the data to be processed 102, and generate processed data 108. For example, system 100 can receive data to be encoded, encode the data to be encoded to generate encoded data, or system 100 can receive data to be decoded, decode the data to be decoded to generate decoded data. In some embodiments, the components in system 100 can be implemented by one or more processors, which can be processors in computing devices or processors in mobile devices (e.g., drones). The processor can be any type of processor, and this embodiment of the invention does not limit this. In some possible designs, the processor can include an encoder, a decoder, or a codec, etc. System 100 may also include one or more memories. The memories can be used to store instructions and data, such as computer-executable instructions for implementing the technical solutions of this embodiment of the invention, data to be processed 102, processed data 108, etc. The memories can be any type of memory, and this embodiment of the invention does not limit this.

[0027] The data to be encoded can include text, images, graphic objects, animation sequences, audio, video, or any other data that needs to be encoded. In some cases, the data to be encoded can include sensing data from sensors, such as vision sensors (e.g., cameras, infrared sensors), microphones, near-field sensors (e.g., ultrasonic sensors, radar), position sensors, temperature sensors, touch sensors, etc. In some cases, the data to be encoded can include information from the user, such as biometric information, which can include facial features, fingerprint scans, retinal scans, voice recordings, DNA samples, etc.

[0028] Figure 2 This is a schematic diagram of a video encoding framework 2 according to an embodiment of this application. For example... Figure 2As shown, after receiving the video to be encoded, each frame in the video is encoded sequentially, starting from the first frame. The current encoded frame mainly undergoes prediction, transformation, quantization, and entropy coding, ultimately outputting the bitstream of the current encoded frame. Correspondingly, the decoding process typically involves decoding the received bitstream in reverse order to recover the video frame information before decoding.

[0029] Specifically, such as Figure 2 As shown, the video encoding framework 2 includes an encoding control module 201, used for decision-making and control actions during the encoding process, as well as parameter selection. For example, as Figure 2 As shown, the encoding control module 201 controls the parameters used in the transformation, quantization, inverse quantization, and inverse transformation, controls the selection of intra-frame mode or inter-frame mode, and controls the parameters of motion estimation and filtering. The control parameters of the encoding control module 201 are also input to the entropy encoding module to be encoded into a part of the encoded bitstream.

[0030] When encoding a current coded frame begins, the frame is divided into 202 blocks. Specifically, it is first sliced, then divided into blocks. Optionally, in one example, the coded frame is divided into multiple non-overlapping largest coding tree units (CTUs). Each CTU can be iteratively divided into a series of smaller coding units (CUs) using a quadtree, binary tree, or ternary tree approach. In some examples, a CU may also contain associated prediction units (PUs) and transform units (TUs), where the PU is the basic unit for prediction, and the TU is the basic unit for transformation and quantization. In some examples, the PU and TU are obtained by dividing a CU into one or more blocks, where one PU contains multiple prediction blocks (PBs) and related syntax elements. In some examples, the PU and TU can be the same, or they can be obtained from CUs using different partitioning methods. In some examples, at least two of the CU, PU, ​​and TU are the same; for example, CU, PU, ​​and TU are not distinguished, and prediction, quantization, and transformation are all performed at the CU level. For ease of description, CTU, CU, or other data units will be referred to as coded blocks in the following text.

[0031] It should be understood that, in the embodiments of this application, the data unit targeted by video encoding can be a frame, stripe, coding tree unit, coding unit, coding block, or any group thereof. The size of the data unit can vary in different embodiments.

[0032] Specifically, such as Figure 2 As shown, after the coded frame is divided into multiple coded blocks, a prediction process is performed to remove spatial and temporal redundancy information in the current coded frame. Currently, commonly used predictive coding methods include intra-frame prediction and inter-frame prediction. Intra-frame prediction uses only the reconstructed information from the current frame to predict the current coded block, while inter-frame prediction uses information from other previously reconstructed frames (also called reference frames) to predict the current coded block. Specifically, in this embodiment, the coding control module 201 is used to decide whether to perform intra-frame prediction or inter-frame prediction.

[0033] When the intra-prediction mode is selected, the intra-prediction process 203 includes obtaining the reconstructed blocks of the coded neighboring blocks around the current coded block as reference blocks. Based on the pixel values ​​of the reference blocks, the prediction mode method is used to calculate the prediction values ​​to generate a prediction block. The corresponding pixel values ​​of the current coded block and the prediction block are subtracted to obtain the residual of the current coded block. The residual of the current coded block is transformed 204, quantized 205, and entropy encoded 210 to form the bitstream of the current coded block. Furthermore, all coded blocks of the current coded frame undergo the above encoding process to form a part of the coded bitstream of the coded frame. In addition, the control and reference data generated in the intra-prediction 203 are also encoded by entropy encoded 210 to form a part of the coded bitstream.

[0034] Specifically, Transform 204 is used to remove the correlation of the residuals of image blocks in order to improve coding efficiency. The transformation of the residual data of the current coding block is usually carried out by two-dimensional discrete cosine transform (DCT) and two-dimensional discrete sine transform (DST). For example, at the coding end, the residual information of the coding block is multiplied by an N×M transformation matrix and its transpose matrix respectively, and the transformation coefficients of the current coding block are obtained after multiplication.

[0035] After generating the transform coefficients, quantization 205 further improves compression efficiency. Quantization yields quantized coefficients, which are then entropy-encoded 210 to obtain the residual bitstream of the current coding block. Entropy encoding methods include, but are not limited to, Context Adaptive Binary Arithmetic Coding (CABAC). Finally, the bitstream obtained from entropy encoding and the encoded mode information are stored or sent to the decoding end. At the encoding end, the quantization result is inverse-quantized 206, and the inverse-quantization result is inverse-transformed 207. After inverse-transformation 207, the reconstructed pixels are obtained using the inverse-transformation result and motion compensation result. Then, the reconstructed pixels are filtered (i.e., loop filtering) 211. After 211, the filtered reconstructed image (belonging to the reconstructed video frame) is output. Subsequently, the reconstructed image can be used as a reference frame image for inter-frame prediction of other frames. In this embodiment, the reconstructed image can also be referred to as the reconstructed image or the reconstructed image.

[0036] Specifically, the encoded neighboring blocks in the intra-frame prediction 203 process are: neighboring blocks that were encoded before the current coded block was encoded. The reconstructed block is obtained by adding the residual generated during the encoding process of these neighboring blocks to the predicted block of these neighboring blocks after transform 204, quantization 205, inverse quantization 206, and inverse transform 207. Correspondingly, inverse quantization 206 and inverse transform 207 are the inverse processes of quantization 206 and transform 204, used to recover the residual data before quantization and transform.

[0037] like Figure 2 As shown, when the inter-frame prediction mode is selected, the inter-frame prediction process includes motion estimation (ME) 208 and motion compensation (MC) 209. Specifically, the encoder can perform motion estimation 208 based on reference frame images in the reconstructed video frame. In one or more reference frame images, it searches for the image block most similar to the current coding block according to a certain matching criterion, using it as the prediction block. The relative displacement between the prediction block and the current coding block is the motion vector (MV) of the current coding block. The residual of the coding block is obtained by subtracting the original value of the pixel of the coding block from the corresponding pixel value of the prediction block. The residual of the current coding block is transformed 204, quantized 205, and entropy encoded 210 to form a part of the encoded bitstream of the coded frame. For the decoder, motion compensation 209 can be performed based on the determined motion vector and the prediction block to obtain the current coding block.

[0038] Among them, such as Figure 2As shown, the reconstructed video frame is the video frame obtained after filtering 211. The reconstructed video frame includes one or more reconstructed images. Filtering 211 is used to reduce compression distortions such as blockiness and ringing during the encoding process. The reconstructed video frame is used to provide a reference frame for inter-frame prediction during the encoding process. During the decoding process, the reconstructed video frame is post-processed and output as the final decoded video.

[0039] Specifically, inter-frame prediction modes can include Advanced Motion Vector Prediction (AMVP) mode, Merge mode, or Skip mode.

[0040] For AMVP mode, motion vector prediction (MVP) can be determined first. After obtaining MVP, the starting point of motion estimation can be determined based on MVP. Motion search is performed near the starting point. After the search is completed, the optimal MV is obtained. The position of the reference block in the reference image is determined by MV. The reference block is subtracted from the current block to obtain the residual block. MV is subtracted from MVP to obtain the motion vector difference (MVD). The index of MVD and MVP is transmitted to the decoder through the bitstream.

[0041] In Merge mode, the MVP can be determined first and directly set as the MV of the current block. To obtain the MVP, an MVP candidate list (merge candidate list) can be constructed. The MVP candidate list can include at least one candidate MVP, and each candidate MVP can correspond to an index. After the encoder selects an MVP from the MVP candidate list, it can write the MVP index into the bitstream. The decoder can then find the MVP corresponding to that index from the MVP candidate list to decode the image block.

[0042] It should be understood that the above process is only one specific implementation of the Merge pattern. The Merge pattern can also have other implementations.

[0043] For example, Skip mode is a special case of Merge mode. After obtaining the MV according to Merge mode, if the encoder determines that the current block and the reference block are basically the same, then there is no need to transmit residual data. Only the index of the MVP needs to be transmitted, and further, a flag can be transmitted to indicate that the current block can be directly obtained from the reference block.

[0044] In other words, the Merge mode is characterized by MV=MVP (MVD=0); while the Skip mode has an additional characteristic: the reconstructed value rec=the predicted value pred (the residual value resi=0).

[0045] For the decoding end, the operations corresponding to those at the encoding end are performed. First, residual information is obtained using entropy decoding, inverse quantization, and inverse transform. Then, based on the decoded bitstream, it is determined whether intra-frame prediction or inter-frame prediction is used for the current image block. If it is intra-frame prediction, prediction information is constructed using the reconstructed image blocks in the current frame according to the intra-frame prediction method. If it is inter-frame prediction, motion information needs to be parsed, and the parsed motion information is used to determine the reference block in the reconstructed image to obtain prediction information. Next, the prediction information and residual information are superimposed, and after filtering, the reconstructed information can be obtained.

[0046] This application mainly relates to the encoding and decoding technology of images or videos based on neural networks (also known as "intelligent encoding and decoding technology"). Therefore, the following is a brief introduction to the relevant content of neural networks.

[0047] A neural network (NN), also known as an "artificial" neural network (ANN), is an information processing system that shares some performance characteristics with biological neural networks. A neural network system consists of many simple, highly interconnected processing components that process information through dynamic state responses to external inputs. These processing components can be thought of as neurons in the human brain, where each perceptron receives multiple inputs and computes a weighted sum of them. In the field of neural networks, perceptrons are considered mathematical models of biological neurons. Furthermore, these interconnected processing components are typically organized in layers. For recognition applications, external inputs may correspond to patterns presented to the network, which communicate with one or more intermediate layers, also called "hidden layers," where the actual processing is done through a weighted "connection" system.

[0048] ANNs can use different architectures to specify which variables and their topological relationships are involved in the network. For example, variables involved in a neural network might be the weights of the connections between neurons, as well as the activity of the neurons. Most artificial neural networks contain some form of "learning rule" that modifies the weights of the connections based on the presented input patterns. In a sense, artificial neural networks learn from instances, much like their biological counterparts.

[0049] Deep Neural Networks (DNNs), or deep multilayer neural networks, correspond to neural networks with multiple levels of interconnected nodes, allowing them to compactly represent highly nonlinear and highly variable functions. However, the computational complexity of DNNs increases rapidly with the number of nodes associated with a large number of layers.

[0050] Currently, one approach to managing and transmitting model parameters of neural network models in neural network-based encoding techniques is to fix the model parameters of the neural network model in a common library file that is available to both the encoder and decoder. The encoder and decoder can obtain the model parameters of the neural network model from this common library file when performing encoding or decoding. However, this approach will be less effective if some encoding tools of the encoder change.

[0051] Another approach is to transmit the model parameters of the neural network model to the decoder via a bitstream. This allows for flexible adjustment of the model parameters according to the encoder's requirements. However, for deep learning neural network models with greater depth, the number of parameters is generally larger, and directly writing them into the bitstream for transmission increases the transmission bandwidth.

[0052] This application provides an encoding method, a decoding method, and an encoding and decoding apparatus that can save transmission bandwidth.

[0053] The following will combine Figure 3 The encoding method 300 provided in the embodiments of this application is described in detail.

[0054] like Figure 3 The diagram shown is a schematic diagram of an encoding method 300 provided in an embodiment of this application. The method 300 may include steps 310-330.

[0055] 310. The image to be encoded is encoded using a neural network-based encoding technique to obtain the first encoded data.

[0056] In this embodiment of the application, using neural network-based encoding technology (also known as "intelligent encoding technology") to encode the image to be encoded can be understood as: using a neural network-based encoding module to replace or supplement some encoding modules in the encoding / decoding framework to encode the image to be encoded, or using a neural network-based encoding framework to replace the original encoding framework to encode the image to be encoded.

[0057] Neural network-based coding technologies mainly include three aspects: hybrid neural network video coding (which embeds neural network coding modules into traditional video coding frameworks instead of traditional coding modules), neural network rate-distortion optimization coding, and end-to-end neural network video coding.

[0058] Optionally, in some embodiments, for at least one of prediction, transformation, quantization, entropy coding, and filtering in the coding process, a coding module based on a neural network model can be used to replace some coding modules in the coding framework to encode the image to be encoded.

[0059] It should be noted that, for filtering during the encoding process, a filtering module based on a neural network model can be added to the existing filtering module to filter the reconstructed image.

[0060] For example, for intra-frame prediction during the encoding process, a neural network-based intra-frame prediction method is compared with the original intra-frame prediction method to determine the optimal intra-frame prediction method for prediction; for inter-frame prediction during the encoding process, neural network-based image super-resolution technology can be used for prediction, which can improve motion estimation performance and thus improve the efficiency of inter-frame prediction.

[0061] For entropy coding in the encoding process, a context probability estimation method based on neural network technology can be used to replace the traditional rule-based context probability prediction model, and applied to the entropy coding process of coefficients or other syntax elements. For filtering in the encoding process, a neural network filter (NNF) technique can be added after deblocking filtering.

[0062] End-to-end neural network video coding completely abandons the traditional hybrid video coding framework, directly inputting video images and related information, and using a neural network-based prediction method for prediction and compression coding.

[0063] 320. The model parameter set of the neural network model contained in the neural network-based encoding technology is compressed and encoded using differential prediction to obtain the second encoded data.

[0064] 330, transmit the first encoded data and the second encoded data.

[0065] Optionally, the neural network model in the embodiments of this application may include an offline trained neural network model or an online trained neural network model, without limitation.

[0066] The neural network model in the embodiments of this application may include DNN, Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) or other neural network variants, and this application does not impose specific limitations on it.

[0067] The model parameters of the neural network model in the embodiments of this application include, but are not limited to, the number of layers of the neural network, the number of neurons, and the weights of the connections between neurons.

[0068] Correspondingly, for the decoding end, the bitstream can be decoded based on the model parameters of the received neural network model. For example... Figure 4 The diagram shown is a schematic diagram of a decoding method 400 provided in an embodiment of this application. The method 400 may include steps 410-430.

[0069] 410, Receive the first encoded data and the second encoded data.

[0070] 420. Decode the second encoded data to obtain the model parameters in the model parameter set of the neural network model.

[0071] 430. The first encoded data is decoded using the model parameters in the model parameter set of the neural network model to obtain the decoded image.

[0072] Optionally, in some embodiments, for at least one of entropy decoding, inverse quantization, inverse transform, predictive reconstruction, or filtering in the decoding process, a decoding module based on a neural network model can be used to replace some decoding modules in the decoding framework to decode the image to be decoded.

[0073] It should be noted that, for filtering during the decoding process, a filtering module based on a neural network model can be added to the existing filtering module to filter the predicted and reconstructed image.

[0074] The solution provided in this application uses differential prediction to compress and encode the model parameter set of the neural network model at the encoding end, and then transmits the second encoded data obtained after compression and encoding, which can save transmission bandwidth.

[0075] Optionally, in some embodiments, the bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to a first threshold.

[0076] The bit width of the model parameter in the embodiments of this application can be understood as the number of bits occupied by the model parameter after it is converted into binary format.

[0077] For example, suppose the model parameter set of the neural network model includes the value 10. The encoder can encode it as "1010". After the value is converted to binary format, it occupies 4 bits, that is, the bit width of the value is 4 bits. Suppose the model parameter set of the neural network model includes the value 5. The encoder can encode it as "101". After the value is converted to binary format, it occupies 3 bits, that is, the bit width of the value is 3 bits.

[0078] If the first threshold is 4, when the encoding end trains the neural network model, the maximum value of the model parameters in the model parameter set of the trained neural network model can be 15; if the first threshold is 8, when the encoding end trains the neural network model, the maximum value of the model parameters in the model parameter set of the trained neural network model can be 255.

[0079] In the embodiments of this application, the first threshold may be a predetermined value agreed by the protocol, or a preset value of the encoding end; the first threshold in the embodiments of this application may be a fixed value, or a variable value, which is not limited herein.

[0080] In the solution provided by this application, the bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to the first threshold, which is beneficial for compression encoding of the model parameter set of the neural network model by means of differential prediction.

[0081] As mentioned above, the neural network model includes an offline-trained neural network model or an online-trained neural network model. In the following, taking these two types of neural network models as examples, the relevant content of compressing and encoding the model parameter set of the neural network model included in the neural network-based encoding technology by means of differential prediction will be introduced respectively.

[0082] Case 1: Compressing and encoding the model parameter set of the online-trained neural network model included in the neural network-based encoding technology by means of differential prediction Optionally, in some embodiments, the neural network model comprises an online-trained neural network model, and said performing compression encoding on the model parameter set of the neural network model included in the neural network-based encoding technology by means of differential prediction comprises: Encoding the model parameter set of the first neural network model; For the model parameter set of the n-th neural network model, encoding the parameter difference of corresponding parameters between the model parameter set of the n-th neural network model and the model parameter set of the m-th neural network model, wherein n is a positive integer greater than 1, and m is a positive integer greater than or equal to 1.

[0083] In the embodiments of this application, m < n, that is, when encoding the model parameter set of the n-th neural network model, it is only necessary to encode the parameter difference of corresponding parameters between the model parameter set of the n-th neural network model and the model parameter set of the m-th neural network model that precedes the n-th neural network model.

[0084] For example, for the model parameter set of the third neural network model, the parameter difference between the corresponding parameters in the model parameter set of the third neural network model and the model parameter set of the second neural network model can be encoded and transmitted to the decoding end; or, for the model parameter set of the third neural network model, the parameter difference between the corresponding parameters in the model parameter set of the third neural network model and the model parameter set of the first neural network model can also be encoded and transmitted to the decoding end; there is no limitation.

[0085] Correspondingly, the decoding end can receive the encoded data of the model parameter set of the first neural network model, and the encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The received encoded data is then decoded to obtain the model parameters in the model parameter set of the first neural network model, and the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The model parameters of the model parameter set of the nth neural network model are obtained by using the parameter difference obtained after decoding and the model parameters in the model parameter set of the mth neural network model.

[0086] It should be noted that, in the embodiments of this application, when the encoding end encodes the model parameter set of the first neural network model, it can encode the model parameter set of the first neural network model separately and transmit it to the decoding end; or it can encode the parameter difference between the model parameters in the model parameter set of the first neural network model and the preset value, and transmit the parameter difference to the decoding end; there is no limitation.

[0087] The solution provided in this application allows the encoder to encode an image using an online-trained neural network model. In addition to encoding the model parameter set of the first neural network model, the encoder can also encode the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model, and transmit the encoded data corresponding to the parameter difference to the decoder, thereby saving transmission bandwidth.

[0088] Optionally, in some embodiments, the parameter difference between the corresponding parameters in the model parameter set encoding the nth neural network model and the model parameter set encoding the mth neural network model includes: If the number of model parameters in the model parameter set of the nth neural network model is less than or equal to the number of model parameters in the model parameter set of the mth neural network model, a first parameter difference is encoded. The first parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model.

[0089] Optionally, in some embodiments, the neural network model includes an online-trained neural network model, and the compression encoding of the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: The set of model parameters for encoding the first neural network model; If the number of model parameters in the model parameter set of the nth neural network model is greater than the number of model parameters in the model parameter set of the mth neural network model, a first parameter difference and a second parameter difference are encoded. The first parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The second parameter difference is the parameter difference between the parameters in the model parameter set of the nth model model other than the model parameter corresponding to the first parameter difference and a preset value.

[0090] In this embodiment of the application, for the model parameters in the model parameter set of the nth neural network model, the encoder can determine the parameter difference during the encoding process based on the number of model parameters included in the model parameter set of the nth neural network model and the number of model parameters included in the model parameter set of the mth neural network model.

[0091] Specifically, if the number of model parameters in the parameter set of the nth neural network model is less than or equal to the number of model parameters in the parameter set of the mth neural network model, the first parameter difference is encoded; if the number of model parameters in the parameter set of the nth neural network model is greater than the number of model parameters in the parameter set of the mth neural network model, the first parameter difference and the second parameter difference are encoded.

[0092] For example, assuming that the model parameter set of the nth neural network model includes 100 model parameters and the model parameter set of the mth neural network model includes 100 model parameters, the encoder can encode the parameter difference of the corresponding parameters among these 100 model parameters, that is, encode the parameter difference of the corresponding parameters among the 100 model parameters included in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model.

[0093] Assuming the parameter set of the nth neural network model contains 80 parameters and the parameter set of the mth neural network model contains 100 parameters, the encoder can encode the parameter differences between the 80 parameters in the parameter set of the nth neural network model and the corresponding 80 parameters in the parameter set of the mth neural network model.

[0094] Assuming the model parameter set of the nth neural network model contains 100 model parameters and the model parameter set of the mth neural network model contains 80 model parameters, the encoder can encode the parameter differences between the corresponding parameters in the model parameter set of the nth neural network model and the 80 model parameters in the model parameter set of the mth neural network model. For the remaining 20 model parameters in the model parameter set of the nth neural network model, the encoder can encode the parameter differences between these 20 model parameters and the preset values.

[0095] If the bit width of the model parameters of the above neural network model is 'a', the preset value can be that when the encoder encodes the remaining 20 model parameters, it can encode the parameter difference between these 20 model parameters and the parameter difference.

[0096] It should be noted that the above preset values ​​can also be other values, such as, etc., without restriction.

[0097] Accordingly, for the decoding end, the decoding end decodes the encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. This may yield a first parameter difference, or a first parameter difference and a second parameter difference. If the first parameter difference is obtained, the decoding end sums the first parameter difference with the corresponding model parameters in the model parameter set of the mth neural network model to obtain the model parameter set of the nth neural network model. If both the first and second parameter differences are obtained, the decoding end sums the first parameter difference with the corresponding model parameters in the model parameter set of the mth neural network model, and sums the second parameter difference with a preset value to obtain the model parameter set of the nth neural network model.

[0098] As mentioned above, for online trained neural network models, the model parameter set of the first neural network model can be encoded, and the parameter differences between the corresponding parameters in the model parameter sets of the nth and mth neural network models can be encoded. The encoding method is different for the model parameters in the model parameter sets of different neural network models. Please refer to the following text for details.

[0099] Optionally, in some embodiments, encoding the model parameter set of the first neural network model includes: encoding the model parameter set of the first neural network model using a first encoding method; The parameter difference between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model is encoded by a second encoding method.

[0100] In this embodiment, the model parameter set of the first neural network model, and the parameter differences between corresponding parameters in the model parameter sets of the nth and mth neural network models can be encoded using different encoding methods. Specifically, the model parameter set of the first neural network model can be encoded using a first encoding method; the parameter differences between corresponding parameters in the model parameter sets of the nth and mth neural network models can be encoded using a second encoding method.

[0101] Correspondingly, for the decoding end, the encoded data of the model parameter set of the first neural network model is decoded using the first decoding method, and the encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model is decoded using the second decoding method. The decoded data is then summed to obtain the model parameter set of the neural network model.

[0102] It should be noted that if the number of model parameters in the parameter set of the nth neural network model is greater than the number of model parameters in the parameter set of the mth neural network model, the encoder can encode the parameter differences between the corresponding parameters in the parameter sets of the nth and mth neural network models using a second encoding method, as well as the parameter differences between the parameters in the nth parameter set (excluding the parameters corresponding to the first parameter difference) and preset values. Correspondingly, the decoder can decode the parameter differences between the parameter sets of the nth and mth neural network models using a second decoding method, as well as the parameter differences between the parameters in the nth parameter set (excluding the parameters corresponding to the first parameter difference) and preset values.

[0103] The solution provided in this application encodes the parameter set of the first neural network model, and the parameter differences between corresponding parameters in the parameter sets of the nth and mth neural network models using different encoding methods. Alternatively, it encodes the parameter differences between the parameter sets of the first and mth neural network models, and the parameter differences between the parameters in the parameter set of the nth model (excluding the parameter corresponding to the first parameter difference) and preset values ​​using different encoding methods, thereby improving the flexibility of encoding.

[0104] Optionally, in some embodiments, the first encoding method is a fixed-length code encoding method, and the second encoding method is a variable-length code encoding method.

[0105] The fixed-length code encoding method in the embodiments of this application can be understood as an encoding method with a fixed bit width; the variable-length code encoding method can be understood as an encoding method with a variable bit width.

[0106] For example, assuming the model parameters of a neural network model have a bit width of 4 bits, the model parameters of the first neural network model's model parameter set can be encoded with a bit width of 4 bits.

[0107] Specifically, if the model parameter set of the first neural network model includes the value 10, the encoder can encode it as "1010"; if the model parameter set of the first neural network model also includes the value 2, the encoder can encode it as "0010". In other words, regardless of the value of the model parameter set of the first neural network model, the encoder encodes it with a 4-bit width.

[0108] For the parameter set of the second neural network model, the encoder encodes the parameter differences between the corresponding parameters in the parameter set of the second neural network model and those in the parameter set of the first neural network model. If the parameter difference includes the value 2, the encoder can encode it as "10", meaning the parameter difference 2 occupies a bit width of 2 bits. If the parameter difference includes the value 5, the encoder can encode it as "101", meaning the parameter difference 5 occupies a bit width of 3 bits. In other words, for the parameter differences between the parameter sets of the second and first neural network models, the encoder encodes different bit widths based on the magnitude of the parameter differences.

[0109] Similarly, for the model parameter set of the nth neural network model, the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model can be encoded using the variable-length code encoding method described above.

[0110] It should be understood that the above values ​​are for illustrative purposes only and other values ​​may also be used, and should not impose any special limitations on this application.

[0111] The scheme provided in this application encodes the model parameters in the model parameter set of the first neural network model using a fixed-length code encoding method, and encodes the parameter differences between the corresponding parameters in the model parameter sets of the nth and mth neural network models using a variable-length code encoding method, which can further save bandwidth.

[0112] Optionally, in some embodiments, m is a parameter fixed to or preset in the encoding end before encoding, or m is determined by the encoding end based on preset rules.

[0113] For the decoding end, m is a parameter that is fixed to or preset in the decoding end, or m is determined based on preset rules.

[0114] Optionally, in some embodiments, m=1, or m=n-1.

[0115] In this embodiment, m can be a parameter fixed or preset in the encoding or decoding end. For example, the encoding end can fix or preset m=1. Then, for the model parameter set of the nth neural network model, the encoding end can encode the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the 1st neural network model. Since m=1 is also fixed or preset in the decoding end, the encoding end does not need to transmit the value of m to the decoding end. After receiving the encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the 1st neural network model, the decoding end can sum the parameter difference with the corresponding parameter in the model parameter set of the 1st neural network model to obtain the model parameters of the model parameter set of the nth neural network model.

[0116] Similarly, if the encoding end has fixed or preset m=n-1, then for the model parameter set of the nth neural network model, the encoding end can encode the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the (n-1)th neural network model. Since m=n-1 is also fixed or preset in the decoding end, the encoding end does not need to transmit the value of m to the decoding end. After receiving the encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the (n-1)th neural network model, the decoding end can sum the parameter difference with the corresponding parameter in the model parameter set of the (n-1)th neural network model to obtain the model parameters of the model parameter set of the nth neural network model.

[0117] The solution provided in this application, by fixing or presetting m at the encoding and decoding ends, can reduce the bits consumed by the encoding end due to the transmission of m, thereby further saving bandwidth.

[0118] Optionally, in some embodiments, m is determined by the encoding end based on preset rules, and the first encoded data further includes first indication information, which is used to indicate the value of m.

[0119] In this embodiment of the application, the encoding end can also determine m based on a preset rule. At the same time, the value of m can be transmitted to the decoding end through the first encoded data so that the decoding end can decode correctly.

[0120] For example, for the model parameter set of the third neural network model, the parameter difference between the corresponding parameters in the model parameter set of the third neural network model and the model parameter set of the first neural network model can be encoded. At the same time, the neural network model used is the first neural network model, which is transmitted to the decoding end, that is, m=1 is transmitted to the decoding end, so as to facilitate correct decoding by the decoding end. For the model parameter set of the third neural network model, the parameter difference between the corresponding parameters in the model parameter set of the third neural network model and the model parameter set of the second neural network model can also be encoded. At the same time, the neural network model used is the second neural network model, which is transmitted to the decoding end, that is, m=2 is transmitted to the decoding end, so as to facilitate correct decoding by the decoding end.

[0121] The solution provided in this application determines m based on preset rules at the encoding end and uses first indication information to indicate the value of m. The first indication information is transmitted to the decoding end through the first encoded data, which can improve the flexibility of encoding and at the same time ensure the correctness of encoding and decoding.

[0122] Optionally, in some embodiments, the preset rules include: For the model parameter set of the nth neural network model, select the neural network model parameter set with the smallest number of bits in the parameter difference encoding the model parameter set of the nth neural network model from multiple neural network models.

[0123] In this application's embodiments, "minimum number of bits" can be understood as occupying the fewest number of bits.

[0124] For example, for the model parameter set of the third neural network model, the parameter difference between the corresponding parameters in the model parameter set of the third neural network model and the model parameter set of the first neural network model can be encoded; alternatively, the parameter difference between the corresponding parameters in the model parameter set of the third neural network model and the model parameter set of the second neural network model can also be encoded. The encoder can make a decision based on minimizing the number of bits required to encode the parameter difference of the model parameter set of the nth neural network model.

[0125] Suppose the parameter set of the third neural network model contains 10 parameters. If the parameter set of the first neural network model is used to encode the parameter 10 of the third neural network model, and the corresponding parameter set of the first neural network model contains 5 parameters, then the parameter difference is 5, which is encoded as "101". If the parameter set of the second neural network model is used to encode the parameter 10 of the third neural network model, and the corresponding parameter set of the second neural network model contains 8 parameters, then the parameter difference is 2, which is encoded as "10". Therefore, the encoder can choose to use the parameter set of the second neural network model to encode the parameter 10 of the third neural network model, that is, to encode the parameter difference 2 between the corresponding parameters in the parameter set of the third neural network model and the parameter set of the second neural network model.

[0126] It should be understood that the above values ​​are for illustrative purposes only and other values ​​may also be used, and should not impose any special limitations on this application.

[0127] The scheme provided in this application allows the encoder to select the neural network model parameter set with the smallest number of bits for the parameter difference among multiple neural network models to encode the model parameter set of the nth neural network model, which can further save bandwidth.

[0128] Scenario 2: Compressing and encoding the model parameter set of an offline-trained neural network model using differential prediction techniques. Optionally, in some embodiments, the neural network model includes multiple offline-trained neural network models, and the compression encoding of the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: Encode the model parameter set of the first neural network model among the plurality of offline-trained neural network models; The parameter difference between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model, wherein the second neural network model is a neural network model other than the first neural network model among the plurality of offline trained neural network models.

[0129] In this embodiment, it is assumed that four neural network models are obtained through offline training, namely neural network model 1, neural network model 2, neural network model 3, and neural network model 4. Assuming that the first neural network model is neural network model 1, the encoder can first encode the model parameter set of neural network model 1. For the model parameter sets of the remaining other neural network models (including neural network model 2, neural network model 3, and neural network model 4), the parameter differences between the corresponding parameters in the model parameter sets of these three neural network models and the model parameter set of neural network model 1 can be encoded respectively.

[0130] Specifically, the encoding end can train different neural network models offline for different quantization parameters (QP). For example, four different neural network models (Neural Network Model 1, Neural Network Model 2, Neural Network Model 3, and Neural Network Model 4) can be trained for quantization parameters of 27, 32, 37, and 45 respectively. During the encoding process of the image to be encoded, the corresponding neural network model can be selected for encoding according to the quantization parameters of the image to be encoded.

[0131] Accordingly, the decoding end can receive encoded data of the model parameter set of the first neural network model; simultaneously, it can receive encoded data of the parameter differences between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model. The received encoded data is then decoded and processed (the decoded parameter differences are summed with the corresponding parameters in the model parameter set of the first neural network model), thereby obtaining the model parameter sets of the first and second neural network models.

[0132] The solution provided in this application allows the encoding end to encode the image to be encoded using an offline-trained neural network model. In addition to encoding the model parameter set of the first neural network model, the encoding end can also encode the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model for other offline-trained neural network models, and transmit the encoded data corresponding to the parameter difference, which can save transmission bandwidth.

[0133] Optionally, in some embodiments, the parameter difference between the model parameter set encoding the second neural network model and the corresponding parameters in the model parameter set of the first neural network model includes: If the number of model parameters in the model parameter set of the second neural network model is less than or equal to the number of model parameters in the model parameter set of the first neural network model, a third parameter difference is encoded, wherein the third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model.

[0134] Optionally, in some embodiments, the neural network model includes multiple offline-trained neural network models, and the compression encoding of the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: Encode the model parameter set of the first neural network model among the plurality of offline-trained neural network models; If the number of model parameters in the model parameter set of the second neural network model is greater than the number of model parameters in the model parameter set of the first neural network model, a third parameter difference and a fourth parameter difference are encoded. The third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model. The fourth parameter difference is the parameter difference between the parameters in the model parameter set of the second neural network model other than the model parameters corresponding to the third parameter difference and a preset value.

[0135] In this embodiment of the application, for the model parameters included in the model parameter set of the second neural network model, the encoder can determine the parameter difference during the encoding process based on the number of model parameters included in the model parameter set of the second neural network model and the number of model parameters included in the model parameter set of the first neural network model.

[0136] Specifically, if the number of model parameters in the model parameter set of the second neural network model is less than or equal to the number of model parameters in the model parameter set of the first neural network model, the first parameter difference is encoded; if the number of model parameters in the model parameter set of the second neural network model is greater than the number of model parameters in the model parameter set of the first neural network model, the first parameter difference and the second parameter difference are encoded.

[0137] For example, assuming that the model parameter set of the second neural network model includes 100 model parameters and the model parameter set of the first neural network model includes 100 model parameters, the encoder can encode the parameter difference of the corresponding parameters among these 100 model parameters, that is, encode the parameter difference of the corresponding parameters among the 100 model parameters included in the model parameter set of the second neural network model and the model parameter set of the first neural network model.

[0138] Assuming the second neural network model has 80 model parameters and the first neural network model has 100 model parameters, the encoder simply encodes the parameter differences between the 80 model parameters in the second neural network model and the corresponding 80 model parameters in the first neural network model.

[0139] Assuming the second neural network model has 100 model parameters and the first neural network model has 80 model parameters, the encoder can encode the parameter differences between the corresponding parameters in the second neural network model's parameter set and the 80 model parameters in the first neural network model's parameter set. For the remaining 20 model parameters in the second neural network model's parameter set, the encoder can encode the parameter differences between these 20 model parameters and preset values.

[0140] If the bit width of the model parameters of the above neural network model is 'a', the preset value can be that when the encoder encodes the remaining 20 model parameters, it can encode the parameter difference between these 20 model parameters and the parameter difference.

[0141] It should be noted that the above preset values ​​can also be other values, such as, etc., without restriction.

[0142] Accordingly, for the decoding end, the encoded data of the parameter differences between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model is decoded. This may yield a third parameter difference, or a third parameter difference and a fourth parameter difference. If a third parameter difference is obtained, the decoding end sums the third parameter difference with the corresponding model parameters in the model parameter set of the first neural network model to obtain the model parameter set of the second neural network model. If a third parameter difference and a fourth parameter difference are obtained, the decoding end sums the third parameter difference with the corresponding model parameters in the model parameter set of the first neural network model, and sums the fourth parameter difference with a preset value to obtain the model parameter set of the second neural network model.

[0143] As mentioned above, for offline trained neural network models, the model parameter set of the first neural network model and the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model can be encoded. The encoding method is different for the model parameters in the model parameter sets of different neural network models. Please refer to the following text for details.

[0144] Optionally, in some embodiments, the model parameter set encoding the first neural network model among the plurality of offline-trained neural network models includes: The model parameter set of the first neural network model is encoded using the first encoding method; The parameter differences between the model parameter set encoding the second neural network model and the corresponding parameters in the model parameter set of the first neural network model include: The parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model are encoded using a second encoding method.

[0145] In this embodiment, the model parameter set of the first neural network model, and the parameter differences between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model can be encoded using different encoding methods. That is, the model parameter set of the first neural network model can be encoded using a first encoding method; the parameter differences between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model can be encoded using a second encoding method.

[0146] Accordingly, for the decoding end, the encoded data of the model parameter set of the first neural network model is decoded using the first decoding method; and the encoded data of the parameter differences between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model are decoded using the second decoding method. The decoded data are then summed to obtain the model parameter set of the neural network model.

[0147] It should be noted that if the number of model parameters in the parameter set of the second neural network model is greater than the number of model parameters in the parameter set of the first neural network model, the encoder can encode the parameter differences between the corresponding parameters in the parameter sets of the second and first neural networks using a second encoding method, as well as the parameter differences between the parameters in the second parameter set (excluding the parameters corresponding to the third parameter difference) and preset values. Correspondingly, the decoder can decode the parameter differences between the corresponding parameters in the parameter sets of the second and first neural networks using a second decoding method, as well as the parameter differences between the parameters in the second parameter set (excluding the parameters corresponding to the third parameter difference) and preset values.

[0148] The solution provided in this application encodes the parameter sets of the first neural network model and the parameter differences between the corresponding parameters in the parameter sets of the second neural network model and the parameter sets of the first neural network model using different encoding methods. Alternatively, it encodes the parameter differences between the parameter sets of the first neural network model and the parameter differences between the corresponding parameters in the parameter sets of the second neural network model and the parameter differences between the parameter sets of the first neural network model and the parameter sets of the second neural network model, as well as the parameter differences between the parameters in the second model parameter set other than the parameter differences corresponding to the third parameter differences and the preset values ​​using different encoding methods. This can improve the flexibility of the encoding.

[0149] Optionally, in some embodiments, the first encoding method is a fixed-length code encoding method, and the second encoding method is a variable-length code encoding method.

[0150] Specifically, if the model parameter set of the first neural network model (such as neural network model 1 above) includes the value 10, the encoder can encode it as "1010"; if the model parameter set of the first neural network model also includes the value 2, the encoder can encode it as "0010". In other words, regardless of the value of the model parameter in the model parameter set of the first neural network model, the encoder encodes it with a 4-bit width.

[0151] For the model parameter set of the second neural network model, the encoder encodes the parameter differences between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model. Assume that the second neural network model includes three neural network models (such as neural network model 2, neural network model 3, and neural network model 4 mentioned above).

[0152] For neural network model 2, if the parameter difference between the corresponding parameters in the parameter sets of neural network model 2 and the first neural network model includes the value 2, the encoder can encode it as "10", meaning the parameter difference 2 occupies a bit width of 2 bits. If the parameter difference between the parameter sets of neural network model 2 and the first neural network model also includes the value 5, the encoder can encode it as "101", meaning the parameter difference 5 occupies a bit width of 3 bits. In other words, for the parameter differences between the parameter sets of neural network model 2 and the first neural network model, the encoder encodes different bit widths according to the magnitude of the value for different parameter differences; that is, the encoder can encode using variable-length code encoding based on the magnitude of the value.

[0153] Similarly, neural network model 3 and neural network model 4 can be encoded using a variable-length code encoding method similar to that of neural network model 2.

[0154] The solution provided in this application encodes the model parameters in the model parameter set of the first neural network model using a fixed-length code encoding method, and encodes the parameter differences between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model using a variable-length code encoding method, which can further save bandwidth.

[0155] Optionally, in some embodiments, the number of model parameters included in the model parameter set of the first neural network model is the neural network model with the fewest number of model parameters included in the model parameter set of the plurality of offline trained neural network models.

[0156] In this embodiment, it is still assumed that the offline trained neural network model includes four neural network models: neural network model 1, neural network model 2, neural network model 3, and neural network model 4. If the model parameter set of neural network model 1 includes 100 model parameters, the model parameter set of neural network model 2 includes 80 model parameters, the model parameter set of neural network model 3 includes 120 model parameters, and the model parameter set of neural network model 4 includes 95 model parameters, then the first neural network model in this application can be neural network model 2.

[0157] The solution provided in this application has a first neural network model whose model parameter set includes the fewest model parameters among multiple offline trained neural network models, which can further save bandwidth.

[0158] Of course, in some embodiments, the first neural network model may be the neural network model whose model parameter set includes the largest number of model parameters among the model parameter sets of multiple offline trained neural network models, or the first neural network model may be any neural network model among multiple offline trained neural network models, without limitation.

[0159] Optionally, in some embodiments, the first encoded data further includes second indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the p-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level, and p is an integer greater than 0.

[0160] In this embodiment of the application, for encoding a target image using an offline trained neural network model, the first encoding data may further include second indication information that identifies the model parameters of the neural network model used when encoding the p-th target image, so as to facilitate correct decoding at the decoding end.

[0161] In this embodiment, the target image can be an image after the image to be encoded has been divided according to any one of the following: video sequence level, group of pictures (GOP) level, or image level (or frame level). The length of a GOP can be defined independently; generally, the images in a video sequence from the current I-frame to the next I-frame can be considered as one GOP.

[0162] To facilitate understanding of the scheme in this application, a brief introduction to GOP is provided first. A GOP consists of a group of consecutive frames, composed of one I-frame and several B-frames and / or P-frames, and is the basic unit of storage for video image encoders and decoders. An I-frame (also called a keyframe) is an intra-coded image frame, which can be understood as a complete preservation of this frame's image; a B-frame is a bidirectional reference frame or bidirectional difference frame, which can be understood as recording the difference between the current frame and the frames before and after it. During decoding, the previous buffered image and the decoded image are obtained, and the final image is obtained by superimposing the data from the previous and after images with the data of the current frame; a P-frame is a forward reference frame or forward prediction frame, which can be understood as the difference between the current frame and the previous frame. During decoding, the previously buffered image is superimposed with the difference defined in the current frame to generate the final image.

[0163] Assume the offline-trained neural network model includes four neural network models: neural network model 1, neural network model 2, neural network model 3, and neural network model 4. Taking a Group of Pictures (GOP) as an example, suppose that for the first GOP, the encoder uses neural network model 1 to encode it. The encoder can transmit the identifier of the model parameters of neural network model 1 used when encoding the first GOP to the decoder, that is, the second indication information indicates the identifier of the model parameters of neural network model 1 used when encoding the first GOP. Suppose that for the second GOP, the encoder uses neural network model 3 to encode it. The encoder can transmit the identifier of the model parameters of neural network model 3 used when encoding the second GOP to the decoder, that is, the second indication information indicates the identifier of the model parameters of neural network model 3 used when encoding the second GOP, and so on.

[0164] In practical applications, neural network models can be retrained based on different application scenarios. During the encoding process, the encoder can choose to use the retrained neural network model or the existing offline trained basic neural network model for encoding, and write the syntax element of whether to use the retrained neural network model into the first encoded data.

[0165] When the encoding end selects the retrained neural network model for encoding, the model parameters of the retrained neural network model can be encoded by means of differential prediction, the encoded data obtained after encoding is transmitted to the decoding end, and at the same time, the syntax elements using the retrained neural network model are encoded and written into the first encoded data; when the encoding end selects the existing basic neural network model trained offline for encoding, it is not necessary to transmit the model parameters of the retrained neural network model to the decoding end, furthermore, the syntax elements that do not use the retrained neural network model can also be encoded and written into the first encoded data.

[0166] The decoding end decodes the identifier indicating whether a retrained neural network model is used from the received first encoded data, that is, performs decoding by using the model parameter set of the existing basic offline-trained neural network model or performs decoding by using the model parameter set of the transmitted retrained neural network model.

[0167] The second indication information in the embodiments of the present application can be transmitted to the decoding end through the first encoded data, which facilitates decoding by the decoding end.

[0168] In some embodiments, the encoding end may not transmit the second indication information, and may determine the adopted neural network model according to the quantization parameter. For example, it is assumed that the encoding end trains 4 different neural network models respectively for quantization parameters 27, 32, 37, and 45, which are neural network model 1, neural network model 2, neural network model 3, and neural network model 4 respectively.

[0169] The encoding end can establish a mapping relationship between quantization parameters and these 4 neural network models. For example, if QP ≤ 30, neural network model 1 can be used for encoding; if 30 < QP ≤ 35, neural network model 2 can be used for encoding; if 35 < QP ≤ 40, neural network model 3 can be used for encoding; if QP > 40, neural network model 4 can be used for encoding.

[0170] It should be understood that the above mapping relationship is only an example, and the encoding end may also establish different mapping relationships, which is not limited.

[0171] For the first GOP, if its quantization parameter is 25, the encoding end can select neural network model 1 to encode it to obtain the first encoded data. After the decoding end receives the first encoded data and decodes to obtain its quantization parameter, it can determine that the neural network model required for decoding the encoded data is neural network model 1 according to the mapping relationship between quantization parameters and neural network models, and uses the neural network model 1 to decode the encoded data corresponding to the first GOP.

[0172] For the second GOP, if its quantization parameter is 41, the encoder can select neural network model 4 to encode it, obtaining the first encoded data. After receiving the first encoded data and decoding it to obtain its quantization parameter, the decoder can determine the neural network model required to decode the encoded data as neural network model 4 based on the mapping relationship between the quantization parameter and the neural network model, and use neural network model 4 to decode the encoded data corresponding to the second GOP.

[0173] It should be noted that the mapping relationship between the quantization parameters and the neural network model can be preset at the encoding and decoding ends; or, the mapping relationship between the quantization parameters and the neural network model can also be transmitted to the decoding end during the encoding process. This mapping relationship can be written into the first encoded data and transmitted to the decoding end along with the first encoded data, or it can be written into the second encoded data and transmitted to the decoding end along with the second encoded data; there are no restrictions.

[0174] The solution provided in this application includes second indication information in the first encoded data transmitted by the encoding end, which identifies the model parameters of the neural network model used when encoding the p-th target image, thus facilitating correct decoding at the decoding end.

[0175] As mentioned above, the encoding and decoding ends can utilize neural network-based encoding techniques to filter the image to be encoded. The following section combines... Figure 5 This describes the filtering of images to be encoded using neural network-based coding techniques.

[0176] like Figure 5 The diagram shown is a schematic diagram of an encoding framework 2 provided in an embodiment of this application.

[0177] In this process, after obtaining the reconstructed pixels using inverse quantization 206 and inverse transform 207, the encoding end can filter the reconstructed pixels 211. During the filtering process, any one or more of the following can be applied to the reconstructed pixels: Deblocking Filter (DF), Non-Non ...

[0178] During the encoding process of the target image, taking the target image as a Group of Pictures (GOP) as an example, when encoding the first GOP of the current image to be encoded, the image of the first GOP is used as the training set for model training based on neural network filtering techniques. The entire training process can be referenced as follows: During the encoding process, the current image to be encoded can be fed into... Figure 5Encoding training is performed within the framework shown. For example, the first GOP is dequantized and inverse transformed to obtain a reconstructed image, which is then subjected to DF, NNF, SAO, and ALF. After all images within the first GOP are encoded, the bitstream of all images within the first GOP and the neural network model trained based on all images within the first GOP using the neural network framework are obtained.

[0179] When encoding the second Group of Pictures (GOPs) of the current image to be encoded, a neural network-based filtering technique is performed using the neural network model trained on the first GOP. The specific filtering process can be described as follows: When encoding the image within the second GOP, the second GOP undergoes inverse quantization and inverse transform to obtain a reconstructed image. This reconstructed image is then fed into a filtering module that deploys the neural network model trained on the first GOP to obtain a filtered reconstructed image. This filtering module includes a DF module, a NNF module, a SAO module, and an ALF module, with the NNF module containing the neural network model trained on the first GOP.

[0180] Similarly, for the p-th GOP, when encoding the p-th GOP, a neural network-based filtering technique is performed using the neural network model trained on the pt-th GOP. The specific filtering process can be referenced as follows: When encoding the image within the p-th GOP, the p-th GOP undergoes inverse quantization and inverse transform to obtain a reconstructed image. This reconstructed image is then fed into a filtering module that deploys the neural network model trained on the pt-th GOP to obtain a filtered reconstructed image. Likewise, this filtering module includes a DF module, a NNF module, a SAO module, and an ALF module, and the NNF module includes the neural network model trained on the pt-th GOP.

[0181] Understandable, Figure 5 The filtering order shown is for illustrative purposes only, and other orders are also possible, such as DF, SAO, NNF, and ALF, and should not be construed as limiting this application.

[0182] Optionally, in some embodiments, the method further includes: dividing the model parameter set of the neural network model to obtain multiple model parameter subsets of the model parameter set of the neural network model; The method of compressing and encoding the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: encoding multiple subsets of model parameters of the neural network model's model parameter set using differential prediction.

[0183] In this embodiment, the encoder can divide the model parameter set of the neural network model into multiple model parameter subsets and encode these multiple model parameter subsets using differential prediction.

[0184] For example, the encoder can divide the model parameter set of the neural network model according to the layers. The parameters generated by each layer in the neural network model structure can be divided into a model parameter subset, and multiple model parameter subsets can be encoded using differential prediction.

[0185] Accordingly, for the decoding end, the received second encoded data is decoded to obtain multiple subsets of model parameters of the model parameter set of the neural network model.

[0186] The solution provided in this application improves the parallelism of neural network model parameter prediction encoding by partitioning the model parameter set of the neural network model and transmitting the resulting subsets of model parameters using differential prediction. Furthermore, the correlation between model parameters in the resulting subsets of model parameters obtained after partitioning the model parameter sets of different neural network models is higher, and transmitting these subsets using differential prediction further saves bandwidth.

[0187] Optionally, in some embodiments, the neural network model includes multiple neural network models, and the encoding of multiple subsets of model parameters of the model parameter set of the neural network model using the differential prediction method includes: For the first subset of model parameters in the q-th neural network model, the first subset of model parameters is encoded using the subset of model parameters in the third neural network model; For the second model parameter subset in the q-th neural network model, the second model parameter subset is encoded using the model parameter subset in the fourth neural network model. The third neural network model and the fourth neural network model are different models among the plurality of neural network models, and q is an integer greater than 0.

[0188] For example, suppose the encoder pre-trains four different neural network models: neural network model 1, neural network model 2, neural network model 3, and neural network model 4. For the first GOP, the encoder can use neural network model 1 to encode the first GOP. When encoding the subset of model parameters of neural network model 1, the encoder can use the subsets of model parameters included in different neural network models to encode it.

[0189] In this embodiment, encoding using subsets of model parameters from different neural network models can be understood as encoding the subsets of model parameters from neural network model 1 using subsets of model parameters from different neural network models. For example, assuming the model parameter set of neural network model 1 is divided into two subsets, a first subset and a second subset, the encoder can encode these two subsets using subsets of model parameters from different neural network models. For instance, the first subset can be encoded using the subset of model parameters from neural network model 2, and the second subset can be encoded using the subset of model parameters from neural network model 3.

[0190] It should be noted that although the model parameter subsets of the first neural network model are encoded separately using different subsets of model parameters from different neural network models, the image to be encoded is still encoded on a per-neural network model basis when using the neural network models. For example, the encoder can encode the first GOP based on neural network model 1 and the second GOP based on neural network model 2.

[0191] Similarly, for the second GOP, the encoder can encode the second GOP using the model parameter set of neural network model 2. When encoding the subset of model parameters of neural network model 2, it can be encoded using the subsets of model parameters included in different neural network models. For example, assuming that the model parameter set of neural network model 2 is divided into two subsets, namely the first subset and the second subset, the encoder can encode these two subsets using the subsets of model parameters of different neural network models. For example, the first subset can be encoded using the subset of model parameters of neural network model 3, and the second subset can be encoded using the subset of model parameters of neural network model 4.

[0192] It should be understood that the number of model parameter subsets into which the model parameter set of the above neural network model is divided is only for illustrative purposes. In some embodiments, the model parameter set of the neural network model can also be divided into multiple model parameter subsets (such as 3 or 4 model parameter subsets). When encoding these multiple model parameter subsets, the encoder can use the model parameters of different neural network models to encode them separately.

[0193] It should also be understood that if the model parameter set of a neural network model is divided into three model parameter subsets, two of the model parameter subsets can be encoded using a third neural network model, and the other model parameter subset can be encoded using a fourth neural network model; alternatively, the third, fourth, and fifth neural network models (which can be network models from multiple neural network models) can be used to encode these three model parameter subsets respectively; there are no restrictions.

[0194] Correspondingly, for the decoding end, after receiving the second encoded data, it decodes it to obtain the residual data of the first model parameter subset and the residual data of the second model parameter subset. It then uses the first model parameter subset, the residual data, and the third neural network model parameter subset to obtain the first model parameter subset, and uses the residual data of the second model parameter subset and the model parameter subset in the fourth neural network model to obtain the second model parameter subset. Finally, it uses the model parameters from the first and second model parameter subsets to decode the first encoded data.

[0195] The solution provided in this application improves the parallelism of neural network model parameter prediction encoding by partitioning the model parameter set of the neural network model and transmitting the first and second subsets of model parameters obtained after partitioning using differential prediction. Furthermore, the correlation between model parameters in the first and second subsets of model parameters obtained after partitioning the model parameter set is higher, and transmitting the first and second subsets of model parameters using differential prediction further saves bandwidth.

[0196] Optionally, in some embodiments, the first encoded data is transmitted via a first communication link, and the second encoded data is transmitted via a second communication link; or, The first encoded data and the second encoded data are transmitted through the first communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

[0197] In this embodiment of the application, the first encoded data and the second encoded data can be transmitted through different communication links; or the first encoded data and the second encoded data can be transmitted through the same communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

[0198] It should be noted that if the first encoded data and the second encoded data are transmitted through different communication links, the first encoded data may also include indication information to indicate the neural network model used for the current encoded image. After receiving the first encoded data, the decoding end can determine the neural network model used for decoding the current encoded data based on the indication information in the first encoded data.

[0199] The solution provided in this application, by transmitting the first and second encoded data through different communication links, can effectively solve the problem of managing and transmitting model parameters of a neural network model. Since the second encoded data is transmitted through the second communication link, bit consumption can be reduced. Furthermore, it can alleviate the challenge of low-latency transmission due to the need for model parameter management and transmission. Transmitting the first and second encoded data through the same communication link can reduce the risk of errors during transmission, and writing the second encoded data into the first encoded data facilitates the management and transmission of model parameters of the neural network model.

[0200] Optionally, in some embodiments, the first communication link and the second communication link have different physical characteristics.

[0201] The physical characteristics in the embodiments of this application may be determined by specific application requirements. For example, they may include transmission delay and / or transmission bandwidth. However, it should be understood that these physical characteristics may also be other characteristics and should not be construed as limiting this application.

[0202] Optionally, in some embodiments, the transmission delay of the first communication link is lower than that of the second communication link, and / or the transmission bandwidth of the second communication link is higher than that of the first communication link.

[0203] In this embodiment of the application, the first communication link has a lower transmission delay, or a lower transmission bandwidth, or both a lower transmission delay and a lower transmission bandwidth compared to the second communication link; in other words, the transmission delay and / or transmission bandwidth of the second communication link is higher than that of the first communication link.

[0204] It should be noted that during the encoding process of the image to be encoded, the encoding end may transmit the first encoded data in real time so that the decoding end can decode it promptly. Therefore, it is advisable to consider using a first communication link with low latency to transmit the first encoded data.

[0205] It should also be noted that during the encoding process of the neural network model encoding the image to be encoded, the model parameters of the neural network model, i.e., the second encoded data, can be transmitted through the second communication link. Since the neural network model has many model parameters, especially for neural network models with more layers, the model parameters are more complex. Therefore, it is advisable to use the second communication link with higher bandwidth for transmission.

[0206] The solution provided in this application transmits first encoded data through a first communication link with low transmission latency and second encoded data through a second communication link with high transmission bandwidth. This solution can effectively address the management and transmission issues of model parameters in neural network models and further reduce the challenges of low-latency transmission due to the requirements for managing and transmitting model parameters in neural network models.

[0207] Optionally, in some embodiments, the first communication link includes a link with a latency less than or equal to a second threshold, and the second communication link includes a link with a bandwidth greater than or equal to a third threshold.

[0208] The second and / or third thresholds in the embodiments of this application can be specified by a protocol or configured by a server; the second and / or third thresholds can be fixed values ​​or continuously adjusted and changing values; there are no restrictions.

[0209] Optionally, in some embodiments, the first communication link includes a link based on a proprietary image transmission protocol or a wireless local area network protocol, and the second communication link includes a link based on a mobile communication protocol.

[0210] Optionally, in some embodiments, the proprietary image transmission protocol includes a Software Definition Radio (SDR) protocol, the wireless local area network protocol includes a Wireless Fidelity (WiFi) protocol, and the mobile communication protocol includes a 4G or 5G protocol.

[0211] It should be noted that, in this embodiment of the application, the private video transmission protocol shows the SDR protocol. The private video transmission protocol may also include other protocols, such as the Open Network Video Interface Forum (ONVIF) protocol, etc., without limitation.

[0212] In addition to the WiFi protocol shown above, the wireless local area network protocol in this application embodiment may also include other protocols, such as Bluetooth, ZigBee, etc., without limitation.

[0213] In addition to the 4G or 5G protocols shown above, the mobile communication protocols in this application embodiment may also include other protocols, such as the future 6G protocol.

[0214] Of course, in some embodiments, the first communication link may also be an SDR-based link and the second communication link may be a WiFi-based link.

[0215] Optionally, in some embodiments, the first communication link includes a private image transmission link, and the second communication link includes a public network transmission link.

[0216] like Figure 6 The diagram illustrates the image transmission using the intelligent encoding technology provided in this embodiment. After acquiring the video, the acquisition end (also known as the encoding end) compresses and encodes it using an encoder and a neural network computing platform to obtain first encoded data and corresponding second encoded data. This first and second encoded data are then transmitted to the display end via a wireless image transmission system. Upon receiving the first and second encoded data, the display end (also known as the decoding end) decodes the data using its decoder and neural network computing platform to obtain the reconstructed video, which is then displayed on the display's screen.

[0217] The aforementioned wireless image transmission system can be a wireless video transmission system, which may include a private image transmission link and a public network transmission link, used to transmit the first encoded data and the second encoded data respectively.

[0218] The private image transmission link in this application embodiment may include a link based on the SDR protocol, a link based on the WiFi protocol, or a link based on the ONVIF protocol; the public network transmission link in this application embodiment may include a link based on the 4G, 5G, or 6G protocol.

[0219] The solution provided in this application transmits the first encoded data through a private image transmission link and the second encoded data through a public network transmission link, which can effectively solve the problem of managing and transmitting model parameters of neural network models.

[0220] Optionally, in some embodiments, the first communication link and / or the second communication link is selected from one or more of the following links: Links based on wireless LAN protocols, links based on mobile communication protocols, and links based on Ethernet protocols.

[0221] In this embodiment of the application, when the encoding end transmits the first encoded data and the second encoded data, it can also flexibly select from one or more links, thereby improving flexibility.

[0222] Optionally, in some embodiments, the first encoded data is transmitted through the first communication link, the second encoded data is transmitted through the second communication link, and the first encoded data further includes third indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the s-th target image, the target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level, and s is an integer greater than 0.

[0223] In this embodiment of the application, if the first encoded data and the second encoded data are transmitted through different communication links, the first encoded data may further include third indication information that identifies the model parameters of the neural network model used when encoding the s-th target image. After receiving the first encoded data, the decoding end can determine the model parameters of the neural network model used when decoding the first encoded data based on the third indication information in the first encoded data, which facilitates correct decoding by the decoding end.

[0224] Optionally, in some embodiments, the method further includes: Convert the model parameters, including those in the model parameter set of the neural network model, into the target format; The method of compressing and encoding the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: The target format is compressed and encoded using the differential prediction method.

[0225] Accordingly, for the decoding end, in step 420 above, decoding the second encoded data to obtain the model parameters in the model parameter set of the neural network model includes: decoding the second encoded data to obtain the target format.

[0226] In step 430 above, decoding the first encoded data using model parameters from the model parameter set of the neural network model includes: converting the target format; and decoding the first encoded data using model parameters from the converted format.

[0227] In this embodiment, after encoding the image to be encoded using a neural network model, the encoding end can perform format conversion on the model parameters of the neural network model to obtain the target format, and then compress and encode the target format using differential prediction, transmitting the compressed target format to the decoding end. Upon receiving the second encoded data, the decoding end decodes it to obtain the target format, then converts the target format to obtain the corresponding model parameters of the neural network model, and uses the converted model parameters to decode the first encoded data.

[0228] Optionally, in some embodiments, the target format includes Neural Network Exchange Format (NNEF) or Open Neural Network Exchange (ONNX) format.

[0229] The NNEF and ONNX formats in this application are two similar open formats used to represent and exchange neural networks between deep learning frameworks and inference engines. At their core, both formats are based on a set of common operations from which networks can be built.

[0230] NNEF reduces the fragmentation of machine learning deployments by using a rich combination of neural network training tools and inference engines to support applications on multiple devices and platforms. Its main goal is to be able to export networks from deep learning frameworks and import them into hardware vendors' inference engines.

[0231] ONNX defines a set of common operators—building blocks for machine learning and deep learning models—and a common file format that allows artificial intelligence (AI) developers to use their models with a variety of frameworks, tools, runtimes, encoders, and decoders.

[0232] Both formats can store neural network models generated by commonly used deep learning frameworks, with the aim of enabling the interaction and universality of neural network models across different deep learning frameworks.

[0233] The solution provided in this application enables the interaction and universality of neural network models across different deep learning frameworks by converting the model parameters of the neural network model into a target format. Furthermore, by using differential prediction to encode the target format obtained after conversion and transmitting the encoded second data, bandwidth can be further saved.

[0234] In this embodiment of the application, the neural network model can be trained in the following ways.

[0235] like Figure 7 The diagram shown is a flowchart illustrating a training neural network model according to an embodiment of this application. (Reference) Figure 7 The image to be encoded is compressed using an encoder and a neural network computing platform to obtain a neural network model. The model parameters of the neural network model are converted to NNEF or ONNX format, and then the model parameters in the converted NNEF or ONNX format are compressed using a differential prediction compression method to obtain the differential prediction encoded data.

[0236] When encoding an image using neural network-based coding techniques, the encoding end can base it on... Figure 8a The process shown is encoded.

[0237] like Figure 8a The diagram shown is a flowchart illustrating the application of intelligent encoding technology in a video encoder according to an embodiment of this application. (Reference) Figure 8aAfter obtaining the encoded data after differential prediction, the encoded data before differential prediction can be obtained through differential prediction decompression. The encoded data before differential prediction is then converted to NNEF or ONNX format to obtain the model parameters of the neural network model. The obtained model parameters of the neural network model are then deployed on the neural network computing platform. When the encoder encodes an image or video, it can encode it in conjunction with the neural network computing platform that has deployed the model parameters of the neural network model to generate the first encoded data.

[0238] In practical implementation, neural network-based encoding techniques can be achieved by modifying traditional encoders. For example, when encoding header information syntax elements, the encoding of intelligent encoding-related syntax elements can be added to the existing header information syntax parameter set. For instance, a syntax element can be added to the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), or Slice Header to identify the switch that controls whether the neural network-based encoding technique is on or off. This syntax element can be added directly to the aforementioned syntax element parameter sets, or it can be optionally added to the user extension data of the aforementioned syntax element parameter sets.

[0239] For the decoding end, it can be based on Figure 8b The process shown is used for decoding.

[0240] like Figure 8b The diagram shown is a flowchart illustrating the application of intelligent encoding technology in a video decoder according to an embodiment of this application. (Reference) Figure 8b After receiving the encoded data after differential prediction, the decoder can obtain the encoded data before differential prediction through differential prediction decompression. The encoded data before differential prediction is then converted to NNEF or ONNX format to obtain the model parameters of the neural network model. The obtained model parameters of the neural network model are then deployed on the neural network computing platform. When the decoder decodes the bitstream of the image or video to be decoded, it can combine the neural network computing platform with the deployed model parameters of the neural network model to decode it and obtain the image or video to be decoded.

[0241] Optionally, in some embodiments, the first encoded data includes fourth indication information, which is used to indicate whether encoding using neural network encoding techniques is enabled.

[0242] In this embodiment, the first encoded data may further include fourth indication information, indicating whether the encoding end has enabled encoding using neural network technology during encoding. After receiving the first encoded data, the decoding end can determine whether to use neural network decoding technology for decoding based on the fourth indication information in the first encoded data.

[0243] If the fourth indication indicates that encoding using neural network technology is enabled, the decoding end will use neural network decoding technology to perform decoding; if the fourth indication indicates that encoding using neural network technology is not enabled, the decoding end will not use neural network decoding technology to perform decoding.

[0244] For example, assuming that “1” and “0” represent the on and off of encoding using neural network technology, respectively, if the fourth indication information in the first encoded data indicates “1”, the decoding end determines to use the decoding technology of neural network based on the indication information; if the fourth indication information in the first encoded data indicates “0”, the decoding end determines not to use the decoding technology of neural network based on the indication information.

[0245] It should be understood that the use of "1" and "0" to indicate the on and off of encoding using neural network technology is merely an illustrative example. Other identifiers (such as "a", "b", etc.) can also be used to indicate the on and off of encoding using neural network technology, and should not impose any special limitations on this application.

[0246] The solution provided in this application adds a fourth indication to the first encoded data to indicate whether encoding using neural network technology is enabled. After receiving the first encoded data, the decoding end determines whether to use neural network decoding technology for decoding based on the fourth indication in the first encoded data, which can further improve the decoding accuracy.

[0247] Optionally, in some embodiments, the compression encoding of the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: The differential prediction method is used to compress and encode some or all of the model parameters in the model parameter set of the neural network model.

[0248] Accordingly, for the decoding end, in step 430 above, the second encoded data is decoded to obtain the model parameters in the model parameter set of the neural network model, including: The second encoded data is decoded to obtain some or all of the model parameters in the model parameter set of the neural network model.

[0249] In this embodiment of the application, when the encoding end compresses and encodes the model parameter set of the neural network model, it can encode some or all of the model parameters in the model parameter set of the neural network model. The partial model parameters may include, but are not limited to, the number of layers and the number of neurons in the neural network. Other model parameters (including but not limited to the weights of connections between neurons) can be preset at the encoding end and the decoding end.

[0250] The decoding end can obtain some model parameters of the neural network model by decoding the second encoded data, and can combine them with other model parameters preset on the decoding end to decode the first encoded data; or, by decoding the second encoded data, it can obtain all model parameters of the neural network model's model parameter set, and use all the model parameters obtained by decoding to decode the first encoded data.

[0251] The solution provided in this application allows the encoding end to encode a portion of the model parameters in the neural network model's parameter set during compression encoding, while other model parameters are preset at both the encoding and decoding ends. This saves transmission bandwidth, and the decoding end can ensure decoding accuracy when decoding the first encoded data. Alternatively, the encoding end can encode all model parameters in the neural network model's parameter set during compression encoding, and the decoding end can ensure decoding accuracy when decoding the first encoded data.

[0252] The above text combined Figures 1-8b The method embodiments of this application are described in detail below, in conjunction with... Figures 9-13 This section describes the apparatus embodiments of this application. The apparatus embodiments correspond to the method embodiments, so any parts not described in detail can be found in the preceding method embodiments.

[0253] Figure 9 An encoding device 900 is provided in one embodiment of this application. The encoding device 900 may include a processor 910.

[0254] Processor 910 is used to encode the image to be encoded using neural network-based coding technology to obtain first encoded data; The model parameter set of the neural network model included in the neural network-based coding technique is compressed and encoded using differential prediction to obtain the second encoded data; Transmit the first encoded data and the second encoded data.

[0255] Optionally, in some embodiments, the bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to a first threshold.

[0256] Optionally, in some embodiments, the neural network model includes an offline-trained neural network model or an online-trained neural network model.

[0257] Optionally, in some embodiments, the neural network model includes an online-trained neural network model, and the processor 910 is further configured to: The set of model parameters for encoding the first neural network model; For the model parameter set of the nth neural network model, the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model is encoded, where n is a positive integer greater than 1 and m is a positive integer greater than or equal to 1.

[0258] Optionally, in some embodiments, the processor 910 is further configured to: If the number of model parameters in the model parameter set of the nth neural network model is less than or equal to the number of model parameters in the model parameter set of the mth neural network model, a first parameter difference is encoded. The first parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model.

[0259] Optionally, in some embodiments, the processor 910 is further configured to: The set of model parameters for encoding the first neural network model; If the number of model parameters in the model parameter set of the nth neural network model is greater than the number of model parameters in the model parameter set of the mth neural network model, a first parameter difference and a second parameter difference are encoded. The first parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The second parameter difference is the parameter difference between the parameters in the model parameter set of the nth model model other than the model parameter corresponding to the first parameter difference and a preset value.

[0260] Optionally, in some embodiments, the processor 910 is further configured to: The model parameter set of the first neural network model is encoded using the first encoding method; The parameter differences between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model are encoded using a second encoding method.

[0261] Optionally, in some embodiments, m is a parameter fixed to or preset in the encoding end before encoding, or m is determined by the encoding end based on preset rules.

[0262] Optionally, in some embodiments, m=1, or m=n-1.

[0263] Optionally, in some embodiments, m is determined by the encoding end based on preset rules, and the first encoded data further includes first indication information, which is used to indicate the value of m.

[0264] Optionally, in some embodiments, the preset rules include: For the model parameter set of the nth neural network model, select the neural network model parameter set with the smallest number of bits in the parameter difference encoding the model parameter set of the nth neural network model from multiple neural network models.

[0265] Optionally, in some embodiments, the neural network model includes multiple offline-trained neural network models, and the processor 910 is further configured to: Encode the model parameter set of the first neural network model among the plurality of offline-trained neural network models; The parameter difference between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model, wherein the second neural network model is a neural network model other than the first neural network model among the plurality of offline trained neural network models.

[0266] Optionally, in some embodiments, the processor 910 is further configured to: If the number of model parameters in the model parameter set of the second neural network model is less than or equal to the number of model parameters in the model parameter set of the first neural network model, a third parameter difference is encoded, wherein the third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model.

[0267] Optionally, in some embodiments, the processor 910 is further configured to: The neural network model includes multiple offline-trained neural network models, and the processor is further used for: Encode the model parameter set of the first neural network model among the plurality of offline-trained neural network models; If the number of model parameters in the model parameter set of the second neural network model is greater than the number of model parameters in the model parameter set of the first neural network model, a third parameter difference and a fourth parameter difference are encoded. The third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model. The fourth parameter difference is the parameter difference between the parameters in the model parameter set of the second neural network model other than the model parameters corresponding to the third parameter difference and a preset value.

[0268] Optionally, in some embodiments, the processor 910 is further configured to: The model parameter set of the first neural network model is encoded using the first encoding method; The parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model are encoded using a second encoding method.

[0269] Optionally, in some embodiments, the number of model parameters included in the model parameter set of the first neural network model is the neural network model with the fewest number of model parameters included in the model parameter set of the plurality of offline trained neural network models.

[0270] Optionally, in some embodiments, the first encoded data further includes second indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the p-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level, and p is an integer greater than 0.

[0271] Optionally, in some embodiments, the first encoding method is a fixed-length code encoding method, and the second encoding method is a variable-length code encoding method.

[0272] Optionally, in some embodiments, the processor 910 is further configured to: The model parameter set of the neural network model is divided to obtain multiple subsets of model parameters of the neural network model. Multiple subsets of model parameters of the neural network model are encoded using the differential prediction method.

[0273] Optionally, in some embodiments, the neural network model includes multiple neural network models, and the processor 910 is further configured to: For the first subset of model parameters in the q-th neural network model, the first subset of model parameters is encoded using the subset of model parameters in the third neural network model; For the second model parameter subset in the q-th neural network model, the second model parameter subset is encoded using the model parameter subset in the fourth neural network model. The third neural network model and the fourth neural network model are different models among the plurality of neural network models, and q is an integer greater than 0.

[0274] Optionally, in some embodiments, the first encoded data is transmitted via a first communication link, and the second encoded data is transmitted via a second communication link; or, The first encoded data and the second encoded data are transmitted through the first communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

[0275] Optionally, in some embodiments, the first encoded data is transmitted through the first communication link, the second encoded data is transmitted through the second communication link, and the first encoded data further includes third indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the s-th target image, the target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level, and s is an integer greater than 0.

[0276] Optionally, in some embodiments, the processor 910 is further configured to: Convert the model parameters, including those in the model parameter set of the neural network model, into the target format; The target format is compressed and encoded using the differential prediction method.

[0277] Optionally, in some embodiments, the target format includes the Neural Network Transform Format (NNEF) or the Open Neural Network Transform Format (ONNX).

[0278] Optionally, in some embodiments, the processor 910 is further configured to: The neural network-based coding technique is used to perform at least one of the following: prediction, transformation, quantization, entropy coding, or filtering on the image to be encoded.

[0279] Optionally, in some embodiments, the first encoded data includes fourth indication information, which is used to indicate whether encoding using neural network encoding techniques is enabled.

[0280] Optionally, in some embodiments, the processor 910 is further configured to: The differential prediction method is used to compress and encode some or all of the model parameters in the model parameter set of the neural network model.

[0281] Figure 10 A decoding device 1000 is provided in one embodiment of this application. The decoding device 1000 may include a processor 1010.

[0282] Processor 1010 is used to receive first encoded data and second encoded data; Decode the second encoded data to obtain the model parameters in the model parameter set of the neural network model; The first encoded data is decoded using the model parameters in the model parameter set of the neural network model to obtain the decoded image.

[0283] Optionally, in some embodiments, the bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to a first threshold.

[0284] Optionally, in some embodiments, the neural network model includes an offline-trained neural network model or an online-trained neural network model.

[0285] Optionally, in some embodiments, the neural network model includes an online-trained neural network model, and the processor 1010 is further configured to: Receive the encoded data of the model parameter set of the first neural network model; The system receives encoded data of the parameter differences between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model, where n is a positive integer greater than 1 and m is a positive integer greater than or equal to 1. Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; The encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model is decoded to obtain the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The model parameters of the nth neural network model are obtained by using the parameter difference and the model parameters in the model parameter set of the mth neural network model.

[0286] Optionally, in some embodiments, the processor 1010 is further configured to: The encoded data of the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model is decoded to obtain the first parameter difference, which is the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model.

[0287] Optionally, in some embodiments, the neural network model includes an online-trained neural network model, and the processor 1010 is further configured to: Receive the encoded data of the model parameter set of the first neural network model; The system receives encoded data of the parameter differences between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model, where n is a positive integer greater than 1 and m is a positive integer greater than or equal to 1. Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; The encoded data of the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model is decoded to obtain a first parameter difference and a second parameter difference. The first parameter difference is the parameter difference between the corresponding parameter in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The second parameter difference is the parameter difference between the parameters in the model parameter set of the nth neural network model or the model parameter set of the mth neural network model, excluding the model parameter corresponding to the first parameter difference, and a preset value. The model parameters of the nth neural network model are obtained by using the first parameter difference and the model parameters in the model parameter set of the mth neural network model, and by using the second parameter difference and the preset value.

[0288] Optionally, in some embodiments, the processor 1010 is further configured to: The encoded data of the model parameter set of the first neural network model is decoded using the first decoding method; The encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model are decoded using the second decoding method.

[0289] Optionally, in some embodiments, m is a parameter fixed to or preset in the decoding end, or m is determined based on preset rules.

[0290] Optionally, in some embodiments, m=1, or m=n-1.

[0291] Optionally, in some embodiments, m is determined based on preset rules, and the first encoded data further includes first indication information, which is used to indicate the value of m.

[0292] Optionally, in some embodiments, the neural network model includes multiple offline-trained neural network models, and the processor 1010 is further configured to: Receive the encoded data of the model parameter set of the first neural network model among the plurality of offline trained neural network models; The system receives encoded data of the parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model. The second neural network model is a neural network model other than the first neural network model among the plurality of offline trained neural network models. Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; Decode the encoded data of the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model to obtain the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model. The model parameter set of the second neural network model is obtained by using the parameter difference and the model parameters in the model parameter set of the first neural network model.

[0293] Optionally, in some embodiments, the processor 1010 is further configured to: The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model is decoded to obtain the third parameter difference, which is the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model.

[0294] Optionally, in some embodiments, the neural network model includes multiple offline-trained neural network models, and the processor 1010 is further configured to: Receive the encoded data of the model parameter set of the first neural network model among the plurality of offline trained neural network models; The system receives encoded data of the parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model. The second neural network model is a neural network model other than the first neural network model among the plurality of offline trained neural network models. Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model is decoded to obtain the third parameter difference and the fourth parameter difference. The third parameter difference is the parameter difference between the corresponding parameter in the model parameter set of the second neural network model and the model parameter set of the first neural network model. The fourth parameter difference is the parameter difference between the parameter in the model parameter set of the second neural network model or the model parameter set of the first neural network model, excluding the model parameter corresponding to the third parameter difference, and a preset value. The model parameter set of the second neural network model is obtained by using the third parameter difference and the model parameters in the model parameter set of the first neural network model, and by using the fourth parameter difference and the preset value.

[0295] Optionally, in some embodiments, the processor 1010 is further configured to: The encoded data of the model parameter set of the first neural network model is decoded using the first decoding method; The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model are decoded using the second decoding method.

[0296] Optionally, in some embodiments, the number of model parameters included in the model parameter set of the first neural network model is the neural network model with the fewest number of model parameters included in the model parameter set of the plurality of offline trained neural network models.

[0297] Optionally, in some embodiments, the first encoded data further includes second indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the p-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level, and p is an integer greater than 0.

[0298] Optionally, in some embodiments, the first decoding method is a fixed-length code decoding method, and the second decoding method is a variable-length code decoding method.

[0299] Optionally, in some embodiments, the processor 1010 is further configured to: The second encoded data is decoded to obtain multiple subsets of model parameters of the neural network model's model parameter set.

[0300] Optionally, in some embodiments, the neural network model includes multiple neural network models, and the processor 1010 is further configured to: Decode the second encoded data to obtain a first subset of model parameters and a second subset of model parameters, and obtain residual data of the first subset of model parameters and residual data of the second subset of model parameters; The first model parameter subset is obtained by using the residual data of the first model parameter subset and the model parameter subset in the third neural network model; The second model parameter subset is obtained by using the residual data of the second model parameter subset and the model parameter subset in the fourth neural network model, wherein the third neural network model and the fourth neural network model are different models among the plurality of neural network models; The first encoded data is decoded using model parameters from the first subset of model parameters and model parameters from the second subset of model parameters.

[0301] Optionally, in some embodiments, the first encoded data is transmitted via a first communication link, and the second encoded data is transmitted via a second communication link; or, The first encoded data and the second encoded data are transmitted through the first communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

[0302] Optionally, in some embodiments, the first encoded data is transmitted through the first communication link, the second encoded data is transmitted through the second communication link, and the first encoded data further includes third indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the s-th target image, the target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level, and s is an integer greater than 0.

[0303] Optionally, in some embodiments, the processor 1010 is further configured to: The second encoded data is decoded to obtain the target format; Convert the target format; The first encoded data is decoded using the model parameters of the converted format.

[0304] Optionally, in some embodiments, the target format includes the Neural Network Transform Format (NNEF) or the Open Neural Network Transform Format (ONNX).

[0305] Optionally, in some embodiments, the processor 1010 is further configured to: The model parameters in the model parameter set of the neural network are used to perform at least one of the following on the first encoded data: entropy decoding, inverse quantization, inverse transformation, prediction reconstruction, or filtering.

[0306] Optionally, in some embodiments, the first encoded data includes fourth indication information, which is used to indicate whether encoding using neural network encoding techniques is enabled.

[0307] Optionally, in some embodiments, the processor 1010 is further configured to: The second encoded data is decoded to obtain some or all of the model parameters in the model parameter set of the neural network model.

[0308] Figure 11 This is a schematic structural diagram of an encoding device provided in another embodiment of this application. Figure 11 The encoding device 1100 shown includes a processor 1110, which can call and run computer programs from memory to implement the encoding method described in the embodiments of this application.

[0309] Optionally, such as Figure 11 As shown, the encoding device 1100 may further include a memory 1120. The processor 1110 can retrieve and run computer programs from the memory 1120 to implement the encoding method in this embodiment.

[0310] The memory 1120 can be a separate device independent of the processor 1110, or it can be integrated into the processor 1110.

[0311] Optionally, such as Figure 11 As shown, the encoding device 1100 may also include a transceiver 1130. The processor 1110 can control the transceiver 1130 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0312] Optionally, the encoding device may be, for example, an encoder or a terminal (including but not limited to mobile phones, cameras, drones, etc.), and the encoding device may implement the corresponding processes in the various encoding methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0313] Figure 12 This is a schematic structural diagram of a decoding device provided in another embodiment of this application. Figure 12 The decoding device 1200 shown includes a processor 1210, which can call and run computer programs from memory to implement the decoding method described in the embodiments of this application.

[0314] Optionally, such as Figure 12 As shown, the decoding device 1200 may further include a memory 1220. The processor 1210 can retrieve and run computer programs from the memory 1220 to implement the decoding method in this embodiment.

[0315] The memory 1220 can be a separate device independent of the processor 1210, or it can be integrated into the processor 1210.

[0316] Optionally, such as Figure 12 As shown, the decoding device 1200 may also include a transceiver 1230. The processor 1210 can control the transceiver 1230 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0317] Optionally, the decoding device may be, for example, a decoder or a terminal (including but not limited to mobile phones, cameras, drones, etc.), and the decoding device may implement the corresponding processes in the various decoding methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0318] Figure 13 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 13 The chip 1300 shown includes a processor 1310, which can call and run computer programs from memory to implement the encoding or decoding methods in the embodiments of this application.

[0319] Optionally, such as Figure 13 As shown, chip 1300 may further include memory 1320. Processor 1310 can retrieve and run computer programs from memory 1320 to implement the encoding or decoding methods in the embodiments of this application.

[0320] The memory 1320 can be a separate device independent of the processor 1310, or it can be integrated into the processor 1310.

[0321] Optionally, the chip 1300 may also include an input interface 1330. The processor 1310 can control the input interface 1330 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0322] Optionally, the chip 1300 may also include an output interface 1340. The processor 1310 can control the output interface 1340 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0323] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0324] It should be understood that the processor in the embodiments of this application may be an integrated circuit image processing system with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0325] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0326] The memory in this embodiment can provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor can be used to execute instructions stored in the memory, and when the processor executes the instructions, the processor can perform the various steps corresponding to the terminal device in the above method embodiments.

[0327] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or as a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor executes the instructions in the memory, combining them with its hardware to complete the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0328] It should also be understood that, in the embodiments of this application, pixels in the image may be located in different rows and / or columns, wherein the length of A may correspond to the number of pixels in the same row included in A, and the height of A may correspond to the number of pixels in the same column included in A. Furthermore, the length and height of A may also be referred to as the width and depth of A, respectively, and the embodiments of this application do not limit this.

[0329] It should also be understood that, in the embodiments of this application, "the boundary spacing distribution with respect to A" can refer to a distance of at least one pixel from the boundary of A, or it can be referred to as "not adjacent to the boundary of A" or "not located at the boundary of A". The embodiments of this application do not limit this. A can be an image, a rectangular region or a sub-image, etc.

[0330] It should also be understood that the above description of the embodiments of this application focuses on highlighting the differences between the various embodiments. Any similarities or differences not mentioned can be referred to each other. For the sake of brevity, they will not be repeated here.

[0331] This application also provides a computer-readable storage medium for storing computer programs.

[0332] Optionally, the computer-readable storage medium can be applied to the encoding or decoding apparatus in the embodiments of this application, and the computer program causes the computer to execute the corresponding processes implemented by the encoding or decoding apparatus in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0333] This application also provides a computer program product, including computer program instructions.

[0334] Optionally, the computer program product can be applied to the encoding or decoding apparatus in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the encoding or decoding apparatus in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0335] This application also provides a computer program.

[0336] Optionally, the computer program can be applied to the encoding or decoding device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the encoding or decoding device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0337] It should be understood that in the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0338] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0339] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0340] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, apparatuses, or units, or they may be electrical, mechanical, or other forms of connection.

[0341] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0342] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0343] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0344] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An encoding method, characterized in that, include: The image to be encoded is encoded using a neural network-based encoding technique to obtain the first encoded data; The second encoded data is obtained by compressing and encoding multiple model parameter sets corresponding to multiple neural network models included in the neural network-based encoding technology using differential prediction; the neural network model includes an online-trained neural network model or multiple offline-trained neural network models. Transmit the first encoded data and the second encoded data; When the neural network model includes an online-trained neural network model, the step of compressing and encoding the multiple model parameter sets corresponding to the multiple neural network models included in the neural network-based encoding technique using differential prediction includes: The set of model parameters for encoding the first neural network model; If the number of model parameters in the parameter set of the nth neural network model is greater than the number of model parameters in the parameter set of the mth neural network model, a first parameter difference and a second parameter difference are encoded. The first parameter difference is the parameter difference between the corresponding parameters in the parameter set of the nth neural network model and the parameter set of the mth neural network model. The second parameter difference is the parameter difference between the parameters in the parameter set of the nth model model (excluding the parameter corresponding to the first parameter difference) and a preset value. Here, n is a positive integer greater than 1, and m is a positive integer greater than or equal to 1. When the neural network model includes multiple offline-trained neural network models, the step of compressing and encoding the multiple model parameter sets corresponding to the multiple neural network models included in the neural network-based encoding technique using differential prediction includes: Encode the model parameter set of the first neural network model among the plurality of offline-trained neural network models; If the number of model parameters in the model parameter set of the second neural network model is greater than the number of model parameters in the model parameter set of the first neural network model, a third parameter difference and a fourth parameter difference are encoded. The third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model. The fourth parameter difference is the parameter difference between the parameters in the model parameter set of the second neural network model other than the model parameters corresponding to the third parameter difference and a preset value. The second neural network model is a neural network model other than the first neural network model among the multiple offline trained neural network models.

2. The method according to claim 1, characterized in that, The bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to the first threshold; the bit width of the model parameter is the number of bits occupied by the model parameter after it is converted into binary format.

3. The method according to claim 1, characterized in that, When the neural network model includes an online-trained neural network model, the step of compressing and encoding the multiple model parameter sets corresponding to the multiple neural network models included in the neural network-based encoding technique using differential prediction further includes: If the number of model parameters in the model parameter set of the nth neural network model is less than or equal to the number of model parameters in the model parameter set of the mth neural network model, a first parameter difference is encoded. The first parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model.

4. The method according to claim 1, characterized in that, The model parameter set encoding the first neural network model includes: The model parameter set of the first neural network model is encoded using the first encoding method; The parameter differences between the corresponding parameters in the parameter set of the nth neural network model and the parameter set of the mth neural network model include: The parameter differences between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model are encoded using a second encoding method.

5. The method according to claim 1, characterized in that, The m is a parameter that is fixed to or preset in the encoding end before encoding, or the m is determined by the encoding end based on preset rules.

6. The method according to claim 1, characterized in that, The m=1, or the m=n-1.

7. The method according to claim 5, characterized in that, The m is determined by the encoding end based on a preset rule, and the first encoded data also includes first indication information, which is used to indicate the value of m.

8. The method according to claim 7, characterized in that, The preset rules include: For the model parameter set of the nth neural network model, select the model parameter set of the neural network model with the smallest number of bits in the parameter difference encoding the model parameter set of the nth neural network model from multiple neural network models.

9. The method according to claim 1, characterized in that, When the neural network model includes multiple offline-trained neural network models, the step of compressing and encoding the multiple model parameter sets corresponding to the multiple neural network models included in the neural network-based encoding technique using differential prediction further includes: If the number of model parameters in the model parameter set of the second neural network model is less than or equal to the number of model parameters in the model parameter set of the first neural network model, a third parameter difference is encoded, wherein the third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model.

10. The method according to claim 1, characterized in that, The model parameter set of the first neural network model among the plurality of offline-trained neural network models includes: The model parameter set of the first neural network model is encoded using the first encoding method; The parameter differences between the model parameter set encoding the second neural network model and the corresponding parameters in the model parameter set of the first neural network model include: The parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model are encoded using a second encoding method.

11. The method according to claim 1, characterized in that, The first neural network model's parameter set includes the neural network model with the fewest number of model parameters among the multiple offline-trained neural network models' parameter sets.

12. The method according to claim 1, characterized in that, The first encoded data also includes second indication information, which is used to indicate the model parameters of the neural network model used when encoding the p-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, or image level, and p is an integer greater than 0.

13. The method according to claim 4 or 10, characterized in that, The first encoding method is a fixed-length code encoding method, and the second encoding method is a variable-length code encoding method.

14. The method according to claim 1, characterized in that, The method further includes: The model parameter set of the neural network model is divided to obtain multiple subsets of model parameters of the neural network model. The method of compressing and encoding the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: Multiple subsets of model parameters of the neural network model are encoded using the differential prediction method.

15. The method according to claim 14, characterized in that, The neural network model includes multiple neural network models, and the multiple subsets of model parameters of the model parameter set of the neural network model encoded using the differential prediction method include: For the first subset of model parameters in the q-th neural network model, the first subset of model parameters is encoded using the subset of model parameters in the third neural network model; For the second model parameter subset in the q-th neural network model, the second model parameter subset is encoded using the model parameter subset in the fourth neural network model. The third neural network model and the fourth neural network model are different models among the plurality of neural network models, and q is an integer greater than 0.

16. The method according to claim 1, characterized in that, The first encoded data is transmitted through a first communication link, and the second encoded data is transmitted through a second communication link; or, The first encoded data and the second encoded data are transmitted through the first communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

17. The method according to claim 16, characterized in that, The first encoded data is transmitted through the first communication link, and the second encoded data is transmitted through the second communication link. The first encoded data also includes third indication information, which is used to indicate the model parameters of the neural network model used when encoding the s-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level. s is an integer greater than 0.

18. The method according to claim 1, characterized in that, The method further includes: Convert the model parameters, including those in the model parameter set of the neural network model, into the target format; The method of compressing and encoding the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: The target format is compressed and encoded using the differential prediction method.

19. The method according to claim 18, characterized in that, The target format includes the Neural Network Transformation Format NNEF or the Open Neural Network Transformation Format ONNX.

20. The method according to claim 1, characterized in that, The encoding of the image to be encoded using neural network-based encoding technology includes: The neural network-based coding technique is used to perform at least one of the following: prediction, transformation, quantization, entropy coding, or filtering on the image to be encoded.

21. The method according to claim 1, characterized in that, The first encoded data includes a fourth indication information, which is used to indicate whether encoding using neural network encoding technology has been enabled.

22. The method according to claim 1, characterized in that, The method of compressing and encoding the model parameter set of the neural network model included in the neural network-based encoding technique using differential prediction includes: The differential prediction method is used to compress and encode some or all of the model parameters in the model parameter set of the neural network model.

23. A decoding method, characterized in that, include: Receive the first encoded data and the second encoded data; The second encoded data is decoded to obtain model parameters in multiple model parameter sets corresponding to multiple neural network models. The multiple model parameter sets corresponding to the multiple neural network models are encoded using differential prediction. The neural network models include online trained neural network models or multiple offline trained neural network models. The first encoded data is decoded using model parameters from the model parameter set of multiple neural network models to obtain the decoded image; When the neural network model includes an online-trained neural network model, receiving the second encoded data includes: Receive the encoded data of the model parameter set of the first neural network model; The system receives encoded data of the parameter differences between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model, where n is a positive integer greater than 1 and m is a positive integer greater than or equal to 1. Decoding the second encoded data to obtain model parameters from multiple model parameter sets corresponding to multiple neural network models includes: Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; The encoded data of the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model is decoded to obtain a first parameter difference and a second parameter difference. The first parameter difference is the parameter difference between the corresponding parameter in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The second parameter difference is the parameter difference between the parameters in the model parameter set of the nth neural network model other than the model parameter corresponding to the first parameter difference and a preset value. Here, n is a positive integer greater than 1, and m is a positive integer greater than or equal to 1. The model parameters of the nth neural network model are obtained by using the first parameter difference and the model parameters in the model parameter set of the mth neural network model, and by using the second parameter difference and the preset value; or, When the neural network model includes multiple offline-trained neural network models, receiving the second encoded data includes: Receive the encoded data of the model parameter set of the first neural network model among the plurality of offline trained neural network models; The system receives encoded data of the parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model. The second neural network model is a neural network model other than the first neural network model among the plurality of offline trained neural network models. Decoding the second encoded data to obtain model parameters from multiple model parameter sets corresponding to multiple neural network models includes: Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model is decoded to obtain a third parameter difference and a fourth parameter difference. The third parameter difference is the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model. The fourth parameter difference is the parameter difference between the parameters in the model parameter set of the second neural network model other than the model parameter corresponding to the third parameter difference and a preset value. The model parameter set of the second neural network model is obtained by using the third parameter difference and the model parameters in the model parameter set of the first neural network model, and by using the fourth parameter difference and the preset value.

24. The method according to claim 23, characterized in that, The bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to the first threshold; the bit width of the model parameter is the number of bits occupied by the model parameter after it is converted into binary format.

25. The method according to claim 23, characterized in that, When the neural network model includes an online-trained neural network model, the step of decoding the second encoded data to obtain model parameters from multiple model parameter sets corresponding to multiple neural network models further includes: The encoded data of the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model is decoded to obtain the first parameter difference, which is the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model. The model parameters of the nth neural network model are obtained by using the first parameter difference and the model parameters in the model parameter set of the mth neural network model.

26. The method according to claim 23, characterized in that, Decoding the encoded data of the model parameter set of the first neural network model includes: The encoded data of the model parameter set of the first neural network model is decoded using the first decoding method; Decoding the encoded data of the parameter differences between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model includes: The encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model are decoded using the second decoding method.

27. The method according to claim 23, characterized in that, The m is a parameter that is fixed to or preset in the decoding end, or the m is determined based on preset rules.

28. The method according to claim 23, characterized in that, The m=1, or the m=n-1.

29. The method according to claim 27, characterized in that, The m is determined based on a preset rule, and the first encoded data also includes first indication information, which is used to indicate the value of m.

30. The method according to claim 23, characterized in that, When the neural network model includes multiple offline-trained neural network models, the step of decoding the second encoded data to obtain model parameters from multiple model parameter sets corresponding to the multiple neural network models further includes: The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model is decoded to obtain the third parameter difference, which is the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model. The model parameter set of the second neural network model is obtained by using the difference of the third parameter and the model parameters in the model parameter set of the first neural network model.

31. The method according to claim 23, characterized in that, Decoding the encoded data of the model parameter set of the first neural network model includes: The encoded data of the model parameter set of the first neural network model is decoded using the first decoding method; Decoding the encoded data of the parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model includes: The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model are decoded using the second decoding method.

32. The method according to claim 23, characterized in that, The first neural network model's parameter set includes the neural network model with the fewest number of model parameters among the multiple offline-trained neural network models' parameter sets.

33. The method according to claim 23, characterized in that, The first encoded data also includes second indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the p-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, or image level, and p is an integer greater than 0.

34. The method according to claim 26 or 31, characterized in that, The first decoding method is a fixed-length code decoding method, and the second decoding method is a variable-length code decoding method.

35. The method according to claim 23, characterized in that, Decoding the second encoded data to obtain the model parameters in the model parameter set of the neural network model includes: The second encoded data is decoded to obtain multiple subsets of model parameters of the neural network model's model parameter set.

36. The method according to claim 35, characterized in that, The neural network model includes multiple neural network models. Decoding the second encoded data to obtain model parameters from the model parameter set of the neural network model includes: The second encoded data is decoded to obtain the residual data of the first model parameter subset and the residual data of the second model parameter subset; The first model parameter subset is obtained by using the residual data of the first model parameter subset and the model parameter subset in the third neural network model; The second model parameter subset is obtained by using the residual data of the second model parameter subset and the model parameter subset in the fourth neural network model, wherein the third neural network model and the fourth neural network model are different models among the plurality of neural network models; Decoding the first encoded data using model parameters from the model parameter set of the neural network model includes: The first encoded data is decoded using model parameters from the first subset of model parameters and model parameters from the second subset of model parameters.

37. The method according to claim 23, characterized in that, The first encoded data is transmitted through a first communication link, and the second encoded data is transmitted through a second communication link; or, The first encoded data and the second encoded data are transmitted through the first communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

38. The method according to claim 37, characterized in that, The first encoded data is transmitted through the first communication link, and the second encoded data is transmitted through the second communication link. The first encoded data also includes third indication information, which is used to indicate the model parameters of the neural network model used when encoding the s-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level. s is an integer greater than 0.

39. The method according to claim 23, characterized in that, Decoding the second encoded data to obtain the model parameters in the model parameter set of the neural network model includes: The second encoded data is decoded to obtain the target format; Decoding the first encoded data using model parameters from the model parameter set of the neural network model includes: Convert the target format; The first encoded data is decoded using the model parameters of the converted format.

40. The method according to claim 39, characterized in that, The target format includes the Neural Network Transformation Format NNEF or the Open Neural Network Transformation Format ONNX.

41. The method according to claim 23, characterized in that, Decoding the first encoded data using model parameters from the model parameter set of the neural network model includes: The model parameters in the model parameter set of the neural network are used to perform at least one of the following on the first encoded data: entropy decoding, inverse quantization, inverse transformation, prediction reconstruction, or filtering.

42. The method according to claim 23, characterized in that, The first encoded data includes a fourth indication information, which is used to indicate whether encoding using neural network encoding technology has been enabled.

43. The method according to claim 23, characterized in that, Decoding the second encoded data to obtain the model parameters in the model parameter set of the neural network model includes: The second encoded data is decoded to obtain some or all of the model parameters in the model parameter set of the neural network model.

44. An encoding device, characterized in that, include: A processor is used to encode the image to be encoded using neural network-based coding techniques to obtain first encoded data; The second encoded data is obtained by compressing and encoding multiple model parameter sets corresponding to multiple neural network models included in the neural network-based encoding technology using differential prediction; the neural network model includes an online-trained neural network model or multiple offline-trained neural network models. Transmit the first encoded data and the second encoded data; When the neural network model includes an online-trained neural network model, the processor is further configured to: The set of model parameters for encoding the first neural network model; If the number of model parameters in the model parameter set of the nth neural network model is greater than the number of model parameters in the model parameter set of the mth neural network model, encode the first parameter difference and the second parameter difference. The first parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The second parameter difference is the parameter difference between the parameters in the model parameter set of the nth model model other than the model parameter corresponding to the first parameter difference and the preset value. or, When the neural network model includes multiple offline-trained neural network models, the processor is further configured to: Encode the model parameter set of the first neural network model among the plurality of offline-trained neural network models; If the number of model parameters in the model parameter set of the second neural network model is greater than the number of model parameters in the model parameter set of the first neural network model, a third parameter difference and a fourth parameter difference are encoded. The third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model. The fourth parameter difference is the parameter difference between the parameters in the model parameter set of the second neural network model other than the model parameters corresponding to the third parameter difference and a preset value.

45. The apparatus according to claim 44, characterized in that, The bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to the first threshold; the bit width of the model parameter is the number of bits occupied by the model parameter after it is converted into binary format.

46. ​​The apparatus according to claim 44, characterized in that, When the neural network model includes an online-trained neural network model, the processor is further configured to: If the number of model parameters in the model parameter set of the nth neural network model is less than or equal to the number of model parameters in the model parameter set of the mth neural network model, a first parameter difference is encoded. The first parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model.

47. The apparatus according to claim 44, characterized in that, The processor is further used for: The model parameter set of the first neural network model is encoded using the first encoding method; The parameter differences between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model are encoded using a second encoding method.

48. The apparatus according to claim 44, characterized in that, The m is a parameter that is fixed to or preset in the encoding end before encoding, or the m is determined by the encoding end based on preset rules.

49. The apparatus according to claim 44, characterized in that, The m=1, or the m=n-1.

50. The apparatus according to claim 48, characterized in that, The m is determined by the encoding end based on a preset rule, and the first encoded data also includes first indication information, which is used to indicate the value of m.

51. The apparatus according to claim 50, characterized in that, The preset rules include: For the model parameter set of the nth neural network model, select the model parameter set of the neural network model with the smallest number of bits in the parameter difference encoding the model parameter set of the nth neural network model from multiple neural network models.

52. The apparatus according to claim 44, characterized in that, When the neural network model includes multiple offline-trained neural network models, the processor is further configured to: If the number of model parameters in the model parameter set of the second neural network model is less than or equal to the number of model parameters in the model parameter set of the first neural network model, a third parameter difference is encoded, wherein the third parameter difference is the parameter difference between the corresponding parameters in the model parameter set of the second neural network model and the model parameter set of the first neural network model.

53. The apparatus according to claim 44, characterized in that, The processor is further used for: The model parameter set of the first neural network model is encoded using the first encoding method; The parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model are encoded using a second encoding method.

54. The apparatus according to claim 44, characterized in that, The first neural network model's parameter set includes the neural network model with the fewest number of model parameters among the multiple offline-trained neural network models' parameter sets.

55. The apparatus according to claim 44, characterized in that, The first encoded data also includes second indication information, which is used to indicate the model parameters of the neural network model used when encoding the p-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, or image level, and p is an integer greater than 0.

56. The apparatus according to claim 47 or 53, characterized in that, The first encoding method is a fixed-length code encoding method, and the second encoding method is a variable-length code encoding method.

57. The apparatus according to claim 44, characterized in that, The processor is further used for: The model parameter set of the neural network model is divided to obtain multiple subsets of model parameters of the neural network model. Multiple subsets of model parameters of the neural network model are encoded using the differential prediction method.

58. The apparatus according to claim 57, characterized in that, The neural network model includes multiple neural network models, and the processor is further used for: For the first subset of model parameters in the q-th neural network model, the first subset of model parameters is encoded using the subset of model parameters in the third neural network model; For the second model parameter subset in the q-th neural network model, the second model parameter subset is encoded using the model parameter subset in the fourth neural network model. The third neural network model and the fourth neural network model are different models among the plurality of neural network models, and q is an integer greater than 0.

59. The apparatus according to claim 44, characterized in that, The first encoded data is transmitted through a first communication link, and the second encoded data is transmitted through a second communication link; or, The first encoded data and the second encoded data are transmitted through the first communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

60. The apparatus according to claim 59, characterized in that, The first encoded data is transmitted through the first communication link, and the second encoded data is transmitted through the second communication link. The first encoded data also includes third indication information, which is used to indicate the model parameters of the neural network model used when encoding the s-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level. s is an integer greater than 0.

61. The apparatus according to claim 44, characterized in that, The processor is further used for: Convert the model parameters, including those in the model parameter set of the neural network model, into the target format; The target format is compressed and encoded using the differential prediction method.

62. The apparatus according to claim 61, characterized in that, The target format includes the Neural Network Transformation Format NNEF or the Open Neural Network Transformation Format ONNX.

63. The apparatus according to claim 44, characterized in that, The processor is further used for: The neural network-based coding technique is used to perform at least one of the following: prediction, transformation, quantization, entropy coding, or filtering on the image to be encoded.

64. The apparatus according to claim 44, characterized in that, The first encoded data includes a fourth indication information, which is used to indicate whether encoding using neural network encoding technology has been enabled.

65. The apparatus according to claim 44, characterized in that, The processor is further used for: The differential prediction method is used to compress and encode some or all of the model parameters in the model parameter set of the neural network model.

66. A decoding device, characterized in that, include: A processor for receiving first encoded data and second encoded data; The second encoded data is decoded to obtain model parameters in multiple model parameter sets corresponding to multiple neural network models. The multiple model parameter sets corresponding to the multiple neural network models are encoded using differential prediction. The neural network models include online trained neural network models or multiple offline trained neural network models. The first encoded data is decoded using model parameters from the model parameter set of multiple neural network models to obtain the decoded image; When the neural network model includes an online-trained neural network model, the processor is further configured to: Receive the encoded data of the model parameter set of the first neural network model; The system receives encoded data of the parameter differences between the model parameter set of the nth neural network model and the corresponding parameters in the model parameter set of the mth neural network model, where n is a positive integer greater than 1 and m is a positive integer greater than or equal to 1. Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; The encoded data of the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model is decoded to obtain a first parameter difference and a second parameter difference. The first parameter difference is the parameter difference between the corresponding parameter in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model. The second parameter difference is the parameter difference between the parameters in the model parameter set of the nth neural network model other than the model parameter corresponding to the first parameter difference and a preset value. Here, n is a positive integer greater than 1, and m is a positive integer greater than or equal to 1. The model parameters of the nth neural network model are obtained by using the first parameter difference and the model parameters in the model parameter set of the mth neural network model, and by using the second parameter difference and the preset value; or, When the neural network model includes multiple offline-trained neural network models, the processor is further configured to: Receive the encoded data of the model parameter set of the first neural network model among the plurality of offline trained neural network models; The system receives encoded data of the parameter differences between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model. The second neural network model is a neural network model other than the first neural network model among the plurality of offline trained neural network models. Decode the encoded data of the model parameter set of the first neural network model to obtain the model parameters in the model parameter set of the first neural network model; The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model is decoded to obtain a third parameter difference and a fourth parameter difference. The third parameter difference is the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model. The fourth parameter difference is the parameter difference between the parameters in the model parameter set of the second neural network model other than the model parameter corresponding to the third parameter difference and a preset value. The model parameter set of the second neural network model is obtained by using the third parameter difference and the model parameters in the model parameter set of the first neural network model, and by using the fourth parameter difference and the preset value.

67. The apparatus according to claim 66, characterized in that, The bit width of the model parameters included in the model parameter set of the neural network model is less than or equal to the first threshold; the bit width of the model parameter is the number of bits occupied by the model parameter after it is converted into binary format.

68. The apparatus according to claim 66, characterized in that, When the neural network model includes an online-trained neural network model, the processor is further configured to: The encoded data of the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model is decoded to obtain the first parameter difference, which is the parameter difference between the model parameter set of the nth neural network model and the corresponding parameter in the model parameter set of the mth neural network model. The model parameters of the nth neural network model are obtained by using the first parameter difference and the model parameters in the model parameter set of the mth neural network model.

69. The apparatus according to claim 66, characterized in that, The processor is further used for: The encoded data of the model parameter set of the first neural network model is decoded using the first decoding method; The encoded data of the parameter difference between the corresponding parameters in the model parameter set of the nth neural network model and the model parameter set of the mth neural network model are decoded using the second decoding method.

70. The apparatus according to claim 66, characterized in that, The m is a parameter that is fixed to or preset in the decoding end, or the m is determined based on preset rules.

71. The apparatus according to claim 66, characterized in that, The m=1, or the m=n-1.

72. The apparatus according to claim 70, characterized in that, The m is determined based on a preset rule, and the first encoded data also includes first indication information, which is used to indicate the value of m.

73. The apparatus according to claim 66, characterized in that, When the plurality of neural network models includes a plurality of offline-trained neural network models, the processor is further configured to: The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model is decoded to obtain the third parameter difference, which is the parameter difference between the model parameter set of the second neural network model and the corresponding parameter in the model parameter set of the first neural network model. The model parameter set of the second neural network model is obtained by using the difference of the third parameter and the model parameters in the model parameter set of the first neural network model.

74. The apparatus according to claim 66, characterized in that, The processor is further used for: The encoded data of the model parameter set of the first neural network model is decoded using the first decoding method; The encoded data of the parameter difference between the model parameter set of the second neural network model and the corresponding parameters in the model parameter set of the first neural network model are decoded using the second decoding method.

75. The apparatus according to claim 66, characterized in that, The first neural network model's parameter set includes the neural network model with the fewest number of model parameters among the multiple offline-trained neural network models' parameter sets.

76. The apparatus according to claim 66, characterized in that, The first encoded data also includes second indication information, which is used to indicate the identifier of the model parameters of the neural network model used when encoding the p-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, or image level, and p is an integer greater than 0.

77. The apparatus according to claim 69 or 74, characterized in that, The first decoding method is a fixed-length code decoding method, and the second decoding method is a variable-length code decoding method.

78. The apparatus according to claim 66, characterized in that, The processor is further used for: The second encoded data is decoded to obtain multiple subsets of model parameters of the neural network model's model parameter set.

79. The apparatus according to claim 78, characterized in that, The processor is further used for: The second encoded data is decoded to obtain the residual data of the first model parameter subset and the residual data of the second model parameter subset; The first model parameter subset is obtained by using the residual data of the first model parameter subset and the model parameter subset in the third neural network model; The second model parameter subset is obtained by using the residual data of the second model parameter subset and the model parameter subset in the fourth neural network model, wherein the third neural network model and the fourth neural network model are different models among the plurality of neural network models; The first encoded data is decoded using model parameters from the first subset of model parameters and model parameters from the second subset of model parameters.

80. The apparatus according to claim 66, characterized in that, The first encoded data is transmitted through a first communication link, and the second encoded data is transmitted through a second communication link; or, The first encoded data and the second encoded data are transmitted through the first communication link, and the second encoded data is written into the first encoded data in the form of encoded header information.

81. The apparatus according to claim 80, characterized in that, The first encoded data is transmitted through the first communication link, and the second encoded data is transmitted through the second communication link. The first encoded data also includes third indication information, which is used to indicate the model parameters of the neural network model used when encoding the s-th target image. The target image is an image after the image to be encoded is divided according to any one of video sequence level, image group level, and image level. s is an integer greater than 0.

82. The apparatus according to claim 66, characterized in that, The processor is further used for: The second encoded data is decoded to obtain the target format; Convert the target format; The first encoded data is decoded using the model parameters of the converted format.

83. The apparatus according to claim 82, characterized in that, The target format includes the Neural Network Transformation Format NNEF or the Open Neural Network Transformation Format ONNX.

84. The apparatus according to claim 66, characterized in that, The processor is further used for: The model parameters in the model parameter set of the neural network are used to perform at least one of the following on the first encoded data: entropy decoding, inverse quantization, inverse transformation, prediction reconstruction, or filtering.

85. The apparatus according to claim 66, characterized in that, The first encoded data includes a fourth indication information, which is used to indicate whether encoding using neural network encoding technology has been enabled.

86. The apparatus according to claim 66, characterized in that, The processor is further used for: The second encoded data is decoded to obtain some or all of the model parameters in the model parameter set of the neural network model.

87. An encoding device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 22.

88. A decoding device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 23 to 43.

89. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the encoding method according to any one of claims 1 to 22.

90. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the decoding method according to any one of claims 23 to 43.

Images