Encoding method, decoding method and system

Abstract

Provided in the embodiments of the present application are an encoding method, a decoding method and a system, which are used for implementing ultra-low code rate satellite transmission of images, so that receiving ends can restore satellite-transmitted images having richer information. The decoding method comprises: acquiring a first character string, wherein the first character string indicates at least one image frame transmitted by a satellite, for example, the first character string may be a character string obtained by encoding features extracted from the at least one image frame; and then decoding the first character string to obtain the at least one image frame.

Description

An encoding method, a decoding method, and a system

[0001] This application claims priority to Chinese Patent Application No. 202410965433.4, filed on July 17, 2024, entitled "An Encoding Method, Decoding Method and System", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communications, and more particularly to an encoding method, a decoding method, and a system. Background Technology

[0003] Satellite transmission is a crucial communication method for terminals. For example, in scenarios without mobile network coverage, the data transmission bandwidth of a terminal is severely limited, necessitating satellite transmission. However, satellite transmission bandwidth is extremely limited, resulting in a very limited amount of data that can be transmitted, making image transmission particularly challenging. Therefore, achieving ultra-low bitrate image recovery in satellite transmission scenarios has become an urgent problem to be solved. Summary of the Invention

[0004] This application provides an encoding method, a decoding method, and a system for satellite transmission of ultra-low bit rate images, enabling the receiving end to recover satellite-transmitted images with richer information.

[0005] In view of this, in a first aspect, this application provides a decoding method, comprising: obtaining a first string, the first string indicating at least one frame of image transmitted by satellite, for example, the first string may be a string obtained by encoding features extracted from the at least one frame of image, thereby compressing the image into a string with a very small space at the encoding end to adapt to the satellite transmission scenario; subsequently decoding the first string to obtain at least one frame of image, the decoding method of the first string may adopt the reverse process corresponding to the encoding end, for example, the first string may be used as the input of a neural network, and the output may be the recovered image transmitted by satellite.

[0006] In this embodiment, features can be extracted from the image and transmitted after encoding to achieve ultra-low bit rate satellite image transmission. Furthermore, in ultra-low bit rate image transmission scenarios, images with richer details can be recovered.

[0007] In one possible implementation, the aforementioned decoding of the first string to obtain at least one frame of image may include: decoding the first string to obtain decoding features; and inputting the decoding features into a super-resolution model to obtain at least one frame of image.

[0008] In this embodiment, the first string can be decoded first to obtain decoded features, which can represent the features in the image. The decoded features are used as input to the super-resolution model, and the super-resolution model is used to restore the image, resulting in an image with richer details and higher fidelity.

[0009] In one possible implementation, the aforementioned inputting the decoded features into the super-resolution model to obtain at least one frame of image may include: performing image restoration on the decoded features to obtain a compressed image; and inputting the compressed image and the decoded features into the super-resolution model to obtain at least one frame of image. In this embodiment, a compressed image may first be obtained by restoration based on the decoded features, and then both the compressed image and the decoded features may be used as input to the super-resolution model, thereby performing super-resolution on the compressed image to restore an image with richer details and textures.

[0010] In one possible implementation, the aforementioned method may further include: obtaining a second string, which indicates salient and non-salient regions in each frame of at least one image, wherein the salient region is the region of interest in each frame, and the non-salient region is the region in each frame other than the salient region. In embodiments of this application, the image may also be divided into salient and non-salient regions to facilitate adaptive image restoration by combining the salient and non-salient regions during image restoration, resulting in an image with better overall performance.

[0011] In one possible implementation, the aforementioned inputting the compressed image and decoded features into the super-resolution model to obtain at least one frame of image may include: inputting the second string, the compressed image, and the decoded features into the super-resolution model, outputting salient region images and non-salient region images; and fusing the salient region images and non-salient region images to obtain at least one frame of image. In this embodiment, the second string, the compressed image, and the decoded features can all be input into the super-resolution model, allowing the super-resolution model to determine the salient and non-salient regions in the image based on the second string. This enables adaptive super-resolution for each salient and non-salient region, outputting salient and non-salient region images respectively, thereby obtaining an image with better overall performance by fusing the salient and non-salient region images.

[0012] In one possible implementation, the aforementioned super-resolution model includes a first module and a second module. The first module outputs images of salient regions, and the second module outputs images of non-salient regions. The first module is trained using a pixel-based loss function and / or a loss function based on image perception results; the second module is trained using a loss function based on generated pixels. In this embodiment, the first module is set for salient regions, and the second module is set for non-salient regions. The first module can be trained using a pixel-based loss function and / or a loss function based on image perception results, aiming to make the output of the first module more faithful to the original image. The second module is trained using a loss function based on generated pixels, aiming to give the output of the second module a certain generative ability, thereby ultimately obtaining an image with more faithful salient regions and richer details in non-salient regions.

[0013] In one possible implementation, the aforementioned first string includes feature encoding and scale encoding. Decoding the first string to obtain decoded features includes: decoding the feature encoding to obtain feature decoding information; decoding the scale encoding to obtain scale decoding information; and reconstructing the decoded features based on the feature decoding information and the scale decoding information. In this embodiment, the specific decoding process of the first string can employ feature decoding and scale decoding for feature reconstruction, thereby outputting the final reconstructed features to support ultra-low bitrate image transmission scenarios based on neural network output features.

[0014] Secondly, this application provides an encoding method, comprising: acquiring at least one frame of image; inputting the at least one frame of image into an image encoder, outputting a first string, the first string indicating the at least one frame of image, the image encoder being used to extract features from the image and encode them; and subsequently transmitting the first string via satellite. In the embodiments of this application, at the encoding end, a neural network can be used to extract features that occupy very little space, and encode these features to output a string that occupies very little space, thereby achieving ultra-low bitrate image transmission.

[0015] In one possible implementation, the aforementioned method further includes: dividing at least one frame of image into salient and non-salient regions, wherein the salient region is the region of interest in the image and the non-salient region is the region in the image other than the salient region; and then transmitting a second string via satellite, which can be used to indicate the salient or non-salient region.

[0016] In this embodiment, the encoding end can divide the image into salient and non-salient regions, generate a string, and transmit it to the decoding end, so that the decoding end can refer to the salient and non-salient regions to perform image restoration and obtain an image with better restoration effect.

[0017] In one possible implementation, the aforementioned image encoder can be used to extract features from at least one frame of image to obtain encoded features; and to encode the encoded features to obtain a first string. In this embodiment of the application, the image encoder can extract features from the image and encode them to output a string that occupies very little space, thereby achieving ultra-low bitrate image transmission.

[0018] In one possible implementation, the aforementioned image encoder is specifically used for: extracting features from an input image to obtain a first feature; extracting features from the first feature to obtain a second feature; obtaining the residual between the first feature and the second feature to obtain a residual distribution; obtaining scale information based on the second feature, the scale information being used to represent the distribution range of the first feature; encoding the second feature to obtain a feature code; and encoding based on the residual distribution and the scale information to obtain a scale code, wherein the first string includes the feature code and the scale code.

[0019] In this embodiment of the application, during the image encoding process, high-level features extracted from the image can be encoded, and the scale can be calculated based on the features, so that the receiving end can reconstruct the high-level features based on the scale to achieve image restoration.

[0020] Thirdly, this application provides a decoding device, comprising:

[0021] The transceiver module is used to acquire a first string, which indicates at least one frame of image transmitted via satellite;

[0022] The processing module is used to decode the first string to obtain at least one frame of image.

[0023] The effects achieved by the third aspect or any optional implementation of the third aspect can be referred to in the description of the first aspect or any optional implementation of the first aspect, and will not be repeated here.

[0024] In one possible implementation, the aforementioned processing module is specifically used to: decode the first string to obtain decoded features; and input the decoded features into the super-resolution model to obtain at least one frame of image.

[0025] In one possible implementation, the aforementioned processing module is specifically used for: performing image restoration on the decoded features to obtain a compressed image; and inputting the compressed image and the decoded features into a super-resolution model to obtain at least one frame of image.

[0026] In one possible implementation, the aforementioned transceiver module is further configured to acquire a second string, which is used to indicate the salient and non-salient regions of each frame of at least one frame of image, wherein the salient region is the region of interest in each frame of image, and the non-salient region is the region in each frame of image other than the salient region.

[0027] In one possible implementation, the aforementioned processing module is specifically used to: input the second string, the compressed image, and the decoded features into the super-resolution model, and output salient region images and non-salient region images; and fuse the salient region images and non-salient region images to obtain at least one frame of image.

[0028] In one possible implementation, the aforementioned super-resolution model includes a first module and a second module. The first module outputs a salient region image, and the second module outputs a non-salient region image. The first module is trained using a pixel-based loss function and / or a loss function based on image perception results; the second module is trained using a loss function based on generated pixels.

[0029] In one possible implementation, the aforementioned processing module is specifically used to: decode the second string to obtain a first feature; decode the third string to obtain a second feature; and reconstruct the first feature based on the second feature to obtain a decoded feature.

[0030] Fourthly, this application provides an encoding device, comprising:

[0031] The transceiver module is used to acquire at least one frame of image;

[0032] The processing module is used to input the input image to the image encoder and output a first string, which indicates at least one frame of image;

[0033] The transceiver module is also used to send the first string via satellite.

[0034] The effects achieved by the fourth aspect or any optional implementation of the fourth aspect can be referred to the description of the second aspect or any optional implementation of the second aspect mentioned above, and will not be repeated here.

[0035] In one possible implementation, the aforementioned processing module is further configured to divide at least one frame of image into salient regions and non-salient regions, wherein the salient region is the region of interest in the input image, and the non-salient region is the region in the input image other than the salient region.

[0036] The transceiver module is also used to transmit a second string via satellite, which is used to indicate a significant or non-significant area.

[0037] In one possible implementation, the aforementioned image encoder is specifically used for: extracting features from each frame of an image in at least one frame to obtain a first feature; extracting features from the first feature to obtain a second feature; obtaining the residual between the first feature and the second feature to obtain a residual distribution; obtaining scale information based on the second feature, the scale information being used to represent the distribution range of the first feature; encoding the second feature to obtain a feature code; and encoding based on the residual distribution and the scale information to obtain a scale code, wherein the first string includes the feature code and the scale code.

[0038] Fifthly, embodiments of this application provide a terminal including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the first aspects above.

[0039] In a sixth aspect, embodiments of this application provide a server including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the second aspects above.

[0040] In a seventh aspect, embodiments of this application provide a terminal including a processor and a memory, wherein the processor and the memory are interconnected via a circuit, and the processor calls program code in the memory to perform processing-related functions in the method shown in any of the second aspects above.

[0041] Eighthly, this application provides a codec system, including a terminal and a server, wherein the terminal is connected to the server and can be used to perform the steps as described in the first aspect or any optional embodiment of the first aspect; the server can be used to perform the steps as described in the second aspect or any optional embodiment of the second aspect.

[0042] Ninthly, embodiments of this application provide a digital processing chip or chip, the chip including a processing unit and a communication interface, the processing unit obtaining program instructions through the communication interface, the program instructions being executed by the processing unit, the processing unit being used to perform processing-related functions as described in any of the first to second aspects above.

[0043] In a tenth aspect, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described in any of the first or second aspects above.

[0044] In the eleventh aspect, embodiments of this application provide a computer program product comprising a computer program / instruction, which, when executed by a processor, causes the processor to perform the method in any of the embodiments of the first or second aspect described above. Attached Figure Description

[0045] Figure 1 is a schematic diagram of the structure of an encoding / decoding system provided in an embodiment of this application;

[0046] Figure 2 is a schematic diagram of another encoding / decoding system provided in an embodiment of this application;

[0047] Figure 3 is a schematic diagram of an application scenario provided by an embodiment of this application;

[0048] Figure 4 is a flowchart illustrating an encoding method provided in an embodiment of this application;

[0049] Figure 5 is a flowchart illustrating a decoding method provided in an embodiment of this application;

[0050] Figure 6 is a schematic diagram of another application scenario provided by an embodiment of this application;

[0051] Figure 7 is a flowchart illustrating the encoding and decoding methods under the system architecture provided in this application embodiment;

[0052] Figure 8 is a schematic diagram of the structure of an image encoder provided in an embodiment of this application;

[0053] Figure 9 is a schematic diagram of the structure of an image decoder provided in an embodiment of this application;

[0054] Figure 10 is a schematic diagram of the structure of an image encoder and an image decoder provided in an embodiment of this application;

[0055] Figure 11 is a schematic diagram of the structure of a super-resolution model provided in an embodiment of this application;

[0056] Figure 12 is a schematic diagram of another super-resolution model provided in an embodiment of this application;

[0057] Figure 13 is a schematic diagram of the training and inference architecture of a super-resolution model provided in an embodiment of this application;

[0058] Figure 14 is a schematic diagram of the inference architecture of a super-resolution model provided in an embodiment of this application;

[0059] Figure 15 is a schematic diagram of an encoding device provided in an embodiment of this application;

[0060] Figure 16 is a schematic diagram of a decoding device provided in an embodiment of this application;

[0061] Figure 17 is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0062] Figure 18 is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0064] This application relates to the fields of artificial intelligence and computer vision. To facilitate understanding, some terms and concepts involved in this application will be introduced first.

[0065] (1) Neural Network

[0066] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit that takes an intercept of 1 as input can be represented as follows:

[0067] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together; that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0068] (2) Deep Neural Networks

[0069] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple intermediate layers. Based on the position of these layers, the internal neural network of a DNN can be divided into three categories: input layer, intermediate layers, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are intermediate layers, or hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0070] Although DNNs appear complex, each layer can be represented as a linear relational expression: in, It is the input vector. It is the output vector. is the offset vector, also known as the bias parameter; w is the weight matrix (also called coefficients); and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is also quite large. The definitions of these parameters in DNNs are as follows: Taking the coefficient w as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0071] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0072] It's important to note that the input layer does not have a W parameter. In deep neural networks, more intermediate layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0073] (3) Convolutional Neural Network

[0074] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0075] The network used for feature extraction mentioned below in this application may include one or more convolutional layers. For example, the network used for feature extraction may be implemented using a CNN.

[0076] (4) Generative Model

[0077] Generative models are models capable of randomly generating observed data, especially given certain implicit parameters. They assign a joint probability distribution to the observed and labeled data sequences. In machine learning, generative models can be used to directly model data (e.g., sampling data based on the probability density function of a variable) or to establish conditional probability distributions between variables. Conditional probability distributions can typically be derived from generative models using Bayes' theorem.

[0078] (5) Generative Adversarial Networks

[0079] Generative adversarial network (GAN) training: This is a mainstream training paradigm for implicit generative models. For a given implicit generative network, a discriminator network is introduced to play a game with the generative network. From an optimization perspective, this is a typical minimax optimization problem. The equilibrium state of the game can correspond to the distance between various distributions, such as the optimal transmission distance and the Jensen-Shannon divergence.

[0080] (6) Loss Function

[0081] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually an initialization process before the first update, pre-configuring parameters for each layer.) For example, if the network's prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, so training the deep neural network becomes a process of minimizing this loss. Common loss functions include mean squared error, cross-entropy, logarithmic, and exponential loss functions. For example, mean squared error can be used as the loss function, defined as... The specific loss function can be selected based on the actual application scenario.

[0082] (7) Backpropagation algorithm

[0083] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0084] (8) Super-resolution

[0085] Super-resolution (SR) is an image enhancement technique that, given one or a set of low-resolution images, restores high-frequency details by learning prior knowledge of the images, leveraging the similarity between images, and complementing information from multiple frames, thereby generating a higher-resolution target image. In applications, super-resolution can be categorized into single-frame super-resolution and video super-resolution based on the number of input images. Super-resolution has significant applications in high-definition television, satellite imagery, and medical imaging.

[0086] (9) Video super-resolution

[0087] Video super-resolution (VSR) is an enhancement technique for video processing, aiming to convert low-resolution video into high-quality, high-resolution video. Based on the number of input frames, video super-resolution can be divided into multi-frame video super-resolution and looping video super-resolution.

[0088] (10) Receptive Field

[0089] In the field of deep neural networks within computer vision, the receptive field is a term used to describe the size of the receptive field of neurons at different locations within a neural network. A larger receptive field value indicates a wider range of the original image that the neuron can access, meaning that the neuron may contain more global and semantically higher-level features; conversely, a smaller value indicates that the features it contains are more local and detailed. The receptive field value can be used to roughly determine the level of abstraction of each layer.

[0090] (11) Entropy coding

[0091] Entropy coding is a coding process that follows the principle of entropy without losing any information. Information entropy is the average amount of information in a source (a measure of uncertainty). Common entropy coding methods include Shannon coding, Huffman coding, and arithmetic coding.

[0092] For example, if the pixel value distribution of each pixel in the predicted image is known, the optimal compression scheme can be obtained using entropy coding. Using entropy coding, an image with probability p can be represented using -log2p bits. For example, an image with probability 1 / 8 requires 3 bits, and an image with probability 1 / 256 requires 8 bits.

[0093] To determine the number of bits for each letter, the algorithm needs to know the probability of each letter appearing as precisely as possible; the model's task is to provide this data. The better the model's predictions, the better the compression result. Furthermore, the model must provide the same data during compression and decompression.

[0094] The static model (or static entropy coding) analyzes the entire text and calculates the probability of each letter before compression. This calculation result is applied to the entire text. The encoding table only needs to be calculated once, so the encoding speed is high, and the result, except for the probability values ​​needed during decoding, is definitely no longer than the original text.

[0095] In this dynamic model, the probability changes continuously throughout the encoding process. This can be achieved through various algorithms, such as:

[0096] Forward dynamics: The probability is calculated based on the already encoded letters, and its probability increases each time a letter is encoded.

[0097] Reverse dynamics: Before encoding, calculate the probability of each letter appearing in the remaining unencoded portion. As encoding progresses, more and more letters disappear, their probability becoming 0, while the probabilities of the remaining letters increase, reducing the number of bits needed to encode them. The code rate continuously decreases until the last letter only requires 0 bits to encode.

[0098] The entropy encoding used in the method provided in this application may include static entropy encoding methods such as tANS or fse, or dynamic encoding may be implemented using a dynamic model.

[0099] (12) Autoencoder model

[0100] An autoencoder model is a neural network that uses the backpropagation algorithm to make the output value equal to the input value. It first compresses the input data into a latent space representation, and then reconstructs the output through this representation.

[0101] Autoencoder models typically include an encoder model and a decoder model. The trained encoder model extracts features from the input image to obtain latent variables. These latent variables are then input into the trained decoder model to output the residual corresponding to the predicted input image. In the embodiments of this application, the image encoder and image decoder mentioned can both be implemented using an encoder.

[0102] The method provided in this application can be applied to scenarios such as image transmission or storage. For example, it can be applied to satellite transmission scenarios, enabling ultra-low bitrate satellite transmission of images. In traditional image transmission or storage scenarios, images can be encoded, and the encoded data can be transmitted or saved to reduce the resource consumption of image transmission or storage. However, during image transmission or storage, image information may be lost due to image encoding. When decoding the image, details may be lost, style distortion, or pseudo-textures may occur during the recovery process. Especially in satellite transmission scenarios, the bandwidth of satellite transmission is usually very limited, allowing only a small number of characters to be transmitted. Therefore, satellite transmission is usually used for transmitting messages, and it is very difficult to implement for image transmission. Ultra-low bitrate compression of images is required to support satellite image transmission. However, under ultra-low bitrate image compression, the image recovery effect is also very limited.

[0103] In existing solutions for image encoding, minimizing the resource consumption of encoded data may lead to distortion or artifacts in the restored image; conversely, preserving image details requires significant resources for storing or transmitting encoded data. Existing solutions cannot recover realistic and natural texture details in ultra-low bitrate image compression scenarios. Clearly, in ultra-low bitrate satellite image transmission scenarios, achieving both minimal resource consumption in image encoding and superior image restoration is difficult.

[0104] Therefore, this application provides an encoding method and a decoding method for realizing ultra-low bit rate satellite transmission of images, enabling the receiving end to achieve better image super-resolution and output a restored image with more faithful details and structure.

[0105] The system architecture provided in this application is described below.

[0106] Referring to Figure 1, a schematic diagram of an encoding / decoding system provided in this application is shown. This system may include an encoding device 11 and a decoding device 12. Optionally, it may also include a storage device 13.

[0107] The encoding device 11 can be used to encode an image using the encoding method provided in this application embodiment, encoding the image into ultra-low bit rate encoded data to reduce the resource consumption of image transmission or storage.

[0108] The decoding device 12 can be used to decode the encoded data of an image using the decoding method provided in the embodiments of this application, perform super-resolution processing on the decoded data, and output the restored image.

[0109] The storage device 13 can be used to store data; when the storage device is deployed in the encoding device 11 or connected to the encoding device 11, the storage device 13 can be used to store the encoded data of the image output by the encoding device 11, and can also store the parameters of the model used by the encoding device during the encoding process; when the storage device 13 is deployed in the decoding device 12 or connected to the decoding device 12, it can store the encoded data that the decoding device 12 needs to decode or the model parameters used during the decoding process.

[0110] In one possible scenario, as shown in Figure 2, the aforementioned encoding and decoding devices can be located in different devices and connected via a communication network.

[0111] After encoding the image, the encoding device transmits the encoded data to the decoding device through a communication network. The decoding device then decodes the received encoded data to obtain the recovered image.

[0112] Specifically, the encoding device may include mobile terminals, tablets, laptops, augmented reality (AR) / virtual reality (VR) devices, in-vehicle terminals, televisions, digital cameras, cellular phones, cameras, smartphones, personal digital assistant (PDA) computers, tablet computers, laptop computers, machine type communication (MTC) terminals, point of sale (POS) terminals, in-vehicle computers, head-mounted devices, wristbands, smartwatches, security equipment, etc., and may also be in various devices such as servers or cloud computing.

[0113] The decoding device may specifically include the aforementioned mobile terminal, or it may include a server or cloud, etc.

[0114] Encoding and decoding devices can be deployed in the same device, such as in the same terminal, server, or server cluster, or they can be deployed in different devices, such as the encoding device being deployed in the user terminal and the decoding device being deployed in different user terminals, servers, or the cloud.

[0115] The encoding device and the decoding device can be connected via a communication network, such as a wireless network or a wired network. For example, this communication network can be a satellite transmission network, such as the Global Navigation Satellite System (GNSS), specifically one or more of the following: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0116] To facilitate understanding, the application scenarios of the encoding and decoding methods provided in this application are illustrated by example.

[0117] The method provided in this application can be applied to satellite transmission scenarios to achieve image transmission via satellite. Traditional satellite transmission can typically only transmit messages with a small number of characters, making it difficult to transmit larger amounts of data, such as high-definition images. The method provided in this application, however, can extract features from an image using a neural network, encode these features, and then transmit them. The decoding end then uses the neural network to reconstruct the image, obtaining a reconstructed high-definition image, thus achieving ultra-low bitrate image transmission, which is applicable to satellite transmission scenarios. For example, as shown in Figure 3, when users communicate using communication software, they can transmit images via wired or wireless networks. To improve the transmission rate and reduce the network resources occupied by transmitted images, the encoding method provided in this application can be used to compress the image, obtaining compressed encoded data, which can then be transmitted via satellite. After receiving the encoded data, the receiving end can use the decoding method provided in this application to decode the encoded data and obtain the reconstructed image. Typically, satellite communication resources are limited in satellite communication scenarios; therefore, the less resources the data to be transmitted, the better. The encoding method provided in this application can achieve ultra-low bitrate encoding, thereby significantly reducing the communication resources occupied by transmitted images. On the decoding end, the method provided in this application embodiment can more accurately restore the details or textures of the image and output a more faithful restored image.

[0118] The method and process provided in this application will be described below in conjunction with the aforementioned system architecture and application scenarios.

[0119] This application provides an encoding method and a decoding method, and the encoding and decoding processes are described below.

[0120] I. Coding Stage

[0121] Referring to Figure 4, a flowchart of an encoding method provided in this application is shown below.

[0122] 401. Obtain at least one frame of image.

[0123] The at least one frame of image can be an image captured by an encoding device or an image read from storage space. The number of images can be one frame or multiple frames. When multiple frames of images are acquired, the multiple frames of images can include images from a video or images captured individually, etc.

[0124] For example, in an image transmission scenario, at least one frame of the image to be transmitted can be an image captured by the encoding end. For instance, the encoding end can be a terminal, which can acquire images through its own image sensor or through a connected image sensor. The user can select to transmit the image at the encoding end to trigger the transmission of the encoded image.

[0125] 402. Input at least one frame of image into the image encoder and output the first string.

[0126] The image encoder can be a pre-trained encoder that can be used to extract features from an image, encode the extracted features, and output the encoded string.

[0127] Specifically, the image encoding involves feature extraction via an image encoder. Leveraging the feature extraction capabilities of a neural network, it outputs a feature sequence that consumes fewer resources, thereby achieving ultra-low bitrate compression of the image. The extracted feature sequence is then encoded and output as a string format supported by satellite transmission; for ease of distinction, this string is referred to as the first string.

[0128] In one possible implementation, the image encoder can be specifically used to: extract features from at least one frame of an image to be transmitted to obtain a first feature; extract features from the first feature to obtain a second feature; obtain the residual between the first feature and the second feature to obtain a residual distribution; obtain scale information based on the second feature, the scale information being used to represent the distribution range of the first feature; encode the second feature to obtain a feature code; and encode based on the residual distribution and the residual information to obtain a scale code, wherein the first string can include the feature code and the scale code.

[0129] Therefore, in this embodiment, an image encoder can be used to encode the image, outputting a string that occupies less space. For example, compared to directly encoding the image using an encoding algorithm, this embodiment uses a neural network to extract feature sequences and encode these feature sequences; the space occupied by the feature sequences is much smaller than the space occupied by the image, so this embodiment can output image codes that occupy very little space, thereby achieving ultra-low bitrate compression and transmission of the image.

[0130] Specifically, the image encoder may employ entropy coding as mentioned above, or other encoding algorithms, such as other encoding algorithms that can output strings, to be applied to satellite transmission scenarios.

[0131] 403. The first string is transmitted via satellite.

[0132] Once the first string is obtained, it can be sent to the receiving end or the decoding end to represent at least one frame of image, thereby transmitting the at least one frame of image at an ultra-low bit rate.

[0133] The receiving end can be an image encoding end or a device for storing the image encoding. For example, in an image transmission scenario, the encoding end can be a user terminal, and the receiving end can be a user terminal or a cloud server; in an image storage scenario, the receiving end can be a device containing storage space, such as the aforementioned storage device 13.

[0134] Specifically, satellite messages can be transmitted, carrying a first string, and other information such as user IDs or descriptive text can also be transmitted to achieve satellite transmission.

[0135] In this application, an image encoder is used for feature extraction to achieve ultra-low bitrate image encoding, outputting image codes in string format with very small file size, thereby enabling transmission or storage of images with smaller file sizes. For example, in satellite transmission scenarios, string transmission is typically supported. The method provided in this application can transmit ultra-low bitrate image codes, thus enabling image or video transmission in satellite transmission scenarios.

[0136] In one possible scenario, to facilitate decoding at the decoding end and obtain a more faithful decoded image, at least one frame of the image to be transmitted can be divided into salient and non-salient regions. The salient region is the region of interest in the image, and the non-salient region is the region in the image excluding the salient region. A second string is also sent to the receiving end, which includes at least one of the information of the salient region or the non-salient region, enabling the decoding end to distinguish between salient and non-salient regions in the image based on this second string. Typically, the requirements for salient and non-salient regions may differ during image decoding. For example, salient regions may require more realistic details or textures, while non-salient regions may require more vivid information. Therefore, after the encoding end distinguishes between salient and non-salient regions, the decoding end can adaptively restore the salient and non-salient regions separately to obtain an image that better meets the requirements.

[0137] Specifically, the second string may include the coordinates of the corner points, center points, or other coordinates representing the location of the salient or non-salient region, or a string encoded from the coordinates.

[0138] Furthermore, the first string and the second string can be sent in the same message or in adjacent messages, depending on the specific application scenario.

[0139] II. Decoding Phase

[0140] Referring to Figure 5, a flowchart of a decoding method provided in an embodiment of this application is shown below.

[0141] 501. Get the first string.

[0142] The first string can be information received from the encoding end and transmitted via satellite, or information read from storage. Taking information transmitted via satellite as an example, the first string can indicate at least one frame of image.

[0143] The first string includes the data output by the pre-trained image encoder after encoding the input image. The generation process of the first string can be referred to the description of the encoding stage above, and will not be repeated here.

[0144] In one possible scenario, satellite messages can be received, that is, messages transmitted via satellite. The first string can be parsed from the satellite message to realize image transmission in a satellite communication scenario.

[0145] In one possible scenario, based on the aforementioned description of the encoding end, the first string can be divided into multiple parts, such as the aforementioned feature encoding and scale encoding. Thus, the image can be indicated through feature encoding and scale encoding, and richer image content can be indicated with less space through implicit representation.

[0146] 502. Decode the first string to obtain at least one frame of image.

[0147] After obtaining the first string, an imager can be used to decode the first string to obtain at least one frame of image. This decoding process can be understood as the reverse of the aforementioned encoding process, and the corresponding decoding process can be deployed according to the encoding process at the encoding end.

[0148] In the aforementioned encoding stage, the image encoder extracts features from the input image and encodes the features, namely the aforementioned feature encoding and scale encoding. Correspondingly, in the decoding stage, the received first string can be decoded, that is, the feature encoding and scale encoding are decoded to obtain feature decoding information and scale decoding information. The obtained decoded features can be used to represent the features in the image, and based on these features, at least one frame of image with richer content can be recovered.

[0149] In this embodiment, the string transmitted via satellite is a low-bitrate string, occupying very little space. The method provided in this embodiment can decode this string, outputting one or more frames of image, thereby achieving richer details and textures in scenarios where ultra-low bitrate image encoding is used for satellite transmission. Currently, instead of directly compressing and transmitting images, the method provided in this embodiment can transmit features extracted from the image, preserving these features during transmission and achieving ultra-low bitrate image transmission with a very high image compression rate. Furthermore, the receiving end can utilize the recovery capabilities of neural networks based on the received features to perform more faithful image restoration.

[0150] In one possible implementation, decoding the first string may include decoding the first string to obtain decoded features, then inputting the decoded features into a super-resolution model to output at least one frame of image. Therefore, the super-resolution capability of the super-resolution model can be utilized to output an image with richer details or textures.

[0151] Furthermore, the decoding process may include recovering the decoded features to obtain a compressed image, and inputting both the compressed image and the decoded features into the super-resolution model to obtain at least one frame of image. Therefore, in this embodiment, the input to the super-resolution model can be set to include multiple parts, so that when super-resolution is performed on the compressed image, the decoded features can be combined to perform super-resolution, so as to output an image containing richer information. That is, by introducing the features in the image into the super-resolution process and using image encoding to guide the super-resolution process, structural fidelity can be achieved while supplementing image details.

[0152] In one possible implementation, if the input image is further divided into salient and non-salient regions at the encoding end, a second string can be obtained, such as by receiving a second string sent by the encoding end or reading a second string from storage space. This second string is used to indicate salient or non-salient regions in the image. The decoded features, the second string, and the compressed image can all be input into the super-resolution model, outputting salient region images and non-salient region images. By fusing the salient and non-salient region images, at least one frame of image can be obtained. In this embodiment, the second string can also be used as input to the super-resolution model, enabling the model to distinguish between salient and non-salient regions in the image and perform targeted super-resolution on the salient and non-salient regions to output a super-resolution image that better meets the requirements.

[0153] In one possible implementation, the super-resolution model can include multiple modules, such as a first module and a second module. The first module outputs images of salient regions, and the second module outputs images of non-salient regions. Therefore, in this embodiment, super-resolution is performed separately for salient and non-salient regions in the super-resolution model, thereby outputting images that better meet the user's needs.

[0154] Furthermore, to ensure that each module has corresponding output capabilities for different regions, during the training phase, the first module is trained using a pixel-based loss function and / or an image perception-based loss function, resulting in a more faithful image output from the first module. The second module is trained using a loss function based on generated pixels, where generated pixels are those generated by the second module during the training phase, thus giving the second module a certain generation capability. Therefore, for salient regions, a more faithful image can be output, while for non-salient regions, textures or details can be generated, resulting in an image with richer content. This makes the final fused image more faithful in salient regions and richer in details or textures in non-salient regions.

[0155] In one possible implementation, corresponding sub-models can be set for salient and non-salient regions respectively, or the aforementioned first and second modules can also be understood as sub-models in the super-resolution model, such as being referred to as the first supramolecular model and the second supramolecular model respectively. The first and second supramolecular models can adaptively process salient and non-salient regions to obtain a super-resolution image that better meets the requirements. For example, the decoded features, compressed image, and salient region information can be input into the first supramolecular model, outputting a salient region image. The first supramolecular model can combine the decoded features to recover the texture or details of objects in the salient regions of the compressed image, thus outputting a salient region image containing richer textures or details. The decoded features, compressed image, and non-salient region information can be input into the second supramolecular model, outputting a non-salient region image. This second supramolecular model can be used to combine the decoded features to generate richer content in the non-salient regions of the compressed image, thereby outputting a non-salient image containing richer content. The salient and non-salient region images are then fused to obtain the super-resolution image. Therefore, in this embodiment, super-resolution is performed on salient and non-salient regions in the compressed image to adaptively enhance the image of salient and non-salient regions and output a super-resolution image that better meets the requirements.

[0156] Furthermore, during the training of the super-resolution model, the first supramolecular model is trained using a pixel-based loss function and / or an image perception result-based loss function, aiming to make the output of the first supramolecular model more faithful; the second supramolecular model is trained using a loss function based on generated pixels, aiming to make the output of the second supramolecular model richer due to its generative ability, thereby ultimately obtaining an image with more faithful salient regions and richer details in non-salient regions.

[0157] The foregoing has introduced the process of the encoding and decoding methods provided in this application. To facilitate understanding, the process of the encoding and decoding methods provided in this application will be further introduced below with specific application scenarios and more detailed examples.

[0158] First, taking the scenario of satellite transmission of images as an example, the application scenario of the method provided in this application can be shown in Figure 6.

[0159] In this scenario, both the sending and receiving ends can be user mobile phones, and the sending and receiving mobile phones communicate with each other through satellites, cloud servers, and communication base stations.

[0160] For example, for terminals such as mobile phones or wearable devices, users use the terminal to take pictures and save them locally. The terminal has an application with satellite transmission function installed, runs the encoding method provided in the embodiments of this application, encodes the image to obtain an ultra-low bit rate image message, and uploads the image message to the cloud server via satellite or other transmission methods according to a predefined message transmission format.

[0161] Satellite: Receives ultra-low bitrate image packets uploaded by mobile devices and sends them to the cloud server.

[0162] Cloud server: This application provides a decoding method that can be deployed to receive image message messages and parse them to obtain decoding features. The image decoder is used to obtain a low-resolution compressed image. The low-resolution compressed image and the decoding features are then input into a super-resolution model to obtain a restored high-resolution image.

[0163] Mobile communication base station: used to receive high-resolution images sent by the message server and send them to the receiving mobile phone.

[0164] Receiving mobile phone: Receives images and displays them locally.

[0165] In some scenarios, the decoding method provided in this application can also be deployed on the receiving mobile phone. After receiving the image message transmitted by satellite, the decoding method provided in the embodiments of this application can be used to decode the image to obtain one or more frames and display them.

[0166] For example, in a specific application scenario, after a user takes a picture with their mobile phone, the sending mobile phone can encode the high-resolution original image (e.g., larger than 10M) using an image encoder and transmit it as an ultra-low bitrate image message to the receiving end via satellite. Upon receiving the ultra-low bitrate image message, the cloud server decodes it using an image decoder, outputting a low-resolution image, i.e., the aforementioned compressed image. Subsequently, a super-resolution model is used to super-resolution the low-resolution image, outputting a super-resolution image, which is then sent to the receiving mobile phone via a communication base station. The receiving mobile phone can then display the super-resolution image on its display interface. In other words, the encoding and decoding methods provided in this application embodiment can achieve ultra-low bitrate transmission of high-resolution images and more faithful image super-resolution, thereby making the image displayed on the receiving mobile phone as close as possible to the image taken by the user on the sending mobile phone, achieving a lower bitrate and more lossless image transmission process.

[0167] For example, satellite communication can be used in scenarios where mobile phone bandwidth is severely limited in areas without mobile network coverage. In such cases, data transmission can be achieved using satellite communication to ensure that images captured by the phone can be transmitted to the external environment. Typical scenarios include wilderness exploration and disaster relief. In another possible scenario, a user may be outside the coverage area of ​​wireless communication. However, since satellite communication has a wider coverage area, the user can use a terminal with satellite transmission capabilities to communicate with other terminals. However, satellite transmission bandwidth is usually very limited. To improve transmission efficiency, the transmitted images need to be compressed at an ultra-low bit rate, reducing the image size to a very small ultra-low bit rate image message before being transmitted to other terminals via satellite.

[0168] The process of the method provided in this application will be further described below in conjunction with the aforementioned application scenarios.

[0169] Referring to Figure 7, a flowchart illustrating the encoding and decoding methods under the system architecture provided in this application embodiment is shown below.

[0170] First, on the encoding side:

[0171] 701. The terminal captures an image and triggers the image to be transmitted via satellite.

[0172] This terminal can include user terminals, such as mobile phones, smartwatches, smart bracelets, or cameras—electronic devices with deployed image sensors. After capturing an image, the user can trigger the image to be transmitted via satellite.

[0173] For example, the terminal could be a mobile phone. Users can take photos with their phones, directly capturing images by calling the phone's camera through an application deployed on the phone, or by saving photos taken with the phone's system camera or other applications to the phone's system album. Then, an application on the phone (such as a communication software installed on the phone) can call the system album to read the images. Subsequently, the user can trigger the sending of the images on the phone's display interface. For example, the phone's display interface can provide the user with options to select the sending method, which could include the option to send via satellite, allowing the user to choose whether to send the image via satellite.

[0174] 702. The terminal runs the image encoder and outputs the first string.

[0175] After the terminal triggers the sending of the image, the pre-trained image encoder can be run. The captured image is used as the input of the image encoder, and the output is the feature after the feature extracted from the image is encoded, which is the first string.

[0176] Specifically, an image encoder can be used to extract features from an image and encode the extracted features to output an image encoding, i.e., the first string. This can be understood as an image encoder containing a feature extraction module and an encoding module. The feature extraction module extracts features from the input image, and the encoding module encodes the features output by the feature extraction module. Compared to directly compressing and encoding the image, this embodiment extracts higher-level features from the image, thereby expressing the image more completely with very little bandwidth, and thus transmitting the image or video with very little bandwidth usage.

[0177] For example, in one possible implementation, the structure of the image encoder can be as shown in Figure 8. The image encoder may include an image feature extraction module 1 (referred to as encoder1), an image feature extraction module 2 (referred to as encoder2), a residual prediction module, a scale parameter calculation module, and an entropy encoder, etc. The entropy encoder can be used to perform one or more entropy encoding operations in the process mentioned in the embodiments of this application; that is, the entropy encoding steps shown in Figure 8 are performed by the entropy encoder.

[0178] The image is input into encoder1, and the output feature is represented as y;

[0179] y is input into encoder2, and the output feature representation is z;

[0180] Input y and z into the residual prediction module and output residual information;

[0181] Input z into the scale parameter calculation module and output scale information, represented as scale;

[0182] Input z into the entropy encoder, and the output string is represented as z_string;

[0183] The residual information and scale information are input into the entropy encoder, and the output string is represented as y_string, where y_string and z_string can be combined to form the first string.

[0184] Specifically, y follows a Gaussian distribution, which typically involves two parameters: mean and scale. The residual prediction module can be used to estimate the mean of y, meaning the residual information can be used to represent the mean of y. The scale parameter calculation module can be used to estimate the scale of y, meaning the scale can represent the distribution range of y, so that the decoding end can estimate the range of y based on the scale obtained from decoding.

[0185] This is equivalent to extracting features from the input image, which conform to a Gaussian distribution, and calculating the mean and scale of the Gaussian distribution that the feature follows. During encoding, the extracted features, as well as the mean and scale, are encoded simultaneously so that the decoding end can recover the features based on comprehensive information and obtain a more faithful restored image.

[0186] 703. The terminal performs saliency detection and outputs the second string.

[0187] To facilitate image restoration at the decoding end, the terminal can also perform saliency detection on the encoding end, that is, divide the image into salient and non-salient regions, and output a second string to represent the salient and non-salient regions.

[0188] The salient region is typically the region of interest, such as the main subject or primary area of ​​an image as perceived by the human eye. The fidelity of this region significantly impacts the user's perception. The non-salient region is the area in the image other than the salient region, such as non-primary or secondary areas of the image as perceived by the human eye. The fidelity requirement for this region is not high. The decoding end can supplement details in this region to improve the user's overall satisfaction with the image.

[0189] Alternatively, the terminal can distinguish between salient and non-salient regions by using image processing algorithms, such as distinguishing regions based on image texture, or by using a pre-trained neural network to divide salient and non-salient regions in the image.

[0190] Significant and non-significant regions can be represented using masks, meaning the second string can include a mask to distinguish between them; alternatively, they can be represented using rectangles, meaning the second string can include the coordinates of the rectangles used to distinguish between significant and non-significant regions. For example, significant and non-significant regions can be represented by pixel-level masks, where each pixel's value is a Boolean value indicating whether the corresponding image region is significant; or, after dividing the image into significant and non-significant regions, rectangles can be used to represent them, such as rectangles representing significant regions or non-significant regions, with the second string carrying the coordinates of the rectangle's corners or center point, for example, the coordinates of the top-left and bottom-right corners, or the coordinates of the bottom-left and top-right corners.

[0191] 704. Send the first string and the second string to the cloud server via satellite.

[0192] The terminal can then send the first and second strings to the cloud server via satellite, using the satellite as a relay station to forward radio waves, enabling data transmission within the satellite's coverage area.

[0193] Specifically, messages can be transmitted via satellite. These messages may include descriptive text, time, coordinates, and padding such as a first string and a second string or a user ID. The message format can be adapted to the satellite that the terminal can connect to. The first string can indicate one or more image frames. In this embodiment, the encoding and decoding process of one image frame is used as an example. In real-world scenarios, multiple decoding processes can be performed for multiple image frames, which will not be elaborated upon here.

[0194] Therefore, in this embodiment, at the encoding end, the features extracted from the image and the parameters corresponding to those features can be encoded, enabling the encoding of high-level semantic information extracted from the image. This results in an encoded string with very small memory footprint, achieving ultra-low bitrate image transmission. Furthermore, the encoding end divides the image into salient and non-salient regions, allowing the decoding end to adaptively decode and restore the image based on these regions, obtaining a more satisfactory restored image.

[0195] On the decoding end:

[0196] It should be noted that the decoding method provided in this application embodiment can be deployed on a cloud server or on a terminal, etc. In the following embodiments of this application, the decoding method provided in this application embodiment is deployed on a cloud server as an example. The steps executed by the cloud server can also be replaced by the terminal, which will not be described in detail here.

[0197] 705. The cloud server runs an image decoder and outputs decoded features and compressed images.

[0198] The cloud server parses the received image packet, obtaining a first string and a second string. Encoding is then performed, and the image decoder runs, using the first string as input to output decoded features and a compressed image. The second string can also be used to determine salient and non-salient regions in the compressed image, facilitating subsequent super-resolution of these regions separately.

[0199] The image decoder can include multiple modules, such as a decoding module, an image restoration module, and a super-resolution model. The decoding module decodes the first string to obtain a feature sequence representing the image, i.e., decoded features. The image restoration module performs image restoration based on this feature sequence, outputting a compressed image. Specifically, the image restoration module can be a pre-trained neural network that outputs the restored image based on the input features.

[0200] For example, the structure of an image decoder can be as shown in Figure 9, which can include an entropy decoder, a scale parameter calculation module, a reconstruction module, and an image restoration module. The entropy decoder can be used to perform the entropy decoding steps, i.e., to perform the entropy decoding steps mentioned in Figure 9 and Figure 10 below.

[0201] Corresponding to Figure 7 above, as shown in Figure 10, the first string includes y_string and z_string. z_string is input to the entropy decoder, and the reconstructed z is output, represented as...

[0202] Will Input the scale parameter calculation module to obtain the scale information, i.e., the aforementioned scale;

[0203] Input the y_string and scale information into the entropy decoder to obtain the residual information;

[0204] residual information and The input is given to the reconstruction module, i.e., the y-reconstruction in Figures 11 and 12, and the output is the reconstructed y-value, i.e., the decoded feature, represented as... Scale information can be used to determine the distribution range of the reconstructed y;

[0205] Will Input the image into the image restoration module to obtain the restored compressed image.

[0206] In the image decoding stage, the output is a compressed image and

[0207] Therefore, in the embodiments of this application, corresponding to the image encoder, the feature sequence of the image can be recovered to obtain a sequence representing the image features and the corresponding compressed image, so as to realize image recovery in ultra-low bit rate transmission scenarios.

[0208] In output compressed image and After that, image super-resolution can be performed, namely steps 707 and 708 below.

[0209] It should be noted that, for salient and non-salient regions in an image, corresponding modules or sub-models can be set separately in the super-resolution model. For example, in this embodiment, sub-models are set separately for salient and non-salient regions for illustrative purposes. In practical applications, these can be replaced with modules set separately in the super-resolution model.

[0210] 706. Input the decoded features, compressed image, and salient region into the first supramolecular model, and output the salient region image.

[0211] For salient regions, a corresponding super-resolution model, namely the first supramolecular model, is set up at the decoding end. The decoded features, compressed image and salient region are input into the first supramolecular model, and the salient region image obtained after super-resolution of the salient region can be output.

[0212] Specifically, a super-resolution model typically includes multiple parts, such as an input module, one or more feature extraction modules (represented as blocks), a post-processing module, and an upsampling module (represented as a pixel shuffle). The decoded features and compressed image are input into one or more feature extraction blocks. Each feature extraction block receives both the output of the previous block and the decoded features, uses this information to perform feature extraction, and outputs the result to the next block. The post-processing module performs image post-processing and uses pixel shuffle to upsample the image, outputting the super-resolution image.

[0213] For example, the structure of a possible super-resolution model can be shown in Figure 11. Here, intro represents the input module, NAFblock represents the feature extraction block, ending represents the post-processing module, pixelshuffle represents the upsampling module, resize represents the upsampling module using an interpolation algorithm, and the decoded features are as described above. As additional supplementary information, each NAFblock is input to enable it to extract richer information.

[0214] During the super-resolution process, the initial compressed image is represented as a quadruple (n, c, h, w), where n: Batch, the batch size, representing the number of images in a batch; c: Channel, the number of channels in an image; h: Height, the height, representing the number of pixels in the vertical dimension of the image; and w: Width, the width, representing the number of pixels in the horizontal dimension of the image. Specifically, (n, c, h, w) can represent the coverage area of ​​the salient region in the image identified using the mask included in the second string.

[0215] The initial compressed image is input into intro, and the output is a quadruple (n, width, h, w), which replaces the number of channels with the width of the image, so that features can be extracted based on the width of the image later.

[0216] Then (n, width, h, w) and Input to NAFblock for feature extraction, output (n, width, h, w).

[0217] Input (n, width, h, w) into the ending for post-processing, and output (n, c*4, h, w).

[0218] Then (n,c*4,h,w) is input into pixelshuffle for upsampling, that is, upsampling is performed using the features of multiple channels to increase the image resolution, and the output is (n,c,2h,2w).

[0219] Meanwhile, (n,c,h,w) is also input to resize to adjust the image size, and outputs (n,c,2h,2w).

[0220] The (n,c,2h,2w) output by the pixelshuffle module and the (n,c,2h,2w) output by the resize module are then merged to output the final (n,c,2h,2w) to complete the super-resolution of the image.

[0221] For example, a possible super-resolution model based on unet structure points can be shown in Figure 12. Similar to the structure and function of the modules in Figure 11, Figure 12 adds a downsampling processing part and a corresponding fusion part to perform super-resolution on the image at different scales to achieve a certain deblurring effect and output a clearer super-resolution image. For example, in Figure 12, a downsampling module is added, denoted as down( / 2), and each down( / 2) module corresponds to one or more NAF blocks. An upsampling module is also set for each scale, denoted as up(*2), to restore the scale of the image.

[0222] Taking the first downsampling as an example, the (n, width, h, w) output from the previous NAFblock layer is input into down( / 2) for downsampling, and the output is (n, width*2, h / 2, w / 2). Then, (n, width*2, h / 2, w / 2) is compared with... As input to NAFblock, the output is (n, width*2, h / 2, w / 2) after feature extraction. The (n, width*2, h / 2, w / 2) at the current scale is upsampled to obtain (n, width*2, h / 2, w / 2) at the next scale, and then fused with the fused (n, width*2, h / 2, w / 2) and... The output features obtained after inputting into NAFblock are input into up(*2), and the output is (n, width, h, w).

[0223] Therefore, in the implementation of this application, the compressed image can be downsampled once or multiple times, thereby filtering out noise in the image and improving the clarity of the output image through downsampling.

[0224] 707. Input the decoded features, compressed image and non-salient region into the second supramolecular model, and output the non-salient region image.

[0225] The difference between step 707 and the aforementioned step 706 is that the second supramolecular model and the first supramolecular model are trained using different loss functions, and the processing part of the second supramolecular model is the non-salient region in the image. The processing flow of the second supramolecular model can be referred to the aforementioned introduction of the first supramolecular model, which will not be repeated here. The following will introduce some differences.

[0226] First, the second supramolecular model performs super-resolution primarily targeting image regions, including regions in the compressed image indicated by the non-salient region mask.

[0227] Secondly, the first supramolecular model and the second supramolecular model have different update methods set during training.

[0228] Typically, for super-resolution of salient regions, the goal is to achieve the highest possible fidelity, such as better restoration of faces, text, or buildings in an image. Therefore, in the training process of the first supramolecular model, this application embodiment can calculate the loss value based on the difference between the pixel-level output value and the ground truth value, and can further increase the loss value for detail settings in the image to train a model with high output fidelity. For example, for training the first supramolecular model, L1 loss between the output value and the ground truth value can be used, and further, perceptual loss for image details can be added to improve the output fidelity of the first supramolecular model.

[0229] For super-resolution of non-salient regions, the goal is usually to enrich image details and improve the user's viewing experience. Therefore, for training the second supramolecular model, an additional generation loss can be added, such as denoted as GAN loss, to measure the richness of detail in the model's output image. This gives the second supramolecular model a certain generative ability to generate details in non-salient regions, making the output image more detailed and improving the user's viewing experience, thus achieving better image restoration in satellite scenes.

[0230] 708. Merge salient and non-salient region images to output a super-resolution image.

[0231] After obtaining the salient region image and the non-salient region image, the salient region image and the non-salient region image can be fused, for example, by superimposing the salient region image and the non-salient region image to output a complete super-resolution image.

[0232] For example, the training and inference architecture for the first and second supramolecular models can be shown in Figure 13. During the training phase, different training methods are used to train the first and second supramolecular models respectively using the training set. A fidelity-first training method is used for the first supramolecular model, i.e., a fidelity-first training method is used for salient regions, while a visual-first training method is used for the second supramolecular model, i.e., a rich detail-first training method is used for non-salient regions. During the inference phase, referring to Figure 14, the salient and non-salient regions are divided after visual saliency detection at the encoding end. The first supramolecular model performs super-resolution on the compressed image for the salient regions, and the second supramolecular model performs super-resolution on the compressed image for the non-salient regions. By fusing the outputs of the first and second supramolecular models, the final high-resolution reconstructed image can be output.

[0233] Therefore, in this embodiment, decoding features, i.e., the image feature sequence obtained from decoding, are introduced into the input of the super-resolution model. These features are combined with the feature sequence extracted from the image to enrich the texture or detail information in the compressed image, thereby outputting an image with better super-resolution performance and achieving high image restoration in ultra-low bitrate image transmission scenarios. Furthermore, super-resolution is performed separately for salient and non-salient regions, outputting high-fidelity images of salient regions and images of non-salient regions with richer content. This results in high restoration of the region of interest in the final output image, and richer details or textures in other regions.

[0234] 709. The cloud server sends the super-resolution image to the receiving terminal.

[0235] After obtaining the super-resolution image by performing steps 705 to 708 on the cloud server, the super-resolution image can be sent to the receiving terminal to complete the ultra-low bit rate image transmission. The receiving terminal can then display the super-resolution image, completing the transmission of one or more frames of images in a satellite communication scenario.

[0236] Therefore, the method provided in this application embodiment can realize image restoration under ultra-low bitrate image transmission, enabling the receiving end to obtain higher-resolution reconstructed images in ultra-low bitrate image transmission scenarios. Especially in satellite transmission scenarios, it realizes the transmission of ultra-low bitrate images or videos, and the receiving end performs image super-resolution based on neural networks and the restored compressed image, so that even in ultra-low bitrate satellite transmission scenarios, very high-resolution images can be obtained.

[0237] The foregoing has described the method flow provided in the embodiments of this application. The apparatus for executing the method in the embodiments of this application is described below.

[0238] Referring to Figure 15, a schematic diagram of an encoding device provided in an embodiment of this application is shown. This encoding device can be specifically deployed in a terminal, or it can also be referred to as a terminal. The encoding device includes:

[0239] Transceiver module 1501 is used to acquire at least one frame of image;

[0240] Processing module 1502 is used to input the input image to the image encoder and output a first string, the first string indicating at least one frame of image;

[0241] The transceiver module 1501 is also used to transmit the first string via satellite.

[0242] In one possible implementation, the aforementioned processing module 1501 is further configured to divide at least one frame of image into salient regions and non-salient regions, wherein the salient region is the region of interest in the input image and the non-salient region is the region in the input image other than the salient region.

[0243] The transceiver module is also used to transmit a second string via satellite, which is used to indicate a significant or non-significant area.

[0244] In one possible implementation, the aforementioned image encoder is specifically used for: extracting features from each frame of an image in at least one frame to obtain a first feature; extracting features from the first feature to obtain a second feature; obtaining the residual between the first feature and the second feature to obtain a residual distribution; obtaining scale information based on the second feature, the scale information being used to represent the distribution range of the first feature; encoding the second feature to obtain a feature code; and encoding based on the residual distribution and the scale information to obtain a scale code, wherein the first string includes the feature code and the scale code.

[0245] Referring to Figure 16, a schematic diagram of a decoding device provided in an embodiment of this application is shown. This decoding device can be deployed on a server or terminal, and may include:

[0246] The transceiver module 1601 is used to acquire a first string, which indicates at least one frame of image transmitted via satellite;

[0247] Processing module 1602 is used to decode the first string to obtain at least one frame of image.

[0248] In one possible implementation, the aforementioned processing module 1602 is specifically used to: decode the first string to obtain decoded features; and input the decoded features into the super-resolution model to obtain at least one frame of image.

[0249] In one possible implementation, the aforementioned processing module 1602 is specifically used for: performing image restoration on the decoded features to obtain a compressed image; and inputting the compressed image and the decoded features into a super-resolution model to obtain at least one frame of image.

[0250] In one possible implementation, the aforementioned transceiver module 1601 is further configured to acquire a second string, which is used to indicate the salient and non-salient regions of each frame of at least one frame of image, wherein the salient region is the region of interest in each frame of image, and the non-salient region is the region in each frame of image other than the salient region.

[0251] In one possible implementation, the aforementioned processing module 1602 is specifically used to: input the second string, the compressed image, and the decoded features into the super-resolution model, and output salient region images and non-salient region images; and fuse the salient region images and non-salient region images to obtain at least one frame of image.

[0252] In one possible implementation, the aforementioned super-resolution model includes a first module and a second module. The first module outputs a salient region image, and the second module outputs a non-salient region image. The first module is trained using a pixel-based loss function and / or a loss function based on image perception results; the second module is trained using a loss function based on generated pixels.

[0253] In one possible implementation, the aforementioned processing module 1602 is further configured to: decode the second string to obtain a first feature; decode the third string to obtain a second feature; and reconstruct the first feature based on the second feature to obtain a decoded feature.

[0254] Both the aforementioned encoding and decoding devices can be deployed in a terminal, which 100 can be a mobile phone, tablet computer, laptop computer, wearable device with wireless communication capabilities (such as a smartwatch), or other portable electronic device. Exemplary embodiments of the portable electronic device include, but are not limited to, those equipped with… Alternatively, it can be a portable electronic device with another operating system. Terminal 100 can also be a vehicle-mounted device, an augmented reality (AR) device, a virtual reality (VR) device, or other mobile device. Of course, in this embodiment, no specific form of the terminal is limited.

[0255] The structure of a terminal provided in this application will be described below by way of example.

[0256] Figure 17 shows an exemplary structural diagram of terminal 100. As shown in Figure 17, terminal 100 includes: an application processor 101, a microcontroller unit (MCU) 103, a memory 105, a modem 107, a radio frequency (RF) module 109, a Wireless-Fidelity (Wi-Fi) module 111, a Bluetooth module 113, a sensor 114, a positioning module 150, and input / output (I / O) devices 115, etc. These components can communicate through one or more communication buses or signal lines. Those skilled in the art will understand that the hardware structure shown in Figure 17 does not constitute a limitation on the mobile phone; terminal 100 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0257] The following section, with reference to Figure 17, provides a detailed description of each component of the terminal 100:

[0258] The application processor 101 is the control center of the terminal 100, connecting various components of the terminal 100 via various interfaces and buses. In some embodiments, the processor 101 may include one or more processing units.

[0259] The memory 105 stores computer programs, such as the operating system 161 and application program 163 shown in FIG. 17. The application processor 101 is configured to execute the computer program in the memory 105 to implement the functions defined by the computer program, such as executing the steps of the encoding or decoding methods described above. The application processor 101 can also execute the operating system 161 to implement various functions of the operating system on the terminal 100. The memory 105 also stores other data besides the computer program, such as data generated during the operation of the operating system 161 and application program 163. The memory 105 is a non-volatile storage medium, generally including main memory and secondary storage. Main memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), or cache. Secondary storage includes, but is not limited to, flash memory, hard disk, optical disk, universal serial bus (USB) disk, etc. The computer program is usually stored on secondary storage, and the processor loads the program from secondary storage into main memory before executing the computer program.

[0260] The memory 105 can be independent and connected to the application processor 101 via a bus; the memory 105 can also be integrated with the application processor 101 into a chip subsystem.

[0261] MCU 103 is a coprocessor used to acquire and process data from sensor 114. MCU 103 has lower processing power and power consumption than application processor 101, but features an "always-on" characteristic, allowing it to continuously collect and process sensor data while application processor 101 is in sleep mode, ensuring normal sensor operation with extremely low power consumption. In one embodiment, MCU 103 can be a sensor hub chip. Sensor 114 can include light sensors, motion sensors, image sensors, infrared sensors, etc. Specifically, image sensors can be used to acquire images; light sensors can include ambient light sensors and proximity sensors, where the ambient light sensor can adjust the brightness of display 151 according to the ambient light level, and the proximity sensor can turn off the power to the display when the terminal 100 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary; sensor 114 can also include other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be described in detail here. The MCU 103 and sensor 114 can be integrated onto the same chip or they can be separate components connected via a bus.

[0262] Modem 107 and RF module 109 constitute the communication subsystem of terminal 100, used to implement the main functions of wireless communication standard protocols such as 3GPP and ETSI. Modem 107 is used for encoding / decoding, signal modulation / demodulation, and equalization. RF module 109 is used for receiving and transmitting wireless signals, and includes, but is not limited to, an antenna, at least one amplifier, a coupler, and a duplexer. RF module 109 works with Modem 107 to implement wireless communication functions. Modem 107 can be a standalone chip or integrated with other chips or circuits to form a system-on-a-chip or integrated circuit. These chips or integrated circuits can be applied to all terminals that implement wireless communication functions, including: mobile phones, computers, laptops, tablets, routers, wearable devices, automobiles, and home appliances. Furthermore, in satellite communication scenarios, RF module 109 can also be used to send messages to satellites. For example, a user can use the image settings on the terminal to capture an image, encode it using the encoding method provided in the aforementioned embodiments of this application through the application processor, output a string indicating the image, and send a message carrying this string in the frequency band covered by satellite communication.

[0263] Terminal 100 can also use Wi-Fi module 111, Bluetooth module 113, etc., for wireless communication. Wi-Fi module 111 provides network access to terminal 100 conforming to Wi-Fi related standard protocols. Terminal 100 can access a Wi-Fi access point through Wi-Fi module 111 and thus access the Internet. In some other embodiments, Wi-Fi module 111 can also act as a Wi-Fi wireless access point, providing Wi-Fi network access to other terminals. Bluetooth module 113 enables short-range communication between terminal 100 and other terminals (such as mobile phones, smartwatches, etc.). In this embodiment, Wi-Fi module 111 can be an integrated circuit or a Wi-Fi chip, and Bluetooth module 113 can be an integrated circuit or a Bluetooth chip.

[0264] The positioning module 150 is used to determine the geographical location of the terminal 100. It is understood that the positioning module 150 may specifically include a receiver for a positioning system such as the Global Positioning System (GPS), BeiDou Navigation Satellite System, or Russian GLONASS.

[0265] The Wi-Fi module 111, Bluetooth module 113, and positioning module 150 can each be a separate chip or integrated circuit, or they can be integrated together. For example, in one embodiment, the Wi-Fi module 111, Bluetooth module 113, and positioning module 150 can be integrated onto the same chip. In another embodiment, the Wi-Fi module 111, Bluetooth module 113, positioning module 150, and MCU 103 can also be integrated into the same chip.

[0266] Input / output devices 115 include, but are not limited to: display 151, touch screen 153, and audio circuitry 155, etc.

[0267] The touchscreen 153 can collect touch events from the user of the terminal 100 on or near it (such as user actions on or near the touchscreen 153 using a finger, stylus, or any suitable object), and send the collected touch events to other devices (such as the application processor 101). User actions near the touchscreen 153 can be termed hover touch; through hover touch, the user can select, move, or drag targets (such as icons) without directly touching the touchscreen 153. Furthermore, the touchscreen 153 can be implemented using various types of touchscreens, including resistive, capacitive, infrared, and surface acoustic wave.

[0268] The display (also called a screen) 151 is used to display information input by the user or information shown to the user. The display can be configured using a liquid crystal display (LCD), an organic light-emitting diode (OLED), or similar methods. The touchscreen 153 can cover the display 151. When the touchscreen 153 detects a touch event, it transmits the information to the application processor 101 to determine the type of touch event. The application processor 101 then provides corresponding visual output on the display 151 based on the type of touch event. Although in Figure 17, the touchscreen 153 and the display 151 are shown as two separate components implementing the input and output functions of the terminal 100, in some embodiments, the touchscreen 153 and the display 151 can be integrated to achieve the input and output functions of the terminal 100. Furthermore, the touchscreen 153 and the display 151 can be configured as a full-panel display on the front of the terminal 100 to achieve a borderless structure.

[0269] Audio circuit 1155, speaker 116, and microphone 117 provide an audio interface between the user and terminal 100. Audio circuit 109 converts received audio data into electrical signals and transmits them to speaker 113, where speaker 113 converts them into sound signals for output. On the other hand, microphone 114 converts collected sound signals into electrical signals, which are received by audio circuit 109 and converted into audio data. The audio data is then transmitted to, for example, another terminal via modem 107 and radio frequency module 109, or output to memory 105 for further processing.

[0270] In addition, the terminal 100 may also have fingerprint recognition functionality. For example, a fingerprint sensor can be configured on the back of the terminal 100 (e.g., below the rear camera) or on the front of the terminal 100 (e.g., below the touchscreen 153). Alternatively, the fingerprint sensor can be integrated into the touchscreen 153 to achieve fingerprint recognition; that is, the fingerprint sensor can be integrated with the touchscreen 153 to realize the fingerprint recognition function of the terminal 100. In this case, the fingerprint sensor is configured in the touchscreen 153, and may be part of the touchscreen 153 or configured in the touchscreen 153 in other ways. The main component of the fingerprint sensor in this embodiment is a fingerprint sensor, which can employ any type of sensing technology, including but not limited to optical, capacitive, piezoelectric, or ultrasonic sensing technologies.

[0271] In one possible implementation, the aforementioned decoding device can also be deployed in a server, as shown in FIG18, which is a schematic diagram of the hardware structure of a server 180 provided in an embodiment of this application. This server 180 can be used to implement the steps of the decoding method provided in the aforementioned embodiment of this application.

[0272] The server 180 shown in Figure 18 may include a processor 1801, a memory 1802, a communication interface 1803, and a bus 1804. The processor 1801, the memory 1802, and the communication interface 1803 can be connected via the bus 1804.

[0273] Processor 1801 is the control center of server 180. It can be a general-purpose central processing unit (CPU) or other general-purpose processors. The general-purpose processor can be a microprocessor or any conventional processor, such as a GPU or NPU, and can be adapted to the actual application scenario.

[0274] As an example, processor 1801 may include one or more CPUs, and may also include other processors, such as the CPU, NPU or GPU shown in Figure 18.

[0275] The memory 1802 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0276] In one possible implementation, the memory 1802 may exist independently of the processor 1801. The memory 1802 can be connected to the processor 1801 via a bus 1804 and is used to store data, instructions, or program code. When the processor 1801 calls and executes the instructions or program code stored in the memory 1802, it can implement the methods provided in the embodiments of this application, such as the steps of the decoding method provided in the foregoing embodiments of this application.

[0277] In another possible implementation, the memory 1802 can also be integrated with the processor 1801.

[0278] Communication interface 1803 is used for server 180 to connect with other devices via a communication network, which may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. Communication interface 1803 may include a receiving unit for receiving data and a transmitting unit for sending data.

[0279] Bus 1804 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 18, but this does not indicate that there is only one bus or one type of bus.

[0280] It should be noted that the structure shown in Figure 18 does not constitute a limitation on server 180. In addition to the components shown in Figure 18, server 180 may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0281] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0282] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0283] This application also provides a computer-readable storage medium storing a program for training a model or performing inference tasks. When the program is run on a computer, it causes the computer to perform all or part of the steps of the decoding method provided in the foregoing embodiments of this application.

[0284] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the aforementioned processor or processor functions, and one or more interfaces. When the digital processing chip integrates a memory, it can perform the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the method steps of any one or more of the foregoing embodiments based on the program code stored in the external memory.

[0285] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0286] The encoding or decoding device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip in the server to execute the steps of the decoding method provided in the foregoing embodiments of this application. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0287] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0288] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0289] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0290] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0291] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0292] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. The term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps in this application does not imply that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved. The division of modules in this application is a logical division. In actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the modules shown or discussed may be through some ports, and the indirect coupling or communication connection between modules may be electrical or other similar forms, which are not limited in this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs.

Claims

1. A decoding method, comprising: The method comprises the following steps: obtaining a first string, the first string being used for indicating at least one frame of image transmitted via a satellite; decoding the first string to obtain the at least one frame of image.

2. The method of claim 1, wherein, The step of decoding the first string to obtain the at least one frame of image comprises the following steps: decoding the first string to obtain a decoding feature; inputting the decoding feature into a super-resolution model to obtain the at least one frame of image.

3. The method of claim 2, wherein, The step of inputting the decoding feature into the super-resolution model to obtain the at least one frame of image comprises the following steps: performing image restoration on the decoding feature to obtain a compressed image; inputting the compressed image and the decoding feature into the super-resolution model to obtain the at least one frame of image.

4. The method of claim 3, wherein, The method further comprises the following steps: obtaining a second string, the second string being used for indicating a salient region and a non-salient region of each frame of image in the at least one frame of image, the salient region being a region of interest in the each frame of image, and the non-salient region being a region other than the salient region in the each frame of image.

5. The method of claim 4, wherein, The step of inputting the compressed image and the decoding feature into the super-resolution model to obtain the at least one frame of image comprises the following steps: inputting the second string, the compressed image and the decoding feature into the super-resolution model to output a salient region image and a non-salient region image; fusing the salient region image and the non-salient region image to obtain the at least one frame of image.

6. The method of claim 5, wherein, The super-resolution model comprises a first module and a second module, the first module outputs the salient region image, the second module outputs the non-salient region image, and the first module is obtained by training using a loss function based on a pixel point and / or a loss function based on an image perception result; the second module is obtained by training using a loss function based on a generated pixel.

7. The method according to any one of claims 2-6, characterized in that, The first string comprises feature encoding and scale encoding, the step of decoding the first string to obtain a decoding feature comprises the following steps: decoding the feature encoding to obtain feature decoding information; decoding the scale encoding to obtain scale decoding information; reconstructing according to the feature decoding information and the scale decoding information to obtain the decoding feature.

8. An encoding method characterized by comprising: The method comprises the following steps: obtaining at least one frame of image; inputting the at least one frame of image into an image encoder to output a first string, the first string being used for indicating the at least one frame of image; transmitting the first string via a satellite.

9. The method of claim 8, wherein, The method further comprises the following steps: dividing the at least one frame of image into a salient region and a non-salient region, the salient region being a region of interest in the at least one frame of image, and the non-salient region being a region other than the salient region in the at least one frame of image; transmitting a second string via a satellite, the second string being used for indicating the salient region or the non-salient region.

10. The method according to claim 8 or 9, characterized in that, The image encoder is used for: extracting a feature from the at least one frame of image to obtain an encoding feature; encoding the encoding feature to obtain the first string.

11. The method of claim 10, wherein, The image encoder is specifically used for: extracting a feature from the at least one frame of image to obtain a first feature; extracting a feature from the first feature to obtain a second feature, the encoding feature including the first feature and the second feature; encoding the second feature to obtain a feature encoding; obtaining a residual between the first feature and the second feature to obtain a residual distribution; obtaining scale information from the second feature, the scale information being used to represent a distribution range of the first feature; encoding the residual distribution and the scale information to obtain scale encoding, the first string encoding including the feature encoding and the scale encoding.

12. A decoding apparatus, characterized by comprising: Comprising: a transceiver module, configured to obtain a first string, the first string indicating at least one frame of image transmitted via a satellite; a processing module, configured to decode the first string to obtain the at least one frame of image.

13. The apparatus of claim 12, wherein, The processing module is specifically configured to: decode the first string to obtain a decoded feature; input the decoded feature into a super-resolution model to obtain the at least one frame of image.

14. The apparatus of claim 12, wherein, The processing module is specifically configured to: perform image restoration on the decoded feature to obtain a compressed image; input the compressed image and the decoded feature into the super-resolution model to obtain the at least one frame of image.

15. The apparatus of claim 14, wherein: the transceiver module is further configured to obtain a second string, the second string being used to indicate a salient region and a non-salient region of each frame of image in the at least one frame of image, the salient region being a region of interest in the each frame of image, and the non-salient region being a region other than the salient region in the each frame of image.

16. The apparatus of claim 15, wherein, The processing module is specifically configured to: input the second string, the compressed image and the decoded feature into the super-resolution model to output a salient region image and a non-salient region image; fuse the salient region image and the non-salient region image to obtain the at least one frame of image.

17. The apparatus of claim 16, wherein, The super-resolution model includes a first module and a second module, the first module outputs the salient region image, and the second module outputs the non-salient region image, the first module being obtained by training using a loss function based on a pixel point and / or a loss function based on an image perception result; the second module being obtained by training using a loss function based on a generated pixel.

18. The apparatus of any one of claims 13-17, wherein, The first string includes feature encoding and scale encoding, and the processing module is specifically configured to: decode the feature encoding to obtain feature decoding information; decode the scale encoding to obtain scale decoding information; reconstruct according to the feature decoding information and the scale decoding information to obtain the decoded feature.

19. An encoding apparatus, comprising: Comprising: a transceiver module, configured to obtain at least one frame of image; a processing module, configured to input the input image into an image encoder to output a first string, the first string indicating the at least one frame of image; the transceiver module is further configured to transmit the first string via a satellite.

20. The apparatus of claim 19, wherein: The processing module is further configured to divide the at least one frame of image into a salient region and a non-salient region, the salient region being a region of interest in the input image, and the non-salient region being a region other than the salient region in the input image. The transceiving module is further configured to transmit a second string via a satellite, the second string being used to indicate the salient region or the non-salient region.

21. The apparatus of claim 18 or 19, wherein, The image encoder is configured to: extract features from the at least one frame of image to obtain encoded features; encode the encoded features to obtain the first string.

22. The apparatus of claim 21, wherein, The image encoder is specifically configured to: extract first features from the at least one frame of image to obtain the first features; extract second features from the first features to obtain the second features, the encoded features comprising the first features and the second features; encode the second features to obtain feature encoding; obtain a residual distribution by obtaining a residual between the first features and the second features; obtain scale information from the second features, the scale information being used to represent a distribution range of the first features; and encode the residual distribution and the scale information to obtain scale encoding, the first string encoding comprising the feature encoding and the scale encoding.

23. A terminal, characterized by The apparatus comprises a processor and a memory coupled to the processor, the memory storing a program, and when the program stored in the memory is executed by the processor, the steps of the method in any one of claims 1-7 are implemented.

24. A server, comprising: The apparatus comprises a processor and a memory coupled to the processor, the memory storing a program, and when the program stored in the memory is executed by the processor, the steps of the method in any one of claims 1-7 are implemented.

25. A terminal, characterized by The apparatus comprises a processor and a memory coupled to the processor, the memory storing a program, and when the program stored in the memory is executed by the processor, the steps of the method in any one of claims 8-11 are implemented.

26. A cloud system, characterized by The apparatus comprises: an encoding device configured to perform the steps of the method in any one of claims 8-11; and a decoding device configured to perform the steps of the method in any one of claims 1-7.

27. A chip, characterized by The apparatus comprises a processing unit and a communication interface, the processing unit obtaining program instructions via the communication interface, and when the program instructions are executed by the processing unit, the steps of the method in any one of claims 1-7 are implemented.

28. A chip, characterized by The apparatus comprises a processing unit and a communication interface, the processing unit obtaining program instructions via the communication interface, and when the program instructions are executed by the processing unit, the steps of the method in any one of claims 8-11 are implemented.

Images