Image encoding and decoding method, and device, storage medium and program product

Abstract

An image encoding and decoding method, and a device, a storage medium and a program product. In the embodiments of the present disclosure, hierarchical feature extraction, feature partitioning and adaptive quantization are combined, image features of an image to be encoded are extracted hierarchically, and on the basis of image features of a plurality of levels and visual quality weights corresponding to a plurality of visual feature regions included in the image features, non-linear quantization processing is performed on an image vector, so as to obtain quantized features of different quantization degrees, which is beneficial to reducing the amount of encoded data and increasing the image compression ratio; furthermore, entropy encoding is performed on the quantized features by using an encoding code rate that adapts to the visual quality weights, wherein the visual quality weights are positively correlated with the encoding code rate.

Description

Image encoding and decoding method, device, storage medium and program product Cross-reference to related applications

[0001] The present disclosure claims priority to Chinese Patent Application No. 202411259509.8, filed on September 9, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the field of image encoding, and in particular, to an image encoding and decoding method, device, storage medium and program product. BACKGROUND

[0003] With the improvement of computing device performance and the rapid development of deep learning technology, some AI-based image encoding methods with superior performance have emerged. Such image encoding methods are implemented based on a neural network-based encoding framework, that is, an image is input into the encoding framework, and inside the encoding framework, feature extraction is performed on the input image, the extracted features are quantized, and then entropy encoding is performed, and finally a binary code stream is obtained.

[0004] In some neural network-based encoding frameworks, an image representation is modeled through a hyper-prior framework, and a VAE is introduced for feature conversion, and a context module is used to assist the encoder and decoder to accurately estimate the distribution, thereby significantly improving the encoding performance. Even on objective indicators such as PSNR and MS-SSIM, it has surpassed traditional encoding methods such as VVC and HEVC.

[0005] Although the neural network-based encoding framework has greatly improved the encoding performance, it still faces the problem of how to improve the image compression ratio as much as possible and reduce the encoding cost while ensuring the image reconstruction quality. SUMMARY

[0006] Aspects of the present disclosure provide an image encoding and decoding method, device, storage medium and program product to improve the image compression ratio as much as possible and reduce the encoding cost while ensuring the image reconstruction quality.

[0007] The embodiment of the present disclosure provides an image coding method applied to a target coding network, and the method comprises the following steps: performing vectorization processing on a to-be-coded image to obtain an image vector; performing hierarchical feature extraction on the to-be-coded image to obtain a plurality of levels of image features, wherein the image features comprise a plurality of visual feature regions, and the plurality of visual feature regions have respective visual quality weights; performing nonlinear quantization processing on the image vector according to the visual quality weights of the plurality of visual feature regions and the plurality of levels of image features to obtain a plurality of quantized features corresponding to the plurality of visual feature regions; and performing entropy coding processing on the plurality of quantized features according to coding code rates adapted to the visual quality weights of the plurality of visual feature regions to obtain a coding bitstream, wherein the visual quality weights are positively correlated with the coding code rates.

[0008] The embodiment of the present disclosure also provides an image decoding method applied to a target decoding network, and the method comprises the following steps: receiving a coding bitstream; inputting the coding bitstream into an entropy decoder in the target decoding network to perform entropy decoding processing on the coding bitstream to obtain image features; inputting the image features into a decoding module in the target decoding network to decode the image features to a pixel space to obtain an intermediate-state decoded image; and inputting the intermediate-state decoded image into a background generation model in the target decoding network to perform background region generation processing on the intermediate-state decoded image to obtain a target decoded image.

[0009] The embodiment of the present disclosure also provides an electronic device comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is coupled to the memory and configured to execute the computer program to implement the steps in the various methods provided by the embodiment of the present disclosure.

[0010] The embodiment of the present disclosure also provides a computer-readable storage medium storing a computer program, which causes a processor to be able to implement the steps in the above method when the computer program is executed by the processor.

[0011] The embodiment of the present disclosure also provides a computer program product comprising computer programs / instructions, which causes a processor to be able to implement the steps in the above method embodiment when the computer programs / instructions are executed by the processor.

[0012] In the embodiments of the present disclosure, the feature hierarchical extraction, feature partitioning and adaptive quantization are combined, the image features of the image to be encoded are extracted hierarchically, the image vector is subjected to nonlinear quantization processing based on the image features of multiple levels and the visual quality weight corresponding to the multiple visual feature regions included in the image features, so as to obtain quantized features of different quantization degrees, which is beneficial to reducing the amount of encoded data and improving the image compression ratio; further, the quantized features are subjected to entropy encoding using the encoding code rate adapted to the visual quality weight, and the visual quality weight is positively correlated with the encoding code rate, which can ensure that the quantized features of the visual feature region with high visual quality weight have high encoding code rate, further reducing the visual distortion caused by encoding compression, and effectively improving the overall visual quality of the image. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are included to provide a further understanding of the present disclosure, constitute a part of the present disclosure and illustrate the illustrative embodiments of the present disclosure and their description serve to explain the present disclosure, and do not constitute improper limitations on the present disclosure. In the drawings:

[0014] FIG. 1 is a flowchart of an image encoding method according to an embodiment of the present disclosure;

[0015] FIG. 2 is a flowchart of an image decoding method according to an embodiment of the present disclosure;

[0016] FIG. 3 is a schematic diagram of a model structure of a target encoding network according to an embodiment of the present disclosure;

[0017] FIG. 4 is a schematic diagram of a model structure of a target encoding network according to another exemplary embodiment of the present disclosure;

[0018] FIG. 5 is a schematic diagram of a model structure of another target encoding network and a target decoding network according to another exemplary embodiment of the present disclosure;

[0019] FIG. 6 is a schematic diagram of a structure of an electronic device according to another exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be described clearly and completely below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without making any creative efforts fall within the scope of protection of the present disclosure.

[0021] It should be noted that in the case of the user information involved in the embodiments of the present disclosure, the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region, and provide corresponding operation portal for the user to choose authorization or rejection. In addition, the various models (including but not limited to language models or large models) involved in the present disclosure are in line with relevant legal and standard regulations.

[0022] The neural network-based image coding still faces the technical problem of how to improve the image compression ratio as much as possible and reduce the coding cost under the premise of ensuring the image reconstruction quality. In the embodiments of the present disclosure, the feature hierarchical extraction, feature partitioning and adaptive quantization are combined, the image features of the to-be-coded image are extracted hierarchically, and the image vector is subjected to nonlinear quantization processing based on the image features of multiple levels and the visual quality weights corresponding to the multiple visual feature regions included in the image features. This can ensure that the quantization level of the visual feature partition in a higher feature level is lower, and the quantization level of the visual feature region with a higher importance in the same feature level is lower, reducing the visual distortion introduced by quantization; while the quantization level of the visual feature partition in a lower feature level is higher, and the quantization level of the visual feature region with a lower importance in the same feature level is higher, which is conducive to reducing the amount of encoded data and improving the image compression ratio. Further, the quantized features are entropy coded using the coding rate adapted to the visual quality weight, which can ensure that the quantized features of the visual feature region with a higher importance correspond to a higher coding rate, further reducing the visual distortion caused by coding compression, effectively improving the overall visual quality of the image, while the quantized features corresponding to the visual feature partition with a lower importance correspond to a lower coding rate, further improving the image compression ratio, reducing the occupation of storage space and the data traffic in the transmission process, and helping to save costs and improve data processing efficiency.

[0023] In addition, on the basis of low-bit-rate compression, the reconstruction process is optimized by a background generation model to improve the quality (such as clarity) of the reconstructed image content, which is closer to the original image.

[0024] In the embodiments of the present disclosure, the image coding is implemented based on a neural network coding framework, which includes a target coding network and a target decoding network. The target coding network and the target decoding network are obtained through end-to-end model training with the goal of rate-distortion optimization. The rate-distortion optimization means that the coding result can achieve the goal of meeting the set requirements of the weighted loss of distortion and code rate, so as to balance the code rate and the distortion. The rate-distortion optimization problem can be characterized by optimizing the loss function, which can be defined as R+λD, i.e., the weighted sum of the code rate (R) and the distortion (D), λ is the Lagrange multiplier, the distortion refers to the difference between the decoded image and the original image, and the code rate refers to the average number of bits required for each pixel of the coded image. Therefore, the optimization loss function can end the model training when the loss function meets the corresponding termination condition.

[0025] In an optional embodiment, the training process of the target coding network and the target decoding network includes: performing end-to-end model training on the initial coding network and the initial decoding network according to the sample original image, the initial coding network is used to code the sample original image to obtain a sample coded stream; the initial decoding network decodes the sample coded stream to obtain a sample decoded image; further, a loss function is calculated, and the model training is ended when the loss function meets the corresponding termination condition, to obtain the target coding network and the target decoding network.

[0026] The target coding network is used to receive the to-be-coded image, and code the to-be-coded image to obtain a coded stream. The coded stream is transmitted to the target decoding network. The target decoding network is used to receive the coded stream, decode the coded stream, and obtain a decoded image. It should be noted that the deployment and implementation of the to-be-coded image, the target coding network and the target decoding network will be different according to different application scenarios.

[0027] For example, in a live streaming scenario, the to-be-coded image is a video frame in a live streaming stream, the target coding network can be deployed on a live streaming server or a cloud server, used to receive the video frame in the live streaming stream, code the video frame to obtain a coded stream, and store the coded stream; the target decoding network is deployed on a terminal device of a live streaming audience, used to decode the coded stream into a video frame after the live streaming application on the terminal device pulls the coded stream from the live streaming server or the cloud server, so as to provide the live streaming application with the video frame for live streaming. Alternatively, the target decoding network can be deployed in the live streaming application as a functional module, or it can be deployed on the terminal device as a library function to provide image decoding services for various application programs on the terminal device, which is not limited.

[0028] For example, in a cloud storage scenario similar to a cloud album, an image uploaded by a user is compressed and stored. When the user needs it, the image can be pulled from the cloud album and decoded for viewing. In this scenario, the image to be encoded is the image uploaded by the user, the target encoding network can be deployed on the cloud album side to receive the image uploaded by the user, encode the image to obtain an encoded code stream, and store the encoded code stream; the target decoding network can be deployed in the album application on the user terminal to decode the encoded code stream pulled from the cloud album, encode the encoded code stream into a corresponding image, and provide the image for the user to view. Alternatively, the target decoding network can also be deployed on the cloud album side to receive the encoded code stream output by the target encoding network, decode the encoded code stream into a corresponding image, and send the image to the user terminal for the user to view.

[0029] Next, how to use the target encoding network and the target decoding network for encoding and decoding is described.

[0030] FIG. 1 is a flowchart of an image encoding method provided by an embodiment of the present disclosure. The encoding method is applied to a target encoding network. As shown in FIG. 1, the method includes S101 to S104.

[0031] S101: Vectorizing a to-be-encoded image to obtain an image vector.

[0032] S102: Hierarchically extracting features of the to-be-encoded image to obtain image features at multiple levels, the image features including multiple visual feature regions, and the multiple visual feature regions having respective visual quality weights.

[0033] S103: According to the visual quality weights of the multiple visual feature regions and the image features at the multiple levels, performing nonlinear quantization processing on the image vector to obtain multiple quantized features corresponding to the multiple visual feature regions.

[0034] S104: According to an encoding code rate adapted to the visual quality weights of the multiple visual feature regions, performing entropy encoding processing on the multiple quantized features to obtain an encoded code stream, wherein the visual quality weights are positively correlated with the encoding code rate.

[0035] In the embodiment of the present disclosure, the to-be-encoded image is an original image, and the to-be-encoded image is vectorized to obtain an image vector, in which process the to-be-encoded image is converted into a latent vector in a latent vector space, so that the to-be-encoded image can be effectively processed and analyzed.

[0036] In this embodiment, the to-be-encoded image is hierarchically extracted to obtain image features at multiple levels. The image features can be represented by a feature map. Different levels of image features refer to image features at different depths, and different levels of image features are captured by hierarchical feature extraction to capture information at different levels of abstraction.

[0037] The image features include a plurality of visual feature regions. The visual feature region refers to a region containing a type of visual feature information. For example, the visual feature region can be a region rich in details and important for image visual quality evaluation, or a region less rich in details and less important for image visual quality evaluation. The plurality of visual feature regions have respective visual quality weights. The visual quality weight is used to reflect the contribution of the visual feature region to the visual quality of the reconstructed image, and can also be regarded as the importance of the visual feature region. The greater the visual quality weight, the higher the importance of the visual feature region, and vice versa. The visual quality weights of different visual feature regions can be the same or different, but the visual quality weights of the plurality of visual feature regions are generally not completely the same. For example, the visual quality weights of some visual feature regions can be the same.

[0038] In an optional embodiment, the image features of any level in the plurality of levels include a plurality of visual feature regions. In another optional embodiment, the image features of some levels in the plurality of levels include a plurality of visual feature regions, and the image features of other levels are not divided into regions. For the image features not divided into regions, the processing can be regarded as one visual feature region, that is, the same quantization level can be used for quantization in the subsequent non-quantization process.

[0039] In this embodiment, the factors affecting the visual quality weight are not limited. For example, the visual quality weight can be determined according to the application scene or image category to which the to-be-encoded image belongs. For example, in a face recognition scene, a higher visual quality weight can be given to the face region in the to-be-encoded image, and in a traffic control scene, a higher visual quality weight can be given to the vehicle region in the to-be-encoded image. For another example, the visual quality weight corresponding to the region with different user interest degrees in the to-be-encoded image can also be assigned according to the interest degree of human vision. For the Region of Interest (ROI), a higher visual quality weight is assigned.

[0040] The visual quality weight of each visual feature region can be determined in combination with the factors affecting the visual quality weight. In this embodiment, the manner of determining the visual quality weight is not limited. In an optional embodiment, the neural network can be used to determine the visual quality weight of each visual feature region. In another optional embodiment, the artificial marking method can be used to determine the visual quality weight of each visual feature region.

[0041] Further, according to the visual quality weights of the plurality of visual feature regions and the image features of the plurality of levels, the image vector is subjected to a non-linear quantization processing to obtain a plurality of quantized features corresponding to the plurality of visual feature regions. The quantization refers to the discretization of the image vector, and the purpose is to compress the precision information of the image, so that the data amount can be significantly reduced, and the higher the quantization level, the more the data amount is reduced. In this embodiment, the image vector is subjected to non-linear quantization, and the non-linear quantization refers to the process of quantizing the image vector by using different quantization levels for the plurality of visual feature regions in the image features of different levels, that is, different quantization levels are used in the whole quantization process, rather than the same quantization level.

[0042] The quantization level is negatively related to the visual quality weight and the level of the level. The higher the visual quality weight of the visual feature region, the lower the quantization level adapted thereto, and the lower the compression ratio of the corresponding quantized feature; on the contrary, the lower the visual quality weight, the higher the quantization level adapted thereto, and the higher the compression ratio of the corresponding quantized feature. In addition, the higher the level of the level to which the visual feature region belongs, the lower the quantization level adapted thereto, and the lower the compression ratio; on the contrary, the lower the level, the higher the quantization level adapted thereto, and the higher the compression ratio.

[0043] Alternatively, the quantization level can be represented according to a quantization parameter QP (Quantization Parameter). The quantization parameter reflects the compression of the image vector, and the smaller the QP, the more details of the image vector are retained; the larger the QP, the more details of the image vector are lost, and the more serious the distortion.

[0044] Further, in order to further compress the redundant information of the quantized feature and improve the compression efficiency, the plurality of quantized features are subjected to an entropy coding processing according to the encoding code rate adapted to the visual quality weight of the plurality of visual feature regions to obtain an encoded code stream. The visual quality weight is positively related to the encoding code rate, the higher the visual quality weight, the higher the encoding code rate adapted thereto, and the lower the compression ratio of the corresponding quantized feature; on the contrary, the lower the visual quality weight, the lower the encoding code rate adapted thereto, and the higher the compression ratio of the corresponding quantized feature.

[0045] In this embodiment, the way of entropy coding is not limited, for example, arithmetic coding or Huffman coding can be used.

[0046] Correspondingly, the disclosure provides an image decoding method, which is applied to a target decoding network, as shown in FIG. 2, the method comprises S201 to S204.

[0047] S201: receiving an encoded code stream.

[0048] S202: Perform entropy decoding processing on the encoded code stream to obtain image features.

[0049] S203: Decode the image features to the image space to obtain an intermediate state decoded image.

[0050] S204: Perform background region generation processing on the intermediate state decoded image to obtain a target decoded image.

[0051] In this embodiment, the received encoded code stream can be obtained by encoding the target network. Further, entropy decoding processing is performed on the encoded code stream to obtain quantized image features, which can be optionally entropy decoded using an entropy decoder in the target decoding network.

[0052] Further, the image features are decoded and restored to the image space to obtain an intermediate state decoded image, where the image space is a vector space. Optionally, the image features can be decoded using a decoding module in the target decoding network.

[0053] The intermediate state decoded image is processed to generate a background region to obtain a target decoded image. Optionally, the background generation model in the target decoding network can be used for generation processing.

[0054] In the embodiments of the present disclosure, the feature hierarchical extraction, feature partitioning, and adaptive quantization are combined, and the image features of the to-be-encoded image are extracted by hierarchical extraction, and the image vector is nonlinearly quantized based on the image features of multiple levels and the visual quality weights corresponding to the multiple visual feature regions in the image features. This can ensure that the quantization level of the visual feature partition in the higher feature level is lower, and the quantization level of the visual feature region with higher importance in the same feature level is lower, reducing the visual distortion introduced by quantization; while the quantization level of the visual feature partition in the lower feature level is higher, and the quantization level of the visual feature region with lower importance in the same feature level is higher, which is conducive to reducing the amount of encoded data and improving the image compression ratio. Further, the quantized features are entropy encoded using an encoding code rate adapted to the visual quality weight, which can ensure that the quantized features of the visual feature region with higher importance correspond to a higher encoding code rate, further reducing the visual distortion caused by encoding compression, effectively improving the overall visual quality of the image, while the quantized features corresponding to the visual feature partition with lower importance correspond to a lower encoding code rate, further improving the image compression ratio, reducing the storage space occupation and the data traffic in the transmission process, which helps to save costs and improve data processing efficiency. In addition, on the basis of low bit rate compression, the background generation model is used to optimize the reconstruction process, improve the quality and clarity of the reconstructed image content, and be closer to the original image.

[0055] The embodiments of the present disclosure do not limit the model architecture of the target encoding network and the target decoding network. The model architecture of the target encoding network and the target decoding network is exemplarily described below, and the working principle of each is described in combination with the given model architecture.

[0056] FIG. 3 is a schematic diagram of a model structure of a target encoding network provided by an embodiment of the present disclosure. The target encoding network 30 structure shown in FIG. 3 is only one optional implementation, and does not limit the embodiments of the present disclosure. As shown in FIG. 3, the target encoding network 30 can include an encoding module 31, a feature extraction network 32, a first region division network 33, a quantization network 34, and an entropy encoder 35. The encoding method of the embodiments of the present disclosure is introduced below in combination with the target encoding network 30 of FIG. 3.

[0057] In the present embodiment, on the one hand, the to-be-encoded image can be input into the encoding module 31 in the target encoding network 30, and the encoding module 31 is configured to encode the to-be-encoded image into a latent vector space to obtain an image vector, which is regarded as a representation of the to-be-encoded image in the latent vector space. The latent vector space is a low-dimensional and compact representation space, and the to-be-encoded image is encoded into the latent vector space to reduce its dimension, and the latent vector space can also capture the key features of the image.

[0058] On the other hand, the image to be encoded is input into the feature extraction network 32 for image feature extraction. As shown in FIG. 3, the feature extraction network 32 includes a plurality of feature extraction layers, such as feature extraction layer 1-feature extraction layer N, and N > 1, N being a natural number; wherein each feature extraction layer is implemented by a convolutional neural network, and respectively includes at least one convolutional layer, as shown in FIG. 3, the feature extraction layer 1 includes convolutional layer 1-convolutional layer s, s > 1, and s being a natural number; the feature extraction layer 2 includes convolutional layer 1-convolutional layer m, m > s > 1, and m being a natural number; the feature extraction layer N includes convolutional layer 1-convolutional layer n, n > m > s > 1, and n being a natural number. The image to be encoded is input into the plurality of feature extraction layers (such as feature extraction layer 1-N) for feature extraction to obtain a plurality of levels of image features (such as image features of levels 1-N); wherein the number of convolutional layers included in different feature extraction layers in the plurality of feature extraction layers is different, for extracting image features of different levels; and the number of convolutional layers is positively correlated with the level of image features, the more the number of convolutional layers, the deeper the level of the extracted image features, the richer the semantic information expressed, and the higher the level of the image features. In an example, the feature extraction network 32 can include 3 feature extraction layers, which are shallow, middle and deep feature extraction layers, but is not limited thereto. Among them, the number of convolutional layers included in different feature extraction layers is different, taking the example of the feature extraction layers including shallow, middle and deep layers, the number of convolutional layers included increases from shallow to middle and then to deep, and the level of the layers increases from shallow to deep. Among them, the shallow feature extraction layer is used to extract low-level features of the image to be encoded, such as basic visual elements such as edges, corner points, etc.; the middle feature extraction layer is used to extract more complex features of the image to be encoded, i.e. intermediate features, which can be textures or partial object structures, etc.; the deep feature extraction layer is used to extract high-level features of the image to be encoded, which can capture the features of the entire object and complex scene, and has stronger semantic information. As can be seen, by setting feature extraction layers of different levels, hierarchical feature extraction can be achieved.

[0059] In the above description, the example that the feature extraction network 32 includes 3 feature extraction layers, which are shallow, middle and deep feature extraction layers, is described, in the present embodiment, the number of feature extraction layers in the feature extraction network 32 is not limited, and the number of convolutional layers included in any feature extraction layer is also not limited, with the increase of the number of convolutional layers, the features gradually evolve from low-level to high-level, which can be determined according to the needs of the encoding task.

[0060] Further, according to the feature semantic information in the image features of multiple levels, multiple visual feature regions are respectively identified from the image features of multiple levels. Wherein, the image features include semantic information corresponding to the features, and the semantic information corresponding to the features is also called feature semantic information. The feature semantic information presented by the features contained in the image features of different levels is also different. For example, low-level features extracted by the shallow feature extraction layer, such as color, texture and shape, are building blocks of the image to be encoded, and the semantic information is low. Compared with low-level features, mid-level features are more complex, such as texture or part of object structure, and mid-level features begin to show more semantic information. Further, high-level features contain highly abstract semantic information, which can be features of entire objects or complex scenes. For example, visual semantic information as an example of high-level features can be used to construct a semantic representation describing objects, relationships and scenes in the image to be encoded. Further, according to the feature semantic information of the image features of multiple levels, multiple visual feature regions can be identified from the image features of multiple levels, also called semantic segmentation of image features. Semantic segmentation aims to divide features in image features expressing the same feature semantic information into the same visual feature region. For example, chairs, faces, tables and computers are different semantic information, and all visual features belonging to "chairs" are divided into the same visual feature region, and all visual features belonging to "faces" are divided into the same visual feature region, and so on.

[0061] In an optional embodiment, the way of identifying multiple visual feature regions from image features of any level according to the feature semantic information of the image features, can only identify visual feature regions according to the feature semantic information in the image features; or can identify visual feature regions from image features of any level in combination with the feature semantic information in other image features of different levels. For example, in the case of identifying image features extracted by the shallow feature extraction layer, high-level features extracted by the deep feature extraction layer can be combined for identification. According to the semantic information corresponding to the high-level features, specific objects and scenes can be identified, so that the identification result of the visual feature region is more accurate.

[0062] In the embodiment, the image features of multiple levels can be respectively input into the first region division network, and multiple visual feature regions can be recognized from the image features according to the feature semantic information of the image features. In an optional implementation, as shown in FIG. 3, the first region division network 33 includes at least one first foreground recognition model, for example, first foreground recognition model 1-first foreground recognition model V, V≥1, and V is a natural number. When V=1, the first region division network 33 shown in FIG. 3 can include one first foreground recognition model (for example, first foreground recognition model 1). Further, in the case of visual feature region division by using the first region division network 33, for any level of image feature, the image feature is respectively input into at least one first foreground recognition model (for example, first foreground recognition model 1-V). For any one first foreground recognition model, at least one foreground feature region in the image feature can be recognized according to the visual semantic information of the image feature; and at least one foreground feature region and a background feature region other than the at least one foreground feature region are taken as multiple visual feature regions.

[0063] In an optional embodiment, the at least one first foreground recognition model can include a first face recognition model, a first text recognition model and a first object recognition model, as shown in FIG. 3, taking the first foreground recognition model 1 as the first face recognition model, the first foreground recognition model 2 as the first text recognition model, and the first foreground recognition model 3 as the first object recognition model as an example, but not constituting a limitation to the embodiment. Further, when the at least one first face recognition model is used to recognize the foreground feature region for the image feature of any level, at least one of the following operations is included: inputting the image feature into the first face recognition model (such as the first foreground recognition model 1 in FIG. 3), according to the visual semantic information of the image feature, recognizing whether the image feature contains a face feature, in the case of recognizing the face feature, determining a face feature region according to the face feature as a foreground feature region; inputting the image feature into the first text recognition model (such as the first foreground recognition model 2 in FIG. 3), according to the visual semantic information of the image feature, recognizing whether the image feature contains a text feature, in the case of recognizing the text feature, further recognizing the position of the text feature, determining a text feature region according to the text feature and the position as a foreground feature region; inputting the image feature into the first object recognition model (such as the first foreground recognition model 3 in FIG. 3), according to the visual semantic information of the image feature, recognizing whether the image feature contains an object feature, in the case of recognizing the object feature, further recognizing the position of the object feature, determining an object feature region according to the object feature and the position as a foreground feature region. As can be seen, different first foreground recognition models correspond to different recognition operations, and different first recognition models can recognize different visual feature regions, for example, the first face recognition model can recognize a face feature region, the first text recognition model can recognize a text feature region, and the first object recognition model can recognize an object feature region.

[0064] It should be noted that in the present embodiment, the number of first foreground recognition models included in the first region division network is not limited, and can be one or more. In the case of one first foreground recognition model, for example, it can be any one of the above-mentioned at least one first foreground recognition model, and the corresponding recognition operation can be performed. In the case of multiple first foreground recognition models, whether the image features contain corresponding regions, such as face feature regions, text feature regions, etc., the identification will be performed through each of the multiple first foreground recognition models. For example, in the case that the at least one first foreground recognition model includes a first face recognition model and a first character recognition model, the image features are input into the first face recognition model and the first character recognition model, respectively, to identify the face feature region and the text feature region, respectively. For another example, in the case that the at least one first foreground recognition model includes a first face recognition model, a first character recognition model and a first object recognition model, the image features are input into the first face recognition model, the first character recognition model and the first object recognition model, respectively, to identify the face feature region, the text feature region and the object feature region, respectively.

[0065] In the present embodiment, based on the division of the image features of multiple levels by the first foreground recognition model, multiple visual feature regions can be obtained. Further, according to the visual quality contribution of the multiple visual feature regions, the visual quality weight is assigned to the multiple visual feature regions, and the visual quality contribution is positively correlated with the visual quality weight, that is, the greater the visual contribution, the greater the visual quality weight assigned. The visual quality contribution refers to the influence degree of any visual feature region on the quality of the output image after encoding. For example, in visual perception, high-level features carry rich semantic information, and it can be considered that the visual contribution of high-level features is high. Accordingly, the semanticity of low-level features is low, and it can be considered that the visual contribution of low-level features is low.

[0066] In the above embodiment, the first foreground recognition model can be used to identify the face feature region, the text feature region and the object feature region, these feature regions have higher requirements for clarity and greater visual contribution, and therefore higher visual quality weight can be assigned to these regions. The visual quality weight is negatively correlated with the quantization level, that is, the higher the visual quality weight, the lower the quantization level adapted thereto, and the lower the compression ratio of the corresponding quantized features, so that the key details of the face feature region, the text feature region and the object feature region can be preserved, ensuring the recognizability and readability of the reconstructed image, and providing higher accuracy and reliability for the automatic processing based on the reconstructed image, such as face recognition, content review, etc.

[0067] Further, the image vector is subjected to a nonlinear quantization processing according to the visual quality weights of the multiple visual feature regions and the image features of the multiple levels, to obtain multiple quantized features corresponding to the multiple visual feature regions. In this embodiment, the visual quality weights of the multiple visual feature regions and the image features of the multiple levels can be input into a quantization network, and inside the quantization network: for the image features of any level, according to the visual quality weights of the multiple visual feature regions in the image features and the level to which the image features belong, the multiple quantization levels corresponding to the multiple visual feature regions in the image features at the any level are determined (as shown in S301 in FIG. 3). The quantization level is negatively correlated with the visual quality weight and the level rank, that is, the higher the visual quality weight, the lower the quantization level, and the higher the level rank, the lower the quantization level. Please refer to the foregoing embodiments. For example, in the foregoing embodiment, the high-level features are extracted by the deep feature extraction layer, and the level rank corresponding to the deep feature extraction layer is high, so the corresponding quantization level is low, and the amount of data reduced by the quantization processing is also small, thereby the high-level features with high visual quality contribution are given a higher retention rate. Correspondingly, the low-level features with low visual quality contribution are extracted by the shallow feature extraction layer, and the level rank of the shallow feature extraction layer is low, so the corresponding quantization level is high, and the amount of data reduced by the quantization processing is also large, thereby the image quality of the encoding output can be ensured while the code rate is reduced.

[0068] In this embodiment, the quantization level is related to the visual quality weight and the level rank of the image features to which the image features belong. The quantization levels of the image features of different levels are different. For the multiple visual feature regions in the image features of the same level, the quantization levels corresponding to the multiple visual feature regions can be determined in combination with the visual quality weights of the multiple visual feature regions, and the quantization levels of different visual feature regions can be the same or different, depending on the level to which each visual feature region belongs and the visual quality weight corresponding to each visual feature region. Alternatively, the quantization level can also be related to the application scenario and / or image category to which the image to be encoded belongs, for example, the image category can be a portrait, a landscape, an animal, and the like, that is, the application scenario and / or image category can determine the quantization level in combination with the level of the image features and the visual quality weight.

[0069] Further, the image vector is nonlinearly quantized according to the image features of the multiple levels and the multiple visual feature regions corresponding to the multiple quantization levels on the multiple levels, to obtain multiple quantized features corresponding to the multiple visual feature regions (as shown in S302 in FIG. 3). In this embodiment, for the image features of any level, the image features and the image vector are matrix calculated according to the multiple visual feature regions in the image features corresponding to the multiple quantization levels on the any level, to obtain multiple sub-quantized features corresponding to the multiple visual feature regions on the any level; further, the multiple sub-quantized features corresponding to the same visual feature region on the multiple levels are fused, to obtain the multiple quantized features corresponding to the multiple visual feature regions.

[0070] The quantization refers to a process of converting continuous numerical values into discrete values. The quantized features can present uneven probability distribution in the value domain, that is, the appearance probability of some features is relatively high, and the appearance probability of other features is relatively low. The entropy coding can assign optimal coding length to the quantized features based on the appearance probability of the quantized features. For example, the quantized features with high appearance frequency can be assigned short coding, and the quantized features with low appearance frequency can be assigned long coding, so as to ensure that the coded bitstream is more compact, to achieve the purpose of minimizing the code rate.

[0071] Further, the multiple quantized features are entropy coded according to the coding code rate adapted to the visual quality weight of the multiple visual feature regions, to obtain the coded bitstream. In this embodiment, the multiple quantized features are respectively assigned the coding code rate according to the visual quality weight of the multiple visual feature regions, the higher the visual quality weight corresponding to the visual feature region is, the higher the coding code rate assigned to the quantized features corresponding to the visual feature region is, the coding code rate assigned to the multiple quantized features is the total code rate of the image to be coded, that is, the target coding code rate, and the sum of the coding code rates assigned to the multiple quantized features is less than or equal to the target coding code rate; further, the multiple quantized features and the coding code rates are input into the entropy encoder, and the multiple quantized features are entropy coded according to the coding code rates of the multiple quantized features, to obtain the coded bitstream (as shown in S303 in FIG. 3).

[0072] In an optional embodiment, the target coding network can also be implemented as a model structure as shown in FIG. 4. As shown in FIG. 4, the target coding network 40 includes a second region division network 41, a feature extraction network 32, an encoding module 31, a quantization network 34, and an entropy encoder 35. The encoding method of the embodiments of the present disclosure will be introduced below in combination with the target coding network 40 of FIG. 4.

[0073] In this embodiment, on one hand, the image to be coded is input into the encoding module 31 in the target coding network, and the image to be coded is coded into the latent vector space, to obtain the image vector.

[0074] In another aspect, the image to be encoded is input into the feature extraction network 32 to perform hierarchical feature extraction on the image to be encoded to obtain image features at multiple levels. In this embodiment, the image to be encoded is input into multiple feature extraction layers (such as feature extraction layers 1-N) in the feature extraction network 32 to perform feature extraction respectively to obtain image features at multiple levels (such as image features at levels 1-N); wherein the multiple feature extraction layers contain different numbers of convolutional layers, and the number of convolutional layers is positively correlated with the level of the image features, and the more the number of convolutional layers, the deeper the level of the image features extracted; according to the visual semantic information of the image features at multiple levels, the multiple visual feature regions are identified from the image features at multiple levels respectively; and according to the visual quality contribution degree of the multiple visual feature regions, visual quality weights are assigned to the multiple visual feature regions, and the visual quality contribution degree is positively correlated with the visual quality weight. The detailed content of the visual quality contribution degree and the visual quality weight can be referred to the foregoing embodiments, which will not be described here.

[0075] The detailed content of hierarchical feature extraction on the image to be encoded by using the feature extraction network can be referred to the foregoing embodiments, which will not be described here.

[0076] Different from the foregoing embodiments, in this embodiment, the image to be encoded is also input into the second region division network 41 to identify multiple image regions from the image to be encoded according to the feature information of the image to be encoded. Wherein, the feature information refers to the feature information contained in the image to be encoded. In this embodiment, the image to be encoded is input into at least one second foreground recognition model in the second region division network 41, for example, second foreground recognition models 1-second foreground recognition model W, W≥1, and W is a natural number, and in the case of W=1, the second region division network 41 shown in FIG. 4 can only include the second foreground recognition model 1; at least one foreground region in the image to be encoded is identified according to the feature information of the image to be encoded; and the at least one foreground region and the background region other than the at least one foreground region are taken as the multiple image regions.

[0077] Further optionally, the at least one second foreground recognition model can include a second face recognition model, a second text recognition model, and a second object recognition model, as shown in FIG. 4, taking the second foreground recognition model 1 as the second face recognition model, the second foreground recognition model 2 as the second text recognition model, and the second foreground recognition model 3 as the second object recognition model as an example, but not constituting a limitation to the present embodiment. Further, in inputting the to-be-encoded image into the at least one second foreground recognition model in the second region division network, according to the feature information of the to-be-encoded image, at least one foreground region in the to-be-encoded image is recognized, including at least one of the following operations: inputting the to-be-encoded image into the second face recognition model, according to the feature information of the to-be-encoded image, face features in the to-be-encoded image are recognized, a face region is determined according to the face features as a foreground region; inputting the to-be-encoded image into the second text recognition model, according to the feature information of the to-be-encoded image, text information and its position in the to-be-encoded image are recognized, a text region is determined according to the text information and its position as a foreground region; inputting the to-be-encoded image into the second object recognition model, according to the feature information of the to-be-encoded image, object positions in the to-be-encoded image are recognized, object regions are determined according to the object positions as foreground regions.

[0078] Similar to the first foreground recognition model in the foregoing embodiments, the second foreground recognition model can also include one or more, and the implementation regarding the one or more can refer to the one or more of the first foreground recognition model in the foregoing embodiments, which will not be described herein again.

[0079] In the present embodiment, in the case of determining the face region by using the second face recognition model, the to-be-encoded image is input into a preprocessing layer in the second face recognition model, which can be used for preprocessing operation on the to-be-encoded image, including but not limited to size adjustment and normalization; wherein the size adjustment can control the size of the input data, reduce the data amount, and can ensure the consistency of the input data; the normalization can standardize the data to a suitable range; further, the normalized image is input into a feature extraction layer in the second face recognition model for feature extraction to obtain the feature information of the normalized image. Optionally, the feature extraction layer includes a convolution layer, the feature information is obtained by the convolution layer, and the feature information refers to the feature information contained in the to-be-encoded image; the feature information of the normalized image is input into a feature detection layer in the second face recognition model for detection of the face region, the feature detection layer is used to accurately recognize the face region, and further feature analysis is performed to determine the specific positions of the eyes, nose, mouth, and other features to obtain the face features, and the face region is determined according to the face features, and then a mask image corresponding to the face region is generated. Thus, the face region can be recognized from the image features at multiple levels based on the mask image corresponding to the face region.

[0080] In the embodiment, in the case of determining the object region by using the second object recognition model, the to-be-encoded image is input into the object detection network in the second object recognition model, the object in the to-be-encoded image is detected according to the feature information of the to-be-encoded image, to obtain the coordinate information of the detection frame, the detection frame corresponds to the region where the object is located, the coordinate information of the detection frame describes the approximate position and range of the object in the to-be-encoded image, which can provide a reference for subsequent image segmentation to improve the accuracy of segmentation, and can also avoid missed segmentation; further, the coordinate information of the detection frame is input into the object segmentation network in the second object recognition model as a condition and the to-be-encoded image, the physical segmentation network can include an image-level context module and a semantic-level context module, which are respectively used to determine the context information at the pixel level and the context information at the semantic level in the to-be-encoded image, wherein the image-level context module is used to capture the global context information (i.e., the context information at the pixel level) of each pixel in the image; the semantic-level context module is used to aggregate the context information of the same category for each pixel under real segmentation supervision (i.e., the context information at the semantic level); further, the similarity between the pixel features of the to-be-encoded image and the context information at the pixel level and the context information at the semantic level is calculated, and the similarity is used to weight and aggregate the context information at the pixel level and the context information at the semantic level to enhance the quality of the pixel feature representation. Further, each pixel is classified based on the enhanced feature representation to determine the category to which each pixel belongs, and further to generate a plurality of mask images corresponding to a plurality of image regions, different image regions identifying different categories in the to-be-encoded image, such as text regions, face regions or object regions. Further, the target encoding network further includes a mask module (not shown in FIG. 4), which can identify a plurality of visual feature regions (such as text feature regions, face feature regions and object feature regions, etc.) from a plurality of hierarchical image features based on a plurality of mask images corresponding to a plurality of image regions (as shown in S401 in FIG. 4).

[0081] Further, the image vector is subjected to non-linear quantization processing according to the visual quality weight of the plurality of visual feature regions and the plurality of hierarchical image features, to obtain a plurality of quantized features corresponding to the plurality of visual feature regions; the plurality of quantized features are subjected to entropy encoding processing according to the encoding code rate adapted to the visual quality weight of the plurality of visual feature regions, to obtain an encoded code stream, wherein the visual quality weight is positively correlated with the encoding code rate. For details of the non-linear quantization processing and the entropy encoding processing, refer to the above embodiment, which will not be repeated here.

[0082] On the basis of the two target encoding networks described above, the target encoding network can further optionally include a context prediction network 36, as shown in FIG. 3 and FIG. 4. In the above embodiment, it is introduced that the entropy encoding is to optimize the allocation of the encoding length according to the probability distribution of the quantized features. Therefore, it is particularly important for the entropy encoding to accurately estimate the probability of the occurrence of the quantized features. The context prediction network 36 can predict the context information of the image to be encoded based on the image vector, and the probability distribution of the quantized features can be predicted based on the context information, and then the appropriate encoding length is allocated for the quantized features. Therefore, the context information can be input into the entropy encoder, so that the entropy encoder allocates the encoding code rate for the plurality of quantized features according to the plurality of quantized features and the context information, and performs entropy encoding processing on the plurality of quantized features to obtain the encoded code stream.

[0083] Different context prediction networks will generate different context information. In this embodiment, the type and number of generated context information are not limited, for example, at least one of Spatial AR context (spatial autoregressive context), Checkerboard context (checkerboard context) and Channel-wise AR context (channel-wise autoregressive context) can be included. The spatial autoregressive context refers to the autoregressive context established in the spatial dimension, which is used to capture the correlation of the local region in the image, and uses the surrounding image features to predict the current features. The checkerboard context refers to a pattern similar to a checkerboard to process image features, which divides the image features into interleaved regions, and selectively uses part of the features to predict the features of other parts. The channel-wise autoregressive context refers to the autoregressive context established between different channels of image features, which can use the features of other channels in the image features to predict the features of the current channel.

[0084] It should be noted that, in the case that the entropy encoder uses specific context information to perform entropy encoding processing to obtain the encoded code stream, the same context information is used for decoding when entropy decoding is performed on the encoded code stream, so as to ensure that the image features can be accurately reconstructed.

[0085] In the following embodiments, the model architecture and working principle of the target decoding network are described.

[0086] The target decoding network is described in combination with the target encoding network 30 shown in FIG. 3, but is not limited thereto. The target decoding network of the present embodiment can also be used in combination with the target encoding network 40 shown in FIG. 4, and is not limited thereto. As shown in FIG. 5, the model architecture of the target decoding network 50 includes an entropy decoder 51, a decoding module 52, and a background generation model 53. In the case where the encoding bitstream of the encoding output of the target encoding network 30 is received, the encoding bitstream is input to the entropy decoder 51 in the target decoding network 50, and entropy decoding processing is performed on the encoding bitstream (as shown in S501 in FIG. 5) to obtain an image feature. Optionally, the entropy decoder 51 receives the context information output by the context prediction network 36 to perform entropy decoding using the same context information as the entropy encoder 35. Further, the image feature is input to the decoding module 52 in the target decoding network 50, and the image feature is decoded to an image space to obtain an intermediate-state decoded image. The intermediate-state decoded image is input to the background generation model 53 in the target decoding network 50, and the background region of the intermediate-state decoded image is generated (as shown in S502 in FIG. 5) to obtain a target decoded image.

[0087] In the present embodiment, to avoid the background region appearing blurred after compression, the background generation model 53 can be used to perform generation processing on the background region, and the image quality of the encoding output can be improved by generating a background region with higher definition. Optionally, the background generation model 53 can be a generative adversarial network (GAN).

[0088] In the present embodiment of the present disclosure, the target encoding network and the target decoding network are obtained based on end-to-end model training with an optimization loss function as the target. The end-to-end model training process is described below taking the target encoding network and the target decoding network shown in FIG. 5 as an example.

[0089] First, sample images are collected, and the initial encoding network and the initial decoding network are subjected to end-to-end model training based on the sample original images. The initial encoding network is used to perform encoding processing on the sample original images to obtain sample encoding bitstreams. The initial decoding network is used to perform decoding processing on the sample encoding bitstreams to obtain sample decoded images. The initial decoding network includes a background generation model and other modules, and is used to perform background region generation processing on an intermediate-state decoded image output by a previous module of the background generation model in the other modules to obtain a sample decoded image. The other modules can include the entropy decoder 51 and the decoding module 52 in the target decoding network 50 described above, and the background generation model is used to perform background generation processing on an intermediate-state decoded image output by the decoding module 52 to obtain a sample decoded image. Further, a loss function is calculated, and the model training is ended when the loss function satisfies a corresponding termination condition to obtain the target encoding network and the target decoding network.

[0090] In each model training, a sample encoding bitstream and a sample decoding bitstream can be obtained, and then a loss function is calculated.

[0091] In the loss function, the image to be encoded is represented as x, and the encoding module and the decoding module are represented as g a and g s The loss function is represented as follows:

[0092] wherein, is the expected value of x under its probability distribution p(x), the encoder g a The image to be encoded x is extracted as a hidden variable.

[0093] In order to improve the compression efficiency, in an optional embodiment, the model training process is divided into two stages. In the first stage, the loss function is calculated according to the input sample image and the sample decoding result without distinguishing the foreground and the background. The types of the loss function include objective loss function and subjective loss function. The objective loss function is used to measure the difference between the sample decoding image and the sample original image. As an objective index, the objective loss function can be represented by pixel-level mean squared error (MSE) or mean absolute error (MAE) and the like. The subjective loss function refers to the similarity between the sample decoding image and the sample original image at the perception level. As a subjective index, the subjective loss function can be represented by learned perceptual image patch similarity (LPIPS). pips Based on this, the loss function of the first stage is denoted as L stage1 , which can be represented as follows: L stage1 = R + λ (MSE + L pips )

[0094] wherein, R represents the encoding rate, MSE represents the mean squared error, and L pips represents the learned perceptual image patch similarity.

[0095] In the second stage, the non-background area in the input sample original image and the non-background area in the sample decoding image calculate the objective loss function (such as MSE), and the background area in the input sample original image and the background area in the sample decoding image calculate the subjective loss function (such as L pips) and the loss function of the background generation model is calculated for the background area in the input sample original image and the background area in the sample decoded image, and optionally, the loss function of the GAN model is calculated in the case where the background generation model adopts the GAN model, and the GAN loss function is not limited and can be MSE, L1 loss function (MAE), etc. The loss function of the second stage is denoted as L stage2 , which can be expressed as follows:

[0096] wherein m i (i = 0, 1…n), i is a natural number, m i is a mask corresponding to the plurality of foreground areas, λ i is a Lagrange multiplier, and i is the number of foreground areas. In addition, 1-∑m i represents the mask of the background area. In the loss function of the second stage, different loss functions are configured for the foreground area and the background area. The objective indicator is used for the foreground area, the subjective indicator is used for the background area, and the loss function of the background generation model is also used for the background area. Thus, in the training process, the objective indicator and the subjective indicator can be considered at the same time, and the background generation model can also be trained at the same time.

[0097] The detailed implementation and beneficial effects of each step in the method of the embodiment have been described in detail in the foregoing embodiments, and will not be described in detail here.

[0098] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 103 can be device A; for example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and the like.

[0099] In addition, in some of the processes described in the foregoing embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed or in parallel without following the order in which they appear in this text. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the descriptions of “first”, “second”, etc. in this text are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. In addition, “first” and “second” are different types.

[0100] FIG. 6 is a structural schematic diagram of an electronic device according to another exemplary embodiment of the present disclosure. As shown in FIG. 6, the electronic device includes a memory 64 and a processor 65.

[0101] The memory 64 is configured to store computer programs and can be configured to store other various data to support operations on the electronic device. Examples of the data include instructions of any application program or method for operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and the like.

[0102] The memory 64 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0103] The processor 65 is coupled to the memory 64 and is configured to execute computer programs in the memory 64 to: perform vectorization processing on a to-be-encoded image to obtain an image vector; perform hierarchical feature extraction on the to-be-encoded image to obtain a plurality of levels of image features, the image features including a plurality of visual feature regions, the plurality of visual feature regions having respective visual quality weights; perform nonlinear quantization processing on the image vector according to the visual quality weights of the plurality of visual feature regions and the plurality of levels of image features to obtain a plurality of quantized features corresponding to the plurality of visual feature regions; and perform entropy coding processing on the plurality of quantized features according to coding rates adapted to the visual quality weights of the plurality of visual feature regions to obtain a coded bitstream, wherein the visual quality weights are positively correlated with the coding rates.

[0104] In an optional embodiment, when performing vectorization processing on a to-be-encoded image to obtain an image vector, the processor 65 is specifically configured to: input the to-be-encoded image into an encoding module in the target encoding network; and encode the to-be-encoded image into a latent vector space to obtain the image vector.

[0105] In an optional embodiment, the target coding network comprises a feature extraction network, and when performing hierarchical feature extraction on the image to be coded to obtain image features at multiple levels, the processor 65 is specifically configured to: input the image to be coded into multiple feature extraction layers in the feature extraction network for feature extraction respectively to obtain image features at multiple levels; wherein the multiple feature extraction layers contain different numbers of convolutional layers, and the number of convolutional layers is positively correlated with the level of the image features; identify the multiple visual feature regions from the image features at the multiple levels respectively according to visual semantic information of the image features at the multiple levels; and assign visual quality weights to the multiple visual feature regions according to visual quality contribution degrees of the multiple visual feature regions, and the visual quality contribution degrees are positively correlated with the visual quality weights.

[0106] In an optional embodiment, the target coding network comprises a first region division network or a second region division network, and when identifying the multiple visual feature regions from the image features at the multiple levels respectively according to visual semantic information of the image features at the multiple levels, the processor 65 is specifically configured to: input the image features at the multiple levels into the first region division network respectively, and identify the multiple visual feature regions from the image features according to the visual semantic information of the image features; or input the image to be coded into the second region division network, identify multiple image regions from the image to be coded according to feature information of the image to be coded, and identify the multiple visual feature regions from the image features at the multiple levels according to mask images corresponding to the multiple image regions.

[0107] In an optional embodiment, when inputting the image features at the multiple levels into the first region division network respectively, and dividing the image into the multiple visual feature regions according to the visual semantic information of the image features, the processor 65 is specifically configured to: for image features at any level, input the image features into at least one first foreground recognition model in the first region division network, and identify at least one foreground feature region in the image features according to the visual semantic information of the image features; and take the at least one foreground feature region and background feature regions other than the at least one foreground feature region as the multiple visual feature regions.

[0108] In an optional embodiment, the at least one first foreground recognition model comprises at least one of a first face recognition model, a first text recognition model, and a first object recognition model; and the processor 65, when inputting the image feature at any level into the at least one first foreground recognition model in the first region division network to recognize at least one foreground feature region in the image feature according to the visual semantic information of the image feature, is specifically configured to perform at least one of the following operations: inputting the image feature at any level into the first face recognition model to recognize a face feature in the image feature according to the visual semantic information of the image feature, determining a face feature region according to the face feature as a foreground feature region; inputting the image feature into the first text recognition model to recognize a text feature and a position of the text feature in the image feature according to the visual semantic information of the image feature, determining a text feature region according to the text feature and the position of the text feature as a foreground feature region; and inputting the image feature into the first object recognition model to recognize an object feature and a position of the object feature in the image feature according to the visual semantic information of the image feature, determining an object feature region according to the object feature and the position of the object feature as a foreground feature region.

[0109] In an optional embodiment, when inputting the image to be encoded into the second region division network to recognize a plurality of image regions from the image to be encoded according to feature information of the image to be encoded, the processor 65 is specifically configured to: input the image to be encoded into at least one second foreground recognition model in the second region division network to recognize at least one foreground region in the image to be encoded according to the feature information of the image to be encoded; and take the at least one foreground region and a background region other than the at least one foreground region as the plurality of image regions.

[0110] In an optional embodiment, the at least one second foreground recognition model comprises at least one of a second face recognition model, a second text recognition model, and a second object recognition model; when the processor 65 inputs the to-be-encoded image into the at least one second foreground recognition model in the second region division network to recognize at least one foreground region in the to-be-encoded image according to the feature information of the to-be-encoded image, the processor 65 is specifically configured to perform at least one of the following operations: inputting the to-be-encoded image into the second face recognition model to recognize a face feature in the to-be-encoded image according to the feature information of the to-be-encoded image, determining a face region as a foreground region according to the face feature; inputting the to-be-encoded image into the second text recognition model to recognize text information and a position thereof in the to-be-encoded image according to the feature information of the to-be-encoded image, determining a text region as a foreground region according to the text information and the position thereof; and inputting the to-be-encoded image into the second object recognition model to recognize an object position in the to-be-encoded image according to the feature information of the to-be-encoded image, determining an object region as a foreground region according to the object position.

[0111] In an optional embodiment, the second face recognition model comprises a preprocessing layer, a feature extraction layer, and a feature detection layer; when the processor 65 inputs the to-be-encoded image into the second face recognition model to recognize a face feature in the to-be-encoded image according to the feature information of the to-be-encoded image, and determines a face region according to the face feature, the processor 65 is specifically configured to: input the to-be-encoded image into the preprocessing layer in the second face recognition model to perform size adjustment and normalization processing on the to-be-encoded image to obtain a normalized image; input the normalized image into the feature extraction layer in the second face recognition model to perform feature extraction to obtain feature information of the normalized image; and input the feature information of the normalized image into the feature detection layer in the second face recognition model to perform face region detection to obtain the face feature and determine the face region according to the face feature.

[0112] In an optional embodiment, the processor 65 is specifically configured to: input the to-be-encoded image into an object detection network in the second object recognition model, detect the object in the to-be-encoded image according to the feature information of the to-be-encoded image, to obtain coordinate information of a detection frame corresponding to a region where the object is located; and input the to-be-encoded image and the coordinate information of the detection frame into an object segmentation network in the second object recognition model, determine pixel-level context information and semantic-level context information in the to-be-encoded image, and segment the to-be-encoded image according to the pixel-level context information and the semantic-level context information in combination with the coordinate information of the detection frame, to obtain the object region.

[0113] In an optional embodiment, the target encoding network comprises a quantization network, and the processor 65 is specifically configured to: input the visual quality weights of the plurality of visual feature regions and the image features of the plurality of levels into the quantization network, and internally: for any level of image feature, determine a plurality of quantization levels corresponding to the plurality of visual feature regions on the any level according to the visual quality weights of the plurality of visual feature regions and the level to which the image feature belongs; wherein the quantization level is positively correlated with the visual quality weight and the level; and perform nonlinear quantization on the image vector according to the image features of the plurality of levels and the plurality of quantization levels corresponding to the plurality of visual feature regions on the plurality of levels, respectively, to obtain a plurality of quantized features corresponding to the plurality of visual feature regions.

[0114] In an optional embodiment, the processor 65 is specifically configured to: for any level of image feature, perform matrix calculation on the image feature and the image vector according to a plurality of quantization levels corresponding to the plurality of visual feature regions on the any level, to obtain a plurality of sub-quantized features corresponding to the plurality of visual feature regions on the any level; and fuse a plurality of sub-quantized features corresponding to the same visual feature region on the plurality of levels, to obtain a plurality of quantized features corresponding to the plurality of visual feature regions.

[0115] In an optional embodiment, the target encoding network comprises an entropy encoder, and the processor 65 is specifically configured to: input the plurality of quantized features and their encoding code rates into the entropy encoder, and perform entropy encoding processing on the plurality of quantized features according to their encoding code rates to obtain the encoded code stream.

[0116] In an optional embodiment, the target encoding network comprises a context prediction network, and the processor 65 is further configured to: input the image vector into the context prediction network, predict the context information of the image to be encoded, and input the context information into the entropy encoder; and the processor 65 is specifically configured to: perform entropy encoding processing on the plurality of quantized features according to their assigned encoding code rates and the context information to obtain the encoded code stream.

[0117] In an optional embodiment, the processor is further configured to: perform end-to-end model training on an initial encoding network and an initial decoding network according to a sample original image, the initial encoding network being configured to perform encoding processing on the sample original image to obtain a sample encoded code stream, and the initial decoding network being configured to perform decoding processing on the sample encoded code stream to obtain a sample decoded image; wherein the initial decoding network comprises a background generation model and other modules, and is configured to perform background region generation processing on an intermediate state decoded image output by a previous module of the background generation model in the other modules to obtain the sample decoded image; calculate a loss function, the loss function comprising at least one of an objective loss function between the sample original image and the sample decoded image, a subjective loss function between the sample original image and the sample decoded image, an objective loss function between non-background regions in the sample original image and the sample decoded image, a subjective loss function between a background region in the sample original image and a background region in the sample decoded image, and a loss function of the background generation model; and end the model training when the loss function meets a corresponding termination condition to obtain the target encoding network and the target decoding network.

[0118] Further, as shown in FIG. 6, the electronic device further includes a communication component 66, a display 67, a power supply component 68, an audio component 69, and other components. The components shown in FIG. 6 are only schematic and do not mean that the electronic device only includes the components shown in FIG. 6. In addition, the components in the dashed box in FIG. 6 are optional components, not mandatory components, and can be determined according to the product form of the working node. The electronic device of the embodiment can be implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, or an IOT device, or a server device such as a general server, a cloud server, or a server array. If the electronic device of the embodiment is implemented as a terminal device such as a desktop computer, a notebook computer, or a smart phone, it can include the components in the dashed box in FIG. 6. If the electronic device of the embodiment is implemented as a server device such as a general server, a cloud server, or a server array, it can not include the components in the dashed box in FIG. 6.

[0119] The embodiment of the present disclosure also provides an electronic device, which has the same or similar implementation structure as the electronic device shown in FIG. 6, and can be implemented with reference to the structure of the electronic device shown in FIG. 6. The electronic device provided by the embodiment is mainly different from the electronic device in the embodiment shown in FIG. 6 in that the processor in the electronic device implements different functions of the computer program stored in the memory. For the electronic device provided by the embodiment, the processor executes the computer program stored in the memory, which can be used to: receive an encoded code stream; input the encoded code stream into an entropy decoder in the target decoding network, perform entropy decoding processing on the encoded code stream to obtain image features; input the image features into a decoding module in the target decoding network, decode the image features to the pixel space to obtain an intermediate state decoded image; and input the intermediate state decoded image into a background generation model in the target decoding network, perform background region generation processing on the intermediate state decoded image to obtain a target decoded image.

[0120] Correspondingly, the embodiment of the present disclosure also provides a computer readable storage medium storing a computer program, which causes the processor to be able to implement the steps in the above method when the computer program is executed by the processor.

[0121] The embodiment of the present disclosure also provides a computer program product, which includes computer programs / instructions, which causes the processor to be able to implement the steps in the above method embodiment when the computer programs / instructions are executed by the processor.

[0122] The above-described memory can be implemented by any type of volatile or nonvolatile memory devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0123] The above-described communication component is configured to facilitate communication between the device in which the communication component is located and other devices in a wired or wireless manner. The device in which the communication component is located can access a wireless network based on a communication standard, such as a WiFi, 2G, 3G, 4G / LTE, 5G, or the like mobile communication network, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast managing system via a broadcast channel. In an example embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, BlueTooth (BT) technology, and other technologies.

[0124] The above-described display includes a screen, which can include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touch or a slide action, but also detect a duration and a pressure related to a touch or a slide operation.

[0125] The power component provides power to various components of the device in which the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which the power component is located.

[0126] The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the device in which the audio component is located is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in a memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0127] Those skilled in the art will understand that embodiments of the present disclosure can be provided as methods, systems, or computer program products. Accordingly, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure can take the form of a computer program product on one or more computer-readable storage media (including, but not limited to, disk memory, Compact Disc Read-Only Memory (CD-ROM), optical memory, and the like) embodying computer usable program code.

[0128] The present disclosure is described herein with reference to the drawings, in which various embodiments of the present disclosure are illustrated. The present disclosure is described herein with reference to flow diagrams and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks.

[0129] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flow diagrams and / or block diagrams block or blocks.

[0130] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0131] In one typical arrangement, the computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0132] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, and / or non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory. The memory is an example of computer readable media.

[0133] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital video disc (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0135] The above embodiments of the present disclosure are only used to illustrate the technical solutions of the present disclosure, and not intended to limit the present disclosure. Although the present disclosure has been described in detail with reference to the embodiments, the technical solutions in the embodiments are only used to explain the present disclosure, and not intended to limit the present disclosure.

Claims

1. An image coding method, wherein, The method applied to a target coding network comprises: vectorizing a to-be-coded image to obtain an image vector; performing hierarchical feature extraction on the to-be-coded image to obtain a plurality of levels of image features, the image features comprising a plurality of visual feature regions, the plurality of visual feature regions having respective visual quality weights; performing nonlinear quantization processing on the image vector according to the visual quality weights of the plurality of visual feature regions and the plurality of levels of image features to obtain a plurality of quantized features corresponding to the plurality of visual feature regions; performing entropy coding processing on the plurality of quantized features according to coding rates adapted to the visual quality weights of the plurality of visual feature regions to obtain a coded bitstream, wherein the visual quality weights are positively correlated with the coding rates.

2. The method of claim 1, wherein, The vectorizing a to-be-coded image to obtain an image vector comprises: inputting the to-be-coded image into a coding module in the target coding network to code the to-be-coded image into a latent vector space to obtain the image vector.

3. The method of claim 1, wherein, The target coding network comprises a feature extraction network, and the performing hierarchical feature extraction on the to-be-coded image to obtain a plurality of levels of image features comprises: inputting the to-be-coded image into a plurality of feature extraction layers in the feature extraction network to perform feature extraction respectively to obtain a plurality of levels of image features; wherein the plurality of feature extraction layers contain different numbers of convolutional layers, and the numbers of the convolutional layers are positively correlated with the levels of the image features; identifying the plurality of visual feature regions respectively from the plurality of levels of image features according to visual semantic information of the plurality of levels of image features; allocating visual quality weights to the plurality of visual feature regions according to visual quality contributions of the plurality of visual feature regions, and the visual quality contributions are positively correlated with the visual quality weights.

4. The method of claim 3, wherein, The target coding network comprises a first region division network or a second region division network, and the identifying the plurality of visual feature regions respectively from the plurality of levels of image features according to visual semantic information of the plurality of levels of image features comprises: inputting the plurality of levels of image features into the first region division network respectively to identify the plurality of visual feature regions from the image features according to the visual semantic information of the image features; or inputting the to-be-coded image into the second region division network to identify a plurality of image regions from the to-be-coded image according to feature information of the to-be-coded image, and identifying the plurality of visual feature regions from the plurality of levels of image features according to mask images corresponding to the plurality of image regions.

5. The method of claim 4, wherein, The inputting the plurality of levels of image features into the first region division network respectively to divide the image into the plurality of visual feature regions according to the visual semantic information of the image features comprises: for any level of image features, inputting the image features into at least one first foreground recognition model in the first region division network to identify at least one foreground feature region in the image features according to the visual semantic information of the image features; The at least one foreground feature region and a background feature region other than the at least one foreground feature region are taken as the plurality of visual feature regions.

6. The method of claim 5, wherein, The at least one first foreground recognition model comprises at least one of a first face recognition model, a first character recognition model and a first object recognition model. For image features of any level, the image features are input into at least one first foreground recognition model in the first region division network, and at least one foreground feature region in the image features is recognized according to visual semantic information of the image features, including at least one of the following operations: For image features of any level, the image features are input into a first face recognition model, and face features in the image features are recognized according to visual semantic information of the image features, and a face feature region is determined according to the face features as a foreground feature region. The image features are input into a first character recognition model, and text features and their positions in the image features are recognized according to visual semantic information of the image features, and a text feature region is determined according to the text features and their positions as a foreground feature region. The image features are input into a first object recognition model, and object features and their positions in the image features are recognized according to visual semantic information of the image features, and an object feature region is determined according to the object features and their positions as a foreground feature region.

7. The method of claim 4, wherein, The to-be-encoded image is input into the second region division network, and a plurality of image regions are recognized from the to-be-encoded image according to feature information of the to-be-encoded image, including: The to-be-encoded image is input into at least one second foreground recognition model in the second region division network, and at least one foreground region in the to-be-encoded image is recognized according to feature information of the to-be-encoded image. The at least one foreground region and a background region other than the at least one foreground region are taken as the plurality of image regions.

8. The method of claim 7, wherein, The at least one second foreground recognition model comprises at least one of a second face recognition model, a second character recognition model and a second object recognition model. The to-be-encoded image is input into at least one second foreground recognition model in the second region division network, and at least one foreground region in the to-be-encoded image is recognized according to feature information of the to-be-encoded image, including at least one of the following operations: The to-be-encoded image is input into a second face recognition model, and face features in the to-be-encoded image are recognized according to feature information of the to-be-encoded image, and a face region is determined according to the face features as a foreground region. The to-be-encoded image is input into a second character recognition model, and text information and their positions in the to-be-encoded image are recognized according to feature information of the to-be-encoded image, and a text region is determined according to the text information and their positions as a foreground region. The to-be-encoded image is input into a second object recognition model, and object positions in the to-be-encoded image are recognized according to feature information of the to-be-encoded image, and an object region is determined according to the object positions as a foreground region.

9. The method of claim 8, wherein, The second face recognition model comprises a preprocessing layer, a feature extraction layer and a feature detection layer; the second face recognition model is input with the image to be encoded, and the feature information of the image to be encoded is used to identify the face feature in the image to be encoded, and the face region is determined according to the face feature. The preprocessing layer in the second face recognition model is input with the image to be encoded, and the image to be encoded is subjected to size adjustment and normalization processing to obtain a normalized image. The feature extraction layer in the second face recognition model is input with the normalized image to extract the feature information of the normalized image. The feature detection layer in the second face recognition model is input with the feature information of the normalized image to detect the face region, so as to obtain the face feature and determine the face region according to the face feature.

10. The method of claim 8, wherein, The second object recognition model is input with the image to be encoded, and the feature information of the image to be encoded is used to identify the object position in the image to be encoded, and the object region is determined according to the object position. The object detection network in the second object recognition model is input with the image to be encoded, and the feature information of the image to be encoded is used to detect the object in the image to be encoded to obtain the coordinate information of the detection frame, and the detection frame corresponds to the region where the object is located. The object segmentation network in the second object recognition model is input with the image to be encoded and the coordinate information of the detection frame to determine the pixel-level context information and the semantic-level context information of the image to be encoded, and the image to be encoded is segmented according to the pixel-level context information and the semantic-level context information in combination with the coordinate information of the detection frame to obtain the object region.

11. The method of claim 1, wherein, The target encoding network comprises a quantization network, and the image vector is subjected to nonlinear quantization processing according to the visual quality weight of the plurality of visual feature regions and the plurality of hierarchical image features to obtain a plurality of quantization features corresponding to the plurality of visual feature regions, comprising: The visual quality weight of the plurality of visual feature regions and the plurality of hierarchical image features are input into the quantization network, and in the quantization network: For any hierarchical image feature, the plurality of quantization levels corresponding to the plurality of visual feature regions on the any hierarchical level are determined according to the visual quality weight of the plurality of visual feature regions and the hierarchical level to which the image feature belongs; wherein the quantization level is negatively correlated with the visual quality weight and the level of the hierarchical level; The image vector is subjected to nonlinear quantization according to the plurality of hierarchical image features and the plurality of quantization levels corresponding to the plurality of visual feature regions on the plurality of hierarchical levels, respectively, to obtain a plurality of quantization features corresponding to the plurality of visual feature regions.

12. The method of claim 11, wherein, The image vector is subjected to nonlinear quantization according to the plurality of hierarchical image features and the plurality of quantization levels corresponding to the plurality of visual feature regions on the plurality of hierarchical levels, respectively, to obtain a plurality of quantization features corresponding to the plurality of visual feature regions, comprising: For any level of image features, according to the multiple quantization levels corresponding to the multiple visual feature regions at the any level, the image features and the image vector are subjected to matrix calculation to obtain multiple sub-quantization features corresponding to the multiple visual feature regions at the any level; The multiple sub-quantization features corresponding to the same visual feature region at the multiple levels are fused to obtain multiple quantization features corresponding to the multiple visual feature regions.

13. The method of claim 1, wherein, The target encoding network comprises an entropy encoder, and the multiple quantization features are subjected to entropy encoding processing according to the encoding code rate adapted to the visual quality weight of the multiple visual feature regions to obtain an encoding code stream, including: According to the visual quality weight of the multiple visual feature regions, the encoding code rate is allocated to the multiple quantization features; The multiple quantization features and the encoding code rate are input into the entropy encoder, and the multiple quantization features are subjected to entropy encoding processing according to the encoding code rate of the multiple quantization features to obtain an encoding code stream.

14. The method of claim 13, wherein, The target encoding network comprises a context prediction network, and the method further comprises: inputting the image vector into the context prediction network to predict the context information of the image to be encoded, and inputting the context information into the entropy encoder; According to the encoding code rate allocated to the multiple quantization features, the multiple quantization features are subjected to entropy encoding processing to obtain an encoding code stream, including: According to the encoding code rate allocated to the multiple quantization features and the context information, the multiple quantization features are subjected to entropy encoding processing to obtain an encoding code stream.

15. The method of any one of claims 1-14, wherein, Further comprising: According to the sample original image, an end-to-end model training is performed on an initial encoding network and an initial decoding network, the initial encoding network is used for encoding processing on the sample original image to obtain a sample encoding code stream; The initial decoding network decodes the sample encoding code stream to obtain a sample decoding image; Wherein, the initial decoding network comprises a background generation model and other modules, and is used for generating a background region of an intermediate state decoding image output by a previous module of the background generation model in the other modules to obtain the sample decoding image; A loss function is calculated, the loss function comprises at least one of an objective loss function between the sample original image and the sample decoding image, a subjective loss function between the sample original image and the sample decoding image, and an objective loss function of a non-background region between the sample original image and the sample decoding image, a subjective loss function between the background region in the sample original image and the background region in the sample decoding image, and a loss function of the background generation model; In the case that the loss function satisfies a corresponding termination condition, the model training is ended to obtain the target encoding network and the target decoding network.

16. An image decoding method, wherein Applied to the target decoding network, the method comprises: Receiving an encoding code stream; Inputting the encoding code stream into an entropy decoder in the target decoding network to perform entropy decoding processing on the encoding code stream to obtain image features; inputting the image feature into a decoding module in the target decoding network, and decoding the image feature to a pixel space to obtain an intermediate state decoded image; inputting the intermediate state decoded image into a background generation model in the target decoding network, and performing a background region generation process on the intermediate state decoded image to obtain a target decoded image.

17. An electronic device, comprising: comprising a memory and a processor, the memory configured to store a computer program, and the processor coupled to the memory and configured to execute the computer program to implement the steps in the method of any one of claims 1-16.

18. A computer readable storage medium storing computer programs / instructions, wherein, The computer program, when executed by the processor, causes the processor to be capable of implementing the steps in the method of any one of claims 1-16.

19. A computer program product, wherein, comprising: The computer program / instructions, when executed by the processor, cause the processor to be capable of implementing the steps in the method of any one of claims 1-16.

Images