Image encoding and decoding method based on content generation, electronic device and storage medium

Abstract

The present application discloses an image encoding and decoding method, electronic device and storage medium based on content generation, the method comprising: determining multiple marking areas of the original image based on the content of the original image, masking the marking areas in a manner matching the encoding and decoding order, and obtaining multiple mask images; wherein the encoding and decoding order is related to the number of pixels in the marking area, and each mask image corresponds to its own marking area; obtaining the mask image to be encoded, determining the entropy probability of the mask image to be encoded based on the entropy probability of the mask image to be encoded and the encoded mask image, and encoding it to the code stream to be transmitted; obtaining the marking area to be decoded, obtaining the reference feature corresponding to the mask image, decoding and reconstructing the code stream to be transmitted based on the reference feature of the marking area to be decoded and the reconstructed marking area, obtaining the reconstructed marking area in turn, and generating a reconstructed image corresponding to the original image. The above scheme is sufficient to improve the accuracy of image encoding and decoding.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image encoding and decoding method, electronic device and storage medium based on content generation. Background Art

[0002] Since the original image occupies a large amount of data, it is usually necessary to encode and then decode the image during the image transmission process to complete the image transmission. However, the existing encoding and decoding process usually encodes and decodes the entire image based on a universal encoding standard. Therefore, the existing technology relies on a single and fixed method in the encoding and decoding process, making it difficult to adaptively encode or finely decode, resulting in low encoding and decoding accuracy. In view of this, how to improve the accuracy of image encoding and decoding has become an urgent problem to be solved. Summary of the invention

[0003] The main technical problem solved by the present application is to provide an image encoding and decoding method, an electronic device and a storage medium based on content generation, which can improve the accuracy of image encoding and decoding.

[0004] To solve the above technical problems, the first aspect of the present application provides an image encoding and decoding method based on content generation, comprising: in response to obtaining an original image to be encoded, based on the content of the original image, determining multiple marked areas of the original image, masking the marked areas in a manner matching the encoding and decoding order, and obtaining multiple mask images; wherein the encoding and decoding order is related to the number of pixels in the marked area, and each mask image corresponds to its own marked area; obtaining the mask image to be encoded according to the encoding and decoding order, determining the entropy probability of the mask image to be encoded based on the entropy probability of the mask image to be encoded and the encoded mask image, and sequentially encoding all the mask images into a code stream to be transmitted; obtaining the marked area to be decoded according to the encoding and decoding order, obtaining a reference feature corresponding to the mask image matching the encoding and decoding order, decoding and reconstructing the code stream to be transmitted based on the reference feature of the marked area to be decoded and the decoded and reconstructed marked area, sequentially obtaining all the reconstructed marked areas matching the encoding and decoding order, and generating a reconstructed image corresponding to the original image.

[0005] To solve the above technical problems, the second aspect of the present application provides an electronic device, which includes: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method described in the first aspect.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium on which program data is stored. When the program data is executed by a processor, the method described in the first aspect is implemented.

[0007] In the above scheme, in the encoding stage, after the original image to be encoded is obtained, the original image is divided into multiple marked areas based on the content of the original image, wherein each marked area corresponds to multiple pixels, and the encoding and decoding order is related to the number of pixels in the mark, and the marked area is masked in a manner matching the encoding and decoding order to obtain multiple mask images corresponding to the original image, each mask image corresponds to its own marked area, and the current mask image to be encoded is obtained in the encoding and decoding order, and the entropy probability of the mask image to be encoded is predicted based on the entropy probabilities of the mask image to be encoded and the encoded mask image to obtain the entropy probability of the mask image to be encoded, so that all the mask images are encoded into the code stream to be transmitted in sequence according to the encoding and decoding order, so that the marked areas corresponding to the mask images are encoded in sequence in the encoding stage, and the subsequent mask image refers to the entropy probability of the previous mask image during encoding, so as to improve the encoding accuracy. In the decoding stage, the current marked area to be decoded is obtained in the coding and decoding order, and the reference features corresponding to the mask image required at this time in the coding and decoding order are obtained. The transmission code stream is decoded and reconstructed based on the reference features of the marked area to be decoded and the decoded and reconstructed marked area in the coding and decoding order, and all reconstructed marked areas matching the coding and decoding order are obtained in turn, which together constitute the reconstructed image corresponding to the original image. Therefore, in the decoding stage, the marked areas are decoded in sequence according to the coding and decoding order and combined with the reference features of the mask image and the decoded and reconstructed marked areas, so that the decoding accuracy is higher and the accuracy of image coding and decoding is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flowchart of an implementation method of an image encoding and decoding method based on content generation of the present application;

[0010] Figure 2 It is a flowchart of another implementation method of the image encoding and decoding method based on content generation of the present application;

[0011] Figure 3 This is a schematic diagram of an application scenario of an implementation method corresponding to the pixel grouping method of the present application;

[0012] Figure 4 This is a schematic diagram of an application scenario of an implementation method of an image encoding and decoding method based on content generation of the present application;

[0013] Figure 5 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0014] Figure 6 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different implementation methods can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0017] The content generation-based image encoding and decoding method provided in the present application is used to encode and decode images, and its corresponding execution subject is a processing unit that can call and process images.

[0018] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an image encoding and decoding method based on content generation of the present application, the method comprising:

[0019] S101: In response to obtaining an original image to be encoded, based on the content of the original image, multiple marked areas of the original image are determined, and the marked areas are masked in a manner matching the encoding and decoding order to obtain multiple mask images; wherein the encoding and decoding order is related to the number of pixels in the marked area, and each mask image corresponds to its own marked area.

[0020] Specifically, in the encoding stage, after the original image to be encoded is obtained, the original image is divided into multiple marked areas based on the content of the original image, where each marked area corresponds to multiple pixels, and the encoding and decoding order is related to the number of pixels in the mark. The marked area is masked in a manner matching the encoding and decoding order to obtain multiple mask images corresponding to the original image, and each mask image corresponds to its own marked area.

[0021] It can be understood that, in the encoding stage, the original image is adaptively divided into a plurality of marked areas based on the content of the original image, and the encoding and decoding order is determined for subsequent sequential decoding.

[0022] It should be noted that, after the original image to be encoded is obtained, the original image is first divided into a plurality of marked areas based on the content of the original image, and each marked area includes a plurality of pixels.

[0023] In some implementation scenarios, scene segmentation is performed on the original image based on the content of the original image, and the original image is divided into multiple marked areas.

[0024] In some implementation scenarios, a preset pixel grouping method is obtained, and the original image is divided into multiple candidate areas using the pixel grouping method. Based on the content of the original image, the candidate areas are adjusted to obtain multiple marked areas corresponding to the original image.

[0025] Furthermore, the coding and decoding order of the marked area is determined based on the number of pixels in the marked area, and at least part of the marked area is masked in a manner matching the coding and decoding order to obtain a mask image corresponding to each marked area in the coding and decoding order.

[0026] It is understandable that the encoding and decoding order can be to start from the marking area with the least number of pixels, so as to decode all marking areas in the order of increasing number of pixels, or to proceed in the order of decreasing number of pixels, or to select the marking area corresponding to a fixed position and then proceed in the order of decreasing and then increasing number of pixels.

[0027] Furthermore, after the coding and decoding sequence is determined, the original image can be masked based on the coding and decoding sequence to obtain a plurality of masked images, wherein the masking method matches the coding and decoding sequence.

[0028] In some implementation scenarios, the marked area following the current marked area in the encoding and decoding order is masked, thereby obtaining multiple mask images.

[0029] In some implementation scenarios, all other marked areas except the current marked area in the encoding and decoding order are masked, thereby obtaining multiple mask images.

[0030] S102: Obtain a mask image to be encoded in a coding and decoding order, determine the entropy probability of the mask image to be encoded based on the entropy probabilities of the mask image to be encoded and the encoded mask image, and sequentially encode all mask images into a code stream to be transmitted.

[0031] Specifically, the current mask image to be encoded is obtained in the encoding and decoding order, and the entropy probability of the mask image to be encoded is predicted based on the entropy probabilities of the mask image to be encoded and the encoded mask image, so as to obtain the entropy probability of the mask image to be encoded, thereby encoding all the mask images into the code stream to be transmitted in sequence according to the encoding and decoding order, so that the marked areas corresponding to the mask images are encoded in sequence in the encoding stage, and the subsequent mask images refer to the entropy probability of the previous mask images during encoding, so as to improve the encoding accuracy.

[0032] It can be understood that the image encoding process is based on entropy coding, and the image decoding process is based on entropy decoding. Entropy coding is a lossless process. Through the constructed probability model, the probability of the sign bit in each feature is calculated and encoded into a binary representation and written into the bit stream. Entropy decoding is the reverse process.

[0033] Optionally, the encoding stage includes at least one of transformation and quantization operations. The transformation mainly uses a convolutional neural network for nonlinear downsampling. The function of the transformation is to use a more compact expression to describe the main features of the original image and reduce the dimension and data volume of the image. Quantization is one of the links in lossy encoding, which integerizes the data to improve the compression rate, thereby obtaining the reduced-dimensional features.

[0034] In some implementation scenarios, a mask image to be encoded is obtained in a coding and decoding order, features of the mask image to be encoded are obtained, the entropy probability of the mask image to be encoded is predicted based on the features of the mask image to be encoded and the entropy probability of a previous encoded mask image adjacent to the mask image to be encoded, the entropy probability of the mask image to be encoded is obtained, all mask images are traversed, and all mask images are encoded into a code stream to be transmitted.

[0035] In some implementation scenarios, a mask image to be encoded is obtained in a coding and decoding order, features of the mask image to be encoded are obtained, and based on the features of the mask image to be encoded and the entropy probabilities of all mask images that have been encoded before the mask image to be encoded, the entropy probability of the mask image to be encoded is predicted to obtain the entropy probability of the mask image to be encoded, all mask images are traversed, and all mask images are encoded into a code stream to be transmitted.

[0036] Optionally, the entropy probability of the mask image is predicted based on a pre-trained probability model, and the probability model corresponds to a pre-trained context model, and the context model guides the probability model to predict the entropy probability of the mask image to be encoded based on the entropy probability of the encoded mask image. Thus, the context model is used to sequentially encode the marked areas corresponding to the mask image in the encoding stage, and the subsequent mask image refers to the entropy probability of the previous mask image during encoding, so as to achieve sequential encoding of the marked areas corresponding to the mask image, thereby improving the encoding accuracy.

[0037] S103: Obtain the marked area to be decoded according to the coding and decoding order, obtain the reference features corresponding to the mask image that matches the coding and decoding order, decode and reconstruct the transmission code stream based on the reference features of the marked area to be decoded and the decoded and reconstructed marked area, obtain all the reconstructed marked areas that match the coding and decoding order in turn, and generate a reconstructed image corresponding to the original image.

[0038] Specifically, in the decoding stage, the current marked area to be decoded is obtained in the coding and decoding order, and the reference features corresponding to the mask image required at this time in the coding and decoding order are obtained. The transmission code stream is decoded and reconstructed based on the reference features of the marked area to be decoded and the decoded and reconstructed marked area in the coding and decoding order, and all the reconstructed marked areas that match the coding and decoding order are obtained in turn, which together constitute the reconstructed image corresponding to the original image.

[0039] It is understandable that, in the decoding stage, the marked areas are decoded in sequence according to the encoding and decoding order and in combination with the reference features of the mask image and the decoded and reconstructed marked areas, so that the decoding accuracy is higher and the image encoding and decoding precision is improved.

[0040] In some implementation scenarios, a marked area to be decoded is obtained in a coding order, a reference feature of a mask image corresponding to the marked area to be decoded is obtained, and based on the reference feature and the decoded and reconstructed marked area, features associated with the marked area are extracted from a code stream to be transmitted for decoding and reconstruction to obtain a reconstructed marked area, all marked areas are traversed to obtain all reconstructed marked areas that match the coding order, and a reconstructed image corresponding to the original image is generated.

[0041] In some implementation scenarios, the marked area to be decoded is obtained in the encoding and decoding order, the reference features of the mask image corresponding to the marked area to be decoded are obtained, the masked area in the mask image is determined, and based on the reference features and the decoded and reconstructed marked area, the features corresponding to the unmasked marked area are extracted from the code stream to be transmitted for decoding and reconstruction to obtain the reconstructed marked area, all marked areas are traversed to obtain all the reconstructed marked areas that match the encoding and decoding order, and a reconstructed image corresponding to the original image is generated.

[0042] Optionally, the marked area is obtained by decoding and reconstructing the transmission code stream based on a pre-trained probability model. The probability model corresponds to a pre-trained context model, and the context model guides the probability model to decode and reconstruct the transmission code stream based on reference features and the decoded and reconstructed marked area, so that the marked areas are decoded in sequence in the decoding stage by using the context model, and the marked areas at the back refer to the decoded and reconstructed marked areas at the front during decoding, so as to realize the sequential decoding of the marked areas corresponding to the mask image, thereby improving the decoding accuracy.

[0043] In the above scheme, in the encoding stage, after the original image to be encoded is obtained, the original image is divided into multiple marked areas based on the content of the original image, wherein each marked area corresponds to multiple pixels, and the encoding and decoding order is related to the number of pixels in the mark, and the marked area is masked in a manner matching the encoding and decoding order to obtain multiple mask images corresponding to the original image, each mask image corresponds to its own marked area, and the current mask image to be encoded is obtained in the encoding and decoding order, and the entropy probability of the mask image to be encoded is predicted based on the entropy probabilities of the mask image to be encoded and the encoded mask image to obtain the entropy probability of the mask image to be encoded, so that all the mask images are encoded into the code stream to be transmitted in sequence according to the encoding and decoding order, so that the marked areas corresponding to the mask images are encoded in sequence in the encoding stage, and the subsequent mask image refers to the entropy probability of the previous mask image during encoding, so as to improve the encoding accuracy. In the decoding stage, the current marked area to be decoded is obtained in the coding and decoding order, and the reference features corresponding to the mask image required at this time in the coding and decoding order are obtained. The transmission code stream is decoded and reconstructed based on the reference features of the marked area to be decoded and the decoded and reconstructed marked area in the coding and decoding order, and all reconstructed marked areas matching the coding and decoding order are obtained in turn, which together constitute the reconstructed image corresponding to the original image. Therefore, in the decoding stage, the marked areas are decoded in sequence according to the coding and decoding order and combined with the reference features of the mask image and the decoded and reconstructed marked areas, so that the decoding accuracy is higher and the accuracy of image coding and decoding is improved.

[0044] See also Figure 2 , Figure 2 1 is a flow chart of another embodiment of the image encoding and decoding method based on content generation of the present application, the method comprising:

[0045] S201: In response to obtaining an original image to be encoded, based on the content of the original image, multiple marked areas of the original image are determined, and the marked areas are masked in a manner matching the encoding and decoding order to obtain multiple mask images; wherein the encoding and decoding order is related to the number of pixels in the marked area, and each mask image corresponds to its own marked area.

[0046] Specifically, when an original image to be encoded is obtained, the original image is divided into a plurality of marked areas based on the content of the original image, and the encoding and decoding order is determined based on the number of pixels in the marked areas.

[0047] Furthermore, the marked area is masked in a manner matching the encoding and decoding order to obtain a plurality of mask images corresponding to the original image.

[0048] In one application, multiple marked areas of the original image are determined based on the content of the original image, including: based on the content of the original image, pixels in the original image are grouped to obtain multiple marked areas composed of interrelated pixels; wherein the number of pixels in at least some of the marked areas is different from each other, and each marked area is set with a marking number matching the number of pixels.

[0049] Specifically, based on the content of the original image, the pixels in the original image are grouped, and the mutually related pixels are collectively formed into a marking area to obtain multiple marking areas. That is, there is a correlation between the pixels in the same marking area, so that the division of the marking area has higher accuracy, wherein the number of pixels in at least some of the marking areas is different from each other.

[0050] Furthermore, a corresponding marking number is set for each marking area based on the number of pixels in each marking area, so as to facilitate distinguishing the marking areas and associate the marking number with the encoding and decoding order.

[0051] It can be understood that pixels in the same marking area are adjacent to at least one other pixel in the marking area, thereby obtaining a marking area with continuous pixel arrangement, so that the features of the marking area have integrity.

[0052] In some implementation scenarios, based on the content of the original image, the pixels in the original image are grouped to obtain multiple marked areas composed of interrelated pixels, including: using a preset pixel grouping method to divide the original image into multiple candidate areas with increasing numbers of pixels; wherein the pixel grouping method corresponds to a grouping extension direction; based on adjacent pixels between every two adjacent candidate areas and at least part of the pixels in the grouping extension direction, the candidate areas are adjusted until all candidate areas are traversed to obtain multiple marked areas composed of interrelated pixels.

[0053] Specifically, in the group extension direction corresponding to the preset pixel grouping method, the original image is divided into multiple candidate areas with increasing numbers of pixels using the pixel grouping method, and starting from the first candidate area, for each two adjacent candidate areas, based on the adjacent pixels between the adjacent candidate areas and at least part of the pixels in the subsequent candidate area in the group extension direction, the candidate area is adjusted so that the correlation between the pixels in the adjusted candidate area is stronger, until all candidate areas are traversed, and all adjusted candidate areas are used as marked areas. Therefore, the pixel grouping method can more efficiently divide the original image into multiple candidate areas, and fine-tune the candidate areas based on the image content to obtain marked areas.

[0054] Optionally, see Figure 3 , Figure 3 This is a schematic diagram of an application scenario of an implementation method corresponding to the pixel grouping method of the present application, wherein the preset pixel grouping method can be a method of surrounding the picture, and the grouping extension direction can be traversed from the periphery to the inside, or in the reverse direction. The preset pixel grouping method can also be a raster scanning method, and the grouping extension direction can be from top to bottom, or from bottom to top, and the present application does not make specific restrictions on this.

[0055] It should be noted that the candidate area is adjusted based on the adjacent pixels between every two adjacent candidate areas and at least part of the pixels in the group extension direction, including: obtaining the similarity between the adjacent pixels and the pixels within a quantity threshold in the group extension direction, and determining the expansion number of the pixels in the group extension direction based on the similarity; in the group extension direction, extending the pixels in the front candidate area to the rear candidate area by the expansion number, and obtaining the marked area corresponding to the front candidate area.

[0056] Specifically, the similarity between adjacent pixels and pixels within a quantity threshold in the group extension direction is obtained, and the number of pixels to be expanded when expanding the pixels in the group extension direction is determined based on the similarity, wherein the quantity threshold can be adaptively matched with the image resolution, is positively correlated, and can also be custom set.

[0057] Furthermore, in the group extension direction, the pixels in the front candidate area are extended to the rear candidate area by an amount, thereby expanding the number of pixels in the front candidate area and obtaining a marked area corresponding to the front candidate area.

[0058] It can be understood that the adjacent pixel is located in the previous candidate area, and in the group extension direction, the adjacent pixel is adjacent to the pixel in the subsequent candidate area, and the pixel within the number threshold is the pixel located after the adjacent pixel in the group extension direction. The candidate area is fine-tuned based on the similarity between the adjacent pixel and the pixel within the number threshold, wherein the fine-tuning process can be expressed as the following formula:

[0059] (1)

[0060] in, M i It refers to the number of pixels in the candidate area corresponding to the current group. k Refers to the number of pixels that are expanded in the current group. P l refers to the adjacent pixels between the current group and the next group, P l+k It refers to the next group k pixels, MSE is the mean square error calculation, Thr is the mean square error threshold, and the similarity between adjacent pixels and other pixels is determined by the difference degree fed back by the mean square error threshold. In addition, k ∈Q, where Q is the quantity threshold.

[0061] In some implementation scenarios, based on the content of the original image, the pixels in the original image are grouped to obtain multiple labeled areas composed of interrelated pixels, including: performing scene segmentation on the original image to obtain multiple segmented areas corresponding to the original image; wherein the pixels contained in the segmented areas are related to the targets in the original image; based on the number of pixels in the segmented areas, matching label numbers are set for the segmented areas to obtain multiple labeled areas composed of interrelated pixels.

[0062] Specifically, the original image is subjected to scene segmentation, and the original image is divided into a plurality of segmented regions associated with the scene, wherein the scene is at least related to the target in the original image, so that the original image is divided into segmented regions corresponding to the target and the background where the target is located, so that the pixels in the segmented regions have a strong correlation. Among them, the scene segmentation method includes but is not limited to a segmentation method based on a fully convolutional neural network and a segmentation method based on a full-resolution residual network. Therefore, scene segmentation is used to more finely divide the original image into a plurality of marked regions, so that the pixels in the marked regions have a strong correlation.

[0063] Furthermore, based on the number of pixels in the segmented area, a matching marker number is set for the segmented area, so that the encoding and decoding order is determined by using the marker number, and the segmented area with the marker number is used as the marker area.

[0064] It should be noted that the marking numbers of the multiple marking areas are set in ascending order according to a positive correlation with the number of pixels in the marking area, and the encoding and decoding order corresponds to the increasing marking numbers. Figure 3 As shown in , the same marking area corresponds to the same marking number, and the marking number is set based on the number of pixels in the marking area.

[0065] Furthermore, the marked area is masked in a manner matching the encoding and decoding order to obtain multiple mask images, including: taking each marking number as the current marking number in turn based on the encoding and decoding order, and masking the marked areas corresponding to all marking numbers after the current marking number to obtain multiple mask images.

[0066] Specifically, each marker number is taken as the current marker number in turn based on the encoding and decoding order. When determining the mask image, the marker areas corresponding to all markers after the current marker number are masked, thereby reducing the interference of other marker areas after the marker area corresponding to the current marker number during the decoding stage, and the decoded part is not masked, so as to provide more reference features for the corresponding marker areas during the decoding stage.

[0067] It can be understood that based on the mask image acquisition method adopted in any implementation scenario, a mask image with dimensions of H×W×C can be finally obtained, where H, W, and C are the height, width, and channel of the image, respectively. The mask image after fine-tuning is subjected to dimensionality reduction transformation to obtain a feature of h×w×c, which is quantized and then directly encoded. Among them, the dimensionality reduction transformation method includes but is not limited to bilinear downsampling, transformation downsampling based on convolutional neural network, and transformation downsampling based on multi-layer perceptron neural network.

[0068] S202: Obtain a mask image to be encoded in a coding and decoding order, determine the entropy probability of the mask image to be encoded based on the entropy probabilities of the mask image to be encoded and the encoded mask image, and sequentially encode all mask images into a code stream to be transmitted.

[0069] Specifically, the current mask image to be encoded is obtained in the encoding and decoding order, and the entropy probability of the mask image to be encoded is predicted based on the entropy probabilities of the mask image to be encoded and the encoded mask image to obtain the entropy probability of the mask image to be encoded, so as to encode all the mask images into the code stream to be transmitted in sequence according to the encoding and decoding order. For related content, please refer to the previous embodiment, and this application will not repeat it here.

[0070] S203: Acquire an auxiliary code stream, and obtain auxiliary features based on the auxiliary code stream; wherein the auxiliary code stream is obtained based on a reconstructed image that is prior to the original image in time sequence.

[0071] Specifically, an auxiliary code stream is acquired, and an auxiliary feature is obtained from the auxiliary code stream, wherein the auxiliary code stream is obtained based on a reconstructed image that is prior to the original image in time sequence.

[0072] For illustration purposes, see Figure 4 , Figure 4This is a schematic diagram of an application scenario of an implementation method of an image encoding and decoding method based on content generation of the present application, wherein the conventional encoding process of the original image includes dimensionality reduction transformation and quantization, and the features after dimensionality reduction are arithmetically encoded, and the conventional decoding process includes arithmetic decoding, inverse quantization and inverse transformation, wherein the entropy encoding and decoding includes an arithmetic codec and an entropy model network, and the entropy model network performs probability prediction on the coded symbols by constructing a probability model, and the arithmetic codec can encode and decode the stream through the probability of each symbol, and the higher the probability, the fewer the number of bits required, therefore, the more accurate the probability model is in predicting the probability of the symbol, and the higher the coding performance of the overall codec.

[0073] Furthermore, the fixed probability model is a simple entropy model, in which parameters are learned offline, so that the reconstructed image before the original image can be used to generate an auxiliary code stream to assist in predicting the current image.

[0074] S204: Obtain the marked area to be decoded according to the coding and decoding order, obtain the reference features corresponding to the mask image matching the coding and decoding order, decode and reconstruct the transmission code stream based on the reference features, auxiliary features and decoded and reconstructed marked area of ​​the marked area to be decoded, obtain all reconstructed marked areas matching the coding and decoding order in turn, and generate a reconstructed image corresponding to the original image.

[0075] Specifically, the marked area to be decoded is obtained in the coding and decoding order, and the reference features corresponding to the mask image required at this time in the coding and decoding order are obtained. The code stream to be transmitted is decoded and reconstructed based on the reference features, auxiliary features and the decoded and reconstructed marked areas in the coding and decoding order, and all reconstructed marked areas matching the coding and decoding order are obtained in turn, which together constitute the reconstructed image corresponding to the original image. Therefore, in the decoding stage, the code stream to be transmitted is decoded and reconstructed with the help of the reference features, auxiliary features and the decoded and reconstructed marked areas, so as to enhance the accuracy of sequential decoding and reconstruction of multiple marked areas.

[0076] It can be understood that the auxiliary code stream encodes and decodes the image as a whole during the encoding and decoding process, and the auxiliary features obtained from the auxiliary code stream can provide more detailed auxiliary information for the decoding of the original image.

[0077] Optionally, the marked area is obtained by decoding and reconstructing the transmission code stream based on a pre-trained probability model, the probability model corresponds to a pre-trained context model, the reference features of each mask image are transmitted to the context model, and the context model guides the probability model to decode and reconstruct the transmission code stream based on the reference features, auxiliary features and the decoded and reconstructed marked area.

[0078] For details, please refer again to Figure 4The entropy model network constructs a probability model to predict the probability of the coded symbol. The context model obtains reference features, auxiliary features, and decoded and reconstructed marked areas, thereby guiding the probability model to predict the probability of the coded symbol. Among them, the context model is built based on the transformer framework, that is, Figure 4 The main principle of the context model based on generative technology is to use the self-attention mechanism to predict the next information through the decoded information, and the auxiliary features after the auxiliary inverse transformation can be directly used as the input of the bidirectional transformer. The bidirectional transformer refers to the decoded information on one side and the enhanced information of the auxiliary code stream on the other side, thereby guiding the probability model to decode and reconstruct the marked areas in sequence with higher accuracy.

[0079] It can be understood that the features of arithmetic decoding are transformed into auxiliary features of the same scale as the decoded features through auxiliary inverse transformation, and the auxiliary features are involved in the calculation of the bidirectional transformer. Among them, the auxiliary code stream encoding and decoding branch corresponding to the context model can be frozen first, and the context model can be trained. After the context model converges, the richness of the training samples is enhanced, and the auxiliary code stream encoding and decoding branch is trained until convergence, and then all modules in the network system are trained, so that it can be used. Figure 4 The network system shown realizes the end-to-end output of the complete process from encoding to decoding of the original image.

[0080] In this embodiment, the original image is divided into multiple candidate areas more efficiently by pixel grouping, and the candidate areas are fine-tuned based on the image content to obtain the marked areas, or the original image is divided into multiple marked areas more finely by scene segmentation, so that the pixels in the marked areas have a strong correlation. The auxiliary code stream is obtained, the auxiliary features are obtained from the auxiliary code stream, the reference features corresponding to the mask image required at this time in the coding and decoding sequence are obtained, and the code stream to be transmitted is decoded and reconstructed based on the reference features, the auxiliary features and the decoded and reconstructed marked areas in sequence according to the coding and decoding sequence, and all the reconstructed marked areas matching the coding and decoding sequence are obtained in sequence, which together constitute the reconstructed image corresponding to the original image. Therefore, in the decoding stage, the code stream to be transmitted is decoded and reconstructed with the help of the reference features, the auxiliary features and the decoded and reconstructed marked areas, so as to enhance the accuracy of decoding and reconstruction of multiple marked areas in sequence.

[0081] See also Figure 5 , Figure 5 It is a structural diagram of an embodiment of an electronic device of the present application, wherein the electronic device 50 includes a memory 501 and a processor 502 coupled to each other, wherein the memory 501 stores program data (not shown), and the processor 502 calls the program data to implement the method in any of the above embodiments. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0082] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program data 600. When the program data 600 is executed by a processor, the method in any of the above embodiments is implemented. For descriptions of related contents, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0083] It should be noted that the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0084] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0086] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A content-generated image encoding and decoding method, characterized in that: The method comprises: In response to obtaining an original image to be encoded, based on the content of the original image, a plurality of marked areas of the original image are determined, and the marked areas are masked in a manner matching a coding and decoding order to obtain a plurality of mask images; wherein the coding and decoding order is related to the number of pixels in the marked area, each of the mask images corresponds to a respective marked area, and the coding and decoding order is performed in the order of increasing number of pixels in the marked area; Acquire the mask image to be encoded according to the encoding and decoding order, determine the entropy probability of the mask image to be encoded based on the entropy probability of the mask image to be encoded and the encoded mask image, and sequentially encode all the mask images into a code stream to be transmitted; Acquire an auxiliary code stream, and obtain an auxiliary feature based on the auxiliary code stream; wherein the auxiliary code stream is obtained based on a reconstructed image that is prior to the original image in time sequence; Acquire the marked area to be decoded according to the coding and decoding order, acquire the reference features corresponding to the mask image matching the coding and decoding order, decode and reconstruct the code stream to be transmitted based on the reference features of the marked area to be decoded, the auxiliary features and the decoded and reconstructed marked area, sequentially obtain all the reconstructed marked areas matching the coding and decoding order, and generate a reconstructed image corresponding to the original image; The entropy probability of the mask image is predicted based on a pre-trained probability model, and the probability model corresponds to a pre-trained context model, and the context model guides the probability model to predict the entropy probability of the mask image to be encoded based on the entropy probability of the encoded mask image; the reference features of each mask image are transmitted to the context model, and the context model guides the probability model to decode and reconstruct the code stream to be transmitted based on the reference features, the auxiliary features and the decoded and reconstructed marked area; The context model is constructed based on a transformer framework and includes a bidirectional transformer, one side of which refers to decoded information and the other side of which refers to information of the auxiliary code stream.

2. The method according to claim 1, characterized in that The step of determining a plurality of marked areas of the original image based on the content of the original image comprises: Based on the content of the original image, pixels in the original image are grouped to obtain a plurality of marking areas consisting of mutually related pixels; wherein the number of pixels in at least some of the marking areas is different from each other, and each of the marking areas is provided with a marking number matching the number of pixels.

3. The method according to claim 2, characterized in that The pixels in the original image are grouped based on the content of the original image to obtain a plurality of marked areas consisting of mutually related pixels, including: Using a preset pixel grouping method, the original image is divided into a plurality of candidate areas with increasing numbers of pixels; wherein the pixel grouping method corresponds to a grouping extension direction; Based on the adjacent pixels between every two adjacent candidate regions and at least part of the pixels in the group extension direction, the candidate regions are adjusted until all candidate regions are traversed to obtain a plurality of marked regions consisting of mutually related pixels.

4. The method according to claim 3, characterized in that The adjusting the candidate regions based on adjacent pixels between every two adjacent candidate regions and at least part of the pixels in the group extension direction includes: Acquire the similarity between the adjacent pixels and the pixels within a quantity threshold in the group extension direction, and determine the expansion quantity of the pixels to be expanded in the group extension direction based on the similarity; In the group extension direction, pixels in the front candidate area are extended to the rear candidate area by the expansion amount to obtain a marked area corresponding to the front candidate area.

5. The method according to claim 2, characterized in that: The pixels in the original image are grouped based on the content of the original image to obtain a plurality of marked areas consisting of mutually related pixels, including: Performing scene segmentation on the original image to obtain a plurality of segmented regions corresponding to the original image; wherein pixels contained in the segmented regions are related to targets in the original image; Based on the number of pixels in the segmented area, a matching label number is set for the segmented area to obtain a plurality of label areas consisting of mutually related pixels.

6. The method according to claim 2, characterized in that The marking numbers of the plurality of marking areas are set in increasing order according to a positive correlation with the number of pixels in the marking areas, and the encoding and decoding order corresponds to the increasing marking numbers; The marked area is masked in a manner matching the encoding and decoding order to obtain a plurality of mask images, including: Based on the encoding and decoding order, each marking number is taken as the current marking number in turn, and the marking areas corresponding to all marking numbers after the current marking number are masked to obtain a plurality of mask images.

7. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images