Image processing method, apparatus, device, medium, and program product

Abstract

This application provides an image processing method, apparatus, device, medium, and program product, which can be applied in the computer field. The method includes: processing the encoded features of an original image using a super-prior decoding network to obtain the standard deviation and mean of pixel values ​​in multiple dimensions of the original image; determining mask features and inverse mask features based on the standard deviations of the pixel values ​​in multiple dimensions; fusing the mask features and encoded features to obtain target transmission features; performing deterministic sampling on the original image based on the mean of the pixel values ​​in multiple dimensions to obtain sampling features; decoding the pixel positions corresponding to the mean based on the sampling features to obtain the pixels corresponding to the pixel positions; processing the sampling features, inverse mask features, and target transmission features using a decoder network to obtain a compressed image; and storing the compressed image in a target buffer.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to an image processing method, apparatus, device, medium, and program product. Background Technology

[0002] In image transmission scenarios such as video conferencing and telemedicine, network bandwidth resources are limited, and low bitrate compression can lead to significant visual distortion. Generative image compression, by introducing a generative model at the receiving end to synthesize missing details, can achieve good reconstruction quality at extremely low bitrates, but it generally does not solve the core problems: how to determine the image information that must be transmitted, and how to determine the image information that needs to be supplemented by the receiving end.

[0003] Existing methods primarily rely on generative adversarial networks (GANs) or diffusion models to achieve information separation. However, the separation boundary is implicitly determined by the training process and cannot be dynamically adjusted based on image content, making it difficult to adapt to the differentiated needs of various transmission scenarios. Furthermore, the receiving end typically requires additional side information to determine the separation criterion, consuming valuable transmission bandwidth resources. Therefore, a content-adaptive image processing method that does not require the transmission of additional side information is needed to improve image transmission efficiency under limited bandwidth conditions. Summary of the Invention

[0004] In view of the above problems, this application provides an image processing method, apparatus, device, medium, and program product.

[0005] According to a first aspect of this application, an image processing method is provided, comprising: processing the encoded features of an original image using a super-prior decoding network to obtain the standard deviation and mean of pixel values ​​in multiple dimensions of the original image; determining mask features and inverse mask features based on the standard deviations of pixel values ​​in multiple dimensions, wherein the mask features represent a first image region in the original image, and the inverse mask features represent a second image region in the original image, wherein the visual semantic importance of the first image region is higher than that of the second image region; fusing the mask features and encoded features to obtain target transmission features; performing deterministic sampling on the original image based on the mean of pixel values ​​in multiple dimensions to obtain sampling features, wherein a decoding end decodes the pixel positions corresponding to the mean based on the sampling features to obtain the pixels corresponding to the pixel positions; processing the sampling features, inverse mask features, and target transmission features using a decoder network to obtain a compressed image, and storing the compressed image in a target buffer.

[0006] According to an embodiment of this application, determining mask features and inverse mask features based on the standard deviations corresponding to pixel values ​​in multiple dimensions of the original image includes: sorting the standard deviations corresponding to pixel values ​​in multiple dimensions of the original image to obtain a sorting result; determining a first standard deviation set and a second standard deviation set from the sorting result based on a preset ratio, wherein the preset ratio represents the number of standard deviations in the first standard deviation set or the second standard deviation set; and processing the pixel values ​​in the original image based on the first standard deviation set and the second standard deviation set to determine the mask features and inverse mask features.

[0007] According to an embodiment of this application, the number of standard deviations in the first standard deviation set is directly proportional to the bitrate of the compressed image.

[0008] According to embodiments of this application, processing pixel values ​​in the original image based on a first standard deviation set and a second standard deviation set to determine mask features and inverse mask features includes: determining a first pixel corresponding to a first standard deviation in the first standard deviation set and a second pixel corresponding to a second standard deviation in the second standard deviation set, based on the mapping relationship between pixels and standard deviations in the original image; updating the pixel value of the first pixel to a first preset value to obtain mask features; and updating the pixel value of the second pixel to a second preset value to obtain inverse mask features.

[0009] According to an embodiment of this application, deterministic sampling is performed on the original image based on the mean values ​​corresponding to pixel values ​​in multiple dimensions to obtain sampling features, including: processing the mean values ​​corresponding to pixel values ​​in multiple dimensions using a deterministic mapping function to obtain multiple sampling coordinates; and extracting pixel values ​​from the sampling pixels in the original image that correspond to the multiple sampling coordinates to obtain sampling features.

[0010] According to an embodiment of this application, the encoding features of the original image are processed using a super-prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image, including: processing the encoding features of the original image using a super-prior coding network to obtain super-coding features; and processing the super-coding features based on the super-prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image.

[0011] A second aspect of this application provides an image processing apparatus, comprising: a standard deviation module, configured to process the encoded features of an original image using a super-prior decoding network to obtain the standard deviation and mean of pixel values ​​in multiple dimensions of the original image; a feature module, configured to determine mask features and inverse mask features based on the standard deviations of pixel values ​​in multiple dimensions, wherein the mask features represent a first image region in the original image, and the inverse mask features represent a second image region in the original image, and the visual semantic importance of the first image region is higher than that of the second image region; a fusion module, configured to fuse the mask features and encoded features to obtain target transmission features; a sampling module, configured to perform deterministic sampling on the original image based on the mean of pixel values ​​in multiple dimensions to obtain sampling features, wherein a decoding end decodes the pixel positions corresponding to the mean based on the sampling features to obtain the pixels corresponding to the pixel positions; and a compressed image obtaining module, configured to process the sampling features, inverse mask features, and target transmission features using a decoder network to obtain a compressed image, and store the compressed image in a target buffer.

[0012] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0013] A fourth aspect of this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0014] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0015] According to embodiments of this application, by processing the encoded features of the original image using a super-prior decoding network, the standard deviations corresponding to pixel values ​​in multiple dimensions can be obtained. Based on the obtained standard deviations, information separation boundaries can be determined. Mask features and inverse mask features can be obtained based on the standard deviations corresponding to pixel values ​​in multiple dimensions. The mask features represent a first image region in the original image, and the inverse mask features represent a second image region in the original image. The visual semantic importance of the first image region is higher than that of the second image region. By fusing the mask features and encoded features, target transmission features can be obtained. Important target transmission features can be transmitted, while other image information is synthesized at the decoding end. Deterministic sampling of the original image based on the mean values ​​corresponding to pixel values ​​in multiple dimensions yields sampling features. The decoding end decodes the pixel positions corresponding to the mean values ​​based on the sampling features to obtain the pixels corresponding to those positions. By processing the sampling features, inverse mask features, and target transmission features using a decoder network, a compressed image can be obtained, and the compressed image is stored in the target buffer. By using masking and inverse masking features determined based on standard deviation, more transmission resources can be allocated to structural regions and more synthesis resources to texture regions in an interpretable manner. This allows for the acquisition of visually natural and detailed compressed images with limited transmission bandwidth. Compared to forward inference of codec neural networks, it has lower time complexity and can meet real-time requirements while ensuring image quality in latency-sensitive image transmission applications. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0017] Figure 1 The illustration shows application scenarios of image processing methods, apparatus, devices, media, and program products according to embodiments of this application.

[0018] Figure 2 A flowchart of an image processing method according to an embodiment of this application is shown.

[0019] Figure 3 A schematic diagram of an image compression model according to an embodiment of this application is shown.

[0020] Figure 4 A structural block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0021] Figure 5 A block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of this application is shown. Detailed Implementation

[0022] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] With the rapid development of internet and mobile communication technologies, the demand for network transmission of image data is increasing daily. In application scenarios such as video conferencing, telemedicine, cloud image storage, and IoT visual sensing, image data needs to be transmitted through network channels with limited bandwidth. Due to the limitation of network bandwidth resources, raw image data often cannot be transmitted directly and needs to be compressed and encoded at the sending end and decompressed and reconstructed at the receiving end to achieve efficient image transmission.

[0027] In low-bandwidth transmission scenarios, the amount of data that a transmission channel can carry is insufficient to fully describe all the details of the original image. Related methods often produce significant compression distortions such as blockiness and ringing at extremely low bit rates, leading to a severe deterioration in the visual quality of the reconstructed image at the receiving end. To address this issue, generative image compression technology introduces a generative model at the receiving end to synthesize the missing detail information, thereby achieving a visually natural reconstructed image even under extremely low bit rate transmission conditions. The core challenge of this approach is how to divide the coded representation of an image into structural information that must be transmitted through the channel and detail information that can be synthesized by the generative model at the receiving end, and how to determine the optimal separation boundary between the two.

[0028] Generative Adversarial Network (GAN)-based methods use a GAN as the generator at the receiver end, implicitly learning through adversarial training which information needs to be transmitted and which can be synthesized. However, the information separation boundary is entirely implicitly determined by the adversarial training process, lacking explicit separation criteria. The compression result is sensitive to the training process and hyperparameters, and it cannot flexibly adjust the ratio between structural and synthesized information during the transmission stage.

[0029] The diffusion model-based approach employs a cascaded pipeline architecture, where the front-end encoder / decoder transmits low-frequency structural information, while the subsequent diffusion model synthesizes high-frequency details. However, its information separation method is statically fixed in the cascaded architecture design and does not adaptively adjust to changes in image content. For texture-rich and structure-dominated images, the separation strategy is identical, failing to fully utilize the statistical characteristics of image content to optimize transmission efficiency. Furthermore, the diffusion model requires multiple iterative sampling steps, resulting in high decoding latency at the receiver, making it difficult to meet the requirements of real-time transmission applications.

[0030] Selective coding methods based on importance graphs introduce additional network branches to predict the importance score of each coding dimension, determining which dimensions need to be transmitted based on the score. However, the ranking criteria for importance scores are implicitly learned through end-to-end training, lacking theoretical basis at the information theory level; the additional importance prediction network branches increase model complexity and computational overhead; and this method is mainly designed for rate-distortion optimization scenarios, failing to fully consider the requirements for perceptual quality and distribution fidelity in generative compression.

[0031] In summary, related generative image compression techniques have the following shortcomings when applied to image transmission scenarios: the information separation boundary cannot be dynamically adjusted according to the image content, resulting in low efficiency in transmission resource allocation; the receiving end requires additional side information to determine the separation criterion, increasing transmission overhead; and it lacks the ability to flexibly control the bitrate during transmission, making it difficult to adapt to dynamically changing network bandwidth conditions. Therefore, a method is needed that can achieve content-adaptive information separation, eliminate the need to transmit additional side information, and support flexible bitrate control during inference, in order to improve image transmission efficiency and reconstruction quality under limited bandwidth conditions.

[0032] In view of this, this application provides an image processing method, apparatus, medium, and program product. The method includes: processing the encoded features of an original image using a super-prior decoding network to obtain the standard deviation and mean of pixel values ​​in multiple dimensions of the original image; determining mask features and inverse mask features based on the standard deviations of the pixel values ​​in multiple dimensions, where the mask features represent a first image region in the original image, and the inverse mask features represent a second image region in the original image, with the visual semantic importance of the first image region being higher than that of the second image region; fusing the mask features and encoded features to obtain target transmission features; performing deterministic sampling on the original image based on the mean of pixel values ​​in multiple dimensions to obtain sampling features; decoding the pixel positions corresponding to the mean based on the sampling features to obtain the pixels corresponding to the pixel positions; processing the sampling features, inverse mask features, and target transmission features using a decoder network to obtain a compressed image, and storing the compressed image in a target buffer.

[0033] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0034] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0035] Figure 1 The illustration shows application scenarios of image processing methods, apparatus, devices, media, and program products according to embodiments of this application.

[0036] like Figure 1 As shown, the application scenario according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0037] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0039] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0040] It should be noted that the image processing method provided in this application embodiment can generally be executed by server 105. Correspondingly, the image processing apparatus provided in this application embodiment can generally be located in server 105. The image processing method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the image processing apparatus provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Figure 2 A flowchart of an image processing method according to an embodiment of this application is shown.

[0043] like Figure 2 As shown, the image processing method of this embodiment includes operations S210 to S250, and the image processing method can be executed by an electronic device.

[0044] In operation S210, the encoded features of the original image are processed using a super prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image.

[0045] In operation S220, the mask features and inverse mask features are determined based on the standard deviations of the pixel values ​​in multiple dimensions.

[0046] In operation S230, the mask features and coding features are fused to obtain the target transmission features.

[0047] In operation S240, deterministic sampling is performed on the original image based on the mean values ​​of pixel values ​​in multiple dimensions to obtain sampling features.

[0048] In operation S250, the decoder network processes the sampling features, inverse mask features and target transmission features to obtain a compressed image, which is then stored in the target buffer.

[0049] By using a super-prior decoding network to process the encoded features of the original image, the standard deviations corresponding to pixel values ​​in multiple dimensions of the original image can be obtained. Dimensions can represent length, width, channels, etc., but are not limited to these; the embodiments of this application do not specifically limit the dimensions. The standard deviation σ can serve as an intrinsic signal measuring the uncertainty of each dimension: dimensions with small σ values ​​indicate that the decoder has high confidence in predicting that dimension, belonging to predictable structural information, and should be transmitted preferentially; dimensions with large σ values ​​indicate that the decoder has high uncertainty in predicting that dimension, belonging to random detail information, and can be sampled and synthesized by the decoder from the conditional prior distribution.

[0050] The mask feature represents the first image region in the original image, where all other regions are empty. The inverse mask feature represents the second image region in the original image, where all other regions are empty. The visual semantic importance of the first image region is higher than that of the second image region. The mask features and the encoded features can be fused to obtain the target transmission features.

[0051] The decoder decodes the pixel positions corresponding to the mean μ based on the sampling features, thus obtaining the pixels corresponding to those positions. By fusing the sampling features and the inverse mask features, the target synthesis features can be determined. The decoder network then processes the target synthesis features and target transmission features to obtain the compressed image.

[0052] Since the distribution of σ varies with the statistical characteristics of the image content, it can automatically identify regions with dense structural information and regions with random textures in the image, and allocate transmission and synthesis resources accordingly. For regions with dense structural information, the system prioritizes the allocation of transmission bandwidth to ensure reconstruction accuracy; for regions with random textures, the decoding end generates a model for synthesis and completion.

[0053] Based on the obtained multiple standard deviations, the encoder and decoder can independently perform deterministic sorting and gating operations, dividing multiple dimensions into a "transmission set" and a "synthesis set". The encoder performs entropy encoding on only the dimensions in the transmission set and writes them into the bitstream; the decoder decodes the dimensions in the transmission set from the bitstream and deterministically samples the dimensions in the synthesis set from the conditional prior distribution. The combination of the two and the resulting data can be processed by the decoder network to obtain a compressed image.

[0054] The transport set (target transport features) and the synthesis set (target synthesis features) are combined into a complete latent representation, which is then fed into the decoder network to obtain a compressed image. The resulting compressed image can be stored in the target buffer so that it can be used directly when needed. The target buffer is not specifically limited here and can be set according to the actual situation.

[0055] This application does not add any additional branches, but instead seeks a criterion shared by both parties within the standard super-prior framework that does not require an increase in bitrate. The standard deviation σ of the super-prior decoding network output reflects the decoding end's prediction uncertainty for each dimension. Therefore, dimensions with smaller σ can be used as structural information that should be transmitted first, while dimensions with larger σ can be used as random detail information that can be supplemented by conditional prior sampling. Taking an image with both building outlines and leaf textures as an example, the low σ dimension corresponding to the outline is written into the bitstream, while the high σ dimension corresponding to the leaves is synthesized by the decoding end, thereby achieving adaptive content compression without adding additional network branches and mask edge information.

[0056] According to embodiments of this application, by processing the encoded features of the original image using a super-prior decoding network, the standard deviations corresponding to pixel values ​​in multiple dimensions can be obtained. Based on the obtained standard deviations, information separation boundaries can be determined. Mask features and inverse mask features can be obtained based on the standard deviations corresponding to pixel values ​​in multiple dimensions. The mask features represent a first image region in the original image, and the inverse mask features represent a second image region in the original image. The visual semantic importance of the first image region is higher than that of the second image region. By fusing the mask features and encoded features, target transmission features can be obtained. Important target transmission features can be transmitted, while other image information is synthesized at the decoding end. Deterministic sampling of the original image based on the mean values ​​corresponding to pixel values ​​in multiple dimensions yields sampling features. The decoding end decodes the pixel positions corresponding to the mean values ​​based on the sampling features to obtain the pixels corresponding to those positions. By processing the sampling features, inverse mask features, and target transmission features using a decoder network, a compressed image can be obtained, and the compressed image is stored in the target buffer. By using masking and inverse masking features determined based on standard deviation, more transmission resources can be allocated to structural regions and more synthesis resources to texture regions in an interpretable manner. This allows for the acquisition of visually natural and detailed compressed images with limited transmission bandwidth. Compared to the forward inference of codec neural networks, it has lower time complexity and can meet real-time requirements while ensuring image quality in latency-sensitive image transmission applications.

[0057] According to an embodiment of this application, determining mask features and inverse mask features based on the standard deviations corresponding to pixel values ​​in multiple dimensions of the original image includes: sorting the standard deviations corresponding to pixel values ​​in multiple dimensions of the original image to obtain a sorting result; determining a first standard deviation set and a second standard deviation set from the sorting result based on a preset ratio, wherein the preset ratio represents the number of standard deviations in the first standard deviation set or the second standard deviation set; and processing the pixel values ​​in the original image based on the first standard deviation set and the second standard deviation set to determine the mask features and inverse mask features.

[0058] By sorting the standard deviations of pixel values ​​across multiple dimensions in the original image, a sorting result can be obtained. The standard deviations can be sorted in ascending order. Given a preset proportion k∈(0,1), a first set of standard deviations and a second set of standard deviations can be determined from the sorting result based on this preset proportion. The preset proportion represents the number of standard deviations in either the first or second set. All standard deviations in the first set are less than those in the second set. For example, with a preset proportion of 30%, based on the ascending sorting result, the first 30% can be used as the first set of standard deviations, and the last 70% as the second set. Processing the pixel values ​​in the original image based on the first and second standard deviation sets allows for the determination of masking features and inverse masking features.

[0059] By independently performing the same sorting and preset ratio operations at both the encoding and decoding ends, completely consistent mask features can be obtained. This mechanism allows the entire transmission bandwidth to be used for encoding the image content itself, without consuming additional bitrate to transmit mask information or synchronization signals. In bandwidth-constrained transmission scenarios, this zero-side information overhead design can improve the transmission efficiency of effective information.

[0060] According to embodiments of this application, the preset ratio actually corresponds to the truncation position after standard deviation sorting. The preset ratio determines "which dimensions with uncertainty levels below a certain threshold must be transmitted and which dimensions with uncertainty levels above a certain threshold can be synthesized." This allows structurally sensitive information such as edges, contours, and geometric relationships, typically corresponding to low standard deviation dimensions, to be transmitted preferentially, which helps ensure the structural stability of the compressed image. Random textures and high-frequency details corresponding to high standard deviation dimensions can be sampled, which helps reduce the bitrate consumption of difficult-to-predict information and improve bitrate utilization efficiency. In addition, the truncation rules corresponding to the preset ratio can also be synchronously reconstructed by the decoding end without the need for additional transmission of mask edge information. By continuously adjusting the preset ratio, a smooth trade-off can be achieved between "transmitting more structure" and "synthesizing more details," thereby forming a single-model, multi-bitrate, and deployable control method.

[0061] According to an embodiment of this application, the number of standard deviations in the first standard deviation set is directly proportional to the bitrate of the compressed image.

[0062] The number of standard deviations in the first standard deviation set is directly proportional to the bitrate of the compressed image. All standard deviations in the first standard deviation set are smaller than those in the second standard deviation set. The preset ratio is a key parameter for controlling the bitrate. Adjusting the preset ratio allows for single-model, multi-bitrate point coverage. Increasing the preset ratio increases the number of standard deviations in the first standard deviation set, thus increasing the bitrate of the compressed image. Increasing the preset ratio allows more dimensions to be included in the transmission set, increasing the bitrate and improving the quality of the reconstructed compressed image by increasing the transmission component; decreasing the preset ratio allows more dimensions to be included in the synthesis set, decreasing the bitrate and increasing the quality of the reconstructed compressed image by increasing the synthesis component.

[0063] The encoding and decoding parties only need to pre-agree on a preset ratio (which can be used as a small amount of metadata transmission in the bitstream header, requiring only a few bits) to achieve compression at different bitrates under the same training model, without needing to train a separate model for each bitrate point. Constraints in the training model can include gating functions, sorting algorithms, pseudo-random number generators, Gaussian transforms, quantization methods, and scanning order. In the gating function, the mask features only take the encoded features and the preset ratio as input, without relying on information unique to the encoder. The sorting algorithm provides stable sorting, with equal standard deviations determined by ascending order of batch, channel, height, and width flattening indices. Quantization method: Uniform scalar quantization is used consistently for both training and inference. Scanning order: Both encoding and decoding process each dimension in the standard order of batch, channel, height, and width flattening.

[0064] By controlling the preset ratio, the system can dynamically adjust the transmission strategy according to the real-time fluctuations in network bandwidth, eliminating the need to train or deploy models separately for each target bit rate, thus reducing the system's storage overhead and operational complexity.

[0065] According to embodiments of this application, processing pixel values ​​in the original image based on a first standard deviation set and a second standard deviation set to determine mask features and inverse mask features includes: determining a first pixel corresponding to a first standard deviation in the first standard deviation set and a second pixel corresponding to a second standard deviation in the second standard deviation set, based on the mapping relationship between pixels and standard deviations in the original image; updating the pixel value of the first pixel to a first preset value to obtain mask features; and updating the pixel value of the second pixel to a second preset value to obtain inverse mask features.

[0066] Based on the mapping relationship between pixels and standard deviations in the original image, the first pixel corresponding to the first standard deviation in the first standard deviation set and the second pixel corresponding to the second standard deviation in the second standard deviation set are determined from the original image. The pixel value of the first pixel can be updated to a first preset value to obtain the mask feature. The pixel value of the second pixel can be updated to a second preset value to obtain the inverse mask feature. For example, the pixel value of the first pixel can be updated to "1" and the second element to "0" to determine the mask feature, thereby preserving the region of visual semantic importance. Alternatively, the pixel value of the first pixel can be updated to "0" and the second element to "1" to determine the inverse mask feature. Multiple dimensions corresponding to the first standard deviation set can be labeled as the transport set, and the rest as the synthesis set.

[0067] According to embodiments of this application, based on the mapping relationship between pixels and standard deviation, mask features and inverse mask features can be determined, achieving accurate separation of visually semantically important regions (transmission set) and background regions (synthesis set), thereby improving feature discriminability and interpretability.

[0068] According to an embodiment of this application, deterministic sampling is performed on the original image based on the mean values ​​corresponding to pixel values ​​in multiple dimensions to obtain sampling features, including: processing the mean values ​​corresponding to pixel values ​​in multiple dimensions using a deterministic mapping function to obtain multiple sampling coordinates; and extracting pixel values ​​from the sampling pixels in the original image that correspond to the multiple sampling coordinates to obtain sampling features.

[0069] By using a deterministic mapping function to process the mean values ​​corresponding to pixel values ​​in multiple dimensions, multiple sampling coordinates can be obtained. Pixel values ​​are then extracted from the sampled pixels in the original image corresponding to these multiple sampling coordinates to obtain sampling features. These sampling features can be output as fixed-dimensional feature vectors, with the number of dimensions determined by the number of sampled pixels and the feature encoding method. Feature encoding methods can include direct concatenation or weighted summation, but are not limited to these; this application does not restrict the feature encoding method.

[0070] A deterministic random seed can be derived based on the quantized values ​​of the supercoding features. A deterministic pseudo-random number generator is then used to sample from the original image to obtain sampled features, which are then used to generate synthetic values. Since both the seed and the generation algorithm are deterministically derived from the supercoding features shared by the encoder and decoder, the same bitstream will always produce the same sampling results.

[0071] Deterministic sampling can utilize other pseudo-random number generation algorithms in conjunction with different Gaussian transform methods. Seed derivation can employ other deterministic hash functions, the key requirement being that the input is a deterministic signal shared by the encoder and decoder.

[0072] According to the embodiments of this application, by using a deterministic mapping function to process the mean values ​​corresponding to pixel values ​​in multiple dimensions, multiple sampling coordinates can be obtained. Pixel values ​​can be extracted from the sampling pixels in the original image that correspond to the multiple sampling coordinates to obtain sampling features, thereby improving the determinism of sampling, enhancing feature stability and reproducibility, and reducing random interference.

[0073] According to an embodiment of this application, the encoding features of the original image are processed using a super-prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image, including: processing the encoding features of the original image using a super-prior coding network to obtain super-coding features; and processing the super-coding features based on the super-prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image.

[0074] The coding features of the original image can be processed using a super-prior coding network to obtain super-coded features and then quantize them.

[0075] The quantized values ​​can be fed into the super-prior decoder network to predict the standard deviation and mean of each dimension i in the encoded features. Let the uncertainty measure (i.e., standard deviation) be the i-th dimension. The smaller the value, the more accurate the prediction; the more important this dimension is for image compression. A larger value indicates higher randomness and can be replaced by sampling. The image separation boundary can be determined by the standard deviation, or by using reconstruction error estimation, mutual information estimation, or ensemble inference variance, as long as it can be calculated deterministically from the information known at the decoding end.

[0076] According to embodiments of this application, a super prior decoding network can be used to predict the standard deviation and mean of each dimension. The standard deviation is used as an uncertainty measure to guide image separation, thereby achieving adaptive partitioning of the transmission set and the synthesis set, and improving compression efficiency and reconstruction quality.

[0077] Figure 3 A schematic diagram of an image compression model according to an embodiment of this application is shown.

[0078] like Figure 3 As shown, the input image x can be processed based on encoding (i.e., encoder) to obtain encoded features y. The encoded features y are then input into a supercoder (i.e., super-prior encoder network) to obtain super-encoded features z. In the encoding segment, the super-encoded features z can be quantized Q to obtain quantized values. Quantization value Inputting the super-resolution (i.e., the super-prior decoding network) yields the standard deviation σ corresponding to the pixel values ​​in multiple dimensions. Inputting these standard deviations into an uncertainty gate yields the mask feature m. Multiplying the encoded feature y and the mask feature m yields the intermediate encoded feature y. sThen encode the intermediate feature y s Quantization (Q) is performed, followed by processing with an arithmetic encoder (AE) and an arithmetic decoder (AD). This yields the target transmission characteristics. .

[0079] At the decoding end, the quantized value can be obtained based on the quantization information of the parametric entropy model, the arithmetic encoder (AE), and the arithmetic decoder (AD). Transmit the data, then based on the quantized values. By inputting the super-resolution (i.e., the super-prior decoding network), we can obtain the standard deviation σ and mean μ corresponding to the pixel values ​​in multiple dimensions. Inputting these standard deviations into an uncertainty gate yields the inverse mask features 1-m. Deterministic sampling is then performed on the original image based on the mean μ corresponding to the pixel values ​​in multiple dimensions, resulting in sampled features. Fusing the inverse mask features and sampled features yields the target synthetic features. The fusion target synthetic features are processed using decoding (i.e., decoder network). and target transmission characteristics Features after This allows us to obtain a reconstructed image (i.e., a compressed image). The decoder can deterministically reconstruct an indeterminate gating system that is completely identical to the encoder, without transmitting any additional side information.

[0080] Uncertain gating can be replaced with sigmoid-based soft gating or a learnable threshold. The bitstream obtained based on the arithmetic encoder AE does not contain any information about the mask feature m, and the mask can be fully recovered at the decoding end without occupying any bitrate.

[0081] Decoding and recovering from the bitstream The same super-prior decoder predicts the standard deviation and mean, and then the gated mask m is reconstructed deterministically using the same sorting rules and preset ratio k as the encoder. Because... The network parameters in the super-prior decoding networks of the same encoding and decoding ends are identical, the sorting algorithm is deterministic, the reconstructed mask is strictly consistent with the encoding end, and there is no need to transmit mask information.

[0082] In this design, the uncertainty gate in both the encoding and decoding stages shares the same parameters, including flattening, sorting, thresholding, masking, and reshaping. The uncertainty gate visualizes the standard deviation distribution and can sort and filter the standard deviations corresponding to pixel values ​​across multiple dimensions. Specifically, it sorts multiple standard deviations in ascending order and then determines a first and second set of standard deviations from the sorted results based on a preset ratio. At the encoding stage, pixel values ​​in the original image are processed based on the first and second standard deviation sets to determine the mask feature m (i.e., the binary mask m). At the decoding stage, pixel values ​​in the original image are processed based on the first and second standard deviation sets to determine the inverse mask feature 1-m.

[0083] Figure 4 A structural block diagram of an image processing apparatus according to an embodiment of this application is shown.

[0084] like Figure 4 As shown, the image processing apparatus 400 of this embodiment includes a standard deviation module 410, a feature module 420, a fusion module 430, a sampling module 440, and a compressed image acquisition module 450.

[0085] The standard deviation module 410 is used to process the encoded features of the original image using a super prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image.

[0086] The feature module 420 is used to determine the mask feature and the inverse mask feature based on the standard deviation of the pixel values ​​in multiple dimensions. The mask feature represents the first image region in the original image, and the inverse mask feature represents the second image region in the original image. The visual semantic importance of the first image region is higher than that of the second image region.

[0087] The fusion module 430 is used to fuse mask features and coding features to obtain target transmission features.

[0088] The sampling module 440 is used to perform deterministic sampling on the original image based on the mean values ​​corresponding to the pixel values ​​in multiple dimensions to obtain sampling features. The decoding end decodes the pixel positions corresponding to the mean values ​​based on the sampling features to obtain the pixels corresponding to the pixel positions.

[0089] The compressed image acquisition module 450 is used to process the sampling features, inverse mask features and target transmission features using the decoder network to obtain a compressed image and store the compressed image in the target buffer.

[0090] According to embodiments of this application, by processing the encoded features of the original image using a super-prior decoding network, the standard deviations corresponding to pixel values ​​in multiple dimensions can be obtained. Based on the obtained standard deviations, information separation boundaries can be determined. Mask features and inverse mask features can be obtained based on the standard deviations corresponding to pixel values ​​in multiple dimensions. The mask features represent a first image region in the original image, and the inverse mask features represent a second image region in the original image. The visual semantic importance of the first image region is higher than that of the second image region. By fusing the mask features and encoded features, target transmission features can be obtained. Important target transmission features can be transmitted, while other image information is synthesized at the decoding end. Deterministic sampling of the original image based on the mean values ​​corresponding to pixel values ​​in multiple dimensions yields sampling features. The decoding end decodes the pixel positions corresponding to the mean values ​​based on the sampling features to obtain the pixels corresponding to those positions. By processing the sampling features, inverse mask features, and target transmission features using a decoder network, a compressed image can be obtained, and the compressed image is stored in the target buffer. By using masking and inverse masking features determined based on standard deviation, more transmission resources can be allocated to structural regions and more synthesis resources to texture regions in an interpretable manner. This allows for the acquisition of visually natural and detailed compressed images with limited transmission bandwidth. Compared to the forward inference of codec neural networks, it has lower time complexity and can meet real-time requirements while ensuring image quality in latency-sensitive image transmission applications.

[0091] Feature module 420 includes: sorting unit, determination unit and processing unit.

[0092] The sorting unit is used to sort the standard deviations of pixel values ​​in multiple dimensions of the original image to obtain the sorting result.

[0093] The determining unit is used to determine the first standard deviation set and the second standard deviation set from the sorting results based on a preset ratio, wherein the preset ratio represents the number of standard deviations in the first standard deviation set or the second standard deviation set.

[0094] The processing unit is used to process the pixel values ​​in the original image based on the first standard deviation set and the second standard deviation set to determine the mask features and the inverse mask features.

[0095] The number of standard deviations in the first standard deviation set is directly proportional to the bitrate of the compressed image.

[0096] The processing unit includes: a pixel determination subunit, a first update subunit, and a second update subunit.

[0097] The pixel determination subunit is used to determine, based on the mapping relationship between pixels and standard deviation in the original image, a first pixel corresponding to the first standard deviation in the first standard deviation set and a second pixel corresponding to the second standard deviation in the second standard deviation set;

[0098] The first update subunit is used to update the pixel value of the first pixel to a first preset value to obtain the mask feature.

[0099] The second update subunit is used to update the pixel value of the second pixel to the second preset value to obtain the inverse mask feature.

[0100] The sampling module 440 includes a sampling coordinate acquisition unit and a sampling feature acquisition unit.

[0101] The sampling coordinates are obtained by using a deterministic mapping function to process the mean values ​​corresponding to the pixel values ​​in multiple dimensions, thereby obtaining multiple sampling coordinates.

[0102] The sampling feature acquisition unit is used to extract pixel values ​​from the sampling pixels corresponding to multiple sampling coordinates in the original image to obtain sampling features.

[0103] The standard deviation module 410 includes: a supercoding unit and a standard deviation unit.

[0104] The supercoding unit is used to process the coding features of the original image using the super-prior coding network to obtain supercoding features.

[0105] The standard deviation unit is used to process the super-coded features based on the super-prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image.

[0106] According to embodiments of this application, any plurality of modules among the standard deviation module 410, feature module 420, fusion module 430, sampling module 440, and compressed image acquisition module 450 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the standard deviation module 410, feature module 420, fusion module 430, sampling module 440, and compressed image acquisition module 450 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the standard deviation module 410, feature module 420, fusion module 430, sampling module 440, and compressed image acquisition module 450 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0107] Figure 5 A block diagram of an electronic device suitable for implementing an image processing method according to an embodiment of this application is shown.

[0108] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in ROM 502 (Read-Only Memory) or a program loaded from storage portion 508 into RAM 503 (Random Access Memory). The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0109] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0110] According to embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0111] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0112] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0113] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the recommended methods provided in the embodiments of this application.

[0114] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0115] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0116] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0117] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0120] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. An image processing method, characterized by, include: The encoded features of the original image are processed using a super prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image. The standard deviations of pixel values ​​in multiple dimensions of the original image are sorted to obtain the sorting results; A first set of standard deviations and a second set of standard deviations are determined from the sorting results based on a preset ratio, wherein the preset ratio represents the number of standard deviations in the first set of standard deviations or the second set of standard deviations. Based on the mapping relationship between pixels and standard deviation in the original image, a first pixel corresponding to the first standard deviation in the first standard deviation set and a second pixel corresponding to the second standard deviation in the second standard deviation set are determined from the original image; The pixel value of the first pixel is updated to a first preset value to obtain a mask feature, wherein the mask feature represents the first image region in the original image; The pixel value of the second pixel is updated to a second preset value to obtain an inverse mask feature. The inverse mask feature represents a second image region in the original image. The visual semantic importance of the first image region is higher than that of the second image region. By fusing the mask features and the encoding features, the target transmission features are obtained; The original image is deterministically sampled based on the mean values ​​of pixel values ​​in multiple dimensions to obtain sampling features. The decoding end decodes the pixel positions corresponding to the mean values ​​based on the sampling features to obtain the pixels corresponding to the pixel positions. The sampling features, the inverse mask features, and the target transmission features are processed using a decoder network to obtain a compressed image, which is then stored in the target buffer.

2. The method according to claim 1, characterized in that, The number of standard deviations in the first set of standard deviations is directly proportional to the bitrate of the compressed image.

3. The method of claim 1, wherein, The original image is deterministically sampled based on the mean values ​​of pixel values ​​in multiple dimensions to obtain sampling features, including: By using a deterministic mapping function to process the mean values ​​corresponding to pixel values ​​in multiple dimensions, multiple sampling coordinates are obtained; Pixel values ​​are extracted from the sampled pixels in the original image that correspond to the multiple sampled coordinates to obtain the sampled features.

4. The method of claim 1, wherein, The encoded features of the original image are processed using a super-prior decoding network to obtain the standard deviation and mean of pixel values ​​in multiple dimensions of the original image, including: The coding features of the original image are processed using a super-prior coding network to obtain super-coded features; The super-coded features are processed using a super-prior decoding network to obtain the standard deviation and mean of pixel values ​​in multiple dimensions of the original image.

5. An image processing apparatus characterized by comprising: The device includes: The standard deviation module is used to process the encoded features of the original image using the super prior decoding network to obtain the standard deviation and mean of the pixel values ​​in multiple dimensions of the original image. The feature module is used to determine mask features and inverse mask features based on the standard deviations of pixel values ​​in multiple dimensions. The mask features represent a first image region in the original image, and the inverse mask features represent a second image region in the original image. The visual semantic importance of the first image region is higher than that of the second image region. The fusion module is used to fuse the mask features and the encoding features to obtain the target transmission features; The sampling module is used to perform deterministic sampling on the original image based on the mean values ​​corresponding to pixel values ​​in multiple dimensions to obtain sampling features. The decoding end decodes the pixel positions corresponding to the mean values ​​based on the sampling features to obtain the pixels corresponding to the pixel positions. A compressed image obtaining module is used to process the sampling features, the inverse mask features, and the target transmission features using a decoder network to obtain a compressed image and store the compressed image in a target buffer, wherein the image processing device is constructed according to any one of claims 1 to 4.

6. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images