An image processing method and system

Abstract

The application discloses an image processing method, which comprises the following steps: determining a quality level corresponding to a first image based on at least a network state parameter and a task demand parameter, wherein the task demand parameter at least comprises a task type; encoding the first image based on the first image and the quality level corresponding to the first image; decoding a received first code stream to generate a second image, wherein the first image is encoded and decoded by a shared variable code rate reversible image codec, and the first code stream at least comprises a task type identifier corresponding to the task type; and inputting the second image into a visual feature reconstruction network to output a perception optimization reconstruction image and / or a task adaptation reconstruction image. The image processing method of the application greatly reduces the storage occupation and model maintenance cost in the image processing process, improves the flexibility and scalability of the image processing method, and can output a reconstruction image meeting the actual scene demand and having high quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image processing method and system. Background Technology

[0002] Image compression techniques typically reduce the bitrate overhead of visual data during transmission and storage through methods such as transformation, quantization, and entropy coding. With the development of scenarios such as intelligent surveillance, autonomous driving, edge perception, and the visual Internet of Things, the users of image data are no longer limited to human vision but also include machine vision models such as classification, object detection, and instance segmentation. Therefore, image compression systems not only need to maintain good subjective reconstruction quality at a limited bitrate but also need to preserve as much edge structure, texture details, and semantic cues as possible for machine vision tasks.

[0003] Therefore, how to provide an image processing method that can be adapted to different types of tasks to output high-quality images that meet the requirements has become one of the urgent problems to be solved. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an image processing method and system to adapt to different types of tasks and generate high-quality images that meet requirements.

[0005] This invention provides an image processing method, comprising: The quality level corresponding to the first image is determined based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least: task type; Encode it based on the first image and its corresponding quality level; The received first bitstream is decoded to generate a second image, wherein the first image is encoded and decoded by a shared variable bitrate reversible image codec, and the first bitstream includes at least: a task type identifier corresponding to the task type; The second image is input into a visual feature reconstruction network to output a perceptually optimized reconstructed image and / or a task-adapted reconstructed image.

[0006] Optionally, encoding the first image based on its corresponding quality level includes: The first image is subjected to a multi-scale reversible transformation to generate latent variables at multiple scales corresponding to it. Based on the quality level corresponding to the first image, latent variables at multiple scales corresponding to the first image are subjected to gain modulation, quantization, and entropy coding to generate a compressed bitstream.

[0007] Optionally, the shared variable bitrate reversible image codec includes: a spatial rearrangement module and an N-level reversible module connected thereto, wherein the N-level reversible modules are cascaded sequentially, and N is a natural number greater than 1. The step of performing a multi-scale reversible transformation on the first image to generate latent variables at multiple corresponding scales includes: The first image is input into the spatial rearrangement module, which outputs spatial rearrangement features. The spatial rearrangement features are input into an N-level reversible module that is cascaded in series to output multiple latent variables corresponding to the first image. The input of the first-level invertible module is the spatial rearrangement feature, and the output includes: the latent variables corresponding to the current scale and the hidden representation of the current scale; The input of any other non-final-level invertible module is the hidden representation of the output of the next-level invertible module. The output includes: the latent variable corresponding to the current level scale and the hidden representation of the current level. The input to the final-level invertible module is the hidden representation of the output of the previous-level invertible module. The output includes two latent variables at different scales.

[0008] Optionally, the step of performing gain modulation, quantization, and entropy coding on latent variables at multiple scales corresponding to the first image based on the quality level corresponding to the first image to generate a compressed bitstream includes: For each scale of latent variable, obtain the residual of the latent variable at that scale, where the residual of the latent variable at that scale is obtained based on the latent variable at that scale and its corresponding mean parameter; Based on the quality level, determine the corresponding gain parameter to modulate the latent variable residual at that scale, and obtain the modulated latent variable residual at that scale. The modulated latent variable residual at this scale is quantized to obtain the scale residual quantization value, and the scale residual quantization value is entropy encoded based on the scale parameter corresponding to the latent variable at this scale to generate a compressed bitstream. The mean and scale parameters of the latent variables at each scale are obtained based on entropy model prediction.

[0009] Optionally, the first bitstream includes the compressed bitstream and a task type identifier, and the step of decoding the received first bitstream to generate the second image includes: The compressed bitstream is parsed to obtain the quality level; The compressed bitstream is entropy decoded to obtain the quantized multi-scale latent variable residuals; Based on the quality level, the quantized multi-scale latent variable residuals are sequentially inversely quantized and inversely gained mapped, and the mean parameter corresponding to the multi-scale latent variable is compensated to obtain the multi-scale latent variable. The multi-scale latent variables are subjected to a stepwise inverse reversible transformation in the reverse order of the multi-scale decomposition at the encoding end to generate a second image.

[0010] Optionally, the task type includes: human visual tasks and / or machine visual tasks, and the visual feature reconstruction network includes: a human perception reconstruction network matched with human visual tasks and / or a task adaptation reconstruction network matched with machine visual tasks; wherein: The human eye perception reconstruction network is used to generate perception-optimized reconstruction images that adapt to human visual perception. The task adaptation and reconstruction network is used to generate task adaptation and reconstruction images based on task supervision constraints.

[0011] Optionally, if the task type is a human visual task, the quality level corresponding to the first image is determined based on the network status score and the task requirement score. If the task type is a machine vision task, the quality level corresponding to the first image is determined based on the network status score, task requirement score, and machine vision task suitability score. Among them, the network state score is associated with the network state parameters, the task requirement score is associated with the task requirement parameters, and the machine task fit score is associated with the evaluation metrics of the machine vision model output image.

[0012] Optionally, the task-adapted reconstructed image is input to the machine vision model performing the machine vision task, and the method further includes: if the evaluation index of the output image of the machine vision model does not reach the preset value, then the quality level corresponding to the first image is adjusted.

[0013] Optionally, the human eye perception reconstruction network and / or the task adaptation reconstruction network include multiple ConvNeXt V2 Blocks connected in series.

[0014] The present invention also provides an image processing system, comprising: The determining unit is configured to determine the quality level corresponding to the first image based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least: task type; A shared variable bitrate reversible image codec is used to encode a first image and its corresponding quality level, and to decode the received first bitstream to generate a second image, wherein the first bitstream includes at least: a task type identifier corresponding to the task type; A visual feature reconstruction network is used as input to the second image and outputs a perceptually optimized reconstructed image and / or a task-adapted reconstructed image.

[0015] The present invention also provides a computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, implement the image processing method described above.

[0016] The present invention also provides an electronic device, including a memory and a processor, wherein the processor executes a computer program in the memory to implement the steps of the above-described image processing method.

[0017] Compared with the prior art, the present invention has at least the following beneficial effects: The image processing method of the present invention first determines a quality level corresponding to a first image based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least a task type. Then, the first image and its corresponding quality level are encoded; the received first bitstream is decoded to generate a second image, wherein the first image is encoded and decoded using a shared variable bitrate reversible image codec, and the first bitstream includes at least a task type identifier corresponding to the task type. Finally, the second image is input into a visual feature reconstruction network to output a perceptually optimized reconstructed image and / or a task-adapted reconstructed image. Since the quality level corresponding to the first image is determined based at least on network state parameters and task requirement parameters, the second image is generated based on the first image and its corresponding quality level, and then the second image is reconstructed, the encoding, decoding, and reconstruction of the first image can be tailored to the specific application scenario to output a high-quality image that meets the actual application scenario and requirements. In addition, this solution, taking into account network transmission conditions and task adaptability, precisely controls the amount of data encoded in the first image and efficiently utilizes network bandwidth and computing resources to generate reconstructed images that match different task types. This improves the quality of reconstructed images to a certain extent, while also achieving an adaptive balance between network bandwidth, reconstruction quality, task accuracy, and computational overhead. Furthermore, it enhances the flexibility of the image processing process.

[0018] By employing a shared variable bitrate reversible image codec and a quality level control mechanism, a single model can cover multiple compression bitrate points (high, medium, and low) without requiring separate training, storage, and deployment of independent models for different bitrates. This effectively reduces model redundancy and significantly lowers storage requirements and model maintenance costs during image processing. Furthermore, the shared variable bitrate reversible image codec allows for flexible switching of compression levels, thereby improving the flexibility of image processing.

[0019] Since the second image (which carries a task type identifier) ​​is processed by a visual feature reconstruction network after being output through a shared variable bit rate reversible image codec, in order to generate reconstructed images that meet the needs of different types of tasks, the output forms of image processing are greatly enriched, and the flexibility and scalability of image processing methods are significantly improved. Furthermore, since the visual feature reconstruction network is adapted to different scenarios, or in other words, the selection of the visual feature reconstruction network is determined based on the task type identifier, it can output reconstructed images that meet the actual needs of the scenario (different task types) and have high quality. Attached Figure Description

[0020] Figure 1 This is a schematic flowchart of the image processing method according to an embodiment of the present invention; Figure 2 This is a partial structural schematic diagram of the shared variable bitrate reversible image codec according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the image processing system according to an embodiment of the present invention. Detailed Implementation

[0021] The present disclosure will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any way. It should be noted that those skilled in the art can make various modifications and improvements to the relevant network structures, training strategies, task types, and parameter settings without departing from the concept of the present disclosure, and all of these should fall within the scope of protection of the present disclosure.

[0022] As described in existing technologies, the use of compressed image data is no longer limited to human viewing. When the compressed image results are directly input into machine vision models, quantization errors, local blurring, and loss of detail can weaken the discriminative information, thus affecting classification, detection, or segmentation performance. Furthermore, current visual collaborative compression schemes are often based on a single bitrate model. When bandwidth fluctuations, storage budget changes, or task requirements switch in practical applications, it is usually necessary to train separately for different bitrate points, and thus store and maintain multiple models, leading to high complexity and deployment costs for image processing systems.

[0023] This invention provides an image processing method that, based on a shared variable bitrate reversible codec, introduces a visual feature reconstruction network. This visual feature reconstruction network can include a human eye perception reconstruction network branch and a task adaptation reconstruction network branch, enabling the same compression backbone to generate compressed bitstreams at different quality levels and serve machine vision tasks such as human eye viewing, classification, object detection, or segmentation at the decoding end.

[0024] Figure 1 This is a schematic flowchart of the image processing method according to an embodiment of the present invention. Figure 1 As shown, the image processing method includes: S10: Determine the quality level corresponding to the first image based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least the task type.

[0025] S11: Encode the first image based on its corresponding quality level.

[0026] S12: Decode the received first bitstream to generate a second image, wherein, by sharing a variable bitrate... The inverse image codec encodes and decodes the first image, and the first bitstream includes at least a task type identifier corresponding to the task type.

[0027] S13: Input the second image into the visual feature reconstruction network to output a perception-optimized reconstructed image and / or a task-adapted reconstructed image.

[0028] In step S10, in this embodiment, network status parameters may include current bandwidth, transmission latency, packet loss rate, buffer utilization, and bitrate budget. Task requirement parameters may include: task type, task priority, real-time requirements, and task accuracy requirements. Task types may include: human vision tasks and machine vision tasks. Machine vision tasks are performed by machine vision models. The quality level q is used to control the bitrate and reconstruction quality during image compression. In this embodiment, the quality level q can be represented by a quality level identifier. The task type can be represented by a task type identifier, which is used to instruct the decoder to select different visual features to reconstruct the network.

[0029] In this embodiment, the first image can be the image to be compressed, such as an RGB image. In this embodiment, the first image is denoted as x. For the first image, different quality levels q correspond to different compression strengths and rate-distortion tradeoffs.

[0030] In this embodiment, the method for determining the quality level of the first image differs depending on the task type. Specifically, if the task type is a human visual task, the quality level corresponding to the first image is determined based on the network state score and the task requirement score.

[0031] If the task type is a machine vision task, the quality level corresponding to the first image is determined based on the network status score, task requirement score, and machine vision task suitability score.

[0032] Among them, the network state score is associated with the network state parameters, the task requirement score is associated with the task requirement parameters, and the machine vision task fit score is associated with the evaluation metrics of the machine vision model output image.

[0033] In this embodiment, the comprehensive score is calculated according to the following formula. S Then, based on the comprehensive score S To determine the quality level corresponding to the first image.

[0034]

[0035] Where N represents the network status score, T represents the task requirement score, F represents the machine vision task suitability score, and α, β, and γ are preset weight parameters that can be determined according to the actual application scenario. Specifically, they can be predetermined through calibration experiments or trial runs before system deployment. For example, they can be adjusted according to different network states, task requirement intensities, and changes in machine vision task suitability to obtain a weight combination suitable for the current application scenario.

[0036] In this embodiment, α can be 0.4, β can be 0.3, and γ can be 0.3.

[0037] In this embodiment, the network state score N can be determined based on at least one of the network state parameters: current bandwidth, transmission latency, packet loss rate, buffer occupancy rate, and bitrate budget. Specifically, a correspondence between each network state parameter and a network state sub-score can be established in advance. For example, current bandwidth, transmission latency, packet loss rate, buffer occupancy rate, and bitrate budget correspond to sub-scores NB, ND, NP, NC, and NR, respectively. When there is only one network state parameter, the sub-score corresponding to that parameter can be directly used as the network state score; when there are multiple network state parameters, a weighted average of the multiple sub-scores can be performed to obtain the network state score N.

[0038] The task requirement score T can be determined based on at least one of the task requirement parameters: task type, task priority, real-time requirement, and task accuracy requirement. For example, if the task type is determined, the task requirement score T for that task type can be directly set to a preset value. Alternatively, a correspondence between task requirement parameters and task requirement sub-scores can be established first. For example, task type, task priority, real-time requirement, and task accuracy requirement can correspond to sub-scores TT, TP, TR, and TA, respectively. When there is only one task requirement parameter, the sub-score corresponding to that parameter can be directly used as the task requirement score; when there are multiple task requirement parameters, a weighted average of the multiple sub-scores can be taken to obtain the corresponding task requirement score T.

[0039] The machine vision task fit score F can be determined based on the evaluation metrics of the machine vision model's output images. The evaluation metrics for the machine vision model's output images differ depending on the task type. For example, if the machine vision task is classification, it can be determined based on the classification confidence of the previous frame or historical images output by the machine vision model. If the machine vision task is object detection, it can be determined based on the detection confidence and / or the number of detection boxes in the previous frame or historical images output by the machine vision model. If the machine vision task is image segmentation, it can be determined based on the stability of the segmented regions in the previous frame or historical images output by the machine vision model. Furthermore, for different types of machine vision tasks, the evaluation metrics for the machine vision model's output images can also include real-time performance requirements, task accuracy requirements, etc., all of which can serve as evaluation metrics for different task types of machine vision model output images. In this embodiment, based on the actual application scenario, the evaluation metric for the machine vision model's output images for that task type can be determined first. If there is only one evaluation metric, the score corresponding to that metric can be used as the machine vision task fit score F. If there is more than one evaluation metric, then determine the score corresponding to each evaluation metric, and take a weighted average of the scores corresponding to each evaluation metric to obtain the machine vision task fit score F.

[0040] S11 is executed, encoding the first image based on its corresponding quality level. In this embodiment, the first image and its corresponding quality level can be input into a shared variable bitrate reversible image codec for encoding to generate a compressed bitstream.

[0041] Specifically, in this embodiment, encoding is performed on the first image based on its corresponding quality level, including: The first image is subjected to a multi-scale reversible transformation to generate latent variables at multiple scales corresponding to it.

[0042] Based on the quality level corresponding to the first image, latent variables at multiple scales corresponding to the first image are subjected to gain modulation, quantization, and entropy coding to generate a compressed bitstream.

[0043] In this embodiment, the shared variable bitrate reversible image codec may include: a spatial rearrangement module and an N-level reversible module connected thereto, wherein the N-level reversible modules are cascaded sequentially, and N is a natural number greater than 1. A multi-scale reversible transformation is performed on the first image to generate latent variables at multiple corresponding scales, including: The first image is input into the spatial rearrangement module, which outputs spatial rearrangement features.

[0044] The spatial rearrangement features are input into N-level reversible modules that are cascaded sequentially to output latent variables at multiple scales corresponding to the first image.

[0045] The input of the first-level reversible module is the spatial rearrangement feature, and the output includes the latent variables corresponding to the current scale and the hidden representation of the current scale.

[0046] The input of any other non-final-level invertible module is the hidden representation of the output of its previous-level invertible module. The output includes: the latent variable corresponding to the current level scale and the hidden representation of the current level scale.

[0047] The input to the final-level invertible module is the hidden representation of the output of the previous-level invertible module. The output includes two latent variables at different scales.

[0048] In this embodiment, the first image is input into the spatial rearrangement module, which only reorganizes the spatial dimensions and regularizes the pixel positions of the first image without losing its core information. By spatially rearranging the first image, the spatial pixel arrangement of the first image can be transformed into a tensor dimension form that adapts to reversible transformation and network coding compression. This allows spatial neighborhood information to be mapped to the channel dimension, decoupling spatial correlation.

[0049] In this embodiment, the N-level reversible module can determine the value of N according to actual needs, such as N being 2, 3, 4, etc. For ease of explanation, the following uses N=4 as an example to illustrate the generation process of latent variables at multiple scales corresponding to the first image in this embodiment of the invention.

[0050] See Figure 2 , Figure 2 This is a partial structural diagram of a shared variable bitrate reversible image codec according to an embodiment of the present invention, as shown below. Figure 2 As shown, the shared variable bitrate reversible image codec includes a spatial rearrangement module and four connected reversible modules. The four reversible modules are cascaded sequentially, i.e., the first-level, second-level, third-level, and fourth-level reversible modules are connected in series. After the first image is input to the spatial rearrangement module, its spatial rearrangement features are output. These features are then input to the first-level reversible module, which outputs the latent variables at the corresponding scale. and hidden representation In this embodiment, except for the final-level reversible module, i.e., the fourth-level reversible module, the output of each other-level reversible module can be divided into two parts along the channel dimension: one part is the latent variable at the scale corresponding to that level of reversible module, and the other part is the hidden representation. See also... Figure 2 The hidden representation output by the first-level reversible module As the input to the second-level reversible module, the output of the second-level reversible module can also be divided into two parts along the channel dimension: one part is the latent variable at the scale corresponding to the second-level reversible module. The other part is its hidden representation. The hidden representation output by the second-level reversible module. As the input to the third-level reversible module, the output of the third-level reversible module is also divided into two parts along the channel dimension: one part is the latent variable at the scale corresponding to the third-level reversible module. The other part is its hidden representation. .

[0051] For the fourth-level reversible module, its input is the hidden representation of the output of the third-level reversible module. The output consists of two latent variables at different scales. and Through the above multi-stage reversible transformation, the output corresponding to the first image can be obtained. A multi-scale set of latent variables ranging from high resolution to low resolution.

[0052] In this embodiment, the above-described reversible transformation can be expressed by the following formula:

[0053] in, Indicates the first i A reversible module, Representing latent variables at different scales, h i This indicates that the middle part is hidden.

[0054] Thus, through the aforementioned reversible transformation, the first image is mapped to a multi-scale latent representation set { } is used for subsequent quantization, entropy encoding, and reversible decoding.

[0055] Next, based on the quality level corresponding to the first image, the latent variables at multiple scales corresponding to the first image are subjected to gain modulation, quantization, and entropy coding respectively to generate a compressed bitstream.

[0056] In this embodiment, based on the quality level corresponding to the first image, latent variables at multiple scales corresponding to the first image are subjected to gain modulation, quantization, and entropy coding respectively to generate a compressed bitstream, including: For each scale of latent variable, the residual of the latent variable at that scale is obtained, where the residual of the latent variable at that scale is obtained based on the latent variable at that scale and its corresponding mean parameter.

[0057] Based on the quality level, determine the corresponding gain parameter to modulate the latent variable residual at that scale, and obtain the modulated latent variable residual at that scale.

[0058] The modulated latent variable residual at that scale is quantized to obtain the quantized value of the scale residual, and the quantized value of the scale residual is entropy encoded based on the scale parameter corresponding to the latent variable at that scale to generate a compressed bitstream.

[0059] The mean and scale parameters of the latent variables at each scale are obtained based on entropy model prediction.

[0060] Specifically, in this embodiment, for the first i Latent variables at various scales First, the entropy model predicts the first... i Individual scale latent variables Corresponding mean parameter Then for the first i Each scale latent variable residual Gain modulation is performed to obtain the modulated first... i Each scale of latent variable residuals.

[0061] The first after quantization modulation i The residuals of the latent variables at the nth scale are used to obtain the nth scale. i Individual scale residual quantization values.

[0062] In this embodiment, quantization can map continuous latent variables to discrete symbols for subsequent entropy coding. In this embodiment, when q corresponds to a higher quality level, the latent variables are more fully gain-modulated before quantization, reducing the equivalent quantization step size and thus achieving a higher bit rate and higher reconstruction quality; when q corresponds to a lower quality level, the equivalent quantization is coarser, resulting in a lower bit rate and a higher compression ratio.

[0063] Next, based on the first i The scaling parameter corresponding to the scale latent variable at the th scale is related to the th ... i Each scale residual quantization value is entropy encoded to generate a compressed bitstream.

[0064] In this embodiment, entropy coding can employ arithmetic coding, range coding, or other coding methods capable of generating compressed bitstreams based on a probability model. The entropy model can, based on decoded low-resolution latent variables or contextual information, [process / resource] the [data / method / etc.]. i The probability distribution of each scale latent variable is estimated, and entropy coding is performed accordingly to generate a compressed bitstream corresponding to the quality level q.

[0065] Execute step S12 to decode the received first bitstream to generate a second image. In this embodiment, the compressed bitstream and the task type identifier are encapsulated together.

[0066] In this embodiment, decoding the received first bitstream to generate a second image includes: parsing the compressed bitstream to obtain the quality level; and decoding based on the quality level and the compressed bitstream to generate a second image.

[0067] In this embodiment, decoding based on the quality level and the compressed bitstream to generate a second image includes: The compressed bitstream is entropy decoded to obtain the quantized multi-scale latent variable residuals.

[0068] Based on the quality level, the quantized multi-scale latent variable residuals are sequentially inversely quantized and inversely gained mapped, and the mean parameter corresponding to the multi-scale latent variable is compensated to obtain the multi-scale latent variable.

[0069] The multi-scale latent variables are subjected to a stepwise inverse reversible transformation in the reverse order of the multi-scale decomposition at the encoding end to generate a second image.

[0070] Specifically, before performing entropy decoding on the compressed bitstream, the compressed bitstream can be parsed to determine the corresponding quality level q. Then, an entropy model is used to perform entropy decoding on the compressed bitstream to obtain the quantized multi-scale latent variable residuals.

[0071] In this embodiment, the latent variable expressions for gain modulation, quantization, and inverse gain mapping in the above process can be obtained by the following formula:

[0072] in, This represents the gain operation corresponding to the quality level q. This represents the inverse gain operation corresponding to the quality level q. Represents the quantization operator. This represents the mean parameter predicted by the entropy model. This represents the quantized latent variable after recovery at quality level q.

[0073] In this embodiment, since a shared variable bitrate reversible image codec is used to generate a second image, it is not necessary to train and deploy multiple independent models for different bitrates. Single-model multi-bitrate compression can be achieved simply by controlling the bitrate output of the same shared compression backbone through the quality level q.

[0074] Thus, through the above process, a second image is generated. This second image can express the main content of the first image. In practical applications, considering that the above process involves quantization and bitrate constraints, problems such as local texture loss, edge blurring, or weakening of task-related structures may exist. Therefore, after generating the second image, the image processing method of this embodiment will further process the second image based on different application scenarios.

[0075] S13 is executed, inputting the second image into the visual feature reconstruction network to output a perceptually optimized reconstructed image and / or a task-adapted reconstructed image.

[0076] As can be seen from the above, the first bitstream includes a compressed bitstream and a task type identifier. Therefore, the second image after decoding also carries the task type identifier. Thus, when the second image is input into the visual feature reconstruction network, the visual feature reconstruction network can determine which type of visual feature reconstruction to perform on the second image based on the task type identifier.

[0077] In this embodiment, for task types including human visual tasks and / or machine visual tasks, the visual feature reconstruction network may include: a human visual perception reconstruction network matching human visual tasks and / or a task adaptation reconstruction network matching machine visual tasks.

[0078] The human eye perception reconstruction network is used to generate perception-optimized reconstructed images that adapt to human visual perception.

[0079] The task adaptation and reconstruction network is used to generate task adaptation and reconstruction images based on task supervision constraints.

[0080] In this embodiment, specifically, if the task type identifier represents a human visual task, the second image is input into the human visual perception reconstruction network to generate a perception-optimized reconstruction image that adapts to human visual perception, or in other words, to generate a reconstruction image suitable for human viewing.

[0081] In this embodiment, the human eye perception reconstruction network can be used to compensate for the quantization error of the second image, alleviate the reconstruction blur of the second image, and restore the local texture and edge details of the second image, thereby obtaining a perception-optimized reconstructed image suitable for human viewing. .

[0082] In this embodiment, the human eye perception reconstruction network may include multiple ConvNeXt V2 Blocks connected in series. Specifically, the human eye perception reconstruction network may include four ConvNeXt V2 Blocks connected in series. In this embodiment, if the input features of the human eye perception reconstruction network (i.e., the features of the second image) are denoted as... F If 0, then the processing of the second image by the human eye perception reconstruction network can be represented as follows:

[0083] in, Indicates the first k One ConvNeXt V2 Block, F out This represents the characteristics output by the human eye perception reconstruction network.

[0084] This output feature F out After output mapping, a reconstructed image for human viewing can be obtained, namely a perceptually optimized reconstructed image.

[0085] In this embodiment, any ConvNeXt V2 Block may include a 7×7 depthwise convolution, layer normalization, a first 1×1 convolution, a GELU activation function, a global response normalization module, a second 1×1 convolution, and residual connections. The process can be represented as follows:

[0086] Where X represents the input feature and Y represents the output feature. This represents a 7×7 depthwise convolution. Representation layer normalization, and These represent two 1×1 convolutions. This represents the GELU activation function. This indicates the global response normalization module.

[0087] In this embodiment, the human eye perception reconstruction network includes multiple ConvNeXt V2 Blocks connected in series, which enables the human eye perception reconstruction network to enhance local texture recovery and channel feature expression capabilities with lower computational overhead, allowing the second image to further recover detailed information and improve the visual reconstruction quality under multi-bitrate conditions.

[0088] In this embodiment, if the task type identifier represents a machine vision task, the second image can be input into the task adaptation and reconstruction network to generate a task-adapted reconstructed image. The task adaptation and reconstruction network has the same or similar network structure as the human eye perception reconstruction network. The function of the task adaptation and reconstruction network is not solely to improve subjective visual quality, but also to make its output image more suitable for subsequent machine vision models to perform tasks such as classification, object detection, or segmentation through task supervision constraints.

[0089] In this embodiment, specifically, the task adaptation and reconstruction network may also include multiple ConvNeXt V2 Blocks connected in series. Any ConvNeXt V2 Block may also include a 7×7 depthwise convolution, layer normalization, a first 1×1 convolution, a GELU activation function, a global response normalization module (GRN module), a second 1×1 convolution, and residual connections.

[0090] Unlike human eye perception reconstruction networks, task adaptation reconstruction networks are primarily trained based on intermediate feature constraints from downstream task models. This makes task adaptation reconstruction networks focus more on target boundaries, regional structures, semantically salient regions, and feature responses that are beneficial for task discrimination.

[0091] In this embodiment, the GRN module is used to adaptively recalibrate features based on the global response intensity of each channel. For intermediate features... We can first calculate the global response for each channel:

[0092] Then normalize the responses of each channel:

[0093] Finally, we obtain the GRN output:

[0094] Where Uc represents the feature map of the c-th channel, ε is a constant to prevent the denominator from being zero, and γ c and β c These are learnable parameters.

[0095] This mechanism allows channels with stronger and more discriminative responses to be further enhanced, while redundant or weak response channels are relatively suppressed, thereby improving the availability of features in subsequent machine vision tasks.

[0096] In this embodiment, after generating a task-adapted reconstruction image through the task adaptation and reconstruction network, it can be input into a downstream machine vision model to perform a machine vision task, thereby obtaining the output image of the machine vision model. Specifically, the machine vision task can be a classification task, an object detection task, an image segmentation task, etc.

[0097] In addition, in this embodiment, for machine vision tasks, if the evaluation index of the image output by the machine vision model does not reach the preset value, the quality level corresponding to the first image can be adjusted.

[0098] In this embodiment, if the prerequisite is that the preset range of the comprehensive score S is 5-8; The quality level set {q1, q2, q3, q4, q5} has corresponding ratings of {2, 4, 6, 8, 10}. Initially, the image is compressed using quality level q1 as the first image. If the calculated comprehensive score S is 2 (not within the preset range), the evaluation index of the machine vision model output image fails to meet the standard when the image compressed at quality level q1 is used for machine vision tasks. Therefore, the quality level of the first image can be increased to q2, and the image is compressed using q2 as the quality level of the first image. If the calculated comprehensive score S is 4 (still not within the preset range), the quality level of the first image can be further increased to q3, and the image is compressed using q3 as the quality level of the first image. If the calculated comprehensive score S is 6, then quality level q3 is the final quality level that meets the rating index of the machine vision model output image. After compressing the image at this quality level, the requirements of the machine vision task can be met, and a high-quality image that conforms to the actual scene can be output.

[0099] The preset range of the comprehensive score S can be obtained in the following way, and this embodiment takes the machine vision task as a classification task as an example for explanation: If the machine vision task is a classification task, the classification confidence score can be used as an evaluation metric for the output image of the machine vision model, with a preset threshold of no less than 0.7 for the classification confidence score. Before system deployment, several calibrated first images can be selected and compressed, decoded, and reconstructed for task adaptation at different quality levels q. Each reconstructed image is then input into the classification model (machine vision model) to obtain its corresponding classification confidence score. Simultaneously, the network state score N, task requirement score T, and machine vision task adaptation score F are obtained at different quality levels q, and the comprehensive score S is calculated according to the aforementioned comprehensive scoring formula. When the task type is determined, the task requirement score T can remain constant. As the quality level q increases, the classification confidence score improves, but this improvement tends to saturate as q increases. The machine vision task adaptation score F increases, but the increased bitrate may lead to increased transmission latency, packet loss rate, buffer usage, or bitrate budget usage, causing the network state score N to decrease. Therefore, the comprehensive score S can form a preset range between the task adaptation benefit and the network transmission cost. If the calibration results show that when the classification confidence level reaches 0.7 or higher and the network status still meets the transmission requirements, the comprehensive score S is mainly distributed between 5 and 8, then 5 to 8 can be determined as the preset interval for the comprehensive score.

[0100] In this embodiment, when the evaluation index of the machine vision model output image does not reach the preset value, the quality level corresponding to the first image can be adjusted so that the image output by the visual feature reconstruction network can better meet the downstream machine vision task, thereby achieving an adaptive balance between network bandwidth, reconstruction quality, task accuracy and computational overhead.

[0101] In other embodiments, if the perception-optimized reconstructed image output by the human eye perception reconstruction network does not meet the needs of human eye viewing, the quality level q corresponding to the first image can be directly adjusted. During this process, the task requirement score T remains unchanged, and closed-loop adjustment is performed only based on the network state score N and the quality level q to quickly converge to the optimal quality level that satisfies the subjective perception of the human eye.

[0102] In this embodiment, the downstream machine vision model can be selected according to the specific task.

[0103] In this embodiment, if the task is an image classification task, the downstream machine vision model can be ResNet-50 or other classification networks.

[0104] If the task is object detection, the downstream machine vision model can be Faster R-CNN or other detection networks.

[0105] If the task is instance segmentation or semantic segmentation, the downstream machine vision model can be Mask R-CNN or other segmentation networks.

[0106] In this embodiment, the downstream machine vision model remains frozen during the training task adaptation and reconstruction network, and is only used to provide task-related feature supervision or output the final task result.

[0107] Specifically, the first image and the task-adapted reconstruction image output by the task-adapted reconstruction network can be respectively input into the same frozen downstream machine vision model, and the intermediate layer features can be extracted. The feature differences between the two are then calculated as the task-related perceptual distortion term of the task adaptation and reconstruction network.

[0108] In this embodiment, for the image classification task, the intermediate features of the first to fourth residual stages of the frozen classification network can be extracted, the mean square error of the first image and the task-adapted reconstructed image on the corresponding layer can be calculated, and the differences of the multi-layer features can be summed or weighted and summed as the loss related to the classification task.

[0109] For object detection or instance segmentation tasks, feature maps of the frozen FPN backbone network at multiple scales can be extracted, the differences between features at the corresponding scales can be calculated, and these differences can be used as the loss for the detection or segmentation task.

[0110] In this embodiment, the training loss of the task adaptation and reconstruction network can be expressed as:

[0111] Where x represents the first image, This represents the task-adapted and reconstructed image output by the task-adapted reconstruction network. This indicates a frozen downstream machine vision model. This indicates task-related perceptual distortion constructed based on intermediate features from downstream machine vision models.

[0112] In this embodiment, the human eye perception reconstruction network and the task adaptation reconstruction network can be trained separately using a phased training strategy. Specifically, for the training of the human eye perception reconstruction network: A shared variable bitrate compression backbone and a human eye perception reconstruction network can be trained together.

[0113] The first image is first compressed and initially reconstructed by a shared variable bitrate reversible codec to output the second image, which is then restored for detail and compensated for errors by a human eye perception reconstruction network.

[0114] The optimization objectives of this training phase include a bitrate term and a reconstruction distortion term, aimed at achieving stable multi-bitrate reconstruction performance across different quality levels q. Its loss function can be expressed as:

[0115] Where Rq represents the bitrate estimation term at quality level q, and λq represents the rate-distortion tradeoff parameter corresponding to quality level q. This represents the distortion term between the first image and the perceptually optimized reconstructed image.

[0116] For training task-adaptive reconstruction networks: During this stage, the shared variable bitrate compressed backbone trained on the human eye perception reconstruction network can be frozen, and the downstream machine vision model can also be frozen, with only the training task adapted to the reconstruction network.

[0117] After the first image is processed through a shared compression backbone to obtain the second image, it is input into the task adaptation and reconstruction network to obtain the task adaptation and reconstruction image. Subsequently, the reconstructed image is input into a frozen downstream machine vision model, and the task-adaptive reconstruction network is optimized by addressing task-related feature distortion. Its loss function can be expressed as:

[0118] Thus, through the aforementioned phased training process, we can obtain a human eye perception reconstruction network and a task adaptation reconstruction network that meet actual requirements.

[0119] Table 1 shows the comparison results of the multi-bitrate reconstruction performance of the image processing method of the present invention under different datasets. As can be seen from Table 1, the image processing method of the present invention can output images with different reconstruction quality at multiple bitrate points. That is, through quality level control, multi-scale latent variable gain modulation and visual feature reconstruction at the decoding end, multi-bitrate compression and reconstruction quality adjustment under a single model can be achieved.

[0120]

[0121] Table 1. Performance comparison of multi-bitrate reconstruction on different datasets. Table 2 shows the performance of the image processing method of the present invention on different machine vision tasks. As can be seen from Table 2, under the condition that the shared variable bitrate compression backbone remains unchanged, the task adaptation reconstruction network can output task adaptation reconstruction images that can be used for downstream machine vision model analysis in classification, detection and segmentation tasks. The image processing method of the present invention can maintain the usability of machine vision tasks under multi-bitrate conditions.

[0122]

[0123] Table 2 Performance comparison results for different machine vision tasks This invention also provides an image processing system, see [link to relevant documentation]. Figure 3 The image processing system includes: The determining unit 10 is used to determine the quality level corresponding to the first image based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least the task type.

[0124] A shared variable bitrate reversible image codec 11 is used to encode a first image and its corresponding quality level, and to decode the received first bitstream to generate a second image, wherein the first bitstream includes at least a task type identifier corresponding to the task type.

[0125] The visual feature reconstruction network 12 is used as input to the second image and outputs a perceptually optimized reconstructed image and / or a task-adapted reconstructed image.

[0126] In this embodiment, the visual feature reconstruction network 12 may include a human eye perception reconstruction network 120 and / or a task adaptation reconstruction network 121; The human eye perception reconstruction network 120 is used to generate a perception-optimized reconstruction image that adapts to human visual perception. The task adaptation and reconstruction network 121 is used to generate task adaptation and reconstruction images based on task supervision constraints. The operation of the image processing system in this embodiment of the invention can be referred to the execution of the image processing method described above, and will not be repeated here.

[0127] Image processing method and image processing system according to embodiments of the present invention: First, a quality level corresponding to the first image is determined based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least the task type. Then, the first image and its corresponding quality level are encoded; the received first bitstream is decoded to generate a second image, wherein the first image is encoded and decoded using a shared variable bitrate reversible image codec, and the first bitstream includes at least a task type identifier corresponding to the task type. Finally, the second image is input into a visual feature reconstruction network to output a perceptually optimized reconstructed image and / or a task-adapted reconstructed image. Since the quality level corresponding to the first image is determined based at least on network state parameters and task requirement parameters, the second image is generated based on the first image and its corresponding quality level, and then the second image is reconstructed, the encoding, decoding, and reconstruction of the first image can be tailored to the specific application scenario to output a high-quality image that meets the actual application scenario and requirements. In addition, this solution, taking into account network transmission conditions and task adaptability, precisely controls the amount of data encoded in the first image and efficiently utilizes network bandwidth and computing resources to generate reconstructed images that match different task types. This improves the quality of reconstructed images to a certain extent, while also achieving an adaptive balance between network bandwidth, reconstruction quality, task accuracy, and computational overhead. Furthermore, it enhances the flexibility of the image processing process.

[0128] By employing a shared variable bitrate reversible image codec and a quality level control mechanism, a single model can cover multiple compression bitrate points (high, medium, and low) without requiring separate training, storage, and deployment of independent models for different bitrates. This effectively reduces model redundancy and significantly lowers storage requirements and model maintenance costs during image processing. Furthermore, the shared variable bitrate reversible image codec allows for flexible switching of compression levels, thereby improving the flexibility of image processing.

[0129] Since the second image (which carries a task type identifier) ​​is processed by a visual feature reconstruction network after being output through a shared variable bit rate reversible image codec, in order to generate reconstructed images that meet the needs of different types of tasks, the output forms of image processing are greatly enriched, and the flexibility and scalability of image processing methods are significantly improved. Furthermore, since the visual feature reconstruction network is adapted to different scenarios, or in other words, the selection of the visual feature reconstruction network is determined based on the task type identifier, it can output reconstructed images that meet the actual needs of the scenario (different task types) and have high quality.

[0130] This invention also provides a computer-readable storage medium storing computer-executable instructions thereon, which, when executed by a processor, implement the image processing method described above.

[0131] According to an embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the processor executes a computer program in the memory to implement the steps of the image processing method described above.

[0132] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. An image processing method, characterized by, include: The quality level corresponding to the first image is determined based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least: task type; Encode it based on the first image and its corresponding quality level; The received first bitstream is decoded to generate a second image, wherein the first image is encoded and decoded by a shared variable bitrate reversible image codec, and the first bitstream includes at least: a task type identifier corresponding to the task type; The second image is input into a visual feature reconstruction network to output a perceptually optimized reconstructed image and / or a task-adapted reconstructed image.

2. The image processing method of claim 1, wherein, The encoding based on the first image and its corresponding quality level includes: The first image is subjected to a multi-scale reversible transformation to generate latent variables at multiple scales corresponding to it. Based on the quality level corresponding to the first image, latent variables at multiple scales corresponding to the first image are subjected to gain modulation, quantization, and entropy coding to generate a compressed bitstream.

3. The method of claim 2, wherein, The shared variable bitrate reversible image codec includes: a spatial rearrangement module and an N-level reversible module connected thereto, wherein the N-level reversible modules are cascaded sequentially, and N is a natural number greater than 1. The step of performing a multi-scale reversible transformation on the first image to generate latent variables at multiple scales includes: The first image is input into the spatial rearrangement module, which outputs spatial rearrangement features. The spatial rearrangement features are input into an N-level reversible module that is cascaded in series to output multiple latent variables corresponding to the first image. The input of the first-level invertible module is the spatial rearrangement feature, and the output includes: the latent variables corresponding to the current scale and the hidden representation of the current scale; The input of any other non-final-level invertible module is the hidden representation of the output of the next-level invertible module. The output includes: the latent variable corresponding to the current level scale and the hidden representation of the current level. The input to the final-level invertible module is the hidden representation of the output of the previous-level invertible module. The output includes two latent variables at different scales.

4. The method of claim 2, wherein, The step of performing gain modulation, quantization, and entropy coding on latent variables at multiple scales corresponding to the first image based on the quality level corresponding to the first image to generate a compressed bitstream includes: For each scale of latent variable, obtain the residual of the latent variable at that scale, where the residual of the latent variable at that scale is obtained based on the latent variable at that scale and its corresponding mean parameter; Based on the quality level, determine the corresponding gain parameter to modulate the latent variable residual at that scale, and obtain the modulated latent variable residual at that scale. The modulated latent variable residual at this scale is quantized to obtain the scale residual quantization value, and the scale residual quantization value is entropy encoded based on the scale parameter corresponding to the latent variable at this scale to generate a compressed bitstream. The mean and scale parameters of the latent variables at each scale are obtained based on entropy model prediction.

5. The image processing method of claim 4, wherein, The first bitstream includes the compressed bitstream and a task type identifier. Decoding the received first bitstream to generate the second image includes: The compressed bitstream is parsed to obtain the quality level; The compressed bitstream is entropy decoded to obtain the quantized multi-scale latent variable residuals; Based on the quality level, the quantized multi-scale latent variable residuals are sequentially inversely quantized and inversely gained mapped, and the mean parameter corresponding to the multi-scale latent variable is compensated to obtain the multi-scale latent variable. The multi-scale latent variables are subjected to a stepwise inverse reversible transformation in the reverse order of the multi-scale decomposition at the encoding end to generate a second image.

6. The method according to claim 1, characterized in that, The task types include: human visual tasks and / or machine visual tasks, and the visual feature reconstruction network includes: a human perception reconstruction network matched with human visual tasks and / or a task adaptation reconstruction network matched with machine visual tasks; wherein: The human eye perception reconstruction network is used to generate perception-optimized reconstruction images that adapt to human visual perception. The task adaptation and reconstruction network is used to generate task adaptation and reconstruction images based on task supervision constraints.

7. The method according to claim 6, characterized in that, If the task type is a human visual task, the quality level corresponding to the first image is determined based on the network status score and the task requirement score. If the task type is a machine vision task, the quality level corresponding to the first image is determined based on the network status score, task requirement score, and machine vision task suitability score. Among them, the network state score is associated with the network state parameters, the task requirement score is associated with the task requirement parameters, and the machine task fit score is associated with the evaluation metrics of the machine vision model output image.

8. The method of claim 7, wherein, The method further includes: if the evaluation index of the image output by the machine vision model does not reach the preset value, then adjusting the quality level corresponding to the first image.

9. The method of claim 6, wherein, The human eye perception reconstruction network and / or the task adaptation reconstruction network include multiple ConvNeXt V2 Blocks connected in series.

10. An image processing system, characterized by include: The determining unit is configured to determine the quality level corresponding to the first image based at least on network state parameters and task requirement parameters, wherein the task requirement parameters include at least: task type; A shared variable bitrate reversible image codec is used to encode a first image and its corresponding quality level, and to decode the received first bitstream to generate a second image, wherein the first bitstream includes at least: a task type identifier corresponding to the task type; A visual feature reconstruction network is used as input to the second image and outputs a perceptually optimized reconstructed image and / or a task-adapted reconstructed image.

Images