Machine vision-oriented image encoding, decoding and compression method and system

Abstract

The application discloses a kind of machine vision-oriented image encoding, decoding and compression method and system, comprising: obtaining the first feature map of the image to be encoded by space-frequency modulation coding network;Each feature value in the first feature map is quantized to integer, and the second feature map is obtained;The distribution parameterization modeling of the second feature map is modeled as Gaussian distribution, and the mean and variance of Gaussian distribution are predicted using entropy model;According to the mean and variance of the predicted Gaussian distribution, the probability of the features of second feature map is estimated, and the arithmetic coding is carried out according to the estimated probability, to obtain the compressed image binary code stream.The application can efficiently extract the information required for the downstream task of machine vision, reduce the redundant information for machine vision, reduce the image compression code rate overhead while effectively improving the accuracy of downstream tasks, and can be applied to any existing end-to-end image compression and decoding network and entropy model network.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and in particular relates to a machine vision-oriented image encoding, decoding and compression method and system in an end-to-end image compression method. BACKGROUND

[0002] Recently, the number of images used for machine vision (such as image classification, object detection and instance segmentation, etc.) is increasing rapidly. In theory, the nonlinear transformation in the deep learning-based image compression model provides more adaptive quantization units for multi-dimensional quantizers, which can overcome the construction limitations of traditional transform coding schemes. However, previous deep learning-based image compression models are usually oriented towards human visual vision, and cannot meet the new scenario of image storage and transmission for machine vision. Therefore, in order to reduce the code rate after image coding and improve the accuracy of downstream tasks for machine vision, machine vision-oriented image compression has become a new important topic.

[0003] Through literature search of the prior art, it is found that Hyomin Choi et al. published "Scalable Image Coding for Humans and Machines" in IEEE Transactions on Image Processing in 2022, which adopts a scalable coding method to truncate the code stream, so that different code streams can correspond to different tasks, such as human visual vision and machine visual vision. However, this method requires the simultaneous training of multiple tasks and multiple decoders, greatly increasing the burden of training and model storage.

[0004] In order to solve this problem, recent research has adopted prompt learning and Transformer, such as Yi-Hsin Chen et al. published "TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception" in Proceedings of the IEEE / CVF International Conference on Computer Vision in 2023, which adds prompts to the encoding and decoding end to migrate the pre-trained compression model for human visual vision, so that it can be used for machine visual vision-oriented image compression, achieving higher efficiency in training cost and model storage.

[0005] However, the method based on prompt learning can only be used for a Transformer-based end-to-end image compression model, and still introduces significant additional computational overhead, and the performance of the downstream task is still poor, and the practicability is weak. SUMMARY

[0006] The present application aims at the above problems, and proposes an image encoding, decoding and compression method and system for machine vision, which can efficiently extract more needed information for machine vision, while reducing the information redundant for machine vision, can reduce the image compression code rate overhead while effectively improving the accuracy of the downstream task, and can also be applied to any existing end-to-end image compression and decoding network and entropy model.

[0007] In a first aspect, the present application provides an image encoding method for machine vision, comprising:

[0008] obtaining a first feature map of the image to be encoded through a spatial-frequency modulation encoding network, wherein the first feature map is a three-dimensional feature map, and the dimensions include a channel dimension, a height dimension and a width dimension;

[0009] quantizing each feature value in the first feature map into an integer to obtain a second feature map;

[0010] parameterizing the distribution of the second feature map as a Gaussian distribution, and predicting the mean and variance of the Gaussian distribution using an entropy model;

[0011] performing probability estimation on the features of the second feature map according to the predicted mean and variance of the Gaussian distribution, and performing arithmetic coding according to the estimated probability to obtain a compressed image binary code stream.

[0012] Optionally, the step of obtaining the first feature map of the image to be encoded through the spatial-frequency modulation encoding network comprises:

[0013] The spatial-frequency modulation encoding network is composed of a plurality of spatial-frequency modulation encoding units, the input of the first spatial-frequency modulation encoding unit is the image to be encoded, the input of the i-th spatial-frequency modulation encoding unit is the output of the i-1-th spatial-frequency modulation encoding unit The output of the last spatial-frequency modulation encoding unit is the first feature map of the image to be encoded;

[0014] The spatial-frequency modulation encoding unit is composed of a basic encoding subunit, a first spatial domain modulation subunit and a first frequency domain modulation subunit, wherein:

[0015] For the i-th spatial-frequency modulation encoding unit, the input is Through the basic encoding subunit, a basic encoding feature map x j is obtained.

[0016] The basic encoding feature map xj As the input of the first spatial modulation subunit, the first spatial modulation feature map SMA is obtained j (x j );

[0017] The basic coding feature map x j As the input of the first frequency domain modulation subunit, the first frequency domain modulation feature map FMA is obtained j (x j );

[0018] Combine the basic encoding feature map x j , the first spatial modulation feature map SMA j (x j ) and the first frequency domain modulation feature map FMA j (x j ), obtain the first space-frequency modulation characteristic diagram As the output of the jth space-frequency modulation coding unit.

[0019] Optionally, the basic coding feature map x j As the input of the first spatial modulation subunit, the first spatial modulation feature map SMA is obtained j (x j ),include:

[0020] The basic encoding feature map x j Input coefficient is The first linear layer of the transformed spatial domain features is obtained

[0021] Will The input coefficient is The second linear layer is used, and the output of the second linear layer is the first depth-separable convolution layer DW_Conv5×5 to obtain the spatial modulation matrix

[0022] Will Through the first ReLU nonlinear mapping layer σ, for For each element of , if the value is greater than 0, Returns the same value, otherwise returns 0;

[0023] Will and Perform Hada code multiplication to multiply the elements of the corresponding positions of the two vectors to obtain the spatial domain modulation result

[0024] Will The input coefficient is The third linear layer obtains the spatial modulation feature map SMA j (x j).

[0025] Optionally, the basic coding feature map x j As the input of the first frequency domain modulation subunit, the first frequency domain modulation feature map FMA is obtained j (x j ),include:

[0026] The basic encoding feature map x j Input coefficient is The fourth linear layer is used and Fourier transform is performed to obtain frequency domain features

[0027] Will It passes through the second depth-wise separable convolution layer DW_Conv3×3, the second ReLU nonlinear mapping layer, and another coefficient The fifth linear layer obtains the frequency domain modulation matrix

[0028] Will Through the third ReLU nonlinear mapping layer, and Perform Hada code multiplication to obtain the frequency domain modulation result

[0029] right Perform inverse Fourier transform and pass the coefficients The sixth linear layer is used to obtain the frequency domain modulation feature map.

[0030] A second aspect of the present invention provides a machine vision-oriented image decoding method for decoding the code obtained by the above-mentioned image encoding method, comprising:

[0031] Obtain compressed image binary code stream;

[0032] Use the entropy model network to predict the mean and variance of the Gaussian model for the second feature map;

[0033] Performing arithmetic decoding on the feature binary code stream according to the mean and variance of the Gaussian model to obtain the second feature map;

[0034] The second feature map is passed through a space-frequency modulation decoding network to obtain a reconstructed image for downstream tasks oriented to machine vision.

[0035] Optionally, the space frequency modulation decoding network has the same structure as the space frequency modulation encoding network. The space frequency modulation decoding network is composed of multiple space frequency modulation decoding units, the input of the first space frequency modulation decoding unit is the image to be decoded, and the input of the jth space frequency modulation decoding unit is the output of the j-1th space frequency modulation decoding unit. The output of the last spatial-frequency modulation decoding unit is the first feature map of the image to be decoded.

[0036] The spatial-frequency modulation decoding unit is composed of a basic decoding subunit, a second spatial modulation subunit and a second frequency modulation subunit.

[0037] For the jth spatial-frequency modulation decoding unit, the input is passed through the basic decoding subunit to obtain a basic decoding feature map.

[0038] The basic decoding feature map is taken as the input of the second spatial modulation subunit to obtain a second spatial modulation feature map.

[0039] The basic decoding feature map is taken as the input of the second frequency modulation subunit to obtain a second frequency modulation feature map.

[0040] The basic decoding feature map, the second spatial modulation feature map and the second frequency modulation feature map are combined to obtain a second spatial-frequency modulation feature map as the output of the jth spatial-frequency modulation decoding unit.

[0041] In a third aspect, the present application provides an image compression method for machine vision, comprising:

[0042] Obtaining an image to be compressed;

[0043] Based on the above-mentioned image encoding method, a compressed image binary code stream is obtained from the image to be encoded.

[0044] Based on the above-mentioned image decoding method, a reconstructed image for machine vision tasks is obtained from the compressed image binary code stream.

[0045] In a fourth aspect, the present application provides an image compression system, which comprises an encoder and a decoder, comprising:

[0046] The encoder: based on the above-mentioned image encoding method, a compressed image binary code stream is obtained from the image to be encoded.

[0047] The decoder: based on the above-mentioned image decoding method, a reconstructed image for machine vision tasks is obtained from the compressed image binary code stream.

[0048] In a fifth aspect, the present application provides an image compression device, which comprises a memory and a processor, the memory stores a program executable by the processor, and the processor can execute the following methods when executing the program:

[0049] - the above-mentioned image encoding method for machine vision, or

[0050] - the above-mentioned image decoding method for machine vision, or

[0051] - the machine vision oriented image compression method described above.

[0052] Compared with the prior art, the embodiment of the present application has at least one of the following beneficial effects:

[0053] The machine vision oriented image encoding, decoding and compression method described above of the present application realizes feature optimization by using space-frequency modulation, and can significantly reduce information redundancy in the spatial and frequency domains.

[0054] The machine vision oriented image encoding, decoding and compression method described above of the present application, compared with the latest method, can improve the accuracy of the reconstructed image for image classification tasks by 30% under the premise of the same code rate overhead, and improves the rate task accuracy performance.

[0055] The machine vision oriented image compression method of the embodiment of the present application can be applied to any existing end-to-end learning image compression codec and entropy encoder, has good flexibility and generalization, and improves the accuracy of the reconstructed image for machine vision oriented downstream tasks. The present application has very strong practical application value, can be applied to different machine vision oriented downstream tasks, and has smaller training cost and model storage overhead. BRIEF DESCRIPTION OF DRAWINGS

[0056] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0057] Figure 1 The flowchart of the machine vision oriented image encoding method in an embodiment of the present application.

[0058] Figure 2 The flowchart of the machine vision oriented image encoding method in a preferred embodiment of the present application.

[0059] Figure 3 The flowchart of the machine vision oriented image decoding method in a preferred embodiment of the present application.

[0060] Figure 4 The module diagram of the machine vision oriented image compression in an embodiment of the present application.

[0061] Figure 5 The flowchart of the machine vision oriented image compression method in a preferred embodiment of the present application. DETAILED DESCRIPTION

[0062] The application will be described in a clear and complete manner below with reference to specific embodiments and drawings. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be pointed out that those skilled in the art can make several modifications and improvements without departing from the concept of the application. These all belong to the protection scope of the application.

[0063] Referring to Figure 1 Fig. 1 shows a flowchart of a machine vision-oriented image encoding method in an embodiment of the application. The embodiment provides a machine vision-oriented image encoding method, which comprises steps S100-S400. Specifically:

[0064] S100, obtaining a first feature map of a to-be-encoded image through a spatial-frequency modulation encoding network;

[0065] In this step, the to-be-encoded image is first determined, and then the first feature map of the to-be-encoded image is obtained through the spatial-frequency modulation encoding network. Specifically, the spatial-frequency modulation encoding network is composed of multiple spatial-frequency modulation encoding units for extracting the first feature map; the spatial-frequency modulation encoding unit is composed of a basic encoding subunit, a spatial domain modulation subunit and a frequency domain modulation subunit; preferably, the basic encoding subunit can be realized by a cascade of a convolution layer with down-sampling and a nonlinear unit.

[0066] In this step, the obtained first feature map is a three-dimensional feature map, and the three-dimensional dimensions include a channel dimension, a height dimension and a width dimension.

[0067] S200, quantizing each feature value in the first feature map into an integer to obtain a second feature map;

[0068] S300, parameterizing each element of the second feature map into a Gaussian distribution, and using an entropy model network to predict the mean and variance of the Gaussian distribution;

[0069] Each element of the second feature map is each feature value of the second feature map, and the probability distribution of each element is the probability distribution corresponding to each element in the second feature map, and each element corresponds to a Gaussian distribution.

[0070] In this step, the entropy model can adopt any existing entropy model, such as a hyper-prior entropy model, a spatial autoregressive entropy model and a channel autoregressive entropy model, etc. Thus, the entire method can be applied to any existing end-to-end image compression and decoding network and entropy model.

[0071] Preferably, the entropy model network can be realized by a hyper-prior network, such as sequentially passing the second feature map through a hyper-encoding neural network to obtain a hyper-prior feature binary code stream, and further passing the hyper-prior feature binary code stream through a hyper-decoding neural network to obtain hyper-prior information.

[0072] S400, performing probability estimation on the features of the second feature map according to the mean and variance of the predicted Gaussian distribution, performing arithmetic coding according to the estimated probability to obtain a feature binary code stream.

[0073] In the above embodiments, the space-frequency modulation coding network can efficiently extract the more needed information of machine vision, while reducing the redundant information for machine vision, thereby reducing the image compression code rate overhead and effectively improving the accuracy of downstream tasks.

[0074] Of course, the above embodiments also do not require strict adherence to the above hyper-prior entropy model and the basic coding unit based on the convolution kernel nonlinear unit, such as using a basic coding unit based on Transformer and an entropy model based on autoregression, which can also achieve the purpose of the present application.

[0075] On the basis of the above embodiments, in order to better reduce information redundancy in the spatial and frequency domains, the space-frequency modulation coding unit is composed of a basic coding subunit, a spatial domain modulation subunit, and a frequency domain modulation subunit. The input of the first space-frequency modulation coding unit is the image to be encoded, and the input of the jth space-frequency modulation coding unit is the output of the (j-1)th space-frequency modulation coding unit. .

[0076] In some preferred embodiments, the space-frequency modulation coding unit performs the following operations:

[0077] For the jth space-frequency modulation coding unit, the input is Through the basic coding subunit, the basic coding feature map x j is obtained.

[0078] The basic coding feature map x j is input into the spatial domain modulation subunit to obtain the spatial domain modulation feature map SMA j (x j ).

[0079] The basic coding feature map x j is input into the frequency domain modulation subunit to obtain the frequency domain modulation feature map FMA j (x j ).

[0080] The basic coding feature map x j , the spatial domain modulation feature map SMA j (x j ), and the frequency domain modulation feature map FMA j (x j ) are combined to obtain the space-frequency modulation feature map as the output of the jth space-frequency modulation coding unit.

[0081] In some preferred embodiments, the base encoding feature map x j As the input of the spatial modulation subunit, the spatial modulation feature map SMA j (x j ) is obtained, including:

[0082] The base encoding feature map x j is input into a linear layer with a coefficient , to obtain the transformed spatial feature The formula is:

[0083]

[0084] The is input into another linear layer with a coefficient , and the output of the linear layer is input into a depth separable convolution layer DW_Conv 5×5 , to obtain the spatial modulation matrix The formula is:

[0085]

[0086] The kernel size of the above depth separable convolution layer DW_Conv 5×5 is preferably 5x5, and of course, in other embodiments, it can also be other sizes.

[0087] The is mapped through a ReLU nonlinear mapping layer σ, for each element of , if the value is greater than 0, the same value is returned, otherwise 0 is returned.

[0088] The is Hadamard multiplied with , the elements of the corresponding positions of the two vectors are multiplied, to obtain the spatial modulation result The formula is:

[0089]

[0090] Wherein, represents Hadamard multiplication;

[0091] The is input into another linear layer with a coefficient , to obtain the spatial modulation feature map SMA j (x j ), the formula is:

[0092]

[0093] The base encoding feature map x j ​As the input of the frequency domain modulation subunit, the frequency domain modulation feature map FMA(x j (x j ), including:

[0094] The base encoding feature map x j is input into a linear layer with a coefficient of , and a Fourier transform is performed to obtain the frequency domain feature The formula is:

[0095]

[0096] Wherein, is the Fourier transform;

[0097] The is sequentially passed through a deep separable convolution layer DW_Conv 3×3 , a ReLU nonlinear mapping layer σ, and another linear layer with a coefficient of , to obtain the frequency domain modulation matrix The formula is:

[0098]

[0099] The kernel size of the above deep separable convolution layer DW_Conv 3×3 is preferably 3*3, and of course, in other embodiments, it can also be other sizes.

[0100] The is passed through a ReLU nonlinear mapping layer σ, and is multiplied by by Hadamard multiplication, to obtain the frequency domain modulation result The formula is:

[0101]

[0102] The inverse Fourier transform is performed on , and another linear layer with a coefficient of is passed through, to obtain the frequency domain modulation feature map FMA(x j (x j ), the formula is:

[0103]

[0104] Wherein, is the inverse Fourier transform.

[0105] In the above embodiments of the application, for the jth space-frequency modulation encoding unit, the input is passed through a base encoding subunit to obtain the base encoding feature map x jThe base encoding subunit can be an encoding unit of any existing end-to-end image compression method, including but not limited to a convolutional layer, a residual block, and a Transformer block. Therefore, the embodiment of the present application can be applied to any existing end-to-end image compression and decoding network and entropy model network.

[0106] Figure 2 A flowchart of the machine vision-oriented image encoding method in a preferred embodiment of the present application. In the preferred embodiment, the machine vision-oriented image encoding can include steps S101-S108, specifically:

[0107] S101, three-dimensional feature extraction: the to-be-encoded image obtains a feature map through the spatial-frequency modulation encoding network, the feature map is a three-dimensional feature, and the three dimensions include a channel dimension, a height dimension, and a width dimension; the spatial-frequency modulation encoding network can refer to the detailed description in the above embodiment.

[0108] S102, hyper-prior feature extraction and quantization: a hyper-encoding neural network is used to obtain hyper-prior features from the three-dimensional features; the hyper-prior features are rounded and quantized into integers;

[0109] S103, hyper-prior feature encoding: a distribution estimation of independent features is established for the hyper-prior features, and arithmetic encoding is performed on the hyper-prior features using the distribution estimation to obtain hyper-prior feature binary code streams;

[0110] S104, hyper-prior feature decoding: arithmetic decoding is performed on the hyper-prior feature binary code streams to obtain recovered hyper-prior features; a hyper-decoding neural network is used to obtain hyper-prior information from the recovered hyper-prior features;

[0111] S105, quantization: the three-dimensional features are rounded and quantized into integers;

[0112] S106, probability entropy modeling: each feature distribution of the three-dimensional features is modeled as a Gaussian distribution, and the hyper-prior information is used to predict the mean and variance of the Gaussian distribution;

[0113] S107, three-dimensional feature encoding: the mean and variance of the predicted Gaussian distribution are used to estimate the probability of each three-dimensional feature, and arithmetic encoding is performed according to the estimated probability to obtain feature binary code streams of the three-dimensional features;

[0114] S108, code stream synthesis: the feature binary code streams are further combined with the hyper-prior feature binary code streams to obtain compressed image binary code streams.

[0115] In the above embodiment of the present application, through the optimization of the characteristics of spatial frequency modulation, the more needed information of machine vision can be efficiently extracted, while the redundant information for machine vision is reduced, the precision of downstream tasks can be effectively improved while reducing the image compression code rate overhead. The parts not described in detail above can be referred to Figure 1 the description in the above embodiment.

[0116] Corresponding to the above-mentioned machine vision-oriented image encoding method, another embodiment of the present application provides a machine vision-oriented image decoding method for decoding the encoding obtained by the above-mentioned machine vision-oriented image encoding method. Specifically, the machine vision-oriented image decoding method comprises:

[0117] M101, obtaining a compressed image binary code stream;

[0118] M102, using an entropy model network to predict the mean and variance of the Gaussian model for the second feature map;

[0119] M103, arithmetically decoding the feature binary code stream according to the mean and variance of the Gaussian model to obtain the second feature map;

[0120] M104, passing the second feature map through a spatial frequency modulation decoding network to obtain a reconstructed image for a machine vision-oriented downstream task.

[0121] The above-mentioned machine vision-oriented image decoding method, the spatial frequency modulation decoding network has the same structure as the spatial frequency modulation encoding network, and the specific implementation techniques are also consistent.

[0122] For example, the spatial domain modulation subunit and the frequency domain modulation subunit of the spatial frequency modulation decoding network are completely consistent with the spatial frequency modulation encoding network. Specifically, the spatial frequency modulation decoding network is composed of multiple spatial frequency modulation decoding units, the input of the first spatial frequency modulation decoding unit is the image to be decoded, and the input of the jth spatial frequency modulation decoding unit is the output of the (j-1) th spatial frequency modulation decoding unit The output of the last spatial frequency modulation decoding unit is the first feature map of the image to be decoded.

[0123] Further, the spatial frequency modulation decoding unit is composed of a basic decoding subunit, a spatial domain modulation subunit and a frequency domain modulation subunit, wherein: for the jth spatial frequency modulation decoding unit, the input is passed through the basic decoding subunit to obtain a basic decoding feature map; the basic decoding feature map is taken as the input of the spatial domain modulation subunit to obtain a spatial domain modulation feature map; the basic decoding feature map is taken as the input of the frequency domain modulation subunit to obtain a frequency domain modulation feature map; the basic decoding feature map, the spatial domain modulation feature map and the frequency domain modulation feature map are combined to obtain a spatial frequency modulation feature map as the output of the jth spatial frequency modulation decoding unit.

[0124] For details not described in detail in the above space-frequency modulation decoding unit and its basic decoding subunit, space-domain modulation subunit, and frequency-domain modulation subunit, please refer to Figure 1 The corresponding description of the space-frequency modulation coding network in the illustrated embodiment will not be repeated here.

[0125] Figure 3 FIG. 1 is a flow chart of a machine vision-oriented image decoding method in a preferred embodiment of the present invention. Figure 3 As shown, in this preferred embodiment, the image decoding method includes steps S201-S205:

[0126] S201, code stream decomposition: decomposing the compressed image binary code stream to obtain a feature binary code stream and a super-prior feature binary code stream;

[0127] S202, super-prior feature decoding: performing arithmetic decoding on the super-prior feature binary code stream to obtain restored super-prior features; applying a super-decoding neural network to the restored super-prior features to obtain super-prior information;

[0128] S203, Probabilistic Entropy Modeling: The structure and parameter values ​​of this module are similar to Figure 2 The probability entropy modeling in the preferred embodiment is exactly the same, wherein the mean and variance of the Gaussian model used to decode each feature in the second feature map can be predicted by using super prior information.

[0129] S204, three-dimensional feature decoding: performing arithmetic decoding on the feature binary code stream according to the mean and variance of the Gaussian model to obtain quantized three-dimensional features.

[0130] S205, image reconstruction: The reconstructed quantized three-dimensional features are decoded through a space-frequency modulation network to obtain a reconstructed image for downstream tasks oriented to machine vision.

[0131] The machine vision-oriented image decoding method of this embodiment is used to correspond to Figure 2 The image coding method for machine vision shown in the figure uses super-prior information to predict the mean and variance of the Gaussian model; the three-dimensional features are passed through the space-frequency modulation decoding network to obtain a reconstructed image for downstream tasks for machine vision, effectively improving the accuracy of downstream tasks.

[0132] Based on the above-mentioned image encoding method and decoding method for machine vision, in another embodiment of the present invention, a method for image compression for machine vision is further provided, comprising:

[0133] M201, obtain the image to be compressed;

[0134] M202, based on the image encoding method of the above embodiment, obtaining a compressed image binary code stream from the image to be encoded;

[0135] M203, based on the image decoding method of the above-mentioned embodiments, obtaining a reconstructed image for a machine vision task from the compressed image binary code stream.

[0136] The image encoding method and the image decoding method of the above-mentioned embodiments M201-M203 are based on a spatial-frequency modulation encoding network and a spatial-frequency modulation decoding network, both of which include a spatial modulation subunit, a frequency modulation subunit, and a basic encoding subunit or a basic decoding subunit. The basic decoding unit and the basic decoding subunit can be the decoding unit of any existing end-to-end image compression method, including but not limited to convolutional layers, residual blocks, and Transformer blocks. In addition, the mean and variance of the Gaussian distribution are predicted using an entropy model. The entropy model can be any entropy model, such as a hyper-prior entropy model, a spatial autoregressive entropy model, and a channel autoregressive entropy model.

[0137] In some embodiments, the parameters of the basic encoding subunit in the image encoding method, the basic decoding subunit in the image decoding method, and the entropy model are obtained through rate-distortion optimization:

[0138]

[0139] wherein the distortion x is the image to be compressed, is the reconstructed image, λ1>0 is a preset parameter for balancing distortion and code rate, and the code rate R1 is obtained according to the Gaussian distribution of each element (each feature value) of the second feature map:

[0140]

[0141] wherein, is the feature of the kth element, is the predicted distribution of the kth element, θ k ={φ k ,ψ k} is a set of learnable parameters.

[0142] Optionally, the parameters of the first spatial modulation subunit and the first frequency modulation subunit in the image encoding method and the second spatial modulation subunit and the second frequency modulation subunit in the image decoding method are obtained through optimization of code rate and task accuracy:

[0143]

[0144] wherein, is a loss function related to the accuracy of the downstream task, λ2>0 is a preset parameter for balancing distortion and task accuracy, and the code rate R2 is calculated in the same way as the code rate R1.

[0145] Corresponding to the above-mentioned machine vision-oriented image compression method, another embodiment of the present application also provides a machine vision-oriented image compression device, referring to Figure 4 According to the above-mentioned machine vision-oriented image compression method, another embodiment of the present application also provides a machine vision-oriented image compression device, referring to

[0146] S301 Acquisition module: sequentially obtain the to-be-compressed image;

[0147] S302 Encoder: encode the to-be-compressed image according to steps S101-S108 to obtain a compressed image binary code stream.

[0148] For example, the space-frequency modulation coding network is composed of 4 basic coding subunits and 3 space domain modulation subunits and 3 frequency domain modulation subunits which are alternately cascaded, the channel number of the extracted three-dimensional feature is 192, the height is 16, and the width is 16.

[0149] S303 Decoder: decompress the compressed image binary code stream according to S201-S205 to obtain a reconstructed image used for a downstream image classification task.

[0150] Wherein, the channel number, height and width of the three-dimensional feature are exactly the same as the encoder. The space-frequency modulation decoding network is composed of 4 basic decoding units and 3 space domain modulation subunits and 3 frequency domain modulation subunits which are alternately cascaded.

[0151] The compression method in the above-mentioned embodiment of the present application is realized based on the machine vision-oriented image coding method shown in Figure 1 、 2 and the machine vision-oriented image coding method shown in Figure 3 The above-mentioned parts not described in detail can refer to the corresponding description in the above-mentioned embodiment, and will not be repeated here.

[0152] The compression method in the above-mentioned embodiment of the present application can efficiently extract the information more needed by machine vision, reduce the redundant information for machine vision, reduce the image compression code rate overhead, and effectively improve the accuracy of the downstream task.

[0153] Based on the same technical concept of the above-mentioned machine vision-oriented image compression device, in other embodiments of the present application, an image compression method is provided, comprising:

[0154] Acquiring a to-be-compressed image;

[0155] Obtaining a compressed image binary code stream by using the machine vision-oriented image coding method in any one of the above-mentioned embodiments;

[0156] The compressed image binary code stream is decompressed by using the machine vision-oriented image decoding method in any of the above embodiments to obtain a reconstructed image for a machine vision-oriented downstream task.

[0157] For the above image compression method, the compression effect can be improved by a model training method. For example, the training set is composed of one million natural images in the ImageNet-train dataset, and the model after training can be used for machine vision-oriented image encoding and decoding.

[0158] Specifically, as shown in Figure 5 The image compression method includes:

[0159] S401, three-dimensional feature extraction: the images in the training set are randomly cropped into images with a resolution size of 256x256, and three-dimensional features are obtained by inputting the images into a space-frequency modulation coding network, wherein: the space-frequency modulation coding network is composed of a basic coding subunit, a combination of a space domain modulation subunit and a frequency domain modulation subunit, a basic coding unit, a combination of a space domain modulation subunit and a frequency domain modulation subunit, a combination of a space domain modulation subunit and a frequency domain modulation subunit, and a basic coding unit are sequentially connected; wherein the basic coding subunit is composed of a convolution layer with down-sampling and a ReLU non-linear unit; the number of three-dimensional feature channels is 192, and the height and width are both 16; of course, the image resolution size and the three-dimensional feature parameters in the above embodiment can be adjusted according to actual conditions, and are not limited to the description in this step, and the parameters in the following steps are also the same. Correspondingly, the basic coding subunit in the space-frequency modulation coding network can also use other network structures, such as Transformer and residual block, and is not limited to the basic coding unit in this step.

[0160] S402, hyper-prior feature extraction and noise addition: inputting the three-dimensional features into a hyper-prior coding neural network to obtain hyper-prior features, to avoid the gradient being zero everywhere caused by quantization operation, adding noise uniformly distributed from -0.5 to 0.5 to the hyper-prior features to simulate actual quantization, wherein: the hyper-prior coding neural network is composed of three convolution layers; of course, the hyper-prior coding neural network can also be other network structure forms, such as three residual modules in cascade, and is not limited to the hyper-prior coding neural network in this step.

[0161] S403, hyper-prior feature code rate estimation: a distribution estimation independent of each feature is established for the hyper-prior features, and a hyper-prior feature code rate estimation is obtained by using the distribution estimation;

[0162] S404, super-prior information acquisition: input the super-prior feature added with noise into a super-prior decoding neural network to obtain super-prior information, wherein the super-prior decoding neural network is composed of three convolutional layers in cascade;

[0163] S405, feature noise addition: to avoid the gradient being 0 everywhere caused by the quantization operation, -0.5 to 0.5 uniformly distributed noise is added to the three-dimensional feature to simulate the actual quantization;

[0164] S406, Gaussian distribution probability modeling: the structure of this module is completely the same as that of the probability entropy modeling in the embodiment shown in Figure 2 ; the super-prior information is used to predict the mean and variance of the Gaussian distribution; of course, the entropy model network can also be in other forms, such as a spatial context autoregressive model and a reconstruction context autoregressive model.

[0165] S407, feature code rate estimation: the mean and variance of the predicted Gaussian distribution are used to obtain the feature code rate estimation;

[0166] S408, total code rate estimation: the super-prior feature code rate estimation and the feature code rate estimation are added to obtain the total code rate estimation;

[0167] S409, image reconstruction: input the three-dimensional feature added with noise into a decoding neural network to obtain a reconstructed image, wherein the spatial-frequency modulation decoding network is composed of a basic encoding subunit, a combination of a spatial modulation subunit and a frequency modulation subunit, a basic encoding subunit, a combination of a spatial modulation subunit and a frequency modulation subunit, a basic encoding subunit, a combination of a spatial modulation subunit and a frequency modulation subunit, and a basic encoding subunit in cascade; wherein the basic encoding subunit is composed of a convolutional layer with upsampling and a ReLU nonlinear unit in cascade; in some possible embodiments, the above-mentioned machine vision-oriented image compression method can be used as an image compression model, which can be further optimized in stages by using the stochastic gradient descent method, in the first stage, the spatial modulation subunit and the frequency modulation subunit are disabled by short connection, and only the basic encoding subunit, the basic decoding subunit and the entropy encoder are optimized. The loss function of the first stage is:

[0168]

[0169] wherein the distortion x is the image to be compressed, is the reconstructed image, λ1>0 is a preset parameter for balancing the distortion and the code rate, and the code rate R1 is obtained according to the Gaussian distribution of each element of the second feature map:

[0170]

[0171] wherein, is the feature of the kth element, is the predicted distribution of the kth element, θ k ={φ k , ψ k} is a set of learnable parameters.

[0172] In the second stage, the base encoding subunit and the base decoding subunit and the entropy encoder are fixed, and the spatial domain modulation subunit and the frequency domain modulation subunit are inserted into the codec, and the parameters thereof are optimized by the following formula:

[0173]

[0174] wherein, is a loss function related to the accuracy of the downstream task, λ2>0 is a preset parameter for balancing distortion and task accuracy, and the code rate R2 is calculated in the same way as the code rate R1 in the first stage.

[0175] In another embodiment of the present application, an image compression system is provided, which comprises an encoder and a decoder, comprising:

[0176] The encoder: based on the image encoding method in any of the above embodiments, obtaining a compressed image binary code stream from the image to be encoded;

[0177] The decoder: based on the image decoding method in any of the above embodiments, obtaining a reconstructed image for machine vision tasks from the compressed image binary code stream.

[0178] It should be noted that the above method and system can correspond to each other. Since the principles of the method and system are similar, the implementation process of the system and the method can be mutually referenced, and will not be repeated here.

[0179] In another embodiment of the present application, an image compression device is provided, which comprises a memory and a processor, the memory stores a program executable by the processor, and the processor can execute the following method when executing the program:

[0180] - the image encoding method for machine vision in any of the above embodiments, or

[0181] - the image decoding method for machine vision in any of the above embodiments, or

[0182] - the image compression method for machine vision in any of the above embodiments.

[0183] Implementation effect:

[0184] In the above embodiment, the image compression model for machine vision is trained using the ImageNet dataset, and in the first stage, lambda is set to 0.0018, 0.0035, 0.0067, and 0.0130 respectively corresponding to four different code rates of the image compression model, and the Adam optimizer is used for 2000000 step optimization, and the learning rate is set to 10 -4 in the first 1800000 steps, then reduced to 3*10 -5 , and finally reduced to 10 -5 at 1950000 steps. In the second stage, lambda is set to 0.0018, 0.0035, 0.0067, and 0.0130 respectively corresponding to four different code rates, and the Adam optimizer is used for 500000 step optimization, and the learning rate is set to 10 -4 .

[0185] In the above embodiment, the image encoder of the six trained image compression models for machine vision is used to compress 50000 images with a resolution of 256*256 in the ImageNet-val test set. The average code rate of the compressed images is 0.0774bpp, 0.1306bpp, 0.2068bpp, and 0.339bpp respectively. In the above embodiment, the image encoder of the four trained image compression models for machine vision is used to compress 50000 images with a resolution of 256*256 in the ImageNet-val test set to obtain the reconstructed images for the vision task. The reconstructed images are used for the downstream image classification task based on ResNet-50, and the classification accuracy is 64.50%, 69.59%, 72.44%, and 74.59% respectively.

[0186] Compared with the method using the benchmark neural network, the image compression method for machine vision provided in the above embodiment of the present application has better rate-distortion performance. Specifically, under the premise of the same code rate overhead, the accuracy of the reconstructed image for the image classification task can be improved by 30%, and the rate task accuracy performance is improved.

[0187] The image compression method for machine vision in the embodiment of the present application can be applied to any existing end-to-end learning image compression codec and entropy encoder, has good flexibility and generalization, and improves the accuracy of the reconstructed image for the downstream task of machine vision. The present application has strong practical application value and can be applied to different downstream tasks of machine vision, and the training cost and model storage overhead are small.

[0188] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, various software modules are stored in memory (such as RAM, ROM, etc.) and executed by one or more general-purpose or special-purpose processors. In a hardware embodiment, various functions are performed by various hardware components. In an embodiment that is a combination of software and hardware, various functions are performed by a combination of software modules and hardware components.

[0189] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the present application. It is to be understood that each flow and / or block in the flowchart and / or block diagrams, as well as combinations of flows and / or blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagrams block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0190] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart and / or block diagrams flow or flows and / or block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart and / or block diagrams flow or flows and / or block or blocks. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified by the flow or flows and / or block or blocks.

[0192] The specific embodiments of the present application have been described. It is to be understood that the application is not limited to the specific devices, methods, or steps described, which can of course vary. The above-described embodiments are meant to be illustrative only and not limiting, as the present application is liable to numerous modifications and adaptations of the application in various fields of applications.

Claims

1. A machine vision-oriented image coding method, characterized in that: include: Obtaining a first feature map of the image to be encoded by a space-frequency modulation coding network, wherein the first feature map is a three-dimensional feature map, and the dimensions include a channel dimension, a height dimension, and a width dimension; quantizing each eigenvalue in the first feature map to an integer to obtain a second feature map; The distribution parameter of the second feature map is modeled as a Gaussian distribution, and the mean and variance of the Gaussian distribution are predicted using an entropy model; Probability estimation is performed on features of the second feature map according to the predicted mean and variance of the Gaussian distribution, and arithmetic coding is performed according to the estimated probability to obtain a compressed image binary code stream; The step of obtaining a first feature map of an image to be encoded by using a space-frequency modulation encoding network includes: The space frequency modulation coding network is composed of multiple space frequency modulation coding units. The input of the first space frequency modulation coding unit is the image to be coded, and the input of the jth space frequency modulation coding unit is the output of the j-1th space frequency modulation coding unit. The output of the last space-frequency modulation encoding unit is the first feature map of the image to be encoded; The space-frequency modulation coding unit is composed of a basic coding subunit, a first space-domain modulation subunit, and a first frequency-domain modulation subunit, wherein: For the jth space-frequency modulation coding unit, the input Through the basic coding subunit, the basic coding feature map x is obtained. j ; The basic coding feature map x j As the input of the first spatial modulation subunit, the first spatial modulation feature map SMA is obtained j (x j ); The basic coding feature map x j As the input of the first frequency domain modulation subunit, the first frequency domain modulation feature map FMA is obtained j (x j ); Combine the basic encoding feature map x j , the first spatial modulation feature map SMA j (x j ) and the first frequency domain modulation feature map FMA j (x j ), obtain the first space-frequency modulation characteristic diagram As the output of the jth space-frequency modulation coding unit.

2. The machine vision-oriented image coding method according to claim 1, characterized in that: The basic coding feature map x j As the input of the first spatial modulation subunit, the first spatial modulation feature map SMA is obtained j (x j ),include: The basic encoding feature map x j Input coefficient is The first linear layer of the transformed spatial domain features is obtained Will The input coefficient is The second linear layer is constructed, and the output of the second linear layer is input into the first depth-separable convolutional layer to obtain the spatial modulation matrix Will Through the first ReLU nonlinear mapping layer σ, for For each element of , if the value is greater than 0, Returns the same value, otherwise returns 0; Will and Perform Hada code multiplication to multiply the elements of the corresponding positions of the two vectors to obtain the spatial domain modulation result Will The input coefficient is The third linear layer obtains the spatial modulation feature map SMA j (x j ).

3. The image coding method for machine vision according to claim 1, characterized in that: The basic coding feature map x j As the input of the first frequency domain modulation subunit, the first frequency domain modulation feature map FMA is obtained j (x j ),include: The basic encoding feature map x j Input coefficient is The fourth linear layer is used and Fourier transform is performed to obtain frequency domain features Will It passes through the second depth-wise separable convolution layer, the second ReLU nonlinear mapping layer, and another coefficient The fifth linear layer obtains the frequency domain modulation matrix Will Through the third ReLU nonlinear mapping layer, and Perform Hada code multiplication to obtain the frequency domain modulation result right Perform inverse Fourier transform and pass the coefficients The sixth linear layer is used to obtain the frequency domain modulation feature map.

4. The image coding method for machine vision according to claim 1, characterized in that: The basic coding subunit is a coding unit of any end-to-end image compression method.

5. The machine vision-oriented image coding method according to any one of claims 1 to 4, characterized in that: The entropy model is any one of a super priori entropy model, a spatial autoregressive entropy model and a channel autoregressive entropy model.

6. A machine vision-oriented image decoding method for decoding the code obtained according to any one of claims 1 to 4, characterized in that: include: Obtain compressed image binary code stream; Use the entropy model network to predict the mean and variance of the Gaussian model for the second feature map; Performing arithmetic decoding on the feature binary code stream according to the mean and variance of the Gaussian model to obtain the second feature map; The second feature map is passed through a space-frequency modulation decoding network to obtain a reconstructed image for downstream tasks oriented to machine vision.

7. The machine vision-oriented image decoding method according to claim 6, characterized in that: The space-frequency modulation decoding network is composed of multiple space-frequency modulation decoding units. The input of the first space-frequency modulation decoding unit is the image to be decoded, and the input of the jth space-frequency modulation decoding unit is the output of the j-1th space-frequency modulation decoding unit. The output of the last space-frequency modulation decoding unit is the first feature map of the image to be decoded; The space-frequency modulation decoding unit is composed of a basic decoding subunit, a second space-domain modulation subunit, a second frequency-domain modulation subunit Composition, of which: For the j-th space-frequency modulation decoding unit, pass the input through the basic decoding subunit to obtain a basic decoding feature map; Using the basic decoding feature map as an input of a second spatial modulation subunit to obtain a second spatial modulation feature map; Using the basic decoding feature map as an input of a second frequency domain modulation subunit to obtain a second frequency domain modulation feature map; The basic decoding feature map, the second spatial domain modulation feature map and the second frequency domain modulation feature map are combined to obtain a second space-frequency modulation feature map as the output of the j-th space-frequency modulation decoding unit.

8. The machine vision-oriented image decoding method according to claim 7, characterized in that: The basic sub-decoding unit is a decoding unit of any existing end-to-end image compression method.

9. A method for image compression for machine vision, characterized in that: include: Get the image to be compressed; Based on the image encoding method according to any one of claims 1 to 5, a compressed image binary code stream is obtained from the image to be encoded; Based on the image decoding method according to any one of claims 6 to 8, a reconstructed image for machine vision tasks is obtained from the compressed image binary code stream.

10. The machine vision-oriented image compression method according to claim 9, characterized in that: The parameters of the basic encoding subunit in the image encoding method, the basic decoding subunit in the image decoding method, and the entropy model are obtained through rate-distortion optimization: Among them, distortion x is the image to be compressed, is the reconstructed image, λ1>0 is a preset parameter for balancing distortion and bit rate, and the bit rate R1 is calculated based on the Gaussian distribution of each element of the second feature map: in, is the feature of the kth element, is the predicted distribution of the kth element, θ k ={φ k ,ψ k } is a set of learnable parameters.

11. The machine vision-oriented image compression method according to claim 9, characterized in that: Parameters of the first spatial domain modulation subunit and the first frequency domain modulation subunit in the image encoding method and the second spatial domain modulation subunit and the second frequency domain modulation subunit in the image decoding method are obtained by optimizing the bit rate and task accuracy: in, The loss function related to the accuracy of the downstream task, λ2>0 is a preset parameter for balancing distortion and task accuracy, and the bit rate R2 is consistent with the bit rate calculation method described in claim 10.

12. An image compression system, comprising an encoder and a decoder, characterized in that: include: Encoder: Based on the image encoding method according to any one of claims 1 to 5, obtain a compressed image binary code stream from the image to be encoded; Decoder: Based on the image decoding method according to any one of claims 6 to 8, a reconstructed image for machine vision tasks is obtained from the compressed image binary code stream.

13. An image compression device, comprising a memory and a processor, wherein the memory stores a program executable by the processor, wherein: When executing the program, the processor may perform any of the following methods: - the image coding method according to any one of claims 1 to 5, or, -The image decoding method according to any one of claims 6 to 8, or, -The image compression method according to any one of claims 9 to 11.

Images