Machine vision coding method and system based on self-supervised learning

Abstract

The invention relates to a machine vision coding method and system based on self-supervised learning, and the method comprises the following steps: randomly sampling image information into sub-blocks, inputting the sub-blocks into a backbone network head part, extracting and transforming a feature channel, and obtaining a first feature; transforming the first feature to obtain a feature of a low-dimensional space, adding uniform noise to the feature of the low-dimensional space through a quantizer to obtain a quantized feature, and reconstructing the compressed feature to obtain a second feature; converting the second feature into a low-dimensional space, increasing uniform noise through a quantizer, reducing redundancy, extracting and coding to obtain side information, decoding the side information, predicting a probability distribution parameter and a code rate of the second feature by adopting a Gaussian mixture entropy model, and reconstructing a dimension of a coding feature as a third feature; and extracting and transforming the dimension of the third feature, extracting convolution features, weighting the convolution features into a thermodynamic diagram, obtaining an effective positive sample through the thermodynamic diagram, and obtaining a coding result. Compared with the prior art, the method has the advantages of low coding complexity, high semantic reliability and the like.

Description

Technical Field

[0001] The present invention relates to the field of image compression, and in particular to a machine vision encoding method and system based on self-supervisory learning. Background Art

[0002] Image compression is a front-end technology for visual interaction, processing and content analysis, and is also a basic technology to support the rapid development of the information age. Image compression aims to eliminate spatial and statistical redundant information in images through algorithms to achieve the purpose of signal compression, thereby storing and transmitting data in a more efficient way. Existing methods for image compression are mainly based on signal fidelity optimization for human vision. In the traditional hybrid coding framework, each manually designed module is independent of each other, and its optimality is difficult to guarantee. In addition, the traditional coding standard uses mean square error as the optimization target, which degrades the semantic information and makes it difficult to efficiently perform machine vision content analysis on compressed images. Unlike traditional coding technology, the end-to-end image compression method does not rely on block division, breaks the coupling constraints between traditional coding tools, and estimates the bit rate with more perfect nonlinear transform coding and more efficient entropy model, which greatly improves the rate-distortion performance. However, the existing end-to-end image compression method aims to efficiently store and reduce the transmission bandwidth, optimize the signal fidelity for human vision, and still does not change the situation that coding reconstruction is difficult to use for advanced semantic analysis.

[0003] In summary, existing image compression methods start from the discrete signal level and use statistical knowledge as a priori to optimize human vision. These methods still face challenges such as unreliable semantic analysis of coding reconstruction, poor robustness, and the inability to fully compress long-range dependencies between distant targets. Therefore, it is necessary to study how to avoid decoding complexity under the condition of reducing transmission bandwidth, so that compressed features can directly serve multiple machine vision tasks, which is of great practical significance. Summary of the invention

[0004] The purpose of the present invention is to provide a machine vision encoding method and system based on self-supervised learning in order to overcome the defects of the above-mentioned prior art in terms of unreliable semantic analysis and high complexity.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A machine vision encoding method based on self-supervised learning includes the following steps:

[0007] After data enhancement, the image information is randomly sampled into sub-blocks, where the sub-blocks with semantic content are positive samples and the sub-blocks without semantic content are negative samples. The positive samples and negative samples are input into the backbone network head to extract and transform the feature channel to obtain the first feature;

[0008] The first feature is transformed by using a redundancy elimination encoder to obtain a feature of a low-dimensional space, uniform noise is added to the feature of the low-dimensional space by a first quantizer to obtain a quantized feature, and the quantized feature is transferred to a redundancy elimination decoder by arithmetic encoding and decoding to reconstruct the compressed feature to obtain a second feature;

[0009] The encoder is used to transform the second feature into a low-dimensional space, and then the second quantizer is used to increase uniform noise to reduce redundancy. The super-a priori encoder extracts and encodes the side information of the second feature after reducing redundancy, and transmits the side information to the super-a priori decoder for decoding through arithmetic encoding and decoding. Then, the probability distribution parameters and bit rate of the second feature are predicted based on the mixed Gaussian entropy model, and then the decoder is used to reconstruct the dimension of the encoded feature to obtain the third feature.

[0010] A convolutional neural network is used to extract and transform the dimension of the third feature. At the same time, the convolutional features are extracted and weighted as a heat map. The heat map is used to guide the selection of valid positive samples of the input image to obtain the encoding result.

[0011] Further, the noise values ​​of the uniform noise added by the first quantizer and the second quantizer are both within an open interval of -0.5 to 0.5.

[0012] The second aspect of the present invention is a machine vision encoding system based on self-supervised learning, comprising a backbone network head, a redundancy elimination module, a compression network and a backbone network tail connected in sequence, and when the system is running, it implements any of the above machine vision encoding methods based on self-supervised learning.

[0013] Furthermore, the backbone network head includes two parts of residual networks, the first part is the Stem layer in the residual convolutional network ResNet50, and the second part is the first layer network in the residual convolutional network ResNet50.

[0014] Furthermore, the backbone network head receives the heat map as a semantic guide to select valid positive samples of the input image.

[0015] Furthermore, the redundancy elimination module includes a redundancy elimination encoder, a first quantizer, a first arithmetic encoder, a first arithmetic decoder and a redundancy elimination decoder, the redundancy elimination encoder and the redundancy elimination decoder are mirror networks, the first quantizer is used to add noise to the features of the low-dimensional space, and the first arithmetic encoder and the first arithmetic decoder are used to compress data into a code stream for storage and transmission.

[0016] Furthermore, the compression network includes an encoder, a super-prior encoder, a super-prior decoder, a second quantizer, a second arithmetic encoder, a second arithmetic decoder and a decoder. The super-prior encoder and the super-prior decoder are used to cooperate with a mixed Gaussian entropy model to predict the probability distribution of encoder output features. The decoder and the encoder are a pair of mirror networks, both of which include two layers of gated recurrent convolutional networks.

[0017] Furthermore, the tail of the backbone network includes the second, third and fourth convolutional neural networks of the residual convolutional network ResNet50. The tail of the backbone network receives the third feature and extracts the semantic content feature as output.

[0018] Furthermore, the last layer of convolutional features in the tail of the backbone network is summed along the channel dimension to obtain a heat map, which is fed back to the input image as a semantic guide to select positive samples with semantic content.

[0019] Furthermore, during system training, the entire system is trained end-to-end in a self-supervised form of contrastive learning, which is divided into three stages. In the first stage, only the backbone network head and the backbone network tail are trained. In the second stage, the weights of the backbone network head and the backbone network tail are fixed, and only the redundancy elimination module is trained. In the third stage, the weights of the backbone network head, the backbone network tail and the redundancy elimination module are fixed, and only the compression network is trained. The loss function during training is the weighted sum of the bit rate of the compressed features and the prediction accuracy of the machine vision analysis task.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1) The redundancy elimination module designed by the present invention can improve the compactness of features in the process of self-supervised learning. At the same time, the compression network involved in the present invention fully considers the dependencies between distant targets, can fully compress the global redundancy of image spatial domain signals, reduces the encoding complexity, can fully retain the semantic content in the process of encoding image signals, and enhances the reliability of encoding features in machine vision task analysis.

[0022] 2) The present invention achieves superior prediction results in three machine vision tasks: image classification, object detection, and instance segmentation, proving that the present method can be applied to multiple machine vision analysis tasks by training the model only once. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the overall architecture of the coding and decoding model of the present invention;

[0024] Figure 2a A redundancy elimination coder / decoder in a redundancy elimination module in a codec model;

[0025] Figure 2b It is the compression network in the encoding and decoding model;

[0026] Figure 3 A schematic diagram showing the comparison of the bit rate and classification accuracy of the present invention and other compression methods on the ImageNet dataset;

[0027] Figure 4 Schematic diagram of comparing the bit rate and instance segmentation performance of the present invention and other compression methods on the COCO 2017 dataset;

[0028] Figure 5a A schematic diagram showing the comparison of the bit rate and target detection performance of the present invention and other compression methods on the PASCAL VOC dataset;

[0029] Figure 5b Schematic diagram of the comparison of bit rate and target detection performance between the present invention and other compression methods on the COCO 2017 dataset; DETAILED DESCRIPTION

[0030] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0031] Example 1

[0032] The present invention proposes a machine vision encoding method based on self-supervised learning, comprising the following steps:

[0033] After data enhancement, the image information is randomly sampled into sub-blocks, where the sub-blocks with semantic content are positive samples and the sub-blocks without semantic content are negative samples. The positive samples and negative samples are input into the backbone network head to extract and transform the feature channel to obtain the first feature;

[0034] The first feature is transformed by using a redundancy elimination encoder to obtain a feature of a low-dimensional space, uniform noise is added to the feature of the low-dimensional space by a first quantizer to obtain a quantized feature, and the quantized feature is transferred to a redundancy elimination decoder by arithmetic encoding and decoding to reconstruct the compressed feature to obtain a second feature;

[0035] The encoder is used to transform the second feature into a low-dimensional space, and then the second quantizer is used to increase uniform noise to reduce redundancy. The super-a priori encoder extracts and encodes the side information of the second feature after reducing redundancy, and transmits the side information to the super-a priori decoder for decoding through arithmetic encoding and decoding. Then, the probability distribution parameters and bit rate of the second feature are predicted based on the mixed Gaussian entropy model, and then the decoder is used to reconstruct the dimension of the encoded feature to obtain the third feature.

[0036] A convolutional neural network is used to extract and transform the dimension of the third feature. At the same time, the convolutional features are extracted and weighted as a heat map. The heat map is used to guide the selection of valid positive samples of the input image to obtain the encoding result.

[0037] To implement the above method, the present invention proposes a machine vision encoding system based on self-supervised learning, such as Figure 1 As shown, including:

[0038] 1) Backbone network head, in the ImageNet dataset, a data-augmented and cropped image of size 224×224 is given as input. The input image is randomly cropped into sub-blocks, and then contrastive learning is used to select sub-blocks from the target area of ​​the same image as positive samples, and sub-blocks from different images as negative samples. The Stem layer and the first layer in ResNet-50 are then used to transform the channels of positive and negative samples and extract features. The Stem layer is a 7×7 convolutional neural network with a stride of 2. The first layer in ResNet-50 contains six 1×1 convolutions, three 3×3 convolutions, and a 2×2 pooling layer. The above steps satisfy the following formula:

[0039] f1=F(x)

[0040] Among them, x represents the input image, F represents the Stem layer and the first layer of the serial connection ResNet-50, and f1 represents the output feature of the backbone network head.

[0041] 2) Redundancy elimination module, receiving the output feature f1 of the backbone network head as input, first transforms f1 into a low-dimensional space using a redundancy elimination encoder, which consists of three layers of convolutional self-attention blocks with a downsampling factor of 2, such as Figure 2a As shown. The redundancy elimination encoder uses two fully connected layers to perform linear transformation on the input features. Then the upper branch uses a 3×3 convolution kernel to perform convolution on the linear transformation features. The convolution result is then subjected to a Hadamard dot product with the linear transformation features of the next branch. The result after the dot product is then subjected to a linear transformation of the fully connected layer. The result of the transformation is then added to the input features, and finally a 3×3 convolution kernel is used to perform downsampling with a stride of 2. The above process is repeated three times to obtain the output y1 of the redundancy elimination encoder. The quantizer Q then adds a uniform noise value between (-0.5, 0.5) to the feature y1 to ensure that the gradient is differentiable during model training. After quantization The arithmetic encoder AE is compressed into a code stream for storage and transmission systems. When features need to be decoded, the arithmetic decoder AD is used to decode the code stream and further pass it to the redundancy elimination decoder to restore the dimension of the encoded features. The redundancy elimination decoder and the redundancy elimination encoder are a pair of mirror networks. It is only necessary to change the downsampling of the convolution self-attention block in the redundancy encoder to upsampling. The above steps satisfy the following formula:

[0042] y1=RE_enc(f1)

[0043]

[0044] Among them, RE_enc is the redundancy elimination encoder, y1 is the output of the redundancy encoder, Q is the quantizer, is the quantified result. The arithmetic encoder AE is compressed into a code stream for use in the storage and transmission system. When the features need to be decoded, the arithmetic decoder AD is used to decode the code stream and further passed to the redundancy elimination decoder RE_dec. f2 is the output of the redundancy elimination decoder.

[0045] 3) The compression network receives the output f2 of the redundancy elimination module as input, and first uses an encoder to transform and encode the input feature f2. The encoder consists of two layers of gated recurrent convolutional networks with a downsampling factor of 2, such as Figure 2b As shown. The working principle of the gated recurrent convolutional network is as follows:

[0046] First, input features (H, W, C are the height, width and number of channels of feature f2 respectively) dimension transformation to Then perform the following transformation:

[0047]

[0048] φ in is a fully connected layer for channel mapping, p0 and l0 respectively represent the average division of the features mapped by the fully connected layer along the channel dimension. Then the following calculation is performed:

[0049] p1=g0(l0)·p0

[0050]

[0051] Among them, g0 is a convolutional network with a convolution kernel of 3×3, · represents the dot product, and the result of the dot product is passed to the fully connected layer φ out Perform dimensional transformation, e is the first-order transformation result of gated convolution. The first-order transformation of gated convolution network is extended to n-order transformation. The expansion method is as follows:

[0052]

[0053] The above formula indicates that the fully connected layer φ in The transformed features are split into n parts along the channel. Then the following calculations are performed:

[0054]

[0055] g m is a convolutional network with a convolution kernel of 3×3, γ is a constant used to scale the output, and h m It is a fully connected layer for channel transformation. The above is the n-order calculation process of gated circular convolution, and its calculation result will be passed to the downsampling convolution network with a stride of 2 and a convolution kernel of 3×3. The above process is repeated twice for the calculation process of the encoder.

[0056] The super-a priori encoder then extracts the side information of the encoded output features. The super-a priori encoder consists of two layers of multi-head attention mechanisms with a downsampling factor of 2. The output of the encoder is then passed to the quantizer Q and the super-a priori encoder for quantization and side information extraction. The quantizer adds uniform noise between (-0.5, 0.5) to the features to ensure that the gradient is differentiable during model training. The super-a priori encoder consists of two layers of multi-head attention mechanisms with a downsampling factor of 2. The output of the super-a priori encoder is then quantized. The above quantized results are compressed into a bitstream using an arithmetic encoder AE for storage and transmission systems. When the image needs to be decoded, the arithmetic decoder AD is used to decode the bitstream. Among them, the result of arithmetic encoding and decoding of the output of the super-a priori encoder will be passed to the super-a priori decoder to cooperate with the mixed Gaussian entropy model to predict the probability distribution of the encoded features. The super-a priori decoder consists of two layers of multi-head attention mechanisms with an upsampling factor of 2, and its output result will be passed to the decoder. The result of the encoder output after arithmetic encoding and decoding will be further passed to the decoder, which is a mirror image of the encoder. It only needs to change the downsampling in the encoder to upsampling. The above calculation process is expressed as:

[0057] y2=Enc(f2)

[0058] z=h a (y2)

[0059]

[0060] Among them, Enc represents the encoder, y2 represents the output of the encoder, and h a represents the super prior encoder, z represents the side information, Q is the quantizer, and Respectively represent the quantization results of y2 and z, h s represents super-a priori decoding, y3 represents the result of super-a priori decoding, Add represents the sum of variables, and Dec represents the decoder.

[0061] 1) At the end of the backbone network, the output f3 of the compression network is received as input, and the second, third, and fourth layers of ResNet-50 are used to extract the semantic information of f3 to migrate to machine vision tasks such as image classification, object detection, and instance segmentation. At the same time, the features such as the output of the tail of the backbone network are summed along the channel dimension, and the summation result is normalized to [0-1] to obtain a heat map. This heat map will guide contrastive learning to select sub-blocks with semantic content in the input image as positive samples. The above calculation process is expressed as:

[0062] f4=F h (f3)

[0063] f5=Heat(f4)

[0064] Among them, F h is the tail of the backbone network, which is a serial connection of the second, third, and fourth layers of the network in ResNet-50. f4 represents the output of the tail of the backbone network, Heat represents the calculation of the heat map of f4, which sums the eigenvalues ​​of f4 in the channel dimension and normalizes the summation result to a closed interval of 0 to 1, and f5 represents the output result of the heat map.

[0065] In order to verify the performance of the above method, the following experiments were designed in this embodiment:

[0066] The proposed model is evaluated on three challenging benchmark datasets: ImageNet, PASCAL VOC and COCO2017 in terms of image classification, object detection and instance segmentation. Rate-distortion performance is a metric, where the bitrate per pixel is measured in bpp and the analytical performance of machine vision tasks represents distortion. The proposed method DICM is compared with the most recent state-of-the-art methods. The experimental results of Top-1(%) vs. bpp on the ImageNet dataset are shown in Figure 2. Figure 3 The experimental results of Ap-mask (%) vs. bpp on the COCO2017 dataset are shown in Figure 4 The experimental results of AP(%) vs. bpp on PASCAL VOC and COCO2017 are shown as follows. Figure 5a and Figure 5b In addition, in order to compare the bit rate gain and prediction performance gain of the DICM method of the present invention with other image compression methods, Tables 1 and 2 list the experimental results of different methods on BD-rate and BD-acc on the PASCAL VOC and COCO2017 datasets with BPG as the benchmark.

[0067] Table 1 Comparison of BD-rate on PASCAL VOC and COCO2017 datasets

[0068]

[0069] Table 2 Comparison results of BD-acc on PASCAL VOC and COCO2017 datasets

[0070]

[0071] Image classification results on the ImageNet dataset: The comparison results of this method with the most advanced image compression methods on the ImageNet dataset are shown in Figure 3 .from Figure 3From the rate-distortion curves, it can be observed that our method achieves the best image classification performance compared with the state-of-the-art image compression methods.

[0072] Instance segmentation results on the COCO2017 dataset: The comparison results of this method with the most advanced image compression methods on instance segmentation on the COCO2017 dataset are shown in Figure 4 , Table 1 and Table 2. The comparison method in the table is based on BPG for quantitative comparison. Figure 4 The results of the rate-distortion curve show that the instance segmentation results of this method are improved at a lower bit rate compared with existing methods. Table 1 confirms that this method can obtain more bit rate gain at similar instance segmentation accuracy compared with other image compression methods, and the specific coding gain reaches -90.01%. Among them, Ins. (COCO) means that the instance segmentation task is performed on COCO2017. In addition, Table 2 also confirms that compared with existing methods, this method can obtain the largest instance segmentation accuracy improvement at similar bit rates. Compared with BPG, the segmentation accuracy of this method is improved by 13.17%.

[0073] The results of the comparison of bit rate and target detection performance on PASCAL VOC and COCO2017 are listed in Figure 5a , Figure 5b , Table 1 and Table 2. Figure 5a The rate-distortion performance of this method and the existing methods on PASCAL VOC is shown. From the figure, it can be observed that this method and the existing methods can achieve the highest target detection performance at a lower bit rate. Figure 5b The rate-distortion performance of this method and existing methods on COCO2017 is demonstrated. Figure 5b It shows that this method can achieve the highest target detection performance at a lower bit rate than existing methods. The Det.(VOC) and Det.(COCO) in the two lists represent target detection tasks performed on the PASCAL VOC and COCO datasets, respectively. Compared with other image compression methods, this method can achieve the largest coding gain and detection performance improvement on the PASCAL VOC dataset, which reaches -74.24% and 3.27% respectively. In addition, compared with other image compression methods, this method can also achieve the largest coding gain and detection performance improvement on the COCO2017 dataset, which reaches -95.17% and 13.89% respectively.

[0074] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A machine vision encoding method based on self-supervised learning, characterized in that: The following steps are involved: After data enhancement, the image information is randomly sampled into sub-blocks, where the sub-blocks with semantic content are positive samples and the sub-blocks without semantic content are negative samples. The positive samples and negative samples are input into the backbone network head to extract and transform the feature channel to obtain the first feature; The first feature is transformed by using a redundancy elimination encoder to obtain a feature of a low-dimensional space, uniform noise is added to the feature of the low-dimensional space by a first quantizer to obtain a quantized feature, and the quantized feature is transferred to a redundancy elimination decoder by arithmetic encoding and decoding to reconstruct the compressed feature to obtain a second feature; The encoder is used to transform the second feature into a low-dimensional space, and then the second quantizer is used to increase uniform noise to reduce redundancy. The super-a priori encoder extracts and encodes the side information of the second feature after reducing redundancy, and transmits the side information to the super-a priori decoder for decoding through arithmetic encoding and decoding. Then, the probability distribution parameters and bit rate of the second feature are predicted based on the mixed Gaussian entropy model, and then the decoder is used to reconstruct the dimension of the encoded feature to obtain the third feature. A convolutional neural network is used to extract and transform the dimension of the third feature. At the same time, the convolutional features are extracted and weighted as a heat map. The heat map is used to guide the selection of valid positive samples of the input image to obtain the encoding result.

2. According to the machine vision encoding method based on self-supervised learning according to claim 1, it is characterized in that: The noise values ​​of the uniform noise added by the first quantizer and the second quantizer are both within the open interval of -0.5 to 0.

5.

3. A machine vision encoding system based on self-supervised learning, characterized in that: The system comprises a backbone network head, a redundancy elimination module, a compression network and a backbone network tail connected in sequence. When the system is running, a machine vision encoding method based on self-supervised learning as described in any one of claims 1-2 is implemented.

4. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The backbone network head includes two parts of residual networks, the first part is the Stem layer in the residual convolutional network ResNet50, and the second part is the first layer network in the residual convolutional network ResNet50.

5. A machine vision encoding system based on self-supervised learning according to claim 4, characterized in that: The backbone network head receives the heat map as a semantic guide to select valid positive samples of the input image.

6. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The redundancy elimination module includes a redundancy elimination encoder, a first quantizer, a first arithmetic encoder, a first arithmetic decoder and a redundancy elimination decoder, wherein the redundancy elimination encoder and the redundancy elimination decoder are mirror networks, the first quantizer is used to add noise to the features of the low-dimensional space, and the first arithmetic encoder and the first arithmetic decoder are used to compress data into a code stream for storage and transmission.

7. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The compression network includes an encoder, a super-a priori encoder, a super-a priori decoder, a second quantizer, a second arithmetic encoder, a second arithmetic decoder and a decoder. The super-a priori encoder and the super-a priori decoder are used to cooperate with a mixed Gaussian entropy model to predict the probability distribution of encoder output features. The decoder and the encoder are a pair of mirror networks, both of which include two layers of gated recurrent convolutional networks.

8. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The tail of the backbone network includes the second, third and fourth convolutional neural networks of the residual convolutional network ResNet50. The tail of the backbone network receives the third feature and extracts the semantic content feature as output.

9. A machine vision encoding system based on self-supervised learning according to claim 8, characterized in that: The last layer of convolutional features in the tail of the backbone network is summed along the channel dimension to obtain a heat map, which is fed back to the input image as a semantic guide to select positive samples with semantic content.

10. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: When the system is trained, the entire system is trained end-to-end in a self-supervised form of contrastive learning, which is divided into three stages. In the first stage, only the backbone network head and the backbone network tail are trained. In the second stage, the weights of the backbone network head and the backbone network tail are fixed, and only the redundancy elimination module is trained. In the third stage, the weights of the backbone network head, the backbone network tail and the redundancy elimination module are fixed, and only the compression network is trained. The loss function during training is the weighted sum of the bit rate of the compression feature and the prediction accuracy of the machine vision analysis task.

Images