A machine vision encoding method and system based on self-supervised learning

Abstract

The present invention relates to a machine vision encoding method and system based on self-supervised learning. The method includes the following steps: randomly sampling image information into sub-blocks, inputting the sub-blocks into a backbone network head to extract and transform feature channels to obtain a first feature; transforming the first feature to obtain a feature in a low-dimensional space, adding uniform noise to the feature in the low-dimensional space through a quantizer to obtain a quantized feature, and reconstructing a compressed feature to obtain a second feature; transforming the second feature to a low-dimensional space, adding uniform noise to the feature in the low-dimensional space through a quantizer to reduce redundancy, extracting and encoding side information, decoding the side information, using a mixed Gaussian entropy model to predict the probability distribution parameters and bit rate of the second feature, and reconstructing the dimension of the encoded feature as a third feature; extracting and transforming the dimension of the third feature, extracting and weighting the convolution feature to form a heat map, obtaining valid positive samples through the heat map, and obtaining an encoding result. Compared with the existing technology, the present invention has the advantages of low encoding complexity and high semantic reliability.

Description

Technical Field

[0001] The present invention relates to the field of image compression, and in particular to a machine vision encoding method and system based on self-supervised learning. Background Art

[0002] Image compression is a front-end technology for visual interaction, processing, and content analysis, and is also a foundational technology supporting the rapid development of the information age. Image compression aims to eliminate spatial and statistical redundancy in images through algorithms to achieve signal compression, thereby enabling more efficient data storage and transmission. Existing image compression methods primarily focus on optimizing signal fidelity for human vision. Traditional hybrid coding frameworks rely on independent, hand-designed modules, making optimality difficult to guarantee. Furthermore, traditional coding standards optimize for mean squared error, which degrades semantic information and makes it difficult to efficiently perform machine vision content analysis on compressed images. Unlike traditional coding techniques, end-to-end image compression methods do not rely on block partitioning, breaking the coupling constraints between traditional coding tools. They use more sophisticated nonlinear transform coding and more efficient entropy models to estimate the bitrate, significantly improving rate-distortion performance. However, existing end-to-end image compression methods, which focus on efficient storage and bandwidth reduction while optimizing signal fidelity for human vision, still make coded reconstruction difficult to use for advanced semantic analysis.

[0003] In summary, existing image compression methods start at the discrete signal level and use statistical knowledge as a priori to optimize human vision. However, these methods still face challenges such as unreliable semantic analysis of encoding and reconstruction, poor robustness, and the inability to fully compress long-range dependencies between distant objects. Therefore, it is highly practical to study how to reduce decoding complexity while reducing transmission bandwidth, so that compressed features can directly serve multiple machine vision tasks. Summary of the Invention

[0004] The purpose of the present invention is to provide a machine vision encoding method and system based on self-supervised learning in order to overcome the defects of the above-mentioned prior art in terms of unreliable semantic analysis and high complexity.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A machine vision encoding method based on self-supervised learning includes the following steps:

[0007] After data enhancement, the image information is randomly sampled into sub-blocks, where the sub-blocks with semantic content are positive samples and the sub-blocks without semantic content are negative samples. The positive and negative samples are input into the backbone network head to extract and transform the feature channel to obtain the first feature;

[0008] The first feature is transformed using a redundancy elimination encoder to obtain a feature of a low-dimensional space, uniform noise is added to the feature of the low-dimensional space using a first quantizer to obtain a quantized feature, and the quantized feature is transferred to a redundancy elimination decoder through arithmetic coding and decoding to reconstruct the compressed feature to obtain a second feature;

[0009] The encoder transforms the second feature into a low-dimensional space, and then uses a second quantizer to add uniform noise to reduce redundancy. The super-prior encoder extracts and encodes the side information of the second feature after redundancy reduction. The side information is passed to the super-prior decoder for decoding through arithmetic coding and decoding. The probability distribution parameters and bit rate of the second feature are then predicted using a mixed Gaussian entropy model. The decoder is then used to reconstruct the dimension of the encoded feature to obtain the third feature.

[0010] A convolutional neural network is used to extract and transform the dimension of the third feature. At the same time, the convolution features are extracted and weighted into a heat map. The heat map is used to guide the selection of valid positive samples of the input image to obtain the encoding result.

[0011] Furthermore, noise values ​​of the uniform noise added by the first quantizer and the second quantizer are both within an open range of -0.5 to 0.5.

[0012] The second aspect of the present invention is a machine vision encoding system based on self-supervised learning, comprising a backbone network head, a redundancy elimination module, a compression network and a backbone network tail connected in sequence, and implementing any of the above machine vision encoding methods based on self-supervised learning when the system is running.

[0013] Furthermore, the backbone network head includes two parts of residual networks, the first part is the Stem layer in the residual convolutional network ResNet50, and the second part is the first layer network in the residual convolutional network ResNet50.

[0014] Furthermore, the backbone network head receives the heat map as semantic guidance to select valid positive samples of the input image.

[0015] Furthermore, the redundancy elimination module includes a redundancy elimination encoder, a first quantizer, a first arithmetic encoder, a first arithmetic decoder and a redundancy elimination decoder. The redundancy elimination encoder and the redundancy elimination decoder are mirror networks. The first quantizer is used to add noise to the features of the low-dimensional space. The first arithmetic encoder and the first arithmetic decoder are used to compress data into a code stream for storage and transmission.

[0016] Furthermore, the compression network includes an encoder, a super-prior encoder, a super-prior decoder, a second quantizer, a second arithmetic encoder, a second arithmetic decoder and a decoder. The super-prior encoder and the super-prior decoder are used to cooperate with the mixed Gaussian entropy model to predict the probability distribution of the encoder output features. The decoder and the encoder are a pair of mirror networks, both of which include two layers of gated recurrent convolutional networks.

[0017] Furthermore, the tail of the backbone network includes the second, third and fourth convolutional neural networks of the residual convolutional network ResNet50. The tail of the backbone network receives the third feature and extracts the semantic content feature as output.

[0018] Furthermore, the convolutional features of the last layer in the backbone network are summed along the channel dimension to obtain a heat map, which is fed back to the input image as a semantic guide to select positive samples with semantic content.

[0019] Furthermore, during system training, the entire system is trained end-to-end in a self-supervised form of contrastive learning, which is divided into three stages. In the first stage, only the backbone network head and the backbone network tail are trained. In the second stage, the weights of the backbone network head and the backbone network tail are fixed, and only the redundancy elimination module is trained. In the third stage, the weights of the backbone network head, the backbone network tail and the redundancy elimination module are fixed, and only the compression network is trained. The loss function during training is the weighted sum of the bit rate of the compressed features and the prediction accuracy of the machine vision analysis task.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1) The redundancy elimination module designed in the present invention can improve the compactness of features in the process of self-supervised learning. At the same time, the compression network involved in the present invention fully considers the dependencies between distant targets, can fully compress the global redundancy of image spatial domain signals, reduce encoding complexity, and can fully retain semantic content in the process of encoding image signals, thereby enhancing the reliability of encoded features in machine vision task analysis.

[0022] 2) The present invention achieves superior prediction results in three machine vision tasks: image classification, object detection, and instance segmentation, proving that the method can be applied to multiple machine vision analysis tasks with only one model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the overall architecture of the coding and decoding model of the present invention;

[0024] Figure 2a A redundancy elimination codec within a redundancy elimination module in a codec model;

[0025] Figure 2b It is the compression network in the encoding and decoding model;

[0026] Figure 3 A schematic diagram comparing the bit rate and classification accuracy of the present invention and other compression methods on the ImageNet dataset;

[0027] Figure 4 Schematic diagram comparing the bit rate and instance segmentation performance of the present invention and other compression methods on the COCO 2017 dataset;

[0028] Figure 5a Schematic diagram comparing the bit rate and object detection performance of the present invention and other compression methods on the PASCAL VOC dataset;

[0029] Figure 5b Schematic diagram comparing the bit rate and object detection performance of the present invention and other compression methods on the COCO 2017 dataset; DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0031] Example 1

[0032] The present invention proposes a machine vision encoding method based on self-supervised learning, comprising the following steps:

[0033] After data enhancement, the image information is randomly sampled into sub-blocks, where the sub-blocks with semantic content are positive samples and the sub-blocks without semantic content are negative samples. The positive and negative samples are input into the backbone network head to extract and transform the feature channel to obtain the first feature;

[0034] The first feature is transformed using a redundancy elimination encoder to obtain a feature of a low-dimensional space, uniform noise is added to the feature of the low-dimensional space using a first quantizer to obtain a quantized feature, and the quantized feature is transferred to a redundancy elimination decoder through arithmetic coding and decoding to reconstruct the compressed feature to obtain a second feature;

[0035] The encoder transforms the second feature into a low-dimensional space, and then uses a second quantizer to add uniform noise to reduce redundancy. The super-prior encoder extracts and encodes the side information of the second feature after redundancy reduction. The side information is passed to the super-prior decoder for decoding through arithmetic coding and decoding. The probability distribution parameters and bit rate of the second feature are then predicted using a mixed Gaussian entropy model. The decoder is then used to reconstruct the dimension of the encoded feature to obtain the third feature.

[0036] A convolutional neural network is used to extract and transform the dimension of the third feature. At the same time, the convolution features are extracted and weighted into a heat map. The heat map is used to guide the selection of valid positive samples of the input image to obtain the encoding result.

[0037] To implement the above method, the present invention proposes a machine vision encoding system based on self-supervised learning, such as Figure 1 Shown, including:

[0038] 1) Backbone network head, in the ImageNet dataset, a data-augmented and cropped image of size 224×224 is given as input. The input image is randomly cropped into sub-blocks, and then contrastive learning is used to select sub-blocks from the target area of ​​the same image as positive samples, and sub-blocks from different images as negative samples. The Stem layer and the first layer in ResNet-50 are then used to transform the channels of positive and negative samples and extract features. The Stem layer is a 7×7 convolutional neural network with a stride of 2. The first layer in ResNet-50 consists of six 1×1 convolutions, three 3×3 convolutions, and a 2×2 pooling layer. The above steps satisfy the following formula:

[0039] f1=F(x)

[0040] Here, x represents the input image, F represents the serial connection between the Stem layer and the first layer of ResNet-50, and f1 represents the output feature of the backbone network head.

[0041] 2) Redundancy elimination module, which receives the output feature f1 of the backbone network head as input, first transforms f1 into a low-dimensional space using a redundancy elimination encoder. The redundancy elimination encoder consists of three layers of convolutional self-attention blocks with a downsampling factor of 2, as shown in Figure 2a As shown. The redundancy elimination encoder uses two fully connected layers to perform linear transformation on the input features. Then the upper branch uses a 3×3 convolution kernel to perform convolution on the linear transformation features. The convolution result is then subjected to a Hadamard dot product with the linear transformation features of the next branch. The result after the dot product is then subjected to a linear transformation of the fully connected layer. The result of the transformation is then added to the input features, and finally a 3×3 convolution kernel is used to perform downsampling with a stride of 2. The above process is repeated three times to obtain the output y1 of the redundancy elimination encoder. The quantizer Q then adds uniform noise with a value between (-0.5, 0.5) to the feature y1 to ensure that the gradient is differentiable during model training. After quantization The arithmetic encoder AE is compressed into a code stream for storage and transmission systems. When features need to be decoded, the arithmetic decoder AD is used to decode the code stream and further pass it to the redundancy elimination decoder to restore the dimension of the encoded features. The redundancy elimination decoder and the redundancy elimination encoder are a pair of mirror networks. Only the downsampling of the convolutional self-attention block in the redundancy encoder needs to be changed to upsampling. The above steps satisfy the following formula:

[0042] y1=RE_enc(f1)

[0043]

[0044] Among them, RE_enc is the redundancy elimination encoder, y1 is the output of the redundancy encoder, Q is the quantizer, is the result after quantization. The arithmetic encoder AE is compressed into a code stream for use in the storage and transmission system. When the features need to be decoded, the arithmetic decoder AD is used to decode the code stream and further pass it to the redundancy elimination decoder RE_dec. f2 is the output of the redundancy elimination decoder.

[0045] 3) Compression network, receiving the output f2 of the redundancy elimination module as input, first uses the encoder to transform the input feature f2, which is composed of two layers of gated recurrent convolutional networks with a downsampling factor of 2, as shown in Figure 2b As shown in Figure 2. The working principle of the gated recurrent convolutional network is as follows:

[0046] First, input features (H, W, C are the height, width and number of channels of feature f2 respectively) dimension transformation to Then perform the following transformation:

[0047]

[0048] φ in It is a fully connected layer for channel mapping. p0 and l0 respectively represent the average division of the features mapped by the fully connected layer along the channel dimension. Then the following calculation is performed:

[0049] p1=g0(l0)·p0

[0050]

[0051] Among them, g0 is a convolutional network with a convolution kernel of 3×3, · represents the dot product, and the result of the dot product is passed to the fully connected layer φ out Perform dimensional transformation, e is the first-order transformation result of gated convolution. Based on the first-order transformation of the gated convolution network, it is extended to n-order transformation. The expansion method is as follows:

[0052]

[0053] The above formula means that the fully connected layer φ in The transformed features are split into n parts along the channel. Then the following calculations are performed:

[0054]

[0055] g m It is a convolutional network with a convolution kernel of 3×3, γ is a constant used to scale the output, and h m is a fully connected layer for channel transformation. The above is the calculation process of the n-th order gated recurrent convolution, and its calculation results are passed to the downsampling convolution network with a stride of 2 and a convolution kernel of 3×3. The above process is repeated twice to form the encoder calculation process.

[0056] The super-a priori encoder then extracts side information from the encoded output features. This super-a priori encoder consists of two layers of multi-head attention mechanisms with a downsampling factor of 2. The encoder output is then passed to a quantizer Q and a super-a priori encoder for quantization and side information extraction. The quantizer adds uniform noise between (-0.5 and 0.5) to the features to ensure gradient differentiability during model training. The super-a priori encoder consists of two layers of multi-head attention mechanisms with a downsampling factor of 2. The super-a priori encoder output is then quantized and compressed into a bitstream using an arithmetic encoder (AE) for storage and transmission. When the image is decoded, the bitstream is decoded using an arithmetic decoder (AD). The results of the arithmetic encoding and decoding of the super-a priori encoder output are passed to a super-a priori decoder, which uses a mixture of Gaussian entropy models to predict the probability distribution of the encoded features. This super-a priori decoder consists of two layers of multi-head attention mechanisms with an upsampling factor of 2, and its output is passed to the decoder. The result of the encoder output after arithmetic encoding and decoding will be further passed to the decoder. The decoder is a mirror image of the encoder, and only the downsampling in the encoder needs to be changed to upsampling. The above calculation process is expressed as:

[0057] y2=Enc(f2)

[0058] z=h a (y2)

[0059]

[0060] Among them, Enc represents the encoder, y2 represents the output of the encoder, h a represents the super prior encoder, z represents the side information, Q is the quantizer, and Represent the quantization results of y2 and z respectively, h s It represents super-prior decoding, y3 represents the result of super-prior decoding, Add represents the sum of variables, and Dec represents the decoder.

[0061] 1) At the end of the backbone network, the output f3 of the compression network is received as input. The second, third, and fourth layers of ResNet-50 are used to extract semantic information from f3 for transfer to machine vision tasks such as image classification, object detection, and instance segmentation. Simultaneously, the features output by the backbone network are summed along the channel dimension, and the summed result is normalized to [0-1] to obtain a heat map. This heat map guides contrastive learning to select sub-blocks with semantic content in the input image as positive samples. The above calculation process is expressed as:

[0062] f4=F h (f3)

[0063] f5=Heat(f4)

[0064] Among them, F h The tail of the backbone network is a serial connection of the second, third, and fourth layers of ResNet-50. f4 represents the output of the tail of the backbone network. Heat represents the calculation of the heat map of f4, which sums the eigenvalues ​​of f4 in the channel dimension and normalizes the sum to a closed interval between 0 and 1. f5 represents the output of the heat map.

[0065] In order to verify the performance of the above method, the following experiments were designed in this embodiment:

[0066] This method evaluates the performance of the proposed model in image classification, object detection, and instance segmentation on three challenging benchmark datasets: ImageNet, PASCAL VOC, and COCO2017. Rate-distortion performance is a metric, where the bitrate per pixel is measured in bits per pixel, and the performance of the machine vision task is expressed as distortion. The proposed method DICM is compared with the most recent state-of-the-art methods. The experimental results of Top-1 (%) vs. bpp on the ImageNet dataset are shown in the figure below. Figure 3 The experimental results of Ap-mask (%) vs. bpp on the COCO2017 dataset are shown in Figure 4 The experimental results of AP(%) vs. bpp on PASCAL VOC and COCO2017 are shown as follows. Figure 5a and Figure 5b In addition, to compare the bit rate gain and prediction performance gain of the proposed DICM method with other image compression methods, Tables 1 and 2 list the experimental results of different methods on BD-rate and BD-acc using BPG as the benchmark on the PASCAL VOC and COCO2017 datasets, respectively.

[0067] Table 1 Comparison of BD-rate on PASCAL VOC and COCO2017 datasets

[0068]

[0069] Table 2 Comparison of BD-acc on PASCAL VOC and COCO2017 datasets

[0070]

[0071] Image classification results on the ImageNet dataset: The comparison results of this method with the most advanced image compression methods on the ImageNet dataset are shown in Figure 3 .from Figure 3From the rate-distortion curves, it can be observed that our method achieves the best image classification performance compared with the state-of-the-art image compression methods.

[0072] Instance segmentation results on the COCO2017 dataset: The comparison results of this method with the most advanced image compression methods on instance segmentation on the COCO2017 dataset are shown in Figure 4 , Table 1 and Table 2. The comparison method in the table is based on BPG for quantitative comparison. Figure 4 The rate-distortion curves show that our method achieves improved instance segmentation results at lower bitrates compared to existing methods. Table 1 confirms that our method achieves greater bitrate gain than other image compression methods at similar instance segmentation accuracy, with a specific coding gain of -90.01%. Ins.(COCO) indicates the instance segmentation task performed on COCO2017. Furthermore, Table 2 confirms that our method achieves the greatest improvement in instance segmentation accuracy compared to existing methods at similar bitrates, achieving a 13.17% improvement in segmentation accuracy compared to BPG.

[0073] The results of target detection on PASCAL VOC and COCO2017 datasets are listed in the comparison of bit rate and target detection performance on PASCAL VOC and COCO2017. Figure 5a 、 Figure 5b , Table 1 and Table 2. Figure 5a The rate-distortion performance of this method and the existing methods on PASCAL VOC is shown. From the figure, we can see that this method and the existing methods can achieve the highest target detection performance at a lower bit rate. Figure 5b The rate-distortion performance of this method and existing methods on COCO2017 is demonstrated. Figure 5b This method demonstrates that it can achieve the highest object detection performance at a lower bitrate than existing methods. The two lists, Det.(VOC) and Det.(COCO), represent object detection tasks performed on the PASCAL VOC and COCO datasets, respectively. Compared to other image compression methods, this method achieves the largest coding gain and detection performance improvement on the PASCAL VOC dataset, reaching -74.24% and 3.27%, respectively. Furthermore, compared to other image compression methods, this method also achieves the largest coding gain and detection performance improvement on the COCO2017 dataset, reaching -95.17% and 13.89%, respectively.

[0074] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A machine vision encoding method based on self-supervised learning, characterized in that: The following steps are involved: After data enhancement, the image information is randomly sampled into sub-blocks, where the sub-blocks with semantic content are positive samples and the sub-blocks without semantic content are negative samples. The positive and negative samples are input into the backbone network head to extract and transform the feature channel to obtain the first feature; The first feature is transformed using a redundancy elimination encoder to obtain a feature of a low-dimensional space, uniform noise is added to the feature of the low-dimensional space using a first quantizer to obtain a quantized feature, and the quantized feature is transferred to a redundancy elimination decoder through arithmetic coding and decoding to reconstruct the compressed feature to obtain a second feature; The encoder transforms the second feature into a low-dimensional space, and then uses a second quantizer to add uniform noise to reduce redundancy. The super-prior encoder extracts and encodes the side information of the second feature after redundancy reduction. The side information is passed to the super-prior decoder for decoding through arithmetic coding and decoding. The probability distribution parameters and bit rate of the second feature are then predicted using a mixed Gaussian entropy model. The decoder is then used to reconstruct the dimension of the encoded feature to obtain the third feature. A convolutional neural network is used to extract and transform the dimension of the third feature. At the same time, the convolution features are extracted and weighted into a heat map. The heat map is used to guide the selection of valid positive samples of the input image to obtain the encoding result.

2. A machine vision encoding method based on self-supervised learning according to claim 1, characterized in that: The noise values ​​of the uniform noise added by the first quantizer and the second quantizer are both within the open range of -0.5 to 0.

5.

3. A machine vision encoding system based on self-supervised learning, characterized in that: The system comprises a backbone network head, a redundancy elimination module, a compression network and a backbone network tail connected in sequence. When the system is running, it implements a machine vision encoding method based on self-supervised learning as described in any one of claims 1-2.

4. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The backbone network head includes two parts of residual networks, the first part is the Stem layer in the residual convolutional network ResNet50, and the second part is the first layer network in the residual convolutional network ResNet50.

5. A machine vision encoding system based on self-supervised learning according to claim 4, characterized in that: The backbone network head receives the heat map as a semantic guide to select valid positive samples of the input image.

6. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The redundancy elimination module includes a redundancy elimination encoder, a first quantizer, a first arithmetic encoder, a first arithmetic decoder and a redundancy elimination decoder. The redundancy elimination encoder and the redundancy elimination decoder are mirror networks. The first quantizer is used to add noise to the features of the low-dimensional space. The first arithmetic encoder and the first arithmetic decoder are used to compress data into a code stream for storage and transmission.

7. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The compression network includes an encoder, a super-prior encoder, a super-prior decoder, a second quantizer, a second arithmetic encoder, a second arithmetic decoder and a decoder. The super-prior encoder and super-prior decoder are used to cooperate with a mixed Gaussian entropy model to predict the probability distribution of the encoder output features. The decoder and encoder are a pair of mirror networks, both of which include two layers of gated recurrent convolutional networks.

8. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: The tail of the backbone network includes the second, third and fourth convolutional neural networks of the residual convolutional network ResNet50. The tail of the backbone network receives the third feature and extracts the semantic content feature as output.

9. A machine vision encoding system based on self-supervised learning according to claim 8, characterized in that: The last layer of convolutional features in the tail of the backbone network is summed along the channel dimension to obtain a heat map, which is fed back to the input image as a semantic guide to select positive samples with semantic content.

10. A machine vision encoding system based on self-supervised learning according to claim 3, characterized in that: During system training, the entire system is end-to-end trained in a self-supervised form of contrastive learning, which is divided into three stages. In the first stage, only the backbone network head and the backbone network tail are trained. In the second stage, the weights of the backbone network head and the backbone network tail are fixed, and only the redundancy elimination module is trained. In the third stage, the weights of the backbone network head, the backbone network tail, and the redundancy elimination module are fixed, and only the compression network is trained. The loss function during training is the weighted sum of the bit rate of the compressed features and the prediction accuracy of the machine vision analysis task.

Images