Image encoding method and system based on vit fusing high-value local information

Abstract

The application provides a ViT-based image coding method and system fusing high-value local information, which comprises the following steps: acquiring an image to be compressed, performing global feature extraction on the image to be compressed based on window division to obtain information between windows; obtaining global information based on the information between windows; performing feature extraction within the window based on the global information to obtain local information; and fusing the global information and the local information based on a self-attention mechanism to obtain fused information, thereby completing the ViT-based image coding method fusing high-value local information. The technical scheme of the application can realize low-computing, low-code-rate and low-loss image compression technology.

Description

Technical Field

[0001] The present invention belongs to the technical field of self-attention mechanism processing image data, and specifically relates to an image encoding method and system based on ViT that integrates high-value local information. Background Art

[0002] Image compression refers to the technology of representing the original pixel matrix with fewer bits, losslessly or losslessly, to reduce the amount of data required to represent the digital image. It is also called image coding.

[0003] Generally speaking, ViT (Vision Transformer) is a reliable and effective method for processing images using the self-attention mechanism. It divides the image into 16x16 windows, then projects each window into a fixed-length vector and feeds it into the transformer. This reduces the input queue length and algorithm complexity. Because ViT uses the entire window as input, local information within the window is lost. When using ViT to extract image features, it optionally incorporates a self-attention mechanism between some windows to compensate for this lost local information. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention discloses an image coding method and system based on ViT that integrates high-value local information. Global information is first generated, and the global information provides guidance for selecting local information for extraction, so that the selected patch blocks are more relevant and the local information extracted within the window is more valuable.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] The image coding method based on ViT and integrating high-value local information includes the following steps:

[0007] Acquire an image to be compressed, and perform global feature extraction based on window division on the image to be compressed to obtain information between windows; and obtain global information based on the information between windows;

[0008] Based on the global information, feature extraction is performed within the window to obtain local information;

[0009] Based on the self-attention mechanism, the global information is fused with the local information to obtain fused information, thereby completing image encoding based on ViT-based fusion of high-value local information.

[0010] Preferably, the image to be compressed is evenly divided into a number of non-overlapping windows, and a class token of the size of a patch is added to each window;

[0011] Based on the self-attention mechanism, feature extraction is performed on the window with the class token added to obtain the class token representing the features of the current window;

[0012] Based on the self-attention mechanism, feature extraction is performed on all class tokens representing the features of the current window to obtain the global information.

[0013] Preferably, based on the global information, a correlation matrix of window-to-window affinities is obtained;

[0014] Based on the correlation matrix, a window that satisfies a preset correlation with each window is selected to obtain a search matrix;

[0015] Based on the retrieval matrix, the self-attention mechanism is used to extract features within the window to obtain the local information.

[0016] Preferably, the method for fusing the global information with the local information is:

[0017] Based on the global information, obtaining a query matrix;

[0018] Based on the local information, obtaining a key matrix;

[0019] Performing a self-attention mechanism on the query matrix and the key matrix to obtain a weight parameter matrix;

[0020] The weight parameter matrix is ​​multiplied by the global information, and the multiplication result is added to the local information to obtain the fusion information.

[0021] The present invention also provides an image coding system based on ViT and integrating high-value local information, which is used to implement the method described above, including:

[0022] A global information acquisition module is used to acquire an image to be compressed, and perform global feature extraction based on window division on the image to be compressed to obtain information between windows; and obtain global information based on the information between windows;

[0023] A local information acquisition module, configured to extract features within a window based on the global information to obtain local information;

[0024] The information fusion module is used to fuse the global information with the local information based on the self-attention mechanism to obtain fused information and complete the image encoding based on ViT fusion of high-value local information.

[0025] Preferably, the global information acquisition module includes:

[0026] A division unit, configured to divide the image to be compressed into a plurality of non-overlapping windows on average, and add a class token of a patch size to each window;

[0027] The first feature extraction unit is used to extract features from the window with the class token added based on the self-attention mechanism to obtain the class token representing the features of the current window;

[0028] The second feature extraction unit is used to extract features of all class tokens representing the features of the current window based on the self-attention mechanism to obtain the global information.

[0029] Preferably, the local information acquisition module includes:

[0030] A correlation matrix acquisition unit, configured to obtain a correlation matrix of window-to-window affinities based on the global information;

[0031] A retrieval matrix acquisition unit, configured to select a window that satisfies a preset correlation with each window based on the correlation matrix to obtain a retrieval matrix;

[0032] The local information acquisition unit is used to extract features within the window based on the retrieval matrix using a self-attention mechanism to obtain the local information.

[0033] Preferably, the information fusion module includes:

[0034] A query matrix acquisition unit, configured to obtain a query matrix based on the global information;

[0035] A key matrix acquisition unit, configured to obtain a key matrix based on the local information;

[0036] A weight parameter matrix acquisition unit is used to perform a self-attention mechanism on the query matrix and the key matrix to obtain a weight parameter matrix;

[0037] An information fusion unit is used to multiply the weight parameter matrix with the global information, and add the multiplication result to the local information to obtain the fused information.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows: the global information extraction step disclosed in the present invention can use only the class token of the number of divided windows as the input of the self-attention mechanism, which can obtain the information of the entire image and reduce the input involved in the calculation. The local feature extraction step selectively extracts the information of the internal patch between windows under the guidance of the window correlation matrix, which reduces the calculation and ensures that the extracted local information comes from more relevant windows. The feature information fusion step fuses the global information with the local information, so that the final result contains the overall information and the local information properly fused at the same latitude, so that the final compression result saves information more comprehensively, the loss of graphic information can be effectively reduced, and at the same time, the computational burden will be reduced because the local information between all windows is not calculated. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 This is a flow chart of an image coding method based on ViT and integrating high-value local information according to an embodiment of the present invention;

[0041] Figure 2 This is a flowchart of global information extraction according to an embodiment of the present invention;

[0042] Figure 3 This is a flow chart of local information extraction according to an embodiment of the present invention;

[0043] Figure 4 This is a flowchart of information fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Example 1

[0047] like Figure 1As shown, the image coding method based on ViT that integrates high-value local information includes the following steps:

[0048] like Figure 2 As shown, S1: obtain the image to be compressed, and perform global feature extraction based on window division on the image to be compressed to obtain information between windows; based on the information between windows, obtain global information;

[0049] Figure 2 In the example, Partition: window division; embedding: converting images into vectors; MSA: self-attention mechanism; N2: the number of partitioned windows; N1: the number of partitioned patches within a window

[0050] A further implementation method is to divide the image to be compressed into several non-overlapping windows and add a class token of the patch size to each window;

[0051] Based on the self-attention mechanism, feature extraction is performed on the window with the class token added to obtain the class token representing the features of the current window;

[0052] Based on the self-attention mechanism, feature extraction is performed on all class tokens representing the features of the current window to obtain global information.

[0053] In this embodiment, if Figure 2 As shown, for the first-level image segmentation, a 256*256 image is divided into 16 64*64 regions, each of which is called a window. For the second-level image segmentation, that is, the window segmentation, a 64*64 window is divided into 16 16*16 regions, each of which is called a patch. The principle of the class token is to add an element of the same size as the sequence element to the head of the self-attention mechanism input sequence. The content is random. After the self-attention mechanism interacts with the input content, its content changes and can represent the characteristics of the input content, representing this content in downstream classification tasks.

[0054] The image is evenly divided into multiple non-overlapping windows. A class token of the same size as a patch is added to each window. A self-attention mechanism is applied only within each window. The input unit for this self-attention mechanism is the patch that is further evenly divided into the window. The added class token is the same size as the patch, with randomly initialized content and placed at the top of the input queue. The class token output by each window represents the characteristics of the current window. A self-attention mechanism is then applied to all class tokens, with the input unit being the class token and the number of partitioned windows. Ultimately, each class token contains information about all other windows, generating global information.

[0055] S2: Based on global information, feature extraction is performed within the window to obtain local information;

[0056] A further embodiment is to obtain a window-to-window affinity correlation matrix based on global information;

[0057] Based on the correlation matrix, select the window that meets the preset correlation with each window to obtain the retrieval matrix;

[0058] In this embodiment, if Figure 3 As shown in the figure, the class token inserted into each window in the previous step replaces the global information of the window features. When generated, the class token has the same dimension as the feature vector within the window, meaning that C is a feature vector of the size of a patch representing the class token. When implementing a global self-attention mechanism, the Q, K, and V tensors are linearly mapped through C. W is the matrix of the self-attention mechanism. When using the self-attention mechanism, three parameter matrices are generated. These three matrices change continuously during training. The process of training the model is essentially the process of continuously revising these three matrices.

[0059] Q=C r W q (1)

[0060] K=C r W k (2)

[0061] V=C r W v (3)

[0062] Afterwards, through Q r and transpose K r The matrix multiplication between them derives the adjacency matrix (correlation matrix) of the window and window affinity

[0063] A r =Q r (K r ) T (4)

[0064] Adjacency matrix A r The entries in measure the semantic relevance between two regions. The next core step is to retain only the top k most closely related regions for each region. That is, using a routing index matrix (retrieval matrix) With row-by-row topk operator.

[0065] I r =topkIndex(A r ) (5)

[0066] Line i r Contains the indices of the k most relevant windows for the i-th window. Each window performs self-attention only with its k most relevant windows, generating local information. The scope is selected over multiple windows, and the input sequence size is a patch.

[0067] Based on the retrieval matrix, the self-attention mechanism is used to extract features within the window and obtain local information.

[0068] S3: Based on the self-attention mechanism, global information is fused with local information to obtain fused information, and image encoding based on ViT that integrates high-value local information is completed.

[0069] A further implementation method is to fuse global information with local information by:

[0070] Based on global information, obtain the query matrix;

[0071] Based on the local information, a key matrix is ​​obtained;

[0072] Apply self-attention mechanism to the query matrix and key matrix to obtain the weight parameter matrix;

[0073] The weight parameter matrix is ​​multiplied by the global information, and the multiplication result is added to the local information to obtain the fused information.

[0074] In this embodiment, after completing the first two tasks, the global information S u The class token and each patch include selectively obtained more valuable local information M. Global information S u It is generated by the attention mechanism between window features S.

[0075]

[0076] q, k, and v represent tensors, and d is a scaling factor. Its purpose is to make the gradient descent more stable during the softmax process and avoid stagnation of model parameter updates due to too small gradients.

[0077] Local information M is the global information S u Under the guidance of , the attention mechanism is performed between the TopK related windows selected in the adjacency matrix.

[0078] V i =V[TopK(S u )] (7)

[0079] M=Attn[V i ]V i (8)

[0080] Next, we need to convert the global information S u Fusing with local information M, since global information is obtained from window features, and local information is obtained from the self-attention mechanism of the patch within the window, the two kinds of information are not in the same order of magnitude. They cannot be added directly. In processing the fusion of the two kinds of information, this study uses the self-attention mechanism to find the relationship weight parameter between the window features and the specific patch within the window, so that the order of magnitude of the product of the global information and the relationship weight parameter is the same as that of the local information. Figure 4 As shown, the specific implementation is to first make the global information S u Multiply by matrix W1 to get query matrix Q S , local information M multiplied by matrix W2 to obtain key matrix K M , based on the obtained Q S , K M Do the self-attention mechanism and get the relationship parameter matrix (weight parameter matrix) λ of the window to all the patch blocks in the window. u Multiply by λ and add to the local information M to get the final fusion result. Here W1 and W2 are the W in formula (2) k , do not reuse W k The reason is that the method is the same each time, but the content of the matrix W obtained after each training is different.

[0081]

[0082] Result=M+λS u (10)

[0083] In the global information extraction step, class tokens are used instead of windows to extract global information. In the local feature extraction step, the window correlation matrix is ​​generated through the global feature information. Under the guidance of the correlation matrix, the information of the internal patch between windows is selectively extracted. In the feature information fusion step, weight parameters are generated through the self-attention mechanism, so that the global information can be fused with the local information at the same level through the weight parameters.

[0084] The principle and method of the self-attention mechanism used in this invention are the same, but the input sequence is different. This invention uses a total of four self-attention mechanisms, namely:

[0085] 1. Extract the class token; the input sequence is a small patch within each window (not across windows), and the class token is added to the head;

[0086] 2. Get global information; the input sequence is the class token produced for each window

[0087] 3. Obtain local information; the input sequence is a patch within the window (can be across windows), but under the guidance of the search matrix, it is not done between all patches.

[0088] 4. Get the weight parameter of the conversion level; the input is two sequences, one is the global feature and the other is the local feature, but for the intermediate state, only the Q provided by the global feature is used. S , and K provided by local features M Fusion to obtain weight parameters.

[0089] Example 2

[0090] The present invention also provides an image coding system based on ViT that integrates high-value local information, which is used to implement an image compression method, including

[0091] The global information acquisition module is used to obtain the image to be compressed, and perform global feature extraction based on window division on the image to be compressed to obtain information between windows; based on the information between windows, obtain global information;

[0092] The local information acquisition module is used to extract features within the window based on global information to obtain local information;

[0093] The information fusion module is used to fuse global information with local information based on the self-attention mechanism to obtain fused information and complete image encoding based on ViT that fuses high-value local information.

[0094] A further embodiment is that the global information acquisition module includes:

[0095] The division unit is used to divide the image to be compressed into several non-overlapping windows and add a class token of the patch size to each window;

[0096] The first feature extraction unit is used to extract features from the window with the class token added based on the self-attention mechanism to obtain the class token representing the features of the current window;

[0097] The second feature extraction unit is used to extract features of all class tokens representing the features of the current window based on the self-attention mechanism to obtain global information.

[0098] A further embodiment is that the local information acquisition module includes:

[0099] A correlation matrix acquisition unit, configured to obtain a correlation matrix of window-to-window affinities based on global information;

[0100] A retrieval matrix acquisition unit is used to select a window that satisfies a preset correlation with each window based on the correlation matrix to obtain a retrieval matrix;

[0101] The local information acquisition unit is used to extract features within the window based on the retrieval matrix and use the self-attention mechanism to obtain local information.

[0102] A further embodiment is that the information fusion module includes:

[0103] A query matrix acquisition unit, configured to obtain a query matrix based on global information;

[0104] A key matrix acquisition unit, configured to obtain a key matrix based on local information;

[0105] The weight parameter matrix acquisition unit is used to perform a self-attention mechanism on the query matrix and the key matrix to obtain the weight parameter matrix;

[0106] The information fusion unit is used to multiply the weight parameter matrix with the global information and add the multiplication result with the local information to obtain fused information.

[0107] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. An image coding method based on ViT that integrates high-value local information, characterized in that: The following steps are involved: Acquire an image to be compressed, and perform global feature extraction based on window division on the image to be compressed to obtain information between windows; Obtain global information based on the information between windows; Based on the global information, feature extraction is performed within the window to obtain local information; Based on the self-attention mechanism, the global information is fused with the local information to obtain fused information, thereby completing image encoding based on ViT that fuses high-value local information. The method for obtaining the local information is: Based on the global information, obtaining a correlation matrix between windows and window affinities; Based on the correlation matrix, a window that satisfies a preset correlation with each window is selected to obtain a search matrix; wherein the correlation matrix selects a window that satisfies a preset correlation with each window by applying a topk operator row by row, wherein the preset correlation refers to the top k most closely related regions; Based on the retrieval matrix, the self-attention mechanism is used to extract features within the window to obtain the local information.

2. The image coding method based on ViT and integrating high-value local information according to claim 1, characterized in that: The method for obtaining the global information is: Divide the image to be compressed into several non-overlapping windows, and add a class token of the patch size to each window; Based on the self-attention mechanism, feature extraction is performed on the window with the class token added to obtain the class token representing the features of the current window; Based on the self-attention mechanism, feature extraction is performed on all class tokens representing the features of the current window to obtain the global information.

3. The image coding method based on ViT and integrating high-value local information according to claim 1, characterized in that: The method of fusing the global information with the local information is: Based on the global information, obtaining a query matrix; Based on the local information, obtaining a key matrix; Performing a self-attention mechanism on the query matrix and the key matrix to obtain a weight parameter matrix; The weight parameter matrix is ​​multiplied by the global information, and the multiplication result is added to the local information to obtain the fusion information.

4. The image coding system based on ViT that integrates high-value local information is characterized by: The method for implementing any one of claims 1 to 3 comprises: A global information acquisition module is used to acquire an image to be compressed, and perform global feature extraction based on window division on the image to be compressed to obtain information between windows; and obtain global information based on the information between windows; A local information acquisition module, configured to extract features within a window based on the global information to obtain local information; An information fusion module is used to fuse the global information with the local information based on a self-attention mechanism to obtain fused information and complete image encoding based on ViT that fuses high-value local information; The local information acquisition module includes: A correlation matrix acquisition unit, configured to obtain a correlation matrix of window-to-window affinities based on the global information; a search matrix acquisition unit, configured to select windows that satisfy a preset correlation with each window based on the correlation matrix to obtain a search matrix; wherein the correlation matrix selects windows that satisfy a preset correlation with each window by applying a topk operator row by row, wherein the preset correlation refers to the top k most closely related regions; The local information acquisition unit is used to extract features within the window based on the retrieval matrix using a self-attention mechanism to obtain the local information.

5. The image coding system based on ViT and integrating high-value local information according to claim 4, characterized in that: The global information acquisition module includes: A division unit, configured to divide the image to be compressed into a plurality of non-overlapping windows on average, and add a class token of a patch size to each window; The first feature extraction unit is used to extract features from the window with the class token added based on the self-attention mechanism to obtain the class token representing the features of the current window; The second feature extraction unit is used to extract features of all class tokens representing the features of the current window based on the self-attention mechanism to obtain the global information.

6. The image coding system based on ViT and integrating high-value local information according to claim 5, characterized in that: The information fusion module includes: A query matrix acquisition unit, configured to obtain a query matrix based on the global information; A key matrix acquisition unit, configured to obtain a key matrix based on the local information; A weight parameter matrix acquisition unit is used to perform a self-attention mechanism on the query matrix and the key matrix to obtain a weight parameter matrix; An information fusion unit is used to multiply the weight parameter matrix with the global information, and add the multiplication result to the local information to obtain the fused information.

Images