Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment. The image processing method comprises the following steps: acquiring N element embedding vectors of N image blocks of an image to be processed, and generating a query matrix, a key matrix and a value matrix of the image to be processed according to the N element embedding vectors; forming a proxy matrix according to the at least one feature matrix of the to-be-processed image; forming a compressed query matrix of the to-be-processed image according to the dot product of the query matrix and the proxy matrix, and forming a compressed key matrix of the to-be-processed image according to the dot product of the proxy matrix and the key matrix; performing matrix multiplication on the compression key matrix and the value matrix to obtain a compression feature matrix, and performing matrix multiplication on the compression query matrix and the compression feature matrix to obtain a fusion feature vector matrix of the to-be-processed image; and inputting the fused feature vector matrix into a multi-layer perceptron to identify the to-be-processed image. According to the embodiment of the invention, the calculation complexity of the image recognition model can be reduced, and the image recognition accuracy is improved.

Description

Background Art

[0002] The Vision Transformer (ViT) is a deep learning model that applies the Transformer architecture to visual recognition tasks. Compared with traditional convolutional neural networks (CNNs), ViT can capture information at any position in the image through a self-attention mechanism, without being limited by the local receptive field, so as to better learn the global features of the image. It is very advantageous for processing large-scale image tasks and tasks that require understanding the overall semantics of the image, such as image classification and scene understanding.

[0003] However, since ViT needs to perform global self-attention calculations on each image block, the computational complexity is high, especially when processing high-resolution images, which requires a lot of computing resources and a long training time. The huge number of parameters and calculations in the model itself brings more computing power overhead and higher hardware requirements, making it difficult to deploy and apply the ViT model on devices with limited computing power.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide an image processing method and an electronic device for reducing the amount of calculation and the amount of parameters of the ViT model, thereby achieving the purpose of compressing the model without reducing the model recognition performance.

[0006] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, comprising: obtaining N element embedding vectors of N image blocks of an image to be processed, generating a query matrix, a key matrix and a value matrix of the image to be processed according to the N element embedding vectors, wherein the dimensions of the query matrix, the key matrix and the value matrix are all n×d, and d is the dimension of a feature vector of each of the image blocks; forming a proxy matrix according to at least one feature matrix of the image to be processed, wherein the feature matrix includes the query matrix, the key matrix, the value matrix and a meta matrix composed of the N element embedding vectors, and the proxy matrix The dimension of the rational matrix is ​​n×d, n is a preset value, 1≤n<N; a compressed query matrix of the image to be processed is formed according to the dot product of the query matrix and the proxy matrix, and a compressed key matrix of the image to be processed is formed according to the dot product of the proxy matrix and the key matrix; a matrix multiplication operation is performed on the compressed key matrix and the value matrix to obtain a compressed feature matrix, and a matrix multiplication operation is performed on the compressed query matrix and the compressed feature matrix to obtain a fused feature vector matrix of the image to be processed; the fused feature vector matrix is ​​input into a multilayer perceptron to identify the image to be processed.

[0007] In an exemplary embodiment of the present disclosure, forming a proxy matrix according to at least one feature matrix of the image to be processed includes: performing a pooling operation on at least one of the feature matrices to obtain a pooled feature matrix of n×d dimensions; and forming the proxy matrix of n×d dimensions according to the pooled feature matrix.

[0008] In an exemplary embodiment of the present disclosure, performing a pooling operation on at least one of the feature matrices to obtain a pooled feature matrix of n×d dimensions includes: performing a pooling operation on the query matrix and the key matrix respectively to obtain a first pooled feature matrix of n×d dimensions corresponding to the query matrix and a second pooled feature matrix of n×d dimensions corresponding to the key matrix; performing bitwise weighted summation on the first pooled feature matrix and the second pooled feature matrix to form the proxy matrix of n×d dimensions.

[0009] In an exemplary embodiment of the present disclosure, performing a pooling operation on at least one of the feature matrices to obtain an n×d-dimensional pooled feature matrix includes: performing a linear transformation on the element matrix to convert it into an N×d-dimensional element feature matrix; performing a pooling operation on each column of the element feature matrix to form n pooling output values; and forming the proxy matrix according to the n pooling output values ​​corresponding to each column of the d columns of the element matrix.

[0010] In an exemplary embodiment of the present disclosure, forming a proxy matrix based on at least one feature matrix of the image to be processed includes: performing clustering calculations on N element embedding vectors in the element matrix to divide the N element embedding vectors into n clusters; and forming the proxy matrix based on the element embedding vector closest to the centroid in each cluster.

[0011] In an exemplary embodiment of the present disclosure, forming a proxy matrix according to at least one characteristic matrix of the image to be processed includes: performing a linear transformation on the element matrix through a preset linear transformation matrix to form the proxy matrix.

[0012] In an exemplary embodiment of the present disclosure, forming a compressed query matrix of the image to be processed according to the dot product of the query matrix and the proxy matrix, and forming a compressed key matrix of the image to be processed according to the dot product of the proxy matrix and the key matrix include: normalizing the dot product of the query matrix and the proxy matrix to form the compressed query matrix; normalizing the dot product of the proxy matrix and the key matrix to form the compressed key matrix.

[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0014] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the image processing method as described in any one of the above is implemented.

[0015] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0016] The disclosed embodiment sets a proxy matrix of lower dimension formed according to the feature matrix of the image to be processed, uses the proxy matrix to perform dimension compression on the query matrix and key matrix of the image to be processed, and uses the value matrix and the compressed key matrix to perform operations before performing the attention calculation. This can greatly reduce the computational complexity of the ViT model's attention calculation for the image to be processed, while improving the model's image recognition capability, reducing the hardware cost of deploying the ViT model, and improving the image processing efficiency of the ViT model, while achieving cost reduction and efficiency and accuracy improvement.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 is a flowchart of an image processing method in an exemplary embodiment of the present disclosure.

[0020] Figure 2 It is a schematic diagram of the proxy matrix participating in dimensionality reduction in one embodiment of the present disclosure.

[0021] Figure 3 It is a schematic diagram of the generation process of the fused feature vector matrix in one embodiment of the present disclosure.

[0022] Figure 4 It is a schematic diagram of the position of the fused feature vector matrix output by the embodiment of the present disclosure in the Vit model.

[0023] Figure 5 It is a schematic diagram for verifying the method of the embodiment of the present disclosure through experiments.

[0024] Figure 6 is a block diagram of an image processing apparatus in an exemplary embodiment of the present disclosure.

[0025] Figure 7 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0027] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0028] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 is a flowchart of an image processing method in an exemplary embodiment of the present disclosure.

[0030] refer to Figure 1 , the image processing method 100 may include:

[0031] Step S1, obtaining N element embedding vectors of N image blocks of an image to be processed, and generating a query matrix, a key matrix and a value matrix of the image to be processed according to the N element embedding vectors, wherein the dimensions of the query matrix, the key matrix and the value matrix are all n×d, where d is the dimension of a feature vector of each image block;

[0032] Step S2, forming a proxy matrix according to at least one feature matrix of the image to be processed, wherein the feature matrix includes the query matrix, the key matrix, the value matrix, and a meta-matrix composed of the N meta-embedded vectors, and the dimension of the proxy matrix is ​​n×d, where n is a preset value, 1≤n<N;

[0033] Step S3, forming a compressed query matrix of the image to be processed according to the dot product of the query matrix and the proxy matrix, and forming a compressed key matrix of the image to be processed according to the dot product of the proxy matrix and the key matrix;

[0034] Step S4, performing a matrix multiplication operation on the compression key matrix and the value matrix to obtain a compression feature matrix, and performing a matrix multiplication operation on the compression query matrix and the compression feature matrix to obtain a fusion feature vector matrix of the image to be processed;

[0035] Step S5: input the fused feature vector matrix into a multi-layer perceptron to identify the image to be processed.

[0036] The disclosed embodiment sets a proxy matrix of lower dimension formed according to the feature matrix of the image to be processed, uses the proxy matrix to perform dimension compression on the query matrix and key matrix of the image to be processed, and uses the value matrix and the compressed key matrix to perform operations before performing the attention calculation. This can greatly reduce the computational complexity of the ViT model's attention calculation for the image to be processed, while improving the model's image recognition capability, reducing the hardware cost of deploying the ViT model, and improving the image processing efficiency of the ViT model, while achieving cost reduction and efficiency and accuracy improvement.

[0037] Next, each step of the image processing method 100 is described in detail.

[0038] In step S1, N element embedding vectors of N image blocks of the image to be processed are obtained, and a query matrix, a key matrix and a value matrix of the image to be processed are generated according to the N element embedding vectors, wherein the dimensions of the query matrix, the key matrix and the value matrix are all n×d, where d is the dimension of the feature vector of each image block.

[0039] The method of the embodiment of the present disclosure can be used to improve the ViT visual model. The ViT visual model mainly includes an input layer, a Transformer encoder and an output layer.

[0040] The image to be processed is processed in the input layer to form a fused feature vector of the image which is input into the Transformer encoder. Finally, the output vector of the Transformer encoder is processed as the input of the output layer to obtain the recognition result or classification result of the image to be processed.

[0041] In the input layer, the image to be processed (also called the input image) is first divided into multiple fixed-size image blocks (patches). For example, a 224x224 pixel image is divided into N 16x16 pixel patches, so that the entire image becomes a grid, and each patch (image block) is calculated as an independent input unit. Next, each patch is flattened into a one-dimensional vector and mapped to a feature vector space of a specific dimension through a linear embedding layer to obtain a patch embedding vector. For example, for a patch of size 16×16×3 pixels, it is flattened into a one-dimensional vector of length (assuming it is an RGB image) 768, and then converted into a patch embedding vector of dimension D (such as D=768) through a linear embedding layer.

[0042] At the same time, since the Transformer architecture itself is insensitive to the order of the input sequence, and the position information of the patch in the image is very important for understanding the image, a position embedding vector is also added. The dimension of the position embedding vector is the same as the patch embedding vector, which is used to encode the position information of each patch in the image to be processed. These position embedding vectors can be fixed (such as generated by a sine-cosine function) or learned. Finally, the patch embedding vector and the position embedding vector corresponding to each patch are added to obtain the meta-embedding vector corresponding to the patch. The meta-embedding vectors of all N patches of the image to be processed are input into the Transformer encoder.

[0043] The Transformer encoder is the core part of the ViT model, which is composed of alternating Multi-Head Self-Attention (MSA) layers and Multi-Layer Perceptron (MLP) blocks.

[0044] Among them, the multi-head self-attention layer can calculate multiple different attention heads in parallel, each head focuses on different aspects of the input sequence, so as to capture the long-range dependencies between image blocks and enable the model to learn the associations between different parts of the image, such as the positional relationship between different parts of an object in the image.

[0045] In the input part of the multi-head self-attention layer, in order to calculate the attention scores between different patches of the image, the meta-embedding vector of the patch must first be linearly transformed to obtain three vectors representing the different contents of the patch. These three vectors are Q (Query), K (Key), and V (Value).

[0046] The query vector (Q) is used to match the key vector (K) to determine which parts need to be focused on in the attention mechanism.

[0047] The key vector (K) is used to match the query vector (Q), and the correlation between them is measured by calculating the dot product.

[0048] The value vector (V) is used to calculate the final attention output after the relevance is determined.

[0049] Specifically, for a meta-embedding vector x corresponding to a patch, Q = xW Q , K = xW K , V = xW V , where W Q , W K , W V , is a learnable weight matrix.

[0050] Specifically, for each patch, the meta-embedding vector obtained after linear embedding and position embedding is transformed into Q, K, and V vectors respectively through linear transformation. These vectors are used as the input of the multi-head self-attention mechanism. After being processed by the multi-head self-attention mechanism, the output is the fused feature vector that integrates the relationship information between each patch. These fused feature vectors will be passed to the subsequent layers, usually the multi-layer perceptron (MLP).

[0051] The query vector Q, key vector K, and value vector V of a patch are all of dimension d, and d can be the dimension after the meta-embedding vector is flattened to one dimension. For example, for a patch of size 16×16×3 pixels, it is flattened into a one-dimensional vector of length (assuming it is an RGB image) 768. After passing through the linear embedding layer, this vector may be converted into a vector of dimension d (for example, d=768). This d-dimensional vector contains the feature information of the patch. In the multi-head self-attention mechanism, the dimensions of the Q, K, and V vectors are all d, and d represents the number of feature dimensions of each vector, which determines the richness and complexity of the features that the model can learn. A higher dimension means that the model has more "degrees of freedom" to learn and represent complex features, but it may also bring higher computational costs and overfitting risks. Therefore, d can be the dimension of the meta-embedding vector flattened to one dimension, or it can be customized by a technician in this field according to actual needs (by setting the embedding model), and the present disclosure does not impose any special restrictions on this.

[0052] Next, the N query vectors Q corresponding to the N patches of the image to be processed are taken as rows to form a query matrix corresponding to the image to be processed. The query matrix has N rows (the i-th row corresponds to the i-th query vector Qi), and the dimension of each row is d (the dimension of Q). The dimension of the final query matrix is ​​N×d. Similarly, the key matrix and value matrix corresponding to the image to be processed are formed, and the dimensions of the key matrix and the value matrix are both N×d. To simplify the expression later, the query matrix is ​​called the Q matrix, the key matrix is ​​called the K matrix, and the value matrix is ​​called the V matrix.

[0053] In related technologies, the Q matrix is ​​dot-producted with the K matrix to obtain an attention score matrix that reflects the correlation between each patch, and then further calculated based on these scores and the V matrix (composed of the V vectors of all patches) to complete information fusion and feature extraction under the self-attention mechanism, thereby realizing self-attention calculation of the image to be processed. Each Q vector in the Q matrix is ​​used to query the key information of other patches in the self-attention mechanism to determine the degree of attention of each patch to other patches. Dot product, also known as scalar product or inner product, is a way of operating on two vectors. For the Q and K matrices, the dot product can be expressed as scores=QK T , which is the product of the transpose of the Q matrix and the K matrix. Since the dimension of the Q matrix is ​​N×d and the dimension of the transpose of the K matrix is ​​d×N, the dimension of the attention score matrix obtained by multiplying the two is N×N, and the computational complexity of the attention calculation is N 2 Since N is usually very large, this step requires high computing performance of the hardware device. In addition, the obtained attention score matrix needs to be further calculated with the N×d-dimensional V matrix, which further increases the amount of calculation and slows down the output speed of the result.

[0054] Therefore, in the embodiment of the present disclosure, the attention calculation process after obtaining the Q matrix, K matrix, and V matrix is ​​improved, which can greatly reduce the computational complexity of this step and improve processing efficiency, while also improving the feature expression ability of the results, improving the final image classification accuracy, and achieving two-way optimization of cost and efficiency.

[0055] In step S2, a proxy matrix is ​​formed according to at least one feature matrix of the image to be processed, wherein the feature matrix includes the query matrix, the key matrix, the value matrix, and a meta-matrix composed of the N meta-embedded vectors, and the dimension of the proxy matrix is ​​n×d, where n is a preset value, 1≤n<N.

[0056] In an exemplary embodiment, a proxy matrix with a smaller dimension is introduced to participate in the self-attention calculation process to reduce the computational complexity. The proxy matrix is ​​formed according to the feature matrix of at least one of the to-be-processed graphics in the element matrix composed of the Q matrix, the K matrix, the V matrix or the element embedding vector, and can express the global features of the to-be-processed image. That is, the proxy matrix can express the global features and has a lower data dimension, which is highly compressed for the global features and realizes a concise expression of the global features.

[0057] The dimension of the proxy matrix is ​​n×d, where n is much smaller than the number of patches N and can be preset by a technician in this field; d is the same as the column dimension d of the Q matrix, the K matrix, and the V matrix to implement subsequent matrix operations on the Q matrix, the K matrix, and the V matrix.

[0058] In an exemplary embodiment, a pooling operation may be performed on at least one feature matrix to obtain a pooling feature matrix of n×d dimensions, and then a proxy matrix of n×d dimensions may be formed according to the pooling feature matrix.

[0059] Pooling is an operation used in deep learning to reduce data dimensions. Its main purpose is to downsample the input data while retaining important feature information in the data. Common types of pooling operations include:

[0060] Max Pooling: In a given window range, the maximum value is selected as the output. For example, for a 1x4 window, the input is [1,2,3,4], and the output after max pooling is [4]. This method can extract the most significant features in the data.

[0061] Average Pooling: Calculate the average value as the output within a given window range. For example, for the above 1x4 window, the output after average pooling is [2.5]. It can smooth the data and reduce the impact of noise.

[0062] When a matrix of dimension N×d is pooled to form a proxy matrix of dimension n×d, for example, the N elements of each column can be pooled according to the set window size [x,1], so that n pooling output values ​​are obtained in one column, and a pooling feature matrix of dimension n×d is formed according to the pooling output of d columns. A proxy matrix of dimension n×d can be formed by linearly transforming the pooling feature matrix or directly using the pooling feature matrix.

[0063] For example, a linear transformation can be directly performed on the N×m-dimensional meta-matrix formed by the N meta-embedding vectors corresponding to the N patches (m is the dimension of the meta-embedding vector) to convert it into an N×d-dimensional meta-feature matrix. Then, a pooling operation is performed on each column of the meta-feature matrix to form n pooled output values. Finally, the proxy matrix is ​​formed according to the n pooled output values ​​corresponding to each column of the d columns of the meta-matrix.

[0064] Alternatively, for the N×m-dimensional element matrix, pooling is first performed on the rows to obtain d row pooling output values, and then pooling is performed on the columns to obtain n column pooling output values ​​(or the order of rows and columns of the above pooling can be swapped), and finally a proxy matrix of n×d dimensions is formed according to the pooling results in both row and column directions.

[0065] Alternatively, clustering calculation is performed on the N meta-embedding vectors in the meta-matrix to divide the N meta-embedding vectors into n clusters; a proxy matrix is ​​formed according to the meta-embedding vector closest to the centroid in each cluster (a total of n one-dimensional meta-embedding vectors, each of which has a dimension of d).

[0066] Alternatively, the element matrix is ​​directly subjected to linear transformation and dimension reduction through a preset linear transformation matrix to form an n×d dimensional proxy matrix.

[0067] There are many other ways to form a proxy matrix based on the meta-matrix, such as randomly sampling n meta-embedding vectors to form a proxy matrix, principal component analysis (PCA), deformable points processing, feature extraction and pooling based on convolutional neural network (CNN), dimensionality reduction through autoencoders, methods based on graph neural network (GNN) (in specific graph construction scenarios), etc. Those skilled in the art can set them according to actual conditions. This disclosure only gives a few examples and will not enumerate them any more.

[0068] When choosing a method for forming a proxy matrix, you can consider whether to retain the location information of each patch. For example, methods based on sampling the complete meta-embedding vector (such as random sampling strategy or sequential sampling strategy), methods based on convolutional neural network (CNN) combined with pooling, methods based on graph neural network (GNN) (in specific graph construction scenarios), etc. can retain the location information of each patch, while clustering-based sampling methods, linear combination dimensionality reduction methods such as principal component analysis (PCA), and autoencoder methods are more difficult to retain the location information of patches. You can choose a method based on the situation.

[0069] Since the meta-matrix has the global features of the image to be processed, the proxy matrix obtained according to the meta-matrix has both a lower dimension and can express the global features.

[0070] In some embodiments, a proxy matrix may also be obtained according to the Q matrix, the K matrix, and the V matrix.

[0071] In an exemplary embodiment, the Q matrix and the K matrix can be pooled respectively to obtain a first pooled feature matrix (n×d dimension) corresponding to the Q matrix and a second pooled feature matrix (n×d dimension) corresponding to the K matrix. Then, the first pooled feature matrix and the second pooled feature matrix are bitwise weighted summed to form a proxy matrix of dimension n×d.

[0072] When weighting by bit (for example, taking the weighted sum of Q11 and K11 to obtain the element A11 of the proxy matrix A), the weights of each matrix can be preset or implemented by a learnable matrix. In addition, weighted sum operations can be performed, or bitwise multiplication can be performed directly, or the bitwise multiplication result can be multiplied by a preset coefficient. Finally, the Q, K, and V matrices can be selected at the same time for weighted operations or matrix product operations.

[0073] There are many ways to form a proxy matrix with a dimension of n×d based on two or more matrices in the Q, K, and V matrices, such as pooling first and then weighting, or weighting first and then pooling, etc., which are not listed here one by one, and technical personnel in this field can set them according to actual conditions.

[0074] Next, a proxy matrix with lower dimensionality is used to reduce the dimensionality of the self-attention calculation process.

[0075] In step S3, a compressed query matrix of the image to be processed is formed according to the dot product of the query matrix and the proxy matrix, and a compressed key matrix of the image to be processed is formed according to the dot product of the proxy matrix and the key matrix.

[0076] Figure 2 It is a schematic diagram of the proxy matrix participating in dimensionality reduction in one embodiment of the present disclosure.

[0077] refer to Figure 2 In an exemplary embodiment, a dot product operation may be first performed on the Q matrix (N×d) and the transposed (d×n) of the proxy matrix A to obtain an intermediate result of dimension N×n, and then the intermediate result is normalized (e.g., normalized by applying a Softmax function) to obtain a compressed query matrix of dimension N×n. The Softmax operation makes each value in the intermediate result between 0 and 1, and the sum is 1.

[0078] Similarly, the dot product calculation is performed on the proxy matrix A (n×d) and the transpose (d×N) of the K matrix to form an n×N intermediate result, and the intermediate result is normalized (for example, applying the Softmax function for normalization) to form a compressed key matrix of n×N dimension.

[0079] Since n is much smaller than N, the computational complexity of these two calculation processes is low, and the calculation results, namely the compressed query matrix and the compressed key matrix, can be obtained quickly. Moreover, since the Q matrix, K matrix, and A matrix can all express global features, the formed compressed query matrix and compressed key matrix can also express global features in a lower dimension, which is equivalent to compressing the features.

[0080] However, if the compressed query matrix and the compressed key matrix are directly subjected to dot product calculation, i.e., self-attention calculation, the computational complexity is still N×N, and the optimization effect is limited. Therefore, the embodiment of the present disclosure is also provided to use the V matrix for processing before the self-attention calculation to further reduce the computational complexity.

[0081] In step S4, matrix multiplication operation is performed on the compressed key matrix and the value matrix to obtain a compressed feature matrix, and matrix multiplication operation is performed on the compressed query matrix and the compressed feature matrix to obtain a fused feature vector matrix of the image to be processed.

[0082] Figure 3 It is a schematic diagram of the generation process of the fused feature vector matrix in one embodiment of the present disclosure.

[0083] Also refer to Figure 2 and Figure 3 After obtaining the compressed query matrix and the compressed key matrix, the compressed key matrix of n×N dimension and the V matrix of N×d dimension are used to perform matrix multiplication to obtain the compressed feature matrix of n×d dimension.

[0084] The V matrix includes the value vector of each patch, where the "value" can be understood as the eigenvalue containing the rich semantic information of each patch. It is an important carrier used by the model to transmit and fuse information in the self-attention mechanism. For example, in an image containing an object, the V vector corresponding to each patch (constituting the V matrix) may contain the color, texture and other characteristic information of the object part represented by the patch. When the V matrix is ​​multiplied, these eigenvalues ​​will be recombined and fused according to the other matrix of the multiplication, so as to obtain an output vector containing more comprehensive semantic information, which is used for subsequent image understanding tasks such as classification and detection.

[0085] Therefore, the compressed feature matrix is ​​the result of multiplying the V matrix according to the compressed key matrix. The eigenvalues ​​of each patch are recombined and fused according to the global feature weights expressed by the compressed key matrix. The obtained compressed feature matrix further integrates the global features.

[0086] Finally, matrix multiplication is performed using the N×n dimensional compressed query matrix and the lower dimensional compressed key matrix (n×d) to further fuse the features of each patch according to the correlation between patches (expressed by the compressed query matrix and the compressed key matrix), forming an N×d dimensional fused feature vector matrix that can express global features and self-attention between patches.

[0087] Since n is much smaller than N, the computational complexity of matrix multiplication or dot product calculation is greatly reduced, and the above calculation process can obtain results very quickly. The specific speed can be set by adjusting the n value (i.e., adjusting the degree of feature compression).

[0088] Although the size of the N×d dimensional fused feature vector matrix is ​​the same as that of the fused feature vector matrix in the related art, the calculation speed has been greatly improved. It can express more global features (multiple use of proxy matrices for expressing global features), achieving a double improvement in expression effect and calculation efficiency.

[0089] Next, in step S5, the fused feature vector matrix is ​​input into a multilayer perceptron to identify the image to be processed.

[0090] Figure 4 It is a schematic diagram of the position of the fused feature vector matrix output by the embodiment of the present disclosure in the Vit model.

[0091] refer to Figure 4 , the ViT model includes an input layer 41, a processing layer 42 and an output layer 43.

[0092] The input of the input layer 41 is the image to be processed, and the output is the N-element embedding vector V0 corresponding to the N patches.

[0093] The processing layer 42 includes a plurality of alternating self-attention layers 421 and multi-layer perceptrons 422, wherein the first self-attention layer 421 is used to obtain the Q, K, and V matrices, and output the fused feature vector matrix 44 to the first multi-layer perceptron 422. The method of the disclosed embodiment can be applied to the first self-attention layer 421, using the Q, K, and V matrices, the element embedding vector V0, etc. to form a proxy matrix, and output the fused feature vector matrix 44.

[0094] Next, the N×d dimensional fused feature vector matrix is ​​input into a multilayer perceptron (MLP) for further processing, and alternates between multiple self-attention layers and MLP layers to extract higher-level image features. In the entire ViT architecture, these processed image features are finally input into the output layer 43.

[0095] The output layer 43 is used to output the image classification results. The output of the last layer will pass through a fully connected layer to predict the category to which the image belongs.

[0096] The method of the embodiment of the present disclosure is verified by experiments below.

[0097] Figure 5 It is a schematic diagram for verifying the method of the embodiment of the present disclosure through experiments.

[0098] refer to Figure 5 , in Experiment 1:

[0099] (1) The agent token attention mechanism module is embedded in the Swin Transformer (hereinafter referred to as Swin), and the model is named Agent-Swin;

[0100] (2) The ImageNet-1K dataset is selected for model training and verification, and the image input size is 224*224;

[0101] (3) The effect comparison shows that the parameter volume of Swin is 29M, the floating point calculation volume is 4.5G, and the classification accuracy is 81.3%; the parameter volume of Agent-Swin is 29M, the floating point calculation volume is 4.5G, and the classification accuracy is 82.6%; when the parameter volume and floating point calculation volume are consistent, the classification accuracy is improved by 1.3 percentage points.

[0102] In Experiment 2:

[0103] (1) The attention mechanism module of the agent token is embedded in CSwin, and the model is named Agent-CSwin;

[0104] (2) The ImageNet-1K dataset is selected for model training and verification, and the image input size is 224*224;

[0105] (3) The results are compared. The parameter size of Swin is 78M, the floating-point calculation size is 15G, and the classification accuracy is 84.2%; the parameter size of Agent-Swin is 72M, the floating-point calculation size is 14.9G, and the classification accuracy is 84.7%. With the parameter size reduced by 6M and the floating-point calculation size reduced by 0.1G, the classification accuracy is improved by 0.5 percentage points.

[0106] It can be seen that the method provided by the embodiment of the present disclosure can greatly improve the classification accuracy of the model for the processed images, and greatly improve the processing efficiency and result output speed of the model, achieving dual optimization of efficiency and effect.

[0107] Moreover, since the proxy matrix participates in the calculation twice during the processing (i.e., in the forward calculation), it will undergo two corresponding backward gradient updates during the training process, which doubles the training and learning speeds, further accelerating the convergence and optimization speeds of the model, and can also improve the training effect and efficiency of the model, further optimizing the model.

[0108] The disclosed embodiment only sets one proxy matrix as a model auxiliary, and the principle is simple; in addition, the proxy matrix is ​​a pluggable module and can be embedded in most ViT series models, and has strong versatility.

[0109] Corresponding to the above method embodiments, the present disclosure also provides an image processing device, which can be used to execute the above method embodiments.

[0110] Figure 6 is a block diagram of an image processing apparatus in an exemplary embodiment of the present disclosure.

[0111] refer to Figure 6 , the image processing device 600 may include:

[0112] A feature matrix extraction module 61 is configured to obtain N element embedding vectors of N image blocks of the image to be processed, and generate a query matrix, a key matrix and a value matrix of the image to be processed according to the N element embedding vectors, wherein the dimensions of the query matrix, the key matrix and the value matrix are all n×d, where d is the dimension of the feature vector of each image block;

[0113] a proxy matrix extraction module 62, configured to form a proxy matrix according to at least one feature matrix of the image to be processed, wherein the feature matrix includes the query matrix, the key matrix, the value matrix, and a meta-matrix composed of the N meta-embedded vectors, and the dimension of the proxy matrix is ​​n×d, where n is a preset value, 1≤n<N;

[0114] a feature matrix compression module 63, configured to form a compressed query matrix of the image to be processed according to a dot product of the query matrix and the proxy matrix, and to form a compressed key matrix of the image to be processed according to a dot product of the proxy matrix and the key matrix;

[0115] A compression matrix fusion module 64 is configured to perform a matrix multiplication operation on the compression key matrix and the value matrix to obtain a compression feature matrix, and perform a matrix multiplication operation on the compression query matrix and the compression feature matrix to obtain a fusion feature vector matrix of the image to be processed;

[0116] The fusion matrix processing module 65 is configured to input the fusion feature vector matrix into a multi-layer perceptron to identify the image to be processed.

[0117] Since the functions of the device 600 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0118] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0119] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0120] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0121] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present invention will be described. Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0122] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 710, at least one storage unit 720, and a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710).

[0123] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform the method shown in the embodiment of the present disclosure.

[0124] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0125] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0126] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0127] The electronic device 700 may also communicate with one or more external devices 800 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0128] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0129] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0130] The program product for implementing the above method according to an embodiment of the present invention can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0131] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0132] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0133] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0134] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0135] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0136] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. An image processing method, characterized in that: include: Obtaining N element embedding vectors of N image blocks of the image to be processed, and generating a query matrix, a key matrix and a value matrix of the image to be processed according to the N element embedding vectors, wherein the dimensions of the query matrix, the key matrix and the value matrix are all n×d, where d is the dimension of the feature vector of each image block; Forming a proxy matrix according to at least one feature matrix of the image to be processed, wherein the feature matrix includes the query matrix, the key matrix, the value matrix, and a meta-matrix composed of the N meta-embedded vectors, and the dimension of the proxy matrix is ​​n×d, where n is a preset value and 1≤n<N; forming a compressed query matrix of the image to be processed according to a dot product of the query matrix and the proxy matrix, and forming a compressed key matrix of the image to be processed according to a dot product of the proxy matrix and the key matrix; Performing a matrix multiplication operation on the compression key matrix and the value matrix to obtain a compression feature matrix, and performing a matrix multiplication operation on the compression query matrix and the compression feature matrix to obtain a fused feature vector matrix of the image to be processed; The fused feature vector matrix is ​​input into a multi-layer perceptron to identify the image to be processed.

2. The image processing method according to claim 1, characterized in that: Forming a proxy matrix according to at least one feature matrix of the image to be processed comprises: Performing a pooling operation on at least one of the feature matrices to obtain an n×d-dimensional pooling feature matrix; The proxy matrix with a dimension of n×d is formed according to the pooled feature matrix.

3. The image processing method according to claim 2, characterized in that: Performing a pooling operation on at least one of the feature matrices to obtain an n×d-dimensional pooling feature matrix includes: Performing pooling operations on the query matrix and the key matrix respectively to obtain a first pooled feature matrix of n×d dimensions corresponding to the query matrix and a second pooled feature matrix of n×d dimensions corresponding to the key matrix; The first pooling feature matrix and the second pooling feature matrix are bitwise weighted summed to form the proxy matrix with a dimension of n×d.

4. The image processing method according to claim 2, wherein: Performing a pooling operation on at least one of the feature matrices to obtain an n×d-dimensional pooling feature matrix includes: Performing a linear transformation on the element matrix to convert it into an N×d-dimensional element feature matrix; Performing a pooling operation on each column of the meta-feature matrix to form n pooling output values; The proxy matrix is ​​formed according to n pooling output values ​​corresponding to each column in the d columns of the element matrix.

5. The image processing method according to claim 1, wherein: Forming a proxy matrix according to at least one feature matrix of the image to be processed comprises: Performing clustering calculation on the N element embedding vectors in the element matrix to divide the N element embedding vectors into n clusters; The proxy matrix is ​​formed according to the meta-embedding vectors closest to the centroid in each cluster.

6. The image processing method according to claim 1, characterized in that: Forming a proxy matrix according to at least one feature matrix of the image to be processed comprises: The element matrix is ​​linearly transformed by a preset linear transformation matrix to form the proxy matrix.

7. The image processing method according to claim 1, characterized in that: The forming of a compressed query matrix of the image to be processed according to the dot product of the query matrix and the proxy matrix, and the forming of a compressed key matrix of the image to be processed according to the dot product of the proxy matrix and the key matrix comprises: normalizing a dot product of the query matrix and the proxy matrix to form the compressed query matrix; The dot product of the proxy matrix and the key matrix is ​​normalized to form the compressed key matrix.

8. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 1 to 7 based on instructions stored in the memory.

9. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images