Image feature extraction method, device and equipment and readable storage medium
By combining the attention mechanism of multiple sets of feature map extraction units and window division methods in neural network model, the problem of high computational complexity and discontinuity of information in high-resolution image processing is solved, and more efficient images are achieved Feature extraction.
Patent Information
- Application Number
- CN202411089178.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-03
AI Technical Summary
The existing ViT method has high computational complexity when processing high-resolution images, resulting in poor results. The self-attention mechanism of sliding windows leads to discontinuity of image semantic information, especially at the corners of the image.
Using multiple sets of feature map extraction units, combining the attention mechanism of the first and second window division methods, feature extraction is performed in the neural network model through improved attention algorithms to ensure the effectiveness of information within the window and between the global ones. transfer.
It reduces the computational complexity, improves the accuracy and reliability of image feature extraction, reduces information breakpoints, and can effectively process high-resolution images.
Smart Images

Figure CN119068205A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image feature extraction, and specifically relates to an image feature extraction method, device, equipment and readable storage medium. Background Art
[0002] With the development of artificial intelligence technology, high-resolution images and large models have been widely used in computer vision tasks such as image classification, object detection, and image filling. Image feature extraction, as a basic link in computer vision tasks, has also been developing and improving.
[0003] Currently, ViT (Vision Transformer) is one of the large models commonly used to handle computer vision tasks. It has good results in tasks such as image object detection and image classification. The idea of ViT is to divide the image into several 16*16 pixel patches. Figure 1 As shown in the figure, the smallest square is a pixel, the gray thick frame is a patch, and in the following figures, the patch is usually displayed as the smallest unit, and the pixel will be hidden. Then the image is serialized into one dimension inside the patch and input into the transformer as a token for feature extraction, and image object detection or classification is realized. In fact, the size of the patch can be any n*n. If n is too large, the larger the patch, the fewer the number of patches, and since there is no self-attention mechanism inside the patch of ViT, the larger the part without information interaction. On the contrary, if n is too small, the smaller the patch, the more the number of patches, and the computational complexity will increase. Therefore, ViT generally selects a patch size of n=16. However, as the image resolution increases, the computational complexity of the ViT method is about the fourth power of the image side length. Therefore, the ViT method works well on 224*224 images for tasks such as object detection and image classification, but when used to process high-resolution images, the effect is not good and the speed is very slow, making it difficult to complete computer vision tasks on high-resolution images.
[0004] To address this problem, a common solution is to introduce window self-attention mechanism and sliding window attention mechanism. This improved method is used in large models such as Swin (Swin transformer).
[0005] The window self-attention mechanism combines several patches together as a local area, i.e., a window. Figure 2As shown in the figure, the gray squares represent patches and the red boxes represent windows. Each patch only interacts with the patches in the window, that is, the patches nearby, and does not interact with the global image. This makes the patch very small, effectively reducing the computational complexity. Under the window self-attention mechanism, the information in a window can only be transmitted within the window, and cannot be transmitted to the global image. The information is still isolated. Therefore, a sliding window self-attention mechanism is added to divide the image into different parts. This allows information to be gradually transmitted to the global image. Figure 3 As shown in the figure, the image is divided once according to the left image, and then divided once according to the right image. The right image is a scroll of half the window size, that is, the rightmost and leftmost are in one window, and the top and bottom are in one window. In this way, the two window division methods are alternated, combined with the window self-attention mechanism, after repeated several times, the information in each patch can be gradually transmitted to the global.
[0006] However, the window attention method of the sliding window self-attention mechanism always uses a fixed method to divide the window, resulting in discontinuity of image semantic information. A large number of pixels at the corners of the window cannot directly perceive the information of adjacent pixels. For example, Figure 4 In the left figure of , the patches near the midpoints of the four sides of each area cannot directly interact with the patches at the diagonals, regardless of the partitioning. Figure 4 The two purple patches shown cannot directly communicate with each other in either partitioning. Summary of the invention
[0007] In order to overcome the problems existing in the above-mentioned related technologies, the present application provides an image feature extraction method, device, equipment and readable storage medium.
[0008] According to a first aspect of an embodiment of the present application, there is provided an image feature extraction method, comprising:
[0009] Perform patch division on the collected image to be processed to obtain an image after patch division;
[0010] Inputting the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed;
[0011] Wherein, the neural network model is constructed using multiple groups of feature map extraction units, each group of feature map extraction units includes: a first feature map extraction module constructed based on the attention mechanism of the first window division method and a second feature map extraction module constructed based on the attention mechanism of the second window division method; the number of rows of each window divided by the first window division method is the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method is the number of rows of each window divided by the second window division method.
[0012] Preferably, said performing patch division on the collected image to be processed includes:
[0013] The image to be processed is divided into a preset number of patches to obtain an image after patch division.
[0014] Preferably, the patch-divided image is input into a pre-established neural network model to obtain a final feature map of the image to be processed, including:
[0015] Inputting the patch-divided image into a first feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a first intermediate feature map;
[0016] Inputting the first intermediate feature map into a second feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a second intermediate feature map;
[0017] Inputting the second intermediate feature map into a first feature map extraction module in a second group of feature map extraction units of the neural network model to obtain a third intermediate feature map;
[0018] Inputting the third intermediate feature map into a second feature map extraction module in a second set of feature map extraction units of the neural network model to obtain a fourth intermediate feature map;
[0019] By analogy, the 2D-2th intermediate feature map is input into the first feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2D-1th intermediate feature map;
[0020] The 2D-1th intermediate feature map is input into the second feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2Dth intermediate feature map, and the 2Dth intermediate feature map is the final feature map of the image to be processed; wherein D is the total number of groups of feature map extraction units.
[0021] Preferably, the first window division method includes:
[0022] A patch of a rows and b columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches;
[0023] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0024] Preferably, the second window division method includes:
[0025] A patch of b rows and a columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches;
[0026] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0027] Preferably, the attention mechanism based on the first window division method includes:
[0028] The feature map input to the attention mechanism based on the first window division method is divided into windows according to the first window division method, and the feature map after the window division according to the first window division method is positionally embedded to obtain a feature map F1;
[0029] Using the improved attention algorithm, attention calculation is performed on the feature map F1 to obtain the feature map output by the attention mechanism based on the first window division method.
[0030] Preferably, the attention mechanism based on the second window division method includes:
[0031] The feature map input to the attention mechanism based on the second window division method is divided into windows according to the second window division method, and the feature map after the window division according to the second window division method is positionally embedded to obtain a feature map F2;
[0032] Using the improved attention algorithm, attention calculation is performed on the feature map F2 to obtain the feature map output by the attention mechanism based on the second window division method.
[0033] Preferably, the improved attention algorithm is used to perform attention calculation on the feature graph F1, including:
[0034] Let the number of attention heads be H, and the size of feature map F1 be B1×G1×C1, where B1 is the batch size, G1 is the number of patches in the third feature map, and C1 is the embedding dimension of feature map F1;
[0035] Performing linear mapping on the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix;
[0036] Rolling the first key matrix to obtain a second key matrix;
[0037] Rolling the first value matrix to obtain a second value matrix;
[0038] The first query matrix, the second key matrix and the second value matrix are used to calculate a feature map output by the attention mechanism based on the first window division method.
[0039] Preferably, the improved attention algorithm is used to perform attention calculation on the feature graph F2, including:
[0040] Let the number of attention heads be H, and the size of feature map F2 be B2×G2×C2, where B2 is the batch size, G2 is the number of patches in the fifth feature map, and C2 is the embedding dimension of feature map F2;
[0041] Performing linear mapping on the feature graph F2 to calculate a second query matrix, a third key matrix and a third value matrix;
[0042] Rolling the third key matrix to obtain a fourth key matrix;
[0043] Rolling the third value matrix to obtain a fourth value matrix;
[0044] The feature map output by the attention mechanism based on the second window division method is calculated using the second query matrix, the fourth key matrix and the fourth value matrix.
[0045] Preferably, the linear mapping of the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix includes:
[0046] Multiplying the feature map F1 by a first preset learnable linear mapping matrix to obtain a first query matrix;
[0047] Multiplying the feature map F1 with a second preset learnable linear mapping matrix to obtain a first key matrix;
[0048] Multiplying the feature map F1 with a third preset learnable linear mapping matrix to obtain a first value matrix;
[0049] The sizes of the first query matrix, the first key matrix, and the first value matrix are all: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0050] Preferably, rolling the first key matrix to obtain the second key matrix includes:
[0051] Scrolling the first key matrix upward by half the height of each window divided by the first window division method to obtain a key matrix K0+;
[0052] Scrolling the first key matrix downward by half the height of each window divided by the first window division method to obtain a key matrix K0-;
[0053] Concatenate the key matrix K0+ and the key matrix K0- in the penultimate dimension to obtain the second key matrix;
[0054] Among them, the size of the second bond matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0055] Preferably, rolling the first value matrix to obtain the second value matrix includes:
[0056] Scrolling the first value matrix upward by half the height of each window divided by the first window division method to obtain a value matrix V0+;
[0057] Scroll the first value matrix downward by half the height of each window divided by the first window division method to obtain a value matrix V0-;
[0058] Concatenate the value matrix V0+ and the value matrix V0- in the penultimate dimension to obtain the second value matrix;
[0059] Among them, the size of the second value matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0060] Preferably, the step of calculating the feature map output by the attention mechanism based on the first window division method by using the first query matrix, the second key matrix and the second value matrix includes:
[0061] The last two dimensions of the first query matrix and the transpose of the last two dimensions of the second key matrix are multiplied to obtain a first window attention matrix, where the size of the first window attention matrix is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, 2ab is the number of patches in each window, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, w is the number of patches in each column of the image after patch division;
[0062] The last two dimensions of the first window attention matrix and the last two dimensions of the second value matrix are multiplied to obtain the feature map output by the attention mechanism based on the first window division method; wherein the size of the feature map output by the attention mechanism based on the first window division method is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and C1 is the embedding dimension of the feature map F1.
[0063] Preferably, performing linear mapping on the feature graph F2 to calculate a second query matrix, a third key matrix and a third value matrix includes:
[0064] Multiplying the feature map F2 by a fourth preset learnable linear mapping matrix to obtain a second query matrix;
[0065] Multiplying the feature map F2 by a fifth preset learnable linear mapping matrix to obtain a third key matrix;
[0066] Multiplying the feature map F2 by a sixth preset learnable linear mapping matrix to obtain a third value matrix;
[0067] The sizes of the second query matrix, the third key matrix, and the third value matrix are all: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0068] Preferably, the step of rolling the third key matrix to obtain a fourth key matrix comprises:
[0069] Scroll the third key matrix upward by half the height of each window divided by the second window division method to obtain a key matrix K0′+;
[0070] Scroll the third key matrix downward by half the height of each window divided by the second window division method to obtain a key matrix K0′-;
[0071] Concatenate the key matrix K0′+ and the key matrix K0′- in the penultimate dimension to obtain the fourth key matrix;
[0072] Among them, the size of the fourth key matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0073] Preferably, rolling the third value matrix to obtain a fourth value matrix includes:
[0074] Scrolling the third value matrix upward by half the height of each window divided by the second window division method to obtain a value matrix V0′+;
[0075] Scroll the third value matrix downward by half the height of each window divided by the second window division method to obtain a value matrix V0′-;
[0076] Concatenate the value matrix V0′+ and the value matrix V0′- in the penultimate dimension to obtain the fourth value matrix;
[0077] Among them, the size of the fourth value matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0078] Preferably, the step of calculating the feature map output by the attention mechanism based on the second window partitioning method by using the second query matrix, the fourth key matrix and the fourth value matrix includes:
[0079] The second window attention matrix is obtained by multiplying the last two dimensions of the second query matrix and the transpose of the last two dimensions of the fourth key matrix, where the size of the second window attention matrix is: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and 2ab is the number of patches in each window;
[0080] The last two dimensions of the second window attention matrix and the last two dimensions of the fourth value matrix are multiplied to obtain the feature map output by the attention mechanism based on the second window division method, wherein the size of the feature map output by the attention mechanism based on the second window division method is: Among them, B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0081] Preferably, the process of acquiring the pre-established neural network model includes:
[0082] Collect historical images and final feature maps corresponding to the historical images;
[0083] Constructing a training set using the historical images and final feature maps corresponding to the historical images;
[0084] The improved Transformer model is trained using the training set to obtain the pre-established neural network model; wherein the improved Transformer model includes: a first feature map extraction module and a second feature map extraction module.
[0085] Preferably, the first feature map extraction module includes: a first normalization layer, an attention mechanism based on a first window division method, a second normalization layer and a first multi-layer perceptron connected in sequence;
[0086] The second feature map extraction module includes: a third normalization layer, an attention mechanism based on the second window division method, a fourth normalization layer and a second multi-layer perceptron connected in sequence.
[0087] Preferably, the input of the first normalization layer in the first group of feature map extraction units is the image after patch division; the input of the first normalization layer in other groups of feature map extraction units except the first group of feature map extraction units is the output of the second multi-layer perceptron in the previous group of feature map extraction units;
[0088] The output of the first normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the first window division method in each group of feature map extraction units;
[0089] The output of the attention mechanism based on the first window division method in each group of feature map extraction units is shortcut-connected with the input of the first normalization layer in each group of feature map extraction units, and then used as the input of the second normalization layer in each group of feature map extraction units;
[0090] The output of the second normalization layer in each group of feature map extraction units is the input of the first multi-layer perceptron in each group of feature map extraction units, and the output of the first multi-layer perceptron in each group of feature map extraction units is the 2d-1th intermediate feature map;
[0091] The 2d-1th intermediate feature map is the input of the third normalization layer in each group of feature map extraction units, and the output of the third normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the second window division method in each group of feature map extraction units;
[0092] The output of the attention mechanism based on the second window division method in each group of feature map extraction units is shortcut-connected with the input of the third normalization layer in each group of feature map extraction units, and then used as the input of the fourth normalization layer in each group of feature map extraction units;
[0093] The output of the fourth normalization layer in each group of feature map extraction units is the input of the second multilayer perceptron in each group of feature map extraction units, and the output of the second multilayer perceptron in each group of feature map extraction units is the 2d-th intermediate feature map, where d∈[1,D], and D is the total number of groups of feature map extraction units.
[0094] According to a second aspect of an embodiment of the present application, there is provided an image feature extraction device, comprising:
[0095] A division unit, used for performing patch division on the collected image to be processed to obtain an image after patch division;
[0096] An acquisition unit, used for inputting the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed;
[0097] Among them, the neural network model is constructed using multiple groups of feature map extraction units, each group of feature map extraction units includes: a first feature map extraction module constructed based on the attention mechanism of the first window division method and a second feature map extraction module constructed based on the attention mechanism of the second window division method; the number of rows of each window divided by the first window division method is the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method is the number of rows of each window divided by the second window division method.
[0098] Preferably, the division unit is specifically used for:
[0099] The image to be processed is divided into a preset number of patches to obtain an image after patch division.
[0100] Preferably, the acquisition unit is specifically used for:
[0101] Inputting the patch-divided image into a first feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a first intermediate feature map;
[0102] Inputting the first intermediate feature map into a second feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a second intermediate feature map;
[0103] Inputting the second intermediate feature map into a first feature map extraction module in a second group of feature map extraction units of the neural network model to obtain a third intermediate feature map;
[0104] Inputting the third intermediate feature map into a second feature map extraction module in a second set of feature map extraction units of the neural network model to obtain a fourth intermediate feature map;
[0105] By analogy, the 2D-2th intermediate feature map is input into the first feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2D-1th intermediate feature map;
[0106] The 2D-1th intermediate feature map is input into the second feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2Dth intermediate feature map, and the 2Dth intermediate feature map is the final feature map of the image to be processed; wherein D is the total number of groups of feature map extraction units.
[0107] Preferably, the first window division method includes:
[0108] A patch of a rows and b columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches;
[0109] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0110] Preferably, the second window division method includes:
[0111] A patch of b rows and a columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches;
[0112] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0113] Preferably, the attention mechanism based on the first window division method includes:
[0114] The feature map input to the attention mechanism based on the first window division method is divided into windows according to the first window division method, and the feature map after the window division according to the first window division method is positionally embedded to obtain a feature map F1;
[0115] Using the improved attention algorithm, attention calculation is performed on the feature map F1 to obtain the feature map output by the attention mechanism based on the first window division method.
[0116] Preferably, the attention mechanism based on the second window division method includes:
[0117] The feature map input to the attention mechanism based on the second window division method is divided into windows according to the second window division method, and the feature map after the window division according to the second window division method is positionally embedded to obtain a feature map F2;
[0118] Using the improved attention algorithm, attention calculation is performed on the feature map F2 to obtain the feature map output by the attention mechanism based on the second window division method.
[0119] Preferably, the improved attention algorithm is used to perform attention calculation on the feature graph F1, including:
[0120] Let the number of attention heads be H, and the size of feature map F1 be B1×G1×C1, where B1 is the batch size, G1 is the number of patches in the third feature map, and C1 is the embedding dimension of feature map F1;
[0121] Performing linear mapping on the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix;
[0122] Rolling the first key matrix to obtain a second key matrix;
[0123] Rolling the first value matrix to obtain a second value matrix;
[0124] The first query matrix, the second key matrix and the second value matrix are used to calculate a feature map output by the attention mechanism based on the first window division method.
[0125] Preferably, the improved attention algorithm is used to perform attention calculation on the feature graph F2, including:
[0126] Let the number of attention heads be H, and the size of feature map F2 be B2×G2×C2, where B2 is the batch size, G2 is the number of patches in the fifth feature map, and C2 is the embedding dimension of feature map F2;
[0127] Performing linear mapping on the feature graph F2 to calculate a second query matrix, a third key matrix and a third value matrix;
[0128] Rolling the third key matrix to obtain a fourth key matrix;
[0129] Rolling the third value matrix to obtain a fourth value matrix;
[0130] The feature map output by the attention mechanism based on the second window division method is calculated using the second query matrix, the fourth key matrix and the fourth value matrix.
[0131] Preferably, the linear mapping of the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix includes:
[0132] Multiplying the feature map F1 by a first preset learnable linear mapping matrix to obtain a first query matrix;
[0133] Multiplying the feature map F1 with a second preset learnable linear mapping matrix to obtain a first key matrix;
[0134] Multiplying the feature map F1 with a third preset learnable linear mapping matrix to obtain a first value matrix;
[0135] The sizes of the first query matrix, the first key matrix, and the first value matrix are all: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0136] Preferably, rolling the first key matrix to obtain the second key matrix includes:
[0137] Scrolling the first key matrix upward by half the height of each window divided by the first window division method to obtain a key matrix K0+;
[0138] Scrolling the first key matrix downward by half the height of each window divided by the first window division method to obtain a key matrix K0-;
[0139] Concatenate the key matrix K0+ and the key matrix K0- in the penultimate dimension to obtain the second key matrix;
[0140] Among them, the size of the second bond matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0141] Preferably, rolling the first value matrix to obtain the second value matrix includes:
[0142] Scrolling the first value matrix upward by half the height of each window divided by the first window division method to obtain a value matrix V0+;
[0143] Scroll the first value matrix downward by half the height of each window divided by the first window division method to obtain a value matrix V0-;
[0144] Concatenate the value matrix V0+ and the value matrix V0- in the penultimate dimension to obtain the second value matrix;
[0145] Among them, the size of the second value matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0146] Preferably, the step of calculating the feature map output by the attention mechanism based on the first window division method by using the first query matrix, the second key matrix and the second value matrix includes:
[0147] The last two dimensions of the first query matrix and the transpose of the last two dimensions of the second key matrix are multiplied to obtain a first window attention matrix, where the size of the first window attention matrix is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, 2ab is the number of patches in each window, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, w is the number of patches in each column of the image after patch division;
[0148] The last two dimensions of the first window attention matrix and the last two dimensions of the second value matrix are multiplied to obtain the feature map output by the attention mechanism based on the first window division method; wherein the size of the feature map output by the attention mechanism based on the first window division method is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and C1 is the embedding dimension of the feature map F1.
[0149] Preferably, performing linear mapping on the feature graph F2 to calculate a second query matrix, a third key matrix and a third value matrix includes:
[0150] Multiplying the feature map F2 by a fourth preset learnable linear mapping matrix to obtain a second query matrix;
[0151] Multiplying the feature map F2 by a fifth preset learnable linear mapping matrix to obtain a third key matrix;
[0152] Multiplying the feature map F2 by a sixth preset learnable linear mapping matrix to obtain a third value matrix;
[0153] The sizes of the second query matrix, the third key matrix, and the third value matrix are all: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0154] Preferably, the step of rolling the third key matrix to obtain a fourth key matrix comprises:
[0155] Scroll the third key matrix upward by half the height of each window divided by the second window division method to obtain a key matrix K0′+;
[0156] Scroll the third key matrix downward by half the height of each window divided by the second window division method to obtain a key matrix K0′-;
[0157] Concatenate the key matrix K0′+ and the key matrix K0′- in the penultimate dimension to obtain the fourth key matrix;
[0158] Among them, the size of the fourth key matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0159] Preferably, rolling the third value matrix to obtain a fourth value matrix includes:
[0160] Scrolling the third value matrix upward by half the height of each window divided by the second window division method to obtain a value matrix V0′+;
[0161] Scroll the third value matrix downward by half the height of each window divided by the second window division method to obtain a value matrix V0′-;
[0162] Concatenate the value matrix V0′+ and the value matrix V0′- in the penultimate dimension to obtain the fourth value matrix;
[0163] Among them, the size of the fourth value matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0164] Preferably, the step of calculating the feature map output by the attention mechanism based on the second window partitioning method by using the second query matrix, the fourth key matrix and the fourth value matrix includes:
[0165] The last two dimensions of the second query matrix and the transpose of the last two dimensions of the fourth key matrix are multiplied to obtain a second window attention matrix, where the size of the second window attention matrix is: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and 2ab is the number of patches in each window;
[0166] The last two dimensions of the second window attention matrix and the last two dimensions of the fourth value matrix are multiplied to obtain the feature map output by the attention mechanism based on the second window division method, wherein the size of the feature map output by the attention mechanism based on the second window division method is: Among them, B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0167] Preferably, the process of acquiring the pre-established neural network model includes:
[0168] Collect historical images and final feature maps corresponding to the historical images;
[0169] Constructing a training set using the historical images and final feature maps corresponding to the historical images;
[0170] The improved Transformer model is trained using the training set to obtain the pre-established neural network model; wherein the improved Transformer model includes: a first feature map extraction module and a second feature map extraction module.
[0171] Preferably, the first feature map extraction module includes: a first normalization layer, an attention mechanism based on a first window division method, a second normalization layer and a first multi-layer perceptron connected in sequence;
[0172] The second feature map extraction module includes: a third normalization layer, an attention mechanism based on the second window division method, a fourth normalization layer and a second multi-layer perceptron connected in sequence.
[0173] Preferably, the input of the first normalization layer in the first group of feature map extraction units is the image after patch division; the input of the first normalization layer in other groups of feature map extraction units except the first group of feature map extraction units is the output of the second multi-layer perceptron in the previous group of feature map extraction units;
[0174] The output of the first normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the first window division method in each group of feature map extraction units;
[0175] The output of the attention mechanism based on the first window division method in each group of feature map extraction units is shortcut-connected with the input of the first normalization layer in each group of feature map extraction units, and then used as the input of the second normalization layer in each group of feature map extraction units;
[0176] The output of the second normalization layer in each group of feature map extraction units is the input of the first multi-layer perceptron in each group of feature map extraction units, and the output of the first multi-layer perceptron in each group of feature map extraction units is the 2d-1th intermediate feature map;
[0177] The 2d-1th intermediate feature map is the input of the third normalization layer in each group of feature map extraction units, and the output of the third normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the second window division method in each group of feature map extraction units;
[0178] The output of the attention mechanism based on the second window division method in each group of feature map extraction units is shortcut-connected with the input of the third normalization layer in each group of feature map extraction units, and then used as the input of the fourth normalization layer in each group of feature map extraction units;
[0179] The output of the fourth normalization layer in each group of feature map extraction units is the input of the second multilayer perceptron in each group of feature map extraction units, and the output of the second multilayer perceptron in each group of feature map extraction units is the 2d-th intermediate feature map, where d∈[1,D], and D is the total number of groups of feature map extraction units.
[0180] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;
[0181] The memory is used to store one or more programs;
[0182] When the one or more programs are executed by the at least one processor, the image feature extraction method is implemented.
[0183] According to a fourth aspect of an embodiment of the present application, a readable storage medium is provided, on which an execution program is stored. When the execution program is executed, the image feature extraction method is implemented.
[0184] The technical solution provided by the present invention has the following beneficial effects:
[0185] The present invention provides an image feature extraction method and device, which performs patch division on a collected image to be processed to obtain a patch-divided image, and inputs the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed, thereby realizing image processing and analysis, reducing computational complexity, and effectively improving the accuracy of computer vision tasks such as target detection and image classification.
[0186] The present invention constructs a neural network model by utilizing a first feature map extraction module constructed based on an attention mechanism of a first window division method and a second feature map extraction module constructed based on an attention mechanism of a second window division method. By setting the number of rows of each window divided by the first window division method to be the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method to be the number of rows of each window divided by the second window division method, it is achieved that under the window attention mechanism, by changing the size and shape of the window division and the range of the window attention, under the premise that the amount of calculation is basically equal to that of the sliding window attention mechanism, the pixel points at the corners of the window in the attention mechanism can also perceive the information of adjacent pixels, which greatly reduces the number of information breakpoints, thereby improving the reliability of image feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0187] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0188] Figure 1 This is a schematic diagram of patch partitioning in traditional ViT processing;
[0189] Figure 2 It is a schematic diagram of traditional window division;
[0190] Figure 3 This is a schematic diagram of the traditional sliding window self-attention mechanism;
[0191] Figure 4 This is a schematic diagram of the defects of the traditional sliding window self-attention mechanism;
[0192] Figure 5 is a flow chart of an image feature extraction method provided by an embodiment of the present invention;
[0193] Figure 6 is a schematic diagram of the observation range of the window provided by an embodiment of the present invention;
[0194] Figure 7 is a schematic diagram of the network structure of the improved Transformer model provided by an embodiment of the present invention;
[0195] Figure 8 is a schematic diagram of implementing an image classification task using an image feature extraction method provided by an embodiment of the present invention;
[0196] Fig. 9 is a structural block diagram of an image feature extraction device provided by an embodiment of the present invention;
[0197] Fig.10 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0198] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the following embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0199] Embodiment 1
[0200] The present invention provides an image feature extraction method, such as Figure 5 As shown, the following steps are included:
[0201] Step 101: performing patch division on the collected image to be processed to obtain an image after patch division;
[0202] Step 102: input the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed;
[0203] Among them, the neural network model is constructed using multiple groups of feature map extraction units, each group of feature map extraction units includes: a first feature map extraction module constructed based on the attention mechanism of the first window division method and a second feature map extraction module constructed based on the attention mechanism of the second window division method; the number of rows of each window divided by the first window division method is the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method is the number of rows of each window divided by the second window division method.
[0204] Furthermore, step 101 includes:
[0205] The image to be processed is divided into a preset number of patches to obtain an image after patch division.
[0206] For example, starting from a corner of the image to be processed, the image to be processed is divided into h*w patches, where the pixels of each patch are p*p, h is the row, and w is the column. In the window attention mechanism, each window can observe n*n patch information centered on itself. Figure 6 As shown, the three thick blue, yellow and green frames represent three windows, and the ranges they can observe are the light blue, light yellow and light green patches respectively.
[0207] Furthermore, in step 102, each time the patch-divided image is input into a pre-established neural network model, a target feature map is obtained, including:
[0208] Step 1021: input the patch-divided image into a first feature map extraction module in a first group of feature map extraction units of a neural network model to obtain a first intermediate feature map;
[0209] Step 1022: input the first intermediate feature map to the second feature map extraction module in the first set of feature map extraction units of the neural network model to obtain a second intermediate feature map;
[0210] Step 1023: input the second intermediate feature map to the first feature map extraction module in the second set of feature map extraction units of the neural network model to obtain a third intermediate feature map;
[0211] Step 1024: input the third intermediate feature map to the second feature map extraction module in the second set of feature map extraction units of the neural network model to obtain a fourth intermediate feature map;
[0212] Step 1025: By analogy, the 2D-2th intermediate feature map is input into the first feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2D-1th intermediate feature map;
[0213] Step 1026: Input the 2D-1th intermediate feature map to the second feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2Dth intermediate feature map, which is the final feature map of the image to be processed; wherein D is the total number of groups of feature map extraction units.
[0214] Furthermore, the first window division method includes:
[0215] A patch of a row and b column is used to form a window. The image after patch division is divided into hw / ab windows. The size of each window is ab patches, and the field of view of each window is 2ab patches.
[0216] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0217] For example, let a = n / 2, let b = n, and use n / 2 rows and n columns of patches to form a window. The image after patch division is divided into 2h / n rows and w / n columns, a total of 2hw / n 2 windows, where the size of each window is n 2 / 2 patches, each window has a field of view of n 2 patches.
[0218] Furthermore, the attention mechanism based on the first window division method includes:
[0219] Step 11: The feature map input to the attention mechanism based on the first window division method is divided into windows according to the first window division method, and the feature map after the window division according to the first window division method is positionally embedded to obtain a feature map F1;
[0220] Step 12: Use the improved attention algorithm to perform attention calculation on the feature map F1 to obtain the feature map output by the attention mechanism based on the first window division method.
[0221] Further, step 12 includes:
[0222] Step 121: Let the number of attention heads be H, and the size of the feature map F1 be B1×G1×C1, where B1 is the batch size, G1 is the number of patches in the third feature map, and C1 is the embedding dimension of the feature map F1;
[0223] Step 122: linearly map the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix;
[0224] Step 123: rolling the first key matrix to obtain a second key matrix;
[0225] Step 124: rolling the first value matrix to obtain a second value matrix;
[0226] Step 125: Using the first query matrix, the second key matrix and the second value matrix, calculate the feature map output by the attention mechanism based on the first window division method.
[0227] It should be noted that the query matrix, key matrix and value matrix obtained in the traditional self-attention mechanism method have the same size. Compared with the traditional self-attention mechanism method, although the method of the attention mechanism based on the first window division method adopted in the present invention obtains the second key matrix and the second value matrix, which are twice as large as the size of the first query matrix, the calculation amount only increases a small amount of scrolling and splicing calculations, thereby reducing the calculation complexity.
[0228] Further, step 122 includes:
[0229] Step 1221: multiplying the feature map F1 by a first preset learnable linear mapping matrix to obtain a first query matrix;
[0230] Step 1222: multiplying the feature map F1 by the second preset learnable linear mapping matrix to obtain a first key matrix;
[0231] Step 1223: multiplying the feature map F1 by the third preset learnable linear mapping matrix to obtain a first value matrix;
[0232] Among them, the sizes of the first query matrix, the first key matrix, and the first value matrix are all: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0233] For example, assuming a=n / 2, b=n, the sizes of the first query matrix, the first key matrix, and the first value matrix are all:
[0234]
[0235] Further, step 123 includes:
[0236] 1231: Scroll the first key matrix upward by half the height of each window divided by the first window division method to obtain a key matrix K0+;
[0237] 1232: Scroll the first key matrix downward by half the height of each window divided by the first window division method to obtain a key matrix K0-;
[0238] 1233: Concatenate the key matrix K0+ and the key matrix K0- in the penultimate dimension to obtain a second key matrix;
[0239] Among them, the size of the second bond matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, and C1 is the embedding dimension of the feature map F1.
[0240] For example, assuming a = n / 2, b = n, then half the window height of each patch is Then scroll the first key matrix upwards patches, get the key matrix K0+; scroll the first key matrix downward patches to obtain the key matrix K0-; concatenate the key matrix K0+ and the key matrix K0- in the penultimate dimension to obtain the second key matrix; the size of the second key matrix is a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0241] Further, step 124 includes:
[0242] Step 1241: Scroll the first value matrix upward by half the height of each window divided by the first window division method to obtain a value matrix V0+;
[0243] Step 1242: Scroll the first value matrix downward by half the height of each window divided by the first window division method to obtain a value matrix V0-;
[0244] Step 1243: concatenate the value matrix V0+ and the value matrix V0- in the penultimate dimension to obtain a second value matrix;
[0245] Among them, the size of the second value matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0246] For example, assuming a = n / 2, b = n, then half the window height of each patch is Then scroll the first value matrix V0 upwards patches, and obtain the value matrix V0+; scroll the first value matrix V0 downwards patches to obtain the value matrix V0-; concatenate the value matrix V0+ and the value matrix V0- in the penultimate dimension to obtain the second value matrix; the size of the second value matrix is
[0247] Further, step 125 includes:
[0248] Step 1251: Multiply the last two dimensions of the first query matrix and the transpose of the last two dimensions of the second key matrix to obtain a first window attention matrix, where the size of the first window attention matrix is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, 2ab is the number of patches in each window, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, w is the number of patches in each column of the image after patch division;
[0249] Step 1252: Multiply the last two dimensions of the first window attention matrix and the last two dimensions of the second value matrix to obtain a feature map output by the attention mechanism based on the first window division method; wherein the size of the feature map output by the attention mechanism based on the first window division method is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and C1 is the embedding dimension of the feature map F1.
[0250] For example, assuming a = n / 2, b = n, the size of the first window attention matrix is: The size of the feature map output by the attention mechanism based on the first window division method is:
[0251] Furthermore, the second window division method includes:
[0252] A patch of b rows and a columns is used to form a window. The image after patch division is divided into hw / ab windows. The size of each window is ab patches, and the field of view of each window is 2ab patches.
[0253] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0254] For example, let a = n, b = n / 2, and use a patch of n rows and n / 2 columns to form a window. The image after patch division is divided into h / n rows and 2w / n columns, a total of 2hw / n 2 windows, where the size of each window is n 2 / 2 patches, each window has a field of view of n 2 patches.
[0255] It can be understood that, from the example in the second window division method (a=n, b=n / 2) and the example in the first window division method (a=n / 2, b=n), the number of rows of each window divided by the first window division method is the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method is the number of rows of each window divided by the second window division method.
[0256] Furthermore, the attention mechanism based on the second window division method includes:
[0257] Step 21: The feature map input to the attention mechanism based on the second window division method is divided into windows according to the second window division method, and the feature map after the window division according to the second window division method is positionally embedded to obtain a feature map F2;
[0258] Step 22: Use the improved attention algorithm to perform attention calculation on the feature map F2 to obtain the feature map output by the attention mechanism based on the second window division method.
[0259] Further, step 22 includes:
[0260] Step 221: Let the number of attention heads be H, and the size of the feature map F2 be B2×G2×C2, where B2 is the batch size, G2 is the number of patches in the fifth feature map, and C2 is the embedding dimension of the feature map F2;
[0261] Step 222: linearly map the feature graph F2 to obtain a second query matrix, a third key matrix, and a third value matrix;
[0262] Step 223: rolling the third key matrix to obtain a fourth key matrix;
[0263] Step 224: rolling the third value matrix to obtain a fourth value matrix;
[0264] Step 225: Use the second query matrix, the fourth key matrix and the fourth value matrix to calculate the feature map output by the attention mechanism based on the second window division method.
[0265] Further, step 222 includes:
[0266] Step 2221: multiply the feature map F2 by the fourth preset learnable linear mapping matrix to obtain a second query matrix;
[0267] Step 2222: multiply the feature map F2 by the fifth preset learnable linear mapping matrix to obtain a third key matrix;
[0268] Step 2223: multiplying the feature map F2 by the sixth preset learnable linear mapping matrix to obtain a third value matrix;
[0269] Among them, the sizes of the second query matrix, the third key matrix, and the third value matrix are all: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0270] For example, assuming a=n, b=n / 2, the sizes of the second query matrix, the third key matrix, and the third value matrix are all:
[0271]
[0272] Further, step 223 includes:
[0273] Step 2231: Scroll the third key matrix upward by half the height of each window divided by the second window division method to obtain a key matrix K0′+;
[0274] Step 2232: Scroll the third key matrix downward by half the height of each window divided by the second window division method to obtain a key matrix K0′-;
[0275] Step 2233: concatenate the key matrix K0′+ and the key matrix K0′- in the penultimate dimension to obtain a fourth key matrix;
[0276] Among them, the size of the fourth key matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0277] For example, assuming a = n, b = n / 2, then half the window height of each patch is Then scroll the third key matrix upwards patches, and obtain the key matrix K0′+; scroll the third key matrix downwards patches to obtain the key matrix K0′-; concatenate the key matrix K0′+ and the key matrix K0′- in the penultimate dimension to obtain the fourth key matrix; the size of the fourth key matrix is
[0278] Further, step 224 includes:
[0279] Step 2241: Scroll the third value matrix upward by half the height of each window divided by the second window division method to obtain a value matrix V0′+;
[0280] Step 2242: Scroll the third value matrix downward by half the height of each window divided by the second window division method to obtain a value matrix V0′-;
[0281] Step 2243: concatenate the value matrix V0′+ and the value matrix V0′- in the second to last dimension to obtain a fourth value matrix;
[0282] Among them, the size of the fourth value matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0283] For example, assuming a = n, b = n / 2, then half the window height of each patch is Then scroll the third value matrix upwards patches, and obtain the value matrix V0′+; scroll the third value matrix downward patches, and obtain the value matrix V0′-; concatenate the value matrix V0′+ and the value matrix V0′- in the penultimate dimension to obtain the fourth value matrix; the size of the fourth value matrix is
[0284] It should be noted that "scrolling up or down by half the height of each window divided by the first window division method" and "scrolling up or down by half the height of each window divided by the second window division method" mean that the self-attention key matrix is the attention of the current window to itself. Scrolling up half the height of the window becomes the current window's attention to "the window after its own window is translated up half the height (such as Figure 6 The yellow box in the middle is the attention key matrix of "the upper half of the light yellow area")", and similarly, scrolling down half the height of the window becomes the current window's attention key matrix of "the window after its own window is translated down half the height (such as Figure 6 The yellow box in the middle is for the "lower half of the light yellow area")", and when put together, it becomes the current window for "a window with the same width and twice the height (such as Figure 6 The yellow box in the figure pays attention to the "light yellow area"). From the formula: the size of the first bond matrix is B1*hw / ab*H*ab*C1. Its second dimension and the fourth dimension together correspond to Figure 6 The number of patches in the matrix is considered as a (h*w) B1*H*C1 tensor. Scroll a / 2 patches upward, that is, the position of the h*w tensor in the figure is moved upward by a / 2 patches, and the tensors of the top a / 2 rows and w columns are scrolled to the bottom. Similarly, scroll a / 2 patches downward, that is, the position of the h*w tensor in the figure is moved downward by a / 2 patches, and the tensors of the bottom a / 2 rows and w columns are scrolled to the top. Then convert the obtained (h*w) B1*H*C1 tensors into B1*hw / ab*H*ab*C1 tensors in the original way. Finally, these two tensors are pieced together in the fourth dimension to become a B1*hw / ab*H*2ab*C1 tensor, which is the second key matrix. The rolling of all key matrices and value matrices is similar.
[0285] Further, step 225 includes:
[0286] Step 2251: Multiply the last two dimensions of the second query matrix and the transpose of the last two dimensions of the fourth key matrix to obtain a second window attention matrix, wherein the size of the second window attention matrix is: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and 2ab is the number of patches in each window;
[0287] Step 2252: Multiply the last two dimensions of the second window attention matrix and the last two dimensions of the fourth value matrix to obtain a feature map output by the attention mechanism based on the second window division method, wherein the size of the feature map output by the attention mechanism based on the second window division method is: Among them, B2 is batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0288] For example, assuming a=n, b=n / 2, the size of the second window attention matrix is: The size of the feature map output by the attention mechanism based on the second window division method is:
[0289] It should be noted that, assuming that matrix M1 (size is a*b) is multiplied by matrix M2 (size is b*c), in the two multiplied matrices, the second scale of the first matrix must be the same as the first scale of the second matrix, then matrix M1 multiplied by matrix M2 will obtain an a*c matrix;
[0290] The two matrices mentioned above are multiplied in the last two dimensions. Assuming that M3 is a w*v*a*b matrix and M4 is a w*v*b*c matrix, their first dimensions are the same and they need to be multiplied as two matrices in the last two dimensions. Then the last two dimensions of matrix M3 and the last two dimensions of matrix M4 are multiplied to obtain a w*v*a*c tensor. This is similar to treating M3 and M4 as w*v a*b matrices and w*v b*c matrices, respectively, multiplying them separately, and then putting the w*v results together.
[0291] Therefore, assuming a = n / 2, b = n, the dimension of the first query matrix Q is The dimension of the second bond matrix K is Consider Q as , K is considered as n 2 ×C1 matrices, the two sets of matrices are operated as follows: the last two dimensions of the first query matrix Q are multiplied by the transpose of the last two dimensions of the second key matrix K to obtain , and merge and concatenate them to get The first window attention matrix attn1. Similarly, the same is true for "multiplying the last two dimensions of the first window attention matrix by the last two dimensions of the second value matrix to obtain the feature map output by the attention mechanism based on the first window partitioning method", "multiplying the last two dimensions of the second query matrix by the transpose of the last two dimensions of the fourth key matrix to obtain the second window attention matrix", and "multiplying the last two dimensions of the second window attention matrix by the last two dimensions of the fourth value matrix to obtain the feature map output by the attention mechanism based on the second window partitioning method".
[0292] Furthermore, the method further includes: Step 100: obtaining a pre-established neural network model;
[0293] Specifically, step 100 includes:
[0294] Step 1001: Collect historical images and final feature maps corresponding to the historical images;
[0295] Step 1002: construct a training set using the historical images and the final feature maps corresponding to the historical images;
[0296] Step 1003: Train the improved Transformer model using the training set to obtain a pre-established neural network model; wherein the improved Transformer model includes: a first feature map extraction module and a second feature map extraction module.
[0297] In some embodiments, step 1003 includes:
[0298] The historical images in the training set are used as input layer training samples of the improved Transformer model, and the final feature maps corresponding to the historical images in the training set are used as output layer training samples of the improved Transformer model. The improved Transformer model is trained to obtain a pre-established neural network model.
[0299] Further, such as Figure 7 As shown, the first feature map extraction module includes: a first normalization layer, an attention mechanism based on a first window division method, a second normalization layer and a first multi-layer perceptron connected in sequence;
[0300] The second feature map extraction module includes: a third normalization layer, an attention mechanism based on the second window division method, a fourth normalization layer and a second multi-layer perceptron connected in sequence.
[0301] Furthermore, the input of the first normalization layer in the first group of feature map extraction units is the image after patch division; the input of the first normalization layer in the other groups of feature map extraction units except the first group of feature map extraction units is the output of the second multi-layer perceptron in the previous group of feature map extraction units;
[0302] The output of the first normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the first window division method in each group of feature map extraction units;
[0303] The output of the attention mechanism based on the first window division method in each group of feature map extraction units is shortcut-connected with the input of the first normalization layer in each group of feature map extraction units, and then used as the input of the second normalization layer in each group of feature map extraction units;
[0304] The output of the second normalization layer in each group of feature map extraction units is the input of the first multi-layer perceptron in each group of feature map extraction units, and the output of the first multi-layer perceptron in each group of feature map extraction units is the 2d-1th intermediate feature map;
[0305] The 2d-1th intermediate feature map is the input of the third normalization layer in each group of feature map extraction units, and the output of the third normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the second window division method in each group of feature map extraction units;
[0306] The output of the attention mechanism based on the second window division method in each group of feature map extraction units is shortcut-connected with the input of the third normalization layer in each group of feature map extraction units, and then serves as the input of the fourth normalization layer in each group of feature map extraction units;
[0307] The output of the fourth normalization layer in each group of feature map extraction units is the input of the second multilayer perceptron in each group of feature map extraction units, and the output of the second multilayer perceptron in each group of feature map extraction units is the 2d-th intermediate feature map, where d∈[1,D], and D is the total number of groups of feature map extraction units.
[0308] The present invention provides an image feature extraction method, which collects an image to be processed, performs patch division on the image to be processed, obtains an image after patch division, and inputs the image after patch division into a pre-established neural network model to obtain a final feature map of the image to be processed. Compared with the traditional ViT method, the computational complexity is significantly reduced from approximately the fourth power of the image side length to the second power, thereby being able to process and analyze high-resolution images; at the same time, compared with the sliding window self-attention method with the same computational complexity, the time for information transmission from each point in the image to the entire image in this method remains unchanged, while the number of blind spots in the image is reduced by half, thereby effectively improving the accuracy of computer vision tasks such as target detection and image classification.
[0309] Based on the image feature extraction method provided by the present invention, computer vision tasks such as target detection and image classification can be further realized.
[0310] To further illustrate the above-mentioned image feature extraction method, the present invention provides a specific example, using the image feature extraction method provided by the present invention to implement image classification tasks and target detection tasks.
[0311] Example 1: Using an image feature extraction method provided by the present invention to implement an image classification task, such as Figure 8 As shown, the following steps are included:
[0312] (1) Patch embedding
[0313] Assume that the size of the input image is 2048*2048*3. If the size of the image to be processed is not 2048*2048*3, scale the image to be processed to this size. The batch size of the model training is B=8.
[0314] Using a patch of 16*16 pixels, the image is divided into 128*128=16384 non-intersecting patches, that is, h=128, w=128.
[0315] Taking the patch embedding dimension as 256, the features of the patch are linearly projected, and the output is Ps, with a size of 8 (batch size) * 16384 (number of patches) * 256 (embedding dimension).
[0316] (2) Enter the first feature map extraction module block1 of the pre-established neural network model
[0317] (2.1) Attention mechanism using the first window division
[0318] Let n = 8, and take the window size as 4*8 = 32 patches. The image is divided into 32*16 = 512 non-overlapping windows.
[0319] The number of attention heads is 16. Calculate the first query matrix Q, the first key matrix K0, and the first value matrix V0 based on window attention. The first query matrix Q, the first key matrix K0, and the first value matrix V0 are all tensors obtained by multiplying the input of block1 by the learnable linear mapping matrix. The first key matrix K0 and the first value matrix V0 are 8*8 window self-attention tensors, whose size is 8 (batch size)*512 (number of windows)*16 (attention heads)*32 (number of patches in the window)*256 (embedding dimension).
[0320] Since the window size is 4*8=32, and the window field of view is 8*8=64 (such as Figure 4 As shown), the sizes of the first query matrix Q, the first key matrix K0, and the first value matrix V0 are all 8 (batch size) * 512 (number of windows) * 16 (attention heads) * 32 (number of patches in a window) * 256 (embedding dimension).
[0321] (2.2) Calculation of the second key matrix K and the second value matrix V
[0322] The second key matrix K and the second value matrix V can be quickly calculated through the first key matrix K0 and the first value matrix V0 of the window self-attention respectively.
[0323] The first key matrix K0 is rolled up by 4 / 2=2 patches (half the window height) to obtain the key matrix K0+; similarly, the first key matrix K0 is rolled down by 2 patches to obtain the key matrix K0-. The key matrix K0+ and the key matrix K0- correspond to the upper half and lower half of the window's field of view (8*8 patches), respectively. Therefore, the key matrix K0+ and the key matrix K0- are concatenated to obtain the second key matrix K.
[0324] Similarly, the second value matrix V can be obtained.
[0325] The first value matrix V0 is rolled up by 4 / 2=2 patches (half the window height) to obtain the value matrix V0+; similarly, the first value matrix V0 is rolled down by 2 patches to obtain the value matrix V0-. The value matrix V0+ and the value matrix V0- correspond to the upper and lower half of the field of view (8*8 patches) of the window, respectively. Therefore, the value matrix V0+ and the value matrix V0- are concatenated to obtain the second value matrix V.
[0326] The sizes of the second key matrix K and the second value matrix V are both 8 (batch size)*512 (number of windows)*16 (attention heads)*64 (number of patches covered by the window's field of view)*256 (embedding dimension).
[0327] Using this calculation method, compared with the traditional self-attention mechanism method (QKV has the same size), although the size of the second key matrix K and the second value matrix V is twice as large as the first query matrix Q, the amount of calculation only increases by a small amount of scrolling and splicing calculations.
[0328] (2.3) Propagate forward in block 1
[0329] In the process of forward propagation, the image to be processed after patch division first passes through the first normalization layer LayerNorm.
[0330] The window attention matrix attn1 = Q*K^T (that is, the transpose of the last two dimensions of Q and the last two dimensions of K, T is the transpose), with a size of 8 (batch size) * 512 (number of windows) * 16 (attention heads) * 32 (number of patches in the window) * 64 (number of patches covered by the window's field of view).
[0331] After window attention, the output is x1 = attn1 * V (i.e. the last two dimensions of attn1 multiplied by the last two dimensions of V), with a size of 8 (batch size) * 512 (number of windows) * 16 (attention heads) * 32 (number of patches in the window) * 256 (embedding dimension), which is the same size as Q.
[0332] The output of block 1 is obtained by shortcutting x1 with the input of block 1, and then passing through the second normalization layer LayerNorm and the first multi-layer perceptron MLP layer in sequence. It has the same size as the input.
[0333] (3) Enter the second feature map extraction module block2 of the pre-established neural network model
[0334] (3.1) Attention mechanism using the second window division
[0335] Let n = 8, and take the window size as 8*4 = 32 patches. The image is divided into 16*32 = 512 non-overlapping windows.
[0336] The number of attention heads is taken as 16. Calculate the second query matrix Q1, the third key matrix K0′ and the third value matrix V0′ based on window attention. The second query matrix Q1, the third key matrix K0′ and the third value matrix V0′ are all tensors obtained by multiplying the input of block2 by the learnable linear mapping matrix. The third key matrix K0′ and the third value matrix V0′ are 8*8 window self-attention tensors, whose size is 8 (batch size)*512 (number of windows)*16 (attention heads)*32 (number of patches in the window)*256 (embedding dimension).
[0337] Since the window size is 8*4=32, and the window field of view is 8*8=64 (such as Figure 4 As shown), the sizes of the second query matrix Q1, the third key matrix K0′ and the third value matrix V0′ are all 8 (batch size) * 512 (number of windows) * 16 (attention heads) * 32 (number of patches in a window) * 256 (embedding dimension).
[0338] (3.2) Calculation of the fourth key matrix K1 and the fourth value matrix V1
[0339] The fourth key matrix K1 and the fourth value matrix V1 can be quickly calculated through the third key matrix K0′ and the third value matrix V0′ of the window self-attention respectively.
[0340] The third key matrix K0′ is rolled up by 4 / 2=2 patches (half the window height) to obtain the key matrix K0′+; similarly, the third key matrix K0′ is rolled down by 2 patches to obtain the key matrix K0′-. The key matrix K0′+ and the key matrix K0′- correspond to the upper and lower halves of the field of view (8*8 patches) of the window, respectively. Therefore, the key matrix K0′+ and the key matrix K0′- are concatenated to obtain the fourth key matrix K1.
[0341] Similarly, the fourth value matrix V1 can be obtained.
[0342] The third value matrix V0′ is rolled up by 4 / 2=2 patches (half the window height) to obtain the value matrix V0′+; similarly, the third value matrix V0′ is rolled down by 2 patches to obtain the value matrix V0′-. The value matrix V0′+ and the value matrix V0′- correspond to the upper and lower half of the field of view (8*8 patches) of the window, respectively. Therefore, the value matrix V0′+ and the value matrix V0′- are concatenated to obtain the fourth value matrix V1.
[0343] The sizes of the fourth key matrix K1 and the fourth value matrix V1 are both 8 (batch size) * 512 (number of windows) * 16 (attention heads) * 64 (number of patches covered by the window's field of view) * 256 (embedding dimension).
[0344] Using this calculation method, compared with the traditional self-attention mechanism method (QKV has the same size), although the size of the fourth key matrix K1 and the fourth value matrix V1 is twice as large as the second query matrix Q1, the calculation amount only increases by a small amount of scrolling and splicing calculations.
[0345] (3.3) Propagate forward in block 2
[0346] During the forward propagation process, the output of block1 first passes through the third normalization layer LayerNorm.
[0347] The window attention matrix attn2 = Q1*K1^T (that is, the transpose of the last two dimensions of Q1 and the last two dimensions of K1 multiplied by each other), with a size of 8 (batch size) * 512 (number of windows) * 16 (attention heads) * 32 (number of patches in the window) * 64 (number of patches covered by the window's field of view).
[0348] After window attention, the output is x2=attn2*V2 (i.e., the last two dimensions of attn2 multiplied by the last two dimensions of V2), with a size of 8 (batch size)*512 (number of windows)*16 (attention heads)*32 (number of patches in the window)*256 (embedding dimension), which is the same size as Q.
[0349] The output of block2 is obtained by shortcutting x with the input of block2, and then passing through the fourth normalization layer LayerNorm layer and the second multi-layer perceptron MLP layer in sequence. It has the same size as the input.
[0350] (4) The image to be processed is input into the pre-established neural network model consisting of d groups of block1 and block2.
[0351] Take d=16, that is, the image to be processed passes through block1 and block2 in each group of feature map extraction units in turn, and the size of the feature map obtained remains 8 (batch size)*512 (number of windows)*16 (attention heads)*32 (number of patches in the window)*256 (embedding dimension), then the final feature map size is 8 (batch size)*512 (number of windows)*16 (attention heads)*32 (number of patches in the window)*256 (embedding dimension).
[0352] So far, the step of image feature extraction has been completed.
[0353] (5) For the image classification task, the final feature map is input into the classifier module, that is, through a fully connected layer FC and softmax, and the output is a one-hot encoded category label. At this point, a complete model structure for image classification is formed.
[0354] It is understandable that in a classification task, if there are m categories, then the one-hot encoding of the i-th category is [0,0,0,…,1,…,0,0], the i-th bit is 1, and the others are 0.
[0355] (6) Target detection tasks
[0356] If you need to complete the image target detection task, you can replace the classifier module with the detection head structure. The final feature map is input to the detection head H, and the output is the target detection result, including target coordinate information (xc, yc, w, h), credibility information (prob), category information (classname), etc. (xc, yc, w, h) can determine a target box in the image, xc, yc are the horizontal and vertical coordinates of the center point of the target box, w and h are the width and height of the target box respectively.
[0357] Embodiment 2
[0358] The present invention also provides an image feature extraction device, such as Fig. 9 As shown, including:
[0359] A division unit, used for performing patch division on the collected image to be processed to obtain an image after patch division;
[0360] An acquisition unit, used to input the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed;
[0361] Among them, the neural network model is constructed using multiple groups of feature map extraction units, each group of feature map extraction units includes: a first feature map extraction module constructed based on the attention mechanism of the first window division method and a second feature map extraction module constructed based on the attention mechanism of the second window division method; the number of rows of each window divided by the first window division method is the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method is the number of rows of each window divided by the second window division method.
[0362] Furthermore, the units are divided into:
[0363] The image to be processed is divided into a preset number of patches to obtain an image after patch division.
[0364] Furthermore, the acquisition unit is specifically used for:
[0365] Inputting the patch-divided image into a first feature map extraction module in a first group of feature map extraction units of a neural network model to obtain a first intermediate feature map;
[0366] Inputting the first intermediate feature map into a second feature map extraction module in a first set of feature map extraction units of a neural network model to obtain a second intermediate feature map;
[0367] Inputting the second intermediate feature map into the first feature map extraction module in the second set of feature map extraction units of the neural network model to obtain a third intermediate feature map;
[0368] Inputting the third intermediate feature map into a second feature map extraction module in a second set of feature map extraction units of the neural network model to obtain a fourth intermediate feature map;
[0369] By analogy, the 2D-2th intermediate feature map is input into the first feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2D-1th intermediate feature map;
[0370] The 2D-1th intermediate feature map is input into the second feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2Dth intermediate feature map, which is the final feature map of the image to be processed; wherein D is the total number of groups of feature map extraction units.
[0371] Furthermore, the first window division method includes:
[0372] A patch of a row and b column is used to form a window. The image after patch division is divided into hw / ab windows. The size of each window is ab patches, and the field of view of each window is 2ab patches.
[0373] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0374] Furthermore, the second window division method includes:
[0375] A patch of b rows and a columns is used to form a window. The image after patch division is divided into hw / ab windows. The size of each window is ab patches, and the field of view of each window is 2ab patches.
[0376] Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0377] Furthermore, the attention mechanism based on the first window division method includes:
[0378] The feature map input to the attention mechanism based on the first window division method is divided into windows according to the first window division method, and the feature map after the window division according to the first window division method is positionally embedded to obtain a feature map F1;
[0379] Using the improved attention algorithm, attention calculation is performed on the feature map F1 to obtain the feature map output by the attention mechanism based on the first window division method.
[0380] Furthermore, the attention mechanism based on the second window division method includes:
[0381] The feature map input to the attention mechanism based on the second window division method is divided into windows according to the second window division method, and the feature map after the window division according to the second window division method is positionally embedded to obtain a feature map F2;
[0382] Using the improved attention algorithm, attention calculation is performed on the feature map F2 to obtain the feature map output by the attention mechanism based on the second window division method.
[0383] Furthermore, the improved attention algorithm is used to perform attention calculation on the feature map F1, including:
[0384] Let the number of attention heads be H, and the size of feature map F1 be B1×G1×C1, where B1 is the batch size, G1 is the number of patches in the third feature map, and C1 is the embedding dimension of feature map F1;
[0385] Performing linear mapping on the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix;
[0386] Roll the first key matrix to obtain a second key matrix;
[0387] Roll the first value matrix to obtain a second value matrix;
[0388] The feature map output by the attention mechanism based on the first window division method is calculated using the first query matrix, the second key matrix and the second value matrix.
[0389] Furthermore, the improved attention algorithm is used to perform attention calculation on the feature map F2, including:
[0390] Let the number of attention heads be H, and the size of feature map F2 be B2×G2×C2, where B2 is the batch size, G2 is the number of patches in the fifth feature map, and C2 is the embedding dimension of feature map F2;
[0391] Performing linear mapping on the feature graph F2, calculating and obtaining a second query matrix, a third key matrix and a third value matrix;
[0392] Roll the third key matrix to obtain a fourth key matrix;
[0393] Roll the third value matrix to obtain a fourth value matrix;
[0394] The second query matrix, the fourth key matrix and the fourth value matrix are used to calculate the feature map output by the attention mechanism based on the second window division method.
[0395] Furthermore, linear mapping is performed on the feature map F1 to obtain a first query matrix, a first key matrix and a first value matrix, including:
[0396] Multiplying the feature map F1 with a first preset learnable linear mapping matrix to obtain a first query matrix;
[0397] Multiplying the feature map F1 with the second preset learnable linear mapping matrix to obtain a first key matrix;
[0398] Multiplying the feature map F1 by a third preset learnable linear mapping matrix to obtain a first value matrix;
[0399] Among them, the sizes of the first query matrix, the first key matrix, and the first value matrix are all: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0400] Further, the first key matrix is scrolled to obtain a second key matrix, including:
[0401] The first key matrix is scrolled upward by half the height of each window divided by the first window division method to obtain a key matrix K0+;
[0402] Scroll the first key matrix downward by half the height of each window divided by the first window division method to obtain a key matrix K0-;
[0403] Concatenate the key matrix K0+ and the key matrix K0- in the penultimate dimension to obtain a second key matrix;
[0404] Among them, the size of the second bond matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0405] Furthermore, the first value matrix is rolled to obtain a second value matrix, including:
[0406] Scroll the first value matrix upward by half the height of each window divided by the first window division method to obtain a value matrix V0+;
[0407] Scroll the first value matrix downward by half the height of each window divided by the first window division method to obtain a value matrix V0-;
[0408] Concatenate the value matrix V0+ and the value matrix V0- in the second to last dimension to obtain a second value matrix;
[0409] Among them, the size of the second value matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0410] Furthermore, the first query matrix, the second key matrix and the second value matrix are used to calculate the feature map output by the attention mechanism based on the first window division method, including:
[0411] The first window attention matrix is obtained by multiplying the last two dimensions of the first query matrix and the transpose of the last two dimensions of the second key matrix, where the size of the first window attention matrix is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and 2ab is the number of patches in each window;
[0412] Multiply the last two dimensions of the first window attention matrix and the last two dimensions of the second value matrix to obtain the feature map output by the attention mechanism based on the first window division method; the size of the feature map output by the attention mechanism based on the first window division method is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0413] Furthermore, linear mapping is performed on the feature graph F2 to calculate the second query matrix, the third key matrix and the third value matrix, including:
[0414] Multiplying the feature map F2 by a fourth preset learnable linear mapping matrix to obtain a second query matrix;
[0415] Multiplying the feature map F2 by the fifth preset learnable linear mapping matrix to obtain a third key matrix;
[0416] Multiplying the feature map F2 by the sixth preset learnable linear mapping matrix to obtain a third value matrix;
[0417] Among them, the sizes of the second query matrix, the third key matrix, and the third value matrix are all: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0418] Furthermore, the third key matrix is scrolled to obtain a fourth key matrix, including:
[0419] The third key matrix is rolled upward by half the height of each window divided by the second window division method to obtain a key matrix K0′+;
[0420] The third key matrix is scrolled down by half the height of each window divided by the second window division method to obtain a key matrix K0′-;
[0421] Concatenate the key matrix K0′+ and the key matrix K0′- in the penultimate dimension to obtain the fourth key matrix;
[0422] Among them, the size of the fourth key matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0423] Furthermore, the third value matrix is rolled to obtain a fourth value matrix, including:
[0424] The third value matrix is scrolled upward by half the height of each window divided by the second window division method to obtain a value matrix V0′+;
[0425] The third value matrix is scrolled down by half the height of each window divided by the second window division method to obtain a value matrix V0′-;
[0426] Concatenate the value matrix V0′+ and the value matrix V0′- in the second to last dimension to obtain the fourth value matrix;
[0427] Among them, the size of the fourth value matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0428] Furthermore, the second query matrix, the fourth key matrix and the fourth value matrix are used to calculate the feature map output by the attention mechanism based on the second window division method, including:
[0429] The second window attention matrix is obtained by multiplying the last two dimensions of the second query matrix and the transpose of the last two dimensions of the fourth key matrix. The size of the second window attention matrix is: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and 2ab is the number of patches in each window;
[0430] Multiply the last two dimensions of the second window attention matrix and the last two dimensions of the fourth value matrix to obtain the feature map output by the attention mechanism based on the second window division method. The size of the feature map output by the attention mechanism based on the second window division method is: Among them, B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
[0431] Furthermore, the device also includes: an establishment unit for establishing a pre-established neural network model; the establishment unit includes:
[0432] An acquisition module, used for acquiring historical images and final feature maps corresponding to the historical images;
[0433] A construction module, used to construct a training set using historical images and final feature maps corresponding to the historical images;
[0434] The training module is used to train the improved Transformer model using the training set to obtain a pre-established neural network model; wherein the improved Transformer model includes: a first feature map extraction module and a second feature map extraction module.
[0435] Furthermore, the first feature map extraction module includes: a first normalization layer, an attention mechanism based on a first window division method, a second normalization layer, and a first multi-layer perceptron connected in sequence;
[0436] The second feature map extraction module includes: a third normalization layer, an attention mechanism based on the second window division method, a fourth normalization layer and a second multi-layer perceptron connected in sequence.
[0437] Furthermore, the input of the first normalization layer in the first group of feature map extraction units is the image after patch division; the input of the first normalization layer in the other groups of feature map extraction units except the first group of feature map extraction units is the output of the second multi-layer perceptron in the previous group of feature map extraction units;
[0438] The output of the first normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the first window division method in each group of feature map extraction units;
[0439] The output of the attention mechanism based on the first window division method in each group of feature map extraction units is shortcut-connected with the input of the first normalization layer in each group of feature map extraction units, and then used as the input of the second normalization layer in each group of feature map extraction units;
[0440] The output of the second normalization layer in each group of feature map extraction units is the input of the first multi-layer perceptron in each group of feature map extraction units, and the output of the first multi-layer perceptron in each group of feature map extraction units is the 2d-1th intermediate feature map;
[0441] The 2d-1th intermediate feature map is the input of the third normalization layer in each group of feature map extraction units, and the output of the third normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the second window division method in each group of feature map extraction units;
[0442] The output of the attention mechanism based on the second window division method in each group of feature map extraction units is shortcut-connected with the input of the third normalization layer in each group of feature map extraction units, and then serves as the input of the fourth normalization layer in each group of feature map extraction units;
[0443] The output of the fourth normalization layer in each group of feature map extraction units is the input of the second multilayer perceptron in each group of feature map extraction units, and the output of the second multilayer perceptron in each group of feature map extraction units is the 2d-th intermediate feature map, where d∈[1,D], and D is the total number of groups of feature map extraction units.
[0444] It can be understood that the device embodiment provided above corresponds to the method embodiment described above, and the corresponding specific contents can be referenced to each other and will not be repeated here.
[0445] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.
[0446] Embodiment 3
[0447] like Fig.10 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.
[0448] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement a corresponding method flow or a corresponding function, so as to implement the steps of an image feature extraction method in the above-mentioned embodiment.
[0449] Embodiment 4
[0450] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in the electronic device for storing programs and data. It can be understood that the storage medium here can include both the built-in storage medium in the electronic device and the extended storage medium supported by the electronic device. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of an image feature extraction method in the above embodiment.
[0451] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0452] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0453] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0454] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0455] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for extracting image features, characterized in that: include: Perform patch division on the collected image to be processed to obtain an image after patch division; Inputting the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed; Among them, the neural network model is constructed using multiple groups of feature map extraction units, each group of feature map extraction units includes: a first feature map extraction module constructed based on the attention mechanism of the first window division method and a second feature map extraction module constructed based on the attention mechanism of the second window division method; the number of rows of each window divided by the first window division method is the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method is the number of rows of each window divided by the second window division method.
2. The method according to claim 1, characterized in that: The patch division of the collected image to be processed includes: The image to be processed is divided into a preset number of patches to obtain an image after patch division.
3. The method according to claim 1, characterized in that Inputting the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed, including: Inputting the patch-divided image into a first feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a first intermediate feature map; Inputting the first intermediate feature map into a second feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a second intermediate feature map; Inputting the second intermediate feature map into a first feature map extraction module in a second group of feature map extraction units of the neural network model to obtain a third intermediate feature map; Inputting the third intermediate feature map into a second feature map extraction module in a second set of feature map extraction units of the neural network model to obtain a fourth intermediate feature map; By analogy, the 2D-2th intermediate feature map is input into the first feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2D-1th intermediate feature map; The 2D-1th intermediate feature map is input into the second feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2Dth intermediate feature map, and the 2Dth intermediate feature map is the final feature map of the image to be processed; wherein D is the total number of groups of feature map extraction units.
4. The method according to claim 1, characterized in that: The first window division method includes: A patch of a rows and b columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches; Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
5. The method according to claim 1, characterized in that The second window division method includes: A patch of b rows and a columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches; Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
6. The method according to claim 1, characterized in that The attention mechanism based on the first window division method includes: The feature map input to the attention mechanism based on the first window division method is divided into windows according to the first window division method, and the feature map after the window division according to the first window division method is positionally embedded to obtain a feature map F1; Using the improved attention algorithm, attention calculation is performed on the feature map F1 to obtain the feature map output by the attention mechanism based on the first window division method.
7. The method according to claim 1, characterized in that The attention mechanism based on the second window division method includes: The feature map input to the attention mechanism based on the second window division method is divided into windows according to the second window division method, and the feature map after the window division according to the second window division method is positionally embedded to obtain a feature map F2; Using the improved attention algorithm, attention calculation is performed on the feature map F2 to obtain the feature map output by the attention mechanism based on the second window division method.
8. The method according to claim 6, characterized in that The improved attention algorithm is used to perform attention calculation on the feature map F1, including: Let the number of attention heads be H, and the size of feature map F1 be B1×G1×C1, where B1 is the batch size, G1 is the number of patches in the third feature map, and C1 is the embedding dimension of feature map F1; Performing linear mapping on the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix; Rolling the first key matrix to obtain a second key matrix; Rolling the first value matrix to obtain a second value matrix; The first query matrix, the second key matrix and the second value matrix are used to calculate a feature map output by the attention mechanism based on the first window division method.
9. The method according to claim 7, characterized in that: The improved attention algorithm is used to perform attention calculation on the feature map F2, including: Let the number of attention heads be H, and the size of feature map F2 be B2×G2×C2, where B2 is the batch size, G2 is the number of patches in the fifth feature map, and C2 is the embedding dimension of feature map F2; Performing linear mapping on the feature graph F2 to calculate a second query matrix, a third key matrix and a third value matrix; Rolling the third key matrix to obtain a fourth key matrix; Rolling the third value matrix to obtain a fourth value matrix; The feature map output by the attention mechanism based on the second window division method is calculated using the second query matrix, the fourth key matrix and the fourth value matrix.
10. The method according to claim 8, characterized in that The linear mapping of the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix includes: Multiplying the feature map F1 by a first preset learnable linear mapping matrix to obtain a first query matrix; Multiplying the feature map F1 with a second preset learnable linear mapping matrix to obtain a first key matrix; Multiplying the feature map F1 with a third preset learnable linear mapping matrix to obtain a first value matrix; The sizes of the first query matrix, the first key matrix, and the first value matrix are all: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
11. The method according to claim 8, characterized in that The step of rolling the first key matrix to obtain a second key matrix includes: Scrolling the first key matrix upward by half the height of each window divided by the first window division method to obtain a key matrix K0+; Scrolling the first key matrix downward by half the height of each window divided by the first window division method to obtain a key matrix K0-; Concatenate the key matrix K0+ and the key matrix K0- in the penultimate dimension to obtain the second key matrix; Among them, the size of the second bond matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
12. The method according to claim 8, characterized in that The step of rolling the first value matrix to obtain a second value matrix includes: Scrolling the first value matrix upward by half the height of each window divided by the first window division method to obtain a value matrix V0+; Scroll the first value matrix downward by half the height of each window divided by the first window division method to obtain a value matrix V0-; Concatenate the value matrix V0+ and the value matrix V0- in the penultimate dimension to obtain the second value matrix; Among them, the size of the second value matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
13. The method according to claim 8, characterized in that The step of calculating the feature map output by the attention mechanism based on the first window division method by using the first query matrix, the second key matrix and the second value matrix includes: The last two dimensions of the first query matrix and the transpose of the last two dimensions of the second key matrix are multiplied to obtain a first window attention matrix, where the size of the first window attention matrix is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, 2ab is the number of patches in each window, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, w is the number of patches in each column of the image after patch division; The last two dimensions of the first window attention matrix and the last two dimensions of the second value matrix are multiplied to obtain the feature map output by the attention mechanism based on the first window division method; wherein the size of the feature map output by the attention mechanism based on the first window division method is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and C1 is the embedding dimension of the feature map F1.
14. The method according to claim 9, characterized in that The linear mapping of the feature graph F2 to obtain a second query matrix, a third key matrix and a third value matrix is performed, including: Multiplying the feature map F2 by a fourth preset learnable linear mapping matrix to obtain a second query matrix; Multiplying the feature map F2 by a fifth preset learnable linear mapping matrix to obtain a third key matrix; Multiplying the feature map F2 by a sixth preset learnable linear mapping matrix to obtain a third value matrix; The sizes of the second query matrix, the third key matrix, and the third value matrix are all: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
15. The method according to claim 9, characterized in that The step of rolling the third key matrix to obtain a fourth key matrix includes: Scroll the third key matrix upward by half the height of each window divided by the second window division method to obtain a key matrix K0′+; Scroll the third key matrix downward by half the height of each window divided by the second window division method to obtain a key matrix K0′-; Concatenate the key matrix K0′+ and the key matrix K0′- in the penultimate dimension to obtain the fourth key matrix; Among them, the size of the fourth key matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
16. The method according to claim 9, characterized in that The step of rolling the third value matrix to obtain a fourth value matrix includes: Scrolling the third value matrix upward by half the height of each window divided by the second window division method to obtain a value matrix V0′+; Scroll the third value matrix downward by half the height of each window divided by the second window division method to obtain a value matrix V0′-; Concatenate the value matrix V0′+ and the value matrix V0′- in the penultimate dimension to obtain the fourth value matrix; Among them, the size of the fourth value matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
17. The method according to claim 9, characterized in that The step of calculating the feature map output by the attention mechanism based on the second window partitioning method by using the second query matrix, the fourth key matrix and the fourth value matrix includes: The last two dimensions of the second query matrix and the transpose of the last two dimensions of the fourth key matrix are multiplied to obtain a second window attention matrix, where the size of the second window attention matrix is: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and 2ab is the number of patches in each window; The last two dimensions of the second window attention matrix and the last two dimensions of the fourth value matrix are multiplied to obtain the feature map output by the attention mechanism based on the second window division method, wherein the size of the feature map output by the attention mechanism based on the second window division method is: Among them, B2 is batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
18. The method according to claim 1, characterized in that The acquisition process of the pre-established neural network model includes: Collect historical images and final feature maps corresponding to the historical images; Constructing a training set using the historical images and final feature maps corresponding to the historical images; The improved Transformer model is trained using the training set to obtain the pre-established neural network model; wherein the improved Transformer model includes: a first feature map extraction module and a second feature map extraction module.
19. The method according to claim 1, characterized in that The first feature map extraction module includes: a first normalization layer, an attention mechanism based on a first window division method, a second normalization layer and a first multi-layer perceptron connected in sequence; The second feature map extraction module includes: a third normalization layer, an attention mechanism based on the second window division method, a fourth normalization layer and a second multi-layer perceptron connected in sequence.
20. The method according to claim 19, characterized in that The input of the first normalization layer in the first group of feature map extraction units is the image after patch division; the input of the first normalization layer in other groups of feature map extraction units except the first group of feature map extraction units is the output of the second multi-layer perceptron in the previous group of feature map extraction units; The output of the first normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the first window division method in each group of feature map extraction units; The output of the attention mechanism based on the first window division method in each group of feature map extraction units is shortcut-connected with the input of the first normalization layer in each group of feature map extraction units, and then used as the input of the second normalization layer in each group of feature map extraction units; The output of the second normalization layer in each group of feature map extraction units is the input of the first multi-layer perceptron in each group of feature map extraction units, and the output of the first multi-layer perceptron in each group of feature map extraction units is the 2d-1th intermediate feature map; The 2d-1th intermediate feature map is the input of the third normalization layer in each group of feature map extraction units, and the output of the third normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the second window division method in each group of feature map extraction units; The output of the attention mechanism based on the second window division method in each group of feature map extraction units is shortcut-connected with the input of the third normalization layer in each group of feature map extraction units, and then used as the input of the fourth normalization layer in each group of feature map extraction units; The output of the fourth normalization layer in each group of feature map extraction units is the input of the second multilayer perceptron in each group of feature map extraction units, and the output of the second multilayer perceptron in each group of feature map extraction units is the 2d-th intermediate feature map, where d∈[1,D], and D is the total number of groups of feature map extraction units.
21. An image feature extraction device, characterized in that: include: A division unit, used for performing patch division on the collected image to be processed to obtain an image after patch division; An acquisition unit, used for inputting the patch-divided image into a pre-established neural network model to obtain a final feature map of the image to be processed; Among them, the neural network model is constructed using multiple groups of feature map extraction units, each group of feature map extraction units includes: a first feature map extraction module constructed based on the attention mechanism of the first window division method and a second feature map extraction module constructed based on the attention mechanism of the second window division method; the number of rows of each window divided by the first window division method is the number of columns of each window divided by the second window division method, and the number of columns of each window divided by the second window division method is the number of rows of each window divided by the second window division method.
22. The device according to claim 21, characterized in that The division unit is specifically used for: The image to be processed is divided into a preset number of patches to obtain an image after patch division.
23. The device according to claim 21, characterized in that The acquisition unit is specifically used for: Inputting the patch-divided image into a first feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a first intermediate feature map; Inputting the first intermediate feature map into a second feature map extraction module in a first group of feature map extraction units of the neural network model to obtain a second intermediate feature map; Inputting the second intermediate feature map into a first feature map extraction module in a second group of feature map extraction units of the neural network model to obtain a third intermediate feature map; Inputting the third intermediate feature map into a second feature map extraction module in a second set of feature map extraction units of the neural network model to obtain a fourth intermediate feature map; By analogy, the 2D-2th intermediate feature map is input into the first feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2D-1th intermediate feature map; The 2D-1th intermediate feature map is input into the second feature map extraction module in the Dth group of feature map extraction units of the neural network model to obtain the 2Dth intermediate feature map, and the 2Dth intermediate feature map is the final feature map of the image to be processed; wherein D is the total number of groups of feature map extraction units.
24. The device according to claim 21, characterized in that The first window division method includes: A patch of a rows and b columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches; Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
25. The device according to claim 21, characterized in that The second window division method includes: A patch of b rows and a columns is used to form a window, and the image after the patch division is divided into hw / ab windows, the size of each window is ab patches, and the field of view of each window is 2ab patches; Among them, a, b, h and w are all positive integers, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
26. The device according to claim 21, characterized in that The attention mechanism based on the first window division method includes: The feature map input to the attention mechanism based on the first window division method is divided into windows according to the first window division method, and the feature map after the window division according to the first window division method is positionally embedded to obtain a feature map F1; Using the improved attention algorithm, attention calculation is performed on the feature map F1 to obtain the feature map output by the attention mechanism based on the first window division method.
27. The device according to claim 21, characterized in that The attention mechanism based on the second window division method includes: The feature map input to the attention mechanism based on the second window division method is divided into windows according to the second window division method, and the feature map after the window division according to the second window division method is positionally embedded to obtain a feature map F2; Using the improved attention algorithm, attention calculation is performed on the feature map F2 to obtain the feature map output by the attention mechanism based on the second window division method.
28. The device according to claim 26, characterized in that The improved attention algorithm is used to perform attention calculation on the feature map F1, including: Let the number of attention heads be H, and the size of feature map F1 be B1×G1×C1, where B1 is the batch size, G1 is the number of patches in the third feature map, and C1 is the embedding dimension of feature map F1; Performing linear mapping on the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix; Rolling the first key matrix to obtain a second key matrix; Rolling the first value matrix to obtain a second value matrix; The first query matrix, the second key matrix and the second value matrix are used to calculate a feature map output by the attention mechanism based on the first window division method.
29. The device according to claim 27, characterized in that The improved attention algorithm is used to perform attention calculation on the feature map F2, including: Let the number of attention heads be H, and the size of feature map F2 be B2×G2×C2, where B2 is the batch size, G2 is the number of patches in the fifth feature map, and C2 is the embedding dimension of feature map F2; Performing linear mapping on the feature graph F2 to calculate a second query matrix, a third key matrix and a third value matrix; Rolling the third key matrix to obtain a fourth key matrix; Rolling the third value matrix to obtain a fourth value matrix; The feature map output by the attention mechanism based on the second window division method is calculated using the second query matrix, the fourth key matrix and the fourth value matrix.
30. The device according to claim 28, characterized in that The linear mapping of the feature graph F1 to obtain a first query matrix, a first key matrix and a first value matrix includes: Multiplying the feature map F1 by a first preset learnable linear mapping matrix to obtain a first query matrix; Multiplying the feature map F1 with a second preset learnable linear mapping matrix to obtain a first key matrix; Multiplying the feature map F1 with a third preset learnable linear mapping matrix to obtain a first value matrix; The sizes of the first query matrix, the first key matrix, and the first value matrix are all: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
31. The device according to claim 28, characterized in that The step of rolling the first key matrix to obtain a second key matrix includes: Scrolling the first key matrix upward by half the height of each window divided by the first window division method to obtain a key matrix K0+; Scrolling the first key matrix downward by half the height of each window divided by the first window division method to obtain a key matrix K0-; Concatenate the key matrix K0+ and the key matrix K0- in the penultimate dimension to obtain the second key matrix; Among them, the size of the second bond matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
32. The device according to claim 28, characterized in that The step of rolling the first value matrix to obtain a second value matrix includes: Scrolling the first value matrix upward by half the height of each window divided by the first window division method to obtain a value matrix V0+; Scroll the first value matrix downward by half the height of each window divided by the first window division method to obtain a value matrix V0-; Concatenate the value matrix V0+ and the value matrix V0- in the penultimate dimension to obtain the second value matrix; Among them, the size of the second value matrix is B1 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C1 is the embedding dimension of the feature map F1, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
33. The device according to claim 28, characterized in that The step of calculating the feature map output by the attention mechanism based on the first window division method by using the first query matrix, the second key matrix and the second value matrix includes: The last two dimensions of the first query matrix and the transpose of the last two dimensions of the second key matrix are multiplied to obtain a first window attention matrix, where the size of the first window attention matrix is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, 2ab is the number of patches in each window, a, b, h and w are all positive integers, a is the number of patches in each row of the window, b is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, w is the number of patches in each column of the image after patch division; The last two dimensions of the first window attention matrix and the last two dimensions of the second value matrix are multiplied to obtain the feature map output by the attention mechanism based on the first window division method; wherein the size of the feature map output by the attention mechanism based on the first window division method is: B1 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and C1 is the embedding dimension of the feature map F1.
34. The device according to claim 29, characterized in that The linear mapping of the feature graph F2 to obtain a second query matrix, a third key matrix and a third value matrix is performed, including: Multiplying the feature map F2 by a fourth preset learnable linear mapping matrix to obtain a second query matrix; Multiplying the feature map F2 by a fifth preset learnable linear mapping matrix to obtain a third key matrix; Multiplying the feature map F2 by a sixth preset learnable linear mapping matrix to obtain a third value matrix; The sizes of the second query matrix, the third key matrix, and the third value matrix are all: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
35. The device according to claim 29, characterized in that The step of rolling the third key matrix to obtain a fourth key matrix includes: Scroll the third key matrix upward by half the height of each window divided by the second window division method to obtain a key matrix K0′+; Scroll the third key matrix downward by half the height of each window divided by the second window division method to obtain a key matrix K0′-; Concatenate the key matrix K0′+ and the key matrix K0′- in the penultimate dimension to obtain the fourth key matrix; Among them, the size of the fourth key matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
36. The device according to claim 29, characterized in that The step of rolling the third value matrix to obtain a fourth value matrix includes: Scrolling the third value matrix upward by half the height of each window divided by the second window division method to obtain a value matrix V0′+; Scroll the third value matrix downward by half the height of each window divided by the second window division method to obtain a value matrix V0′-; Concatenate the value matrix V0′+ and the value matrix V0′- in the penultimate dimension to obtain the fourth value matrix; Among them, the size of the fourth value matrix is B2 is the batch size, is the number of windows, H is the number of attention heads, 2ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
37. The device according to claim 29, characterized in that The step of calculating the feature map output by the attention mechanism based on the second window partitioning method by using the second query matrix, the fourth key matrix and the fourth value matrix includes: The last two dimensions of the second query matrix and the transpose of the last two dimensions of the fourth key matrix are multiplied to obtain a second window attention matrix, where the size of the second window attention matrix is: B2 is the batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, and 2ab is the number of patches in each window; The last two dimensions of the second window attention matrix and the last two dimensions of the fourth value matrix are multiplied to obtain the feature map output by the attention mechanism based on the second window division method, wherein the size of the feature map output by the attention mechanism based on the second window division method is: Among them, B2 is batch size, is the number of windows, H is the number of attention heads, ab is the number of patches in each window, C2 is the embedding dimension of the feature map F2, a, b, h and w are all positive integers, b is the number of patches in each row of the window, a is the number of patches in each column of the window, a is less than b, h is the number of patches in each row of the image after patch division, and w is the number of patches in each column of the image after patch division.
38. The device according to claim 21, characterized in that The acquisition process of the pre-established neural network model includes: Collect historical images and final feature maps corresponding to the historical images; Constructing a training set using the historical images and final feature maps corresponding to the historical images; The improved Transformer model is trained using the training set to obtain the pre-established neural network model; wherein the improved Transformer model includes: a first feature map extraction module and a second feature map extraction module.
39. The device according to claim 21, characterized in that The first feature map extraction module includes: a first normalization layer, an attention mechanism based on a first window division method, a second normalization layer and a first multi-layer perceptron connected in sequence; The second feature map extraction module includes: a third normalization layer, an attention mechanism based on the second window division method, a fourth normalization layer and a second multi-layer perceptron connected in sequence.
40. The device according to claim 39, characterized in that The input of the first normalization layer in the first group of feature map extraction units is the image after patch division; the input of the first normalization layer in other groups of feature map extraction units except the first group of feature map extraction units is the output of the second multi-layer perceptron in the previous group of feature map extraction units; The output of the first normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the first window division method in each group of feature map extraction units; The output of the attention mechanism based on the first window division method in each group of feature map extraction units is shortcut-connected with the input of the first normalization layer in each group of feature map extraction units, and then used as the input of the second normalization layer in each group of feature map extraction units; The output of the second normalization layer in each group of feature map extraction units is the input of the first multi-layer perceptron in each group of feature map extraction units, and the output of the first multi-layer perceptron in each group of feature map extraction units is the 2d-1th intermediate feature map; The 2d-1th intermediate feature map is the input of the third normalization layer in each group of feature map extraction units, and the output of the third normalization layer in each group of feature map extraction units is the input of the attention mechanism based on the second window division method in each group of feature map extraction units; The output of the attention mechanism based on the second window division method in each group of feature map extraction units is shortcut-connected with the input of the third normalization layer in each group of feature map extraction units, and then used as the input of the fourth normalization layer in each group of feature map extraction units; The output of the fourth normalization layer in each group of feature map extraction units is the input of the second multilayer perceptron in each group of feature map extraction units, and the output of the second multilayer perceptron in each group of feature map extraction units is the 2d-th intermediate feature map, where d∈[1,D], and D is the total number of groups of feature map extraction units.
41. An electronic device, characterized in that: include: at least one processor and memory; The memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the image feature extraction method according to any one of claims 1 to 20 is implemented.
42. A readable storage medium, characterized in that An execution program is stored thereon, and when the execution program is executed, the image feature extraction method as described in any one of claims 1 to 20 is implemented.
