Image feature extraction method, and device and readable storage medium
By combining local features and global features in the feature extraction model, using the local feature extraction structure to extract pixel spatial position information, the problem of insufficient sensitivity of ViT structure to pixel spatial position information is solved, the accuracy and accuracy of target detection are improved, and the expansion of the Application scenarios.
Patent Information
- Application Number
- PCT/CN2023/114651
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-15
- Filing Date
- 2023-08-24
- Publication Date
- 2024-11-21
AI Technical Summary
The ViT structure pays too much attention to global information when extracting image features, resulting in weak sensitivity to pixel spatial position information, limiting its application scenarios, especially in object detection tasks, resulting in the accuracy and detection accuracy of the position determination of the target in the image. Lower.
By introducing a combination of multiple local feature extraction structures and ViT structures in the feature extraction model, the global features of the ViT structure are enhanced by using the features output by the local feature extraction structure that are sensitive to pixel spatial position information. The specific steps include inputting the original image into the features The model is extracted, and the local feature extraction structure and ViT structure are processed in sequence, combining local features and global features to improve the spatial position sensitivity of the features.
It improves the applicability and application scenarios of ViT structure, improves the accuracy and detection accuracy of pixel spatial positions in the object detection task, and expands the application scope of feature extraction models.
Smart Images

Figure CN2023114651_21112024_PF_FP_ABST
Abstract
Description
Image feature extraction method, device and readable storage medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 15, 2023, with application number 202310545790.0 and invention name “Image feature extraction method, device and readable storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of image processing, and in particular to an image feature extraction method, device, and readable storage medium. Background Art
[0003] Image feature extraction is an image processing method that uses a mathematical vector (also known as an image feature) to represent the content of an image. It can map high-dimensional image information into a low-dimensional feature space, facilitating the implementation of various tasks such as classification and object detection based on the obtained image features. Image features include local features and global features. Local features primarily describe parts of the image, while global features primarily describe the image as a whole. The Vision Transformer (ViT) structure is often used to extract global image features through a self-attention mechanism.
[0004] However, when the ViT structure uses the self-attention mechanism to extract image features, it focuses too much on the global information of the image, making the extracted global features less sensitive to pixel spatial position information. This in turn makes the ViT structure less applicable, which limits its application scenarios. For example, for target detection tasks, it is necessary not only to determine the target to be identified in the image, but also to determine the pixel spatial position of the target in the image. Since the global features extracted by the ViT structure are less sensitive to pixel spatial position information, when implementing target detection tasks based on the global features extracted by the ViT structure, the accuracy of determining the pixel spatial position of the target in the image is low, resulting in low target detection precision.
[0005] Summary of the Invention
[0006] This application provides an image feature extraction method, device, and readable storage medium that can solve the problem of poor applicability of the ViT structure due to the weak sensitivity of the global features extracted by the ViT structure to the pixel spatial position information. The technical solution is as follows:
[0007] A method for extracting image features is provided, the method comprising:
[0008] Obtain the original image of the feature to be extracted;
[0009] Inputting the original image into a feature extraction model to perform feature extraction, and outputting a first image feature of the original image;
[0010] In which, the feature extraction model includes multiple local feature extraction structures and ViT structures connected in sequence; the ViT structure is used to determine the first image feature based on the second image feature and the third image feature, the second image feature is the image feature output by the upper layer structure of the ViT structure, the third image feature is the image feature output by the target local feature extraction structure among the multiple local feature extraction structures, the target local feature extraction structure includes one or more of the multiple local feature extraction structures except the upper layer structure of the ViT structure, and the third image feature is a local feature sensitive to pixel spatial position information.
[0011] The beneficial effects of the technical solution provided by the embodiments of the present application are:
[0012] In an embodiment of the present application, the original image of the feature to be extracted is first obtained, and then the original image is input into the feature extraction model for feature extraction, and the first image feature of the original image is output. Among them, the feature extraction model includes a plurality of local feature extraction structures and a ViT structure connected in sequence, and the ViT structure is used to determine the first image feature based on the second image feature and the third image feature. The second image feature is the image feature output by the upper layer structure of the ViT structure, and the third image feature is the image feature output by the target local feature extraction structure among the multiple local feature extraction structures. The target local feature extraction structure includes one or more of the multiple local feature extraction structures except the upper layer structure of the ViT structure. The third image feature is a local feature sensitive to pixel spatial position information. Since the third image feature is a local feature sensitive to pixel spatial position information, the first image feature determined by the ViT structure based on the third image feature also has a strong pixel spatial position sensitivity, that is, the ViT structure uses the local feature sensitive to pixel spatial position information to enhance the sensitivity of the highly abstract global feature to the spatial position, which can improve the applicability of the ViT structure and expand the application scenario of the ViT structure, that is, expand the application scenario of the feature extraction model. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] FIG1 is a flow chart of an image feature extraction method provided in an embodiment of the present application;
[0014] FIG2 is a schematic diagram of the structure of a feature extraction model provided in an embodiment of the present application;
[0015] FIG3 is a schematic diagram of the structure of another feature extraction model provided in an embodiment of the present application;
[0016] FIG4 is a schematic diagram of a connection between a convolution structure and a ViT structure provided in an embodiment of the present application;
[0017] FIG5 is a schematic diagram of a structure for implementing a target detection task provided by an embodiment of the present application;
[0018] FIG6 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0019] Before explaining the embodiments of the present application in detail, the application scenarios of the embodiments of the present application are first explained.
[0020] The ViT (Vision Transformer) structure is often used to extract global image features through the self-attention mechanism. However, since the ViT structure perceives a global field of view when extracting features, it loses some sensitivity to pixel spatial position information. Therefore, the ViT structure has poor applicability and limited application scenarios.
[0021] Based on this, an embodiment of the present application proposes an image feature extraction method, which can extract features from the original image through a feature extraction model, wherein the feature extraction model includes multiple local feature extraction structures and a ViT structure connected in sequence, and the ViT structure is used to determine the first image feature based on the second image feature output by the upper layer structure of the ViT structure and the third image feature output by the target local feature extraction structure among the multiple local feature extraction structures. In this way, the ViT structure can utilize the sensitivity of the third image feature to the pixel spatial position information, so that the output first image feature also has a strong pixel spatial position sensitivity, thereby improving the applicability of the ViT structure and expanding the application scenarios of the ViT structure, i.e., the feature extraction model.
[0022] Please refer to Figure 1, which is a flowchart of an image feature extraction method provided in an embodiment of the present application. The method can be applied to a computer device, which can be a terminal, server, or embedded device, and the terminal can be a desktop computer or tablet computer. The method includes the following steps:
[0023] Step 101: A computer device obtains an original image from which features are to be extracted.
[0024] Among them, the original image is any two-dimensional RGB image collected by visible light, which is an image including the target to be classified or the target to be detected. In this way, the original image can be firstly feature extracted by the method described below, and then the target can be detected, classified or segmented based on the extracted image features, thereby realizing the target detection task, target classification task or target segmentation task.
[0025] For example, in a street scene, the original image is an image captured by a surveillance camera, and the target to be detected, classified, or segmented is a pedestrian, vehicle, or other target. It should be noted that the street scene is merely an example and does not limit the application scenarios of this application. For example, this application can also be applied to face recognition scenarios, in which the original image is an image captured by a face recognition camera, and the target to be detected, classified, or segmented is a face.
[0026] In step 102 , the computer device inputs the original image into a feature extraction model to perform feature extraction, and outputs a first image feature of the original image.
[0027] The feature extraction model is a pre-trained model, comprising a plurality of local feature extraction structures and a ViT structure connected in sequence. The structure corresponding to the ViT structure among the plurality of local feature extraction structures is a target local feature extraction structure, and the target local feature structure is not a structure above the ViT structure. The ViT structure corresponds to one or more of the plurality of local feature extraction structures, and the correspondence between the ViT structure and the plurality of local feature extraction structures can be pre-set.
[0028] For example, in a street scene, the first image features output by the feature extraction model include pixel features of objects such as pedestrians or vehicles.
[0029] In one example, the ViT structure corresponds to a local feature extraction structure among multiple local feature extraction structures. For example, please refer to Figure 2, which is a structural diagram of a feature extraction model provided in an embodiment of the present application. As shown in Figure 2, the feature extraction model includes a first local feature extraction structure, a second local feature extraction structure, and a ViT structure. The ViT structure can be pre-set to correspond to the first local feature extraction structure, that is, the target local feature extraction structure corresponding to the ViT structure is the first local feature extraction structure, and the first local feature extraction structure is not the upper layer structure of the ViT structure.
[0030] In another example, the ViT structure corresponds to multiple local feature extraction structures in the multiple local feature extraction structures. For example, if the feature extraction model includes three local feature extraction structures and the ViT structure, the ViT structure can be pre-set to correspond to the other two local feature extraction structures in the three local feature extraction structures except the upper layer structure of the ViT structure, that is, the target local feature extraction structures corresponding to the ViT structure are the other two local feature extraction structures in the three local feature extraction structures except the upper layer structure of the ViT structure.
[0031] As an example of the present application, the number of ViT structures can be one or more. In the case where the number of ViT structures is multiple, the multiple ViT structures can correspond one to one with the multiple local feature extraction structures. For example, please refer to Figure 3, which is a structural diagram of another feature extraction model provided by an embodiment of the present application. The feature extraction model includes a first local feature extraction structure, a second local feature extraction structure, a first ViT structure, and a second ViT structure. The first ViT structure is pre-set to correspond to the first local feature extraction structure, and the second ViT structure is pre-set to correspond to the second local feature extraction structure, that is, the target local feature extraction structure corresponding to the first ViT structure is the first local feature extraction structure, and the target local feature extraction structure corresponding to the second ViT structure is the second local feature extraction structure.
[0032] It should be noted that Figures 2 and 3 above are merely examples and do not limit the number of local feature extraction structures and ViT structures, nor do they limit the correspondence between local feature extraction structures and ViT structures. For example, if there are multiple ViT structures, a ViT structure among the multiple ViT structures may correspond to multiple local feature extraction structures.
[0033] Among them, each local feature extraction structure in the multiple local feature extraction structures is used to extract features from the image to obtain shallow image features (or local features) that are sensitive to pixel position information, so that the ViT structure can use the shallow image features output by its corresponding target local feature extraction structure to enhance the sensitivity of highly abstract global features to pixel spatial position information, thereby expanding the applicability and application scenarios of the ViT structure.
[0034] The ViT structure is used to determine the first image feature based on the second image feature and the third image feature. The second image feature is the image feature output by the upper layer structure of the ViT structure. The third image feature is the image feature output by the target local feature extraction structure among multiple local feature extraction structures. The target local feature extraction structure includes one or more of the multiple local feature extraction structures except the upper layer structure of the ViT structure. The target local feature extraction structure corresponds to the ViT structure. The first image feature is a global feature, and the third image feature is a local feature that is sensitive to pixel spatial position information.
[0035] For example, there are multiple ViT structures, and the multiple ViT structures correspond one-to-one to the multiple local feature extraction structures. The input of each ViT structure in the multiple ViT structures includes the output of the upper layer structure of each ViT structure and the output of the target local feature extraction structure corresponding to each ViT structure. The target local feature extraction structure corresponding to each ViT structure is different from the upper layer structure of each ViT structure. In this case, the computer device can input the original image into the feature extraction model, and perform feature extraction through each local feature structure in the multiple local feature extraction structures in turn; then, feature extraction is performed through each ViT structure in the multiple ViT structures in turn, based on the feature image output by the upper layer structure of each ViT structure and the image features extracted by the target local feature extraction structure corresponding to each ViT structure, to determine the first image feature.
[0036] For example, as shown in FIG3 , there are two ViT structures, and the feature extraction model includes a first local feature extraction structure, a second local feature extraction structure, a first ViT structure, and a second ViT structure. The upper layer structure of the first ViT structure is the second local feature extraction structure, and the upper layer structure of the second ViT structure is the first ViT structure. The first ViT structure corresponds to the first local feature extraction structure, and the second ViT structure corresponds to the second local feature extraction structure, that is, the target local feature extraction structure corresponding to the first ViT structure is the first local feature extraction structure, and the target local feature extraction structure corresponding to the second ViT structure is the second local feature extraction structure. In this case, the first ViT structure is used to determine image features based on the image features output by the second local feature extraction structure and the image features output by the first local feature extraction structure, and the second ViT structure is used to determine the first image features based on the image features output by the first ViT structure and the image features output by the second local feature extraction structure.
[0037] For another example, the number of ViT structures is one, and the ViT structure corresponds to one or more of the multiple local feature extraction structures, that is, the target local feature extraction structure is one or more of the multiple local feature extraction structures corresponding to the ViT structure, and the input of the ViT structure includes the output of the upper layer structure of the ViT structure and the output of the target local feature extraction structure corresponding to the ViT structure. For example, as shown in Figure 2, the feature extraction model includes a first local feature extraction structure, a second local feature extraction structure and a ViT structure, the upper layer structure of the ViT structure is the second local feature extraction structure, and the ViT structure corresponds to the first local feature extraction structure, that is, the target local feature extraction structure corresponding to the ViT structure is the first local feature extraction structure. In this case, the ViT structure is used to determine the first image feature based on the image features output by the second local feature extraction structure and the image features output by the first local feature extraction structure.
[0038] As an example, the specific implementation of inputting the original image into the feature extraction model for feature extraction and outputting the first image feature of the original image may include the following two sub-steps:
[0039] A1. The computer device inputs the original image into the feature extraction model, and performs feature extraction through each local feature extraction structure in a plurality of local feature extraction structures in turn.
[0040] Among them, multiple local feature extraction structures are connected in series, and the multiple local feature extraction structures are used to extract features from the image to obtain corresponding local features. For example, as shown in Figure 2, the feature extraction model includes a first local feature extraction model, a second local feature extraction model, and a ViT model. After the original image is input into the feature extraction model, the first local feature extraction model is used to extract features from the original image to obtain a first local feature; then the second local feature extraction model is used to extract features from the first local feature extraction model output again to obtain a second local feature.
[0041] As an example, as shown in Figure 2, the resolution of the original image is H×W×3, and the network base width of the first local feature extraction structure and the second local feature extraction structure is C. After each local feature extraction structure, the resolution of the input image is reduced by four times, that is, every four pixel features of the input image are merged into one pixel feature through the local feature extraction structure, and the dimension of the pixel feature is doubled after each local feature extraction structure. In this case, the feature extraction model first extracts features from the original image through the first local feature extraction structure, and the output first local feature is [C, H / 2, W / 2]. The second local feature extraction model then extracts features from the first local feature, and the output second local feature is [2C, H / 4, W / 4]. Where H, W, and C are positive integers; H and W represent the height and width of the original image, respectively; [C, H / 2, W / 2] means that the first local feature includes (H×W) / 4 pixel features, and the dimension of each pixel feature is C; [2C, H / 4, W / 4] means that the second local feature includes (H×W) / 16 pixel features, and the dimension of each pixel feature is 2C.
[0042] As an example, the target local feature extraction structure among the multiple local feature extraction structures is a convolution structure. The characteristic of the convolution structure is that the extracted image features have a strong sensitivity to pixel spatial position information. In addition, the embodiments of the present application do not limit the relevant parameters of the convolution structure.
[0043] For example, the target local feature extraction structure is a convolution structure, and the convolution structure includes multiple convolution modules. In this case, the input of the ViT structure may include the image features output by the upper layer structure of the ViT structure, and the image features output by at least one convolution module in the target local feature extraction structure. For example, since the similarity of the image features output by similar convolution modules in the convolution structure is high, and the similarity between the first convolution module and the last convolution module is low, in order to reduce unnecessary redundant calculations, the input of the ViT structure may include the image features output by the upper layer structure of the ViT structure, and the image features output by the first convolution module and the image features output by the last convolution module in the target local feature extraction structure.
[0044] For example, please refer to Figure 4, which is a connection diagram of a convolution structure and a ViT structure provided in an embodiment of the present application. As shown in Figure 4, the convolution structure includes multiple convolution modules, and the ViT structure includes multiple ViT modules; the input of the first ViT module among the multiple ViT modules includes the image features output by the upper layer structure of the ViT structure, the image features output by the first convolution module, and the image features output by the last convolution module; the input of other ViT modules except the first ViT module among the multiple ViT modules includes the image features output by the previous module of the other ViT modules, the image features output by the first convolution module, and the image features output by the last convolution module. In this case, when feature extraction is performed based on the feature extraction model, there is an interactive process between the local features extracted by the local feature extraction structure (convolution structure) and the global features extracted by the ViT structure, so that the ViT structure can more effectively utilize the sensitivity of local features to pixel spatial position information, thereby making the global features extracted by the ViT model have strong pixel spatial position sensitivity.
[0045] For example, the number of pixel features included in the image features output by each convolution module in the multiple convolution modules is the same as the dimension of each pixel feature.
[0046] A2. The computer device determines the first image feature according to the second image feature and the third image feature through the ViT structure.
[0047] For example, as shown in Figure 2, the feature extraction model includes a first local feature extraction structure, a second local feature extraction structure and a ViT structure. The target local feature extraction structure is the first local feature extraction structure corresponding to the ViT structure. The second image feature is the local feature output by the second local feature extraction structure, and the third image feature is the local feature output by the first local feature extraction structure.
[0048] Among them, there is a pixel space position mapping relationship between each pixel feature in the second image feature and the Y pixel features in the third image feature. Since the essence of the local feature extraction structure and the ViT structure is to extract image features by successively reducing the image resolution, there is a pixel space position mapping relationship between the image features extracted by the local feature extraction structure and the ViT structure. For example, as shown in Figure 2, the resolution of the original image is H×W×3, and the network base width of the first local feature extraction structure, the second local feature extraction structure and the ViT structure is C. After each local feature extraction structure or ViT structure, the resolution of the input image is reduced by four times, that is, every four pixel features of the input image are merged into one pixel feature through the local feature extraction structure or the ViT structure. In this case, each pixel feature in the first local feature output by the first local feature extraction structure corresponds to 4 pixel features in the original image; each pixel feature in the second local feature output by the second local feature extraction structure corresponds to 4 pixel features in the first local feature, corresponding to 16 pixel features in the original image; each pixel in the first image feature output by the ViT structure corresponds to 4 pixels in the second local feature, corresponds to 16 pixels in the first local feature, and corresponds to 32 pixels in the original image.
[0049] As an example, a computer device may first perform bilinear interpolation on the third image feature to reduce the image resolution of the third image feature, and then perform feature fusion on each pixel feature in the third image feature after bilinear interpolation with the corresponding pixel feature in the second image feature to obtain the first image feature. The image resolutions of the third image feature, the second image feature, and the first image feature after bilinear interpolation are the same, and the corresponding pixel feature in the second image feature refers to the pixel feature in the second image feature that has the same pixel position as each pixel feature in the third image feature after bilinear interpolation. Because linear interpolation destroys relevant information about the pixel features in the third image feature, which may include information such as pixel spatial position and grayscale, although the third image feature is highly sensitive to pixel spatial position information, the third image feature after bilinear interpolation is less sensitive to pixel spatial position, and thus the obtained first image feature is also less sensitive to pixel spatial position.
[0050] As another example, by using the ViT structure, determining the first image feature according to the second image feature and the third image feature can also be achieved through the following two sub-steps:
[0051] B1. The computer device uses the ViT structure to fuse each pixel feature in the second image feature with Y pixel features in the third image feature according to the pixel space position mapping relationship to obtain a first attention matrix.
[0052] Among them, the first attention matrix is used to describe the degree of attention that the ViT structure pays to each pixel feature in the second image feature based on the third image feature.
[0053] Among them, feature fusion refers to fusing (combining) the relevant information of Y pixel features in the third image feature into the corresponding 1 pixel feature in the second image feature. Since the third image feature is a local feature that is sensitive to the pixel spatial position information, the pixel features in the second image feature indicated by the first attention matrix obtained after fusion also have strong pixel spatial position sensitivity, that is, the first attention matrix is used to describe the degree of attention of the ViT structure to each corresponding pixel feature in the second image feature based on the corresponding Y pixel features in the third image feature, and the pixel features in the second image feature described by the first attention matrix have strong pixel spatial position sensitivity.
[0054] As an example, a computer device can use the ViT structure to first determine a query vector group based on the second image feature; then determine a first key vector group based on the third image feature; then, based on the pixel spatial position mapping relationship, align each pixel feature included in the query vector group with the Y pixel features included in the first key vector group; finally, perform feature fusion based on the aligned query vector group and the aligned first key vector group to obtain a first attention matrix. The query vector group is used to indicate each pixel feature to be queried in the second image feature, and the first key vector group is used to indicate relevant information of each pixel feature in the third image feature. The relevant information may include information such as pixel spatial position and grayscale.
[0055] For example, based on the second image feature, the query vector group can be determined by the following formula (1):
[0056] Among them, q i-1 is the query vector group, W q is the learnable parameter of the feature extraction model, x i-1 is the second image feature.
[0057] For example, according to the third image feature, the first key position vector group can be determined by the following formula (2):
[0058] Among them, k skip is the first key vector group, is the learnable parameter of the feature extraction model, s xkip is the third image feature.
[0059] For example, the computer device first rearranges all pixel features included in the first key vector group in units of Y pixel features through the ViT structure, and then adjusts the vector dimension of the query vector group based on the vector dimension of the rearranged first key vector group. The rearranged first key vector group includes Z pixel-unit features, each of the Z pixel-unit features includes Y pixel features, where Z is a positive integer, and the vector dimension of the adjusted query vector group is the same as the vector dimension of the rearranged first key vector group.
[0060] As an example, as shown in Figure 2, the resolution of the original image is H×W×3, the network base width of the first local feature extraction structure, the second local feature extraction structure and the ViT structure is C, the resolution of the input image is reduced by four times after each local feature extraction structure or ViT structure, the dimension of the pixel feature is doubled after each local feature extraction structure or ViT structure, and the dimension of the attention head when the ViT structure adopts the self-attention mechanism is d, the second image feature x i-1 The second local feature output by the second local feature extraction structure, the second image feature x i-1 The vector dimension is [2C, H×W / 4]; the third image feature x skip The first local feature is the output of the first local feature extraction structure, and the third image feature x skip The vector dimension is [C, H×W / 4]. In this case, the second image feature x i-1 A pixel feature in corresponds to the third image feature x skip The 4 (Y is equal to 4) pixel features in the second image are used to determine the query vector group q i-1 The vector dimension is [2C, H×W / 16], [2C, H×W / 16] refers to the query vector group q i-1 It includes H×W / 16 pixel features, and the dimension of each pixel feature is 2C; the first key vector group k determined according to the third image feature skip The vector dimension is [C, H×W / 4], [C, H×W / 4] refers to the first key vector group k skip It includes H×W / 4 pixel features, and the dimension of each pixel feature is C.
[0061] Based on this, the computer device uses the ViT structure to first take 4 pixel features as units and divide the first key vector group k according to the attention head dimension d skip(the vector dimension is [C, H×W / 4]) and the H×W / 4 pixel features included are rearranged, and the vector dimension of the rearranged first key vector group is [H×W / 16, 4, C / d, d]; then, according to the vector dimension of the rearranged first key vector group [H×W / 16, 4, C / d, d] and the attention head dimension d, the query vector group q is i-1 The vector dimension of the query vector group (the vector dimension is [2C, H×W / 16)) is adjusted, and the vector dimension of the adjusted query vector group is [H×W / 16, 1, 2C / d, d].
[0062] Among them, [H×W / 16, 4, C / d, d] means that the rearranged first key vector group includes H×W / 16 (Z is equal to H×W / 16) pixel unit features, each pixel unit feature includes 4 pixel features, and includes C / d attention head dimensions d; [H×W / 16, 4, C / d, d] means that the adjusted query vector group includes H×W / 16 pixel features and includes 2C / d attention head dimensions d; and each pixel feature included in the adjusted query vector group corresponds one-to-one to each pixel unit feature included in the rearranged first key vector group, that is, each pixel feature included in the adjusted query vector group has a pixel space mapping relationship with the pixel unit feature included in the rearranged first key vector group.
[0063] For example, the computer device determines the first attention matrix based on the adjusted query vector group and the rearranged first key vector group. For example, the computer device can determine the first attention matrix based on the adjusted query vector group and the rearranged first key vector group using the following formula (3): attn skip =q″ i-1 @k″ skip .transpose(-1, -2) (3)
[0064] Among them, attn skip is the first attention matrix, q″ i-1 is the adjusted query vector group, @ indicates matrix multiplication of the last two dimensions of the matrix, k″ skip is the first key vector group after rearrangement, and transpose(-1,-2) represents the transposition operation on the last two dimensions of the matrix.
[0065] As an example, the first key vector group (k″) after rearrangement skip ) has a vector dimension of [H×W / 16, 4, C / d, d], and the adjusted query vector group (q″ i-1) has a vector dimension of [H×W / 16, 4, C / d, d], the computer device first performs a two-dimensional and three-dimensional merger on the rearranged first key vector group to obtain a merged first key vector group, the vector dimension of the merged first key vector group is [H×W / 16, 4, C / d, d], according to the merged first key vector group (vector dimension is [H×W / 16, 4, C / d, d]) and the adjusted query vector group (vector dimension is [H×W / 16, 4, C / d, d]), the first attention matrix attn is obtained by the above formula (3) skip The vector dimensions are [H×W / 16, 1, 2C / d, 4C / d].
[0066] Based on this, the first attention matrix is used to describe the degree of attention of the ViT structure to the corresponding each pixel feature in the second image feature (adjusted query vector group) based on each pixel unit feature (Y pixel features) in the third image feature (rearranged first key vector group), and based on the sensitivity of each pixel unit feature (Y pixel features) in the third image feature (rearranged first key vector group) to the pixel spatial position information, each pixel feature in the second image feature described by the first attention matrix has a strong pixel spatial position sensitivity, that is, the second image feature described by the first attention matrix has a strong pixel spatial position sensitivity.
[0067] B2. The computer device determines the first image feature through the ViT structure according to the first attention matrix, the second image feature and the third image feature.
[0068] Since the first attention matrix is used to describe the degree of attention of the ViT structure to the corresponding each pixel feature in the second image feature based on each pixel unit feature in the third image feature, and the second image feature described by the first attention matrix has a strong pixel space position sensitivity, the first image feature determined according to the first attention matrix also has a strong pixel space position sensitivity, that is, the ViT structure in the embodiment of the present application can use local features that are sensitive to position information to enhance the sensitivity of highly abstract global features to position information.
[0069] As an example, a computer device can use the ViT structure to first determine a first output feature vector group based on the third image feature; then determine a second key vector group and a second output feature vector group based on the second image feature; then determine a second attention matrix based on the query vector group and the second key vector group, and finally determine the first image feature based on the first attention matrix, the second attention matrix, the first output feature vector group and the second output feature vector group. The second key vector group is used to indicate relevant information about each pixel feature in the second image feature, and the relevant information may include information such as pixel spatial position and grayscale. The second attention matrix is used to describe the degree of attention that the ViT structure pays to each pixel feature in the second image feature based on the second image feature.
[0070] For example, based on the third image feature, the first output feature vector group can be determined by the following formula (4):
[0071] Among them, v skip is the first key vector group, is the learnable parameter of the feature extraction model, x skip is the third image feature.
[0072] For example, according to the second image feature, the second key position vector group and the second output feature vector group can be determined by the following formula (5):
[0073] Among them, k i-1 is the second key vector group, W k is the learnable parameter of the feature extraction model, x i-1 is the second image feature, v i-1 is the second output feature vector group, W v are the learnable parameters of the feature extraction model.
[0074] For example, based on the query vector group and the second key vector group, the second attention matrix can be determined by the following formula (6): i-1 =q i-1 @k i-1 .transpose(-1, -2) (6)
[0075] Among them, attn i-1 is the second attention matrix, q i-1 To query the vector group, @ indicates matrix multiplication of the last two dimensions of the matrix, k i-1 is the second key vector group, and transpose(-1,-2) means transposing the last two dimensions of the matrix.
[0076] As an example, as shown in Figure 2, the resolution of the original image is H×W×3, the network base width of the first local feature extraction structure, the second local feature extraction structure and the ViT structure is C, the resolution of the input image is reduced by four times after each local feature extraction structure or ViT structure, the dimension of the pixel feature is doubled after each local feature extraction structure or ViT structure, and the dimension of the attention head when the ViT structure adopts the self-attention mechanism is d, the second image feature x i-1 is [2C, H / 4, W / 4], the third image feature x skip = [C, H / 2, W / 2]. In this case, according to the third image feature x skip Determine the first output feature vector group v skip The vector dimension is [C, H×W / 4]; according to the second image feature x i-1 Determine the second key vector group k i-1 and the second output feature vector group v i-1 The vector dimensions are all [2C, H×W / 16]; then according to the attention head dimension d, the first key vector group k skip , the first output feature vector group x skip , query vector group q i-1 , the second key vector group k i-1 and the second output feature vector group v i-1 After rearrangement, the vector dimensions of the rearranged first key vector group and the rearranged first output feature vector group are both [C / d, H×W / 4, d], and the vector dimensions of the rearranged query vector group, the rearranged second key vector group and the rearranged second output feature vector group are all [2C, H×W / 16, d]. Finally, according to the rearranged query vector group and the rearranged second key vector group, the second attention matrix attn is determined by the above formula (6): i-1 The vector dimensions are [2C / d, H×W / 16, H×W / 16].
[0077] As an example, according to the first attention matrix attn skip , the second attention matrix attn i-1 , the first output feature vector group v skip and the second output feature vector group v i-1 , determine the first image feature v i , including: according to the first attention matrix attn skip and the second attention matrix attn i-1 , determine the normalized third attention matrix attn combin ; Then normalize the third attention matrix attn combinDecompose and get the fourth attention matrix attn′ skip and the fifth attention matrix attn′ i-1 ; Then according to the fourth attention matrix attn′ skip and the rearranged first output feature vector group, determine the third output feature vector group; then according to the fifth attention matrix attn′ i-1 and the rearranged second output feature vector group, determine the fourth output feature vector group; finally, determine the first image feature v according to the third output feature vector group and the fourth output feature vector group i .
[0078] For example, first based on the second attention matrix attn i-1 The vector dimension is [2C / d, H×W / 16, H×W / 16], for the first attention matrix attn skip The vector dimension of the first attention matrix is adjusted to [H×W / 16, 1, 2C / d, 4C / d]. The vector dimension of the adjusted first attention matrix is [2C / d, H×W / 16, 4C / d]. Then, according to the second attention matrix attn i-1 And the adjusted first attention matrix, determine the third attention matrix attn combin For example, according to the second attention matrix attn i-1 And the adjusted first attention matrix, the third attention matrix attn can be determined by the following formula (7) combin : attn combin =soft max(cat(attn i-1 ,attn″ skip ,dim=-1),dim=-1) (7)
[0079] Among them, attn combin is the third attention matrix, Softmax(,dim=-1) means normalizing the merged attention matrix along the last dimension, attn i-1 is the second attention matrix, attn″ skip is the adjusted first attention matrix, cat(attn i-1 ,attn″ skip , dim=-1) means attn i-1 and attn″ skip Merge by the last dimension.
[0080] For example, the third attention matrix attn can be calculated by the following formula (8): combin Decompose and get the fourth attention matrix attn′ skip and the fifth attention matrix attn′i-1 : attn′ i-1 attn′ skip =chunk(attn combin , [H×W / 16, 4C / d], dim=-1) (8)
[0081] Among them, attn′ skip is the fourth attention matrix, attn′ i-1 is the fifth attention matrix, attn combin is the third attention matrix, chunk(attn combin , [H×W / 16, 4C / d], dim=-1) means attn combin Decompose it into H×W / 16 and 4C / d dimensions. Among them, the fourth attention matrix attn′ skip The vector dimension is The fifth attention matrix attn′ i-1 The vector dimensions are [2C / d, H×W / 16, H×W / 16].
[0082] For example, according to the fourth attention matrix attn′ skip And the rearranged first output feature vector group can be used to determine the third output feature vector group by the following formula (9):
[0083] v″ skip =attn″ skip @v′ skip (9)
[0084] Among them, v″ skip is the third output feature vector group, the third output feature vector group v″ skip The vector dimension is [2C / d, H×W / 16, d], attn′ skip is the fourth attention matrix, @ represents matrix multiplication of the last two dimensions of the matrix, v′ skip is the first output feature vector group after rearrangement.
[0085] For example, according to the fifth attention matrix attn′ i-1 And the rearranged second output feature vector group, the fourth output feature vector group can be determined by the following formula (10): i-1 =attn′ i-1 @v′ i-1 (10)
[0086] Among them, v″ i-1 is the fourth output feature vector group, the fourth output feature vector group v″ i-1The vector dimension is [2C / d, H×W / 16, d], attn′ i-1 is the fifth attention matrix, @ represents matrix multiplication of the last two dimensions of the matrix, v′ i-1 is the second output feature vector group after rearrangement.
[0087] For example, according to the third output feature vector group and the fourth output feature vector group, the first image feature can be determined by the following formula (11): i-1 +W pose (v″ i-1 +v″ skip ) T ) x i =x′ i FFN(x′ i ) (11)
[0088] Among them, x i-1 is the second image feature, W pose is the learnable parameter of the feature extraction model, v″ i-1 is the fourth output feature vector group, v″ skip is the third output feature vector group, x i It is the first image feature, and FFN() is a commonly used structure filled with zeros in the ViT structure.
[0089] As an example, when the input of the ViT structure includes the image features output by the upper layer structure of the ViT structure, and the image features output by the first convolution module and the image features output by the last convolution module in the target local feature extraction structure, before the above-mentioned step A2, the computer device can determine the third image feature through the ViT structure based on the image features output by the first convolution module and the image features output by the last convolution module in the target local feature extraction structure.
[0090] For example, as shown in Figure 4, the vector dimensions of the image features output by the first convolution module and the last convolution module in the target local feature extraction structure are both [C, H×W / 4]. According to the image features output by the first convolution module and the image features output by the last convolution module in the target local feature extraction structure, the vector dimension of the third image feature is determined to be [C, H×W / 2].
[0091] As an example, the above steps B1-B2 can also be performed interactively. For example, the computer device can determine the first key vector group and the first output feature vector group based on the third image feature through the ViT structure; then determine the query vector group, the second key vector group and the second output feature vector group based on the second image feature; then, according to the pixel space position mapping relationship, align each pixel feature included in the query vector group with the Y pixel features included in the first key vector group, perform feature fusion based on the aligned query vector group and the aligned first key vector group, and obtain a first attention matrix; then, determine the second attention matrix based on the query vector group and the second key vector group; finally, determine the first image feature based on the first attention matrix, the second attention matrix, the first output feature vector group and the second output feature vector group.
[0092] It should be noted that the above-mentioned step A2 is only discussed using the example that the feature extraction model includes one ViT structure. When the feature extraction model includes multiple ViT structures, the specific implementation of "passing through each ViT structure in turn, based on the feature image output by the upper layer structure of each ViT structure, and the image features extracted by the local feature extraction structure corresponding to each ViT structure, to determine the first image feature" is similar to the above-mentioned step A2. That is, those skilled in the art can infer the specific implementation of "passing through each ViT structure in turn, based on the feature image output by the upper layer structure of each ViT structure, and the image features extracted by the local feature extraction structure corresponding to each ViT structure, to determine the first image feature" through the above-mentioned step A2, which will not be repeated here.
[0093] As an example, as shown in Figures 2 and 3, the feature extraction model also includes a preprocessing structure. Before performing feature extraction in sequence through each of the multiple local feature extraction structures, the computer device also processes the original image through the preprocessing structure, and then uses the processed original image as the input of the first local feature extraction structure in the multiple local feature extraction structures, and performs feature extraction in sequence through each of the multiple local feature extraction structures. Among them, preprocessing is used to eliminate irrelevant information in the original image, restore useful real information, enhance the detectability of relevant information and simplify data to the maximum extent. Preprocessing can include grayscale, geometric transformation and image enhancement and other processing. Preprocessing can change the resolution of the original image or not. The embodiment of the present application does not limit the specific implementation of preprocessing.
[0094] It is worth mentioning that since the third image feature is a local feature that is sensitive to pixel spatial position information, the first image feature determined by the ViT structure based on the third image feature also has a strong sensitivity to pixel spatial position, that is, the ViT structure uses the local feature that is sensitive to pixel spatial position information to enhance the sensitivity of the highly abstract global feature to pixel spatial position information. This can improve the applicability of the ViT structure and expand the application scenarios of the ViT structure, that is, expand the application scenarios of the feature extraction model. For example, through the above steps 101-102, a first image feature (global feature) that is highly sensitive to pixel spatial position can be obtained, and then when the target detection task is implemented based on the obtained first image feature that is highly sensitive to pixel spatial position, the accuracy of determining the pixel spatial position of the target in the image is high, and the precision of target detection is high. In addition, since the ViT structure uses the shallow image features output by the local feature extraction structure, no additional feature extraction steps and additional overhead are introduced in the process of enhancing the sensitivity of the highly abstract global feature to pixel spatial position information through the ViT structure.
[0095] As an example, before the above step 102, the computer device may also train the feature extraction model to be trained to obtain a feature extraction model. For example, the training is divided into two stages. In the first training stage, the feature extraction model to be trained is coarsely trained. Specifically, training data is obtained based on the first target task, and the training data is used as the input of the feature extraction model to be trained. The gradient descent algorithm is used to coarsely train the untrained learnable parameters in the target loss function to obtain a coarsely trained feature extraction model. The coarsely trained feature extraction model includes coarsely trained learnable parameters. Then, in the second training stage, the coarsely trained feature extraction model is finely trained. Specifically, sample images and sample annotation information are obtained based on the second target task. The sample images and sample annotation information are used as the input of the coarsely trained feature extraction model. The gradient descent algorithm is used to finely train the coarsely trained learnable parameters in the target loss function to obtain a feature extraction model. The feature extraction model includes finely trained learnable parameters, for example, the learnable parameters in the above formulas (1) to (11).
[0096] Among them, the first target task can be a target detection task, a target classification task or a target segmentation task, and the second target task can be a target detection task, a target classification task or a target segmentation task. The first target task and the second target task can be the same or different.
[0097] The training data is the dataset corresponding to the first target task. For example, if the first target task is object detection, the training data can be datasets such as COCO or PASCAL. If the first target task is object classification, the training data can be datasets such as ImageNet-1K or ImageNet-22K. If the first target task is object segmentation, the training data can be datasets such as ADE20K or SAM-1B.
[0098] Sample images are multiple images collected in the feature extraction model application scenario. For example, in a street scene, the sample images are multiple images captured by a surveillance camera that include objects to be detected, classified, or segmented. The sample annotation information is the information obtained by manually or computer-generated annotation of the objects to be detected, classified, or segmented in the sample images.
[0099] As an example, after step 102 above, the computer device may also apply the image features obtained by the feature extraction model to target detection tasks, target classification tasks, or target segmentation tasks. For example, the computer device may perform image processing based on the extracted first image features, or perform image processing based on the image features output by multiple local feature extraction structures and the first image features, or perform image processing based on the image features output by multiple local feature extraction structures and the image features output by the ViT structure, to obtain corresponding image processing results, thereby achieving the corresponding tasks. The image processing may be target detection, target classification, or target segmentation, and the image features output by the ViT structure include the first image features.
[0100] For example, the image processing is target detection, and the computer device can perform target detection based on the image features output by multiple local feature extraction structures and the first image features. The corresponding image processing results obtained are the size and position of the target and the coverage frame of the target that may exist at each pixel position included in each image feature, thereby achieving the target detection task. Since the first image feature is highly sensitive to the pixel spatial position information, when the target detection task is achieved based on the first image feature, the accuracy of determining the pixel spatial position of the target in the image is high, and the precision of target detection is high. Among them, the method for performing target detection can be a method such as Two-stage, SPP-net or Faster R-CNN based on a neural network. The embodiment of the present application does not limit the method for performing target detection.
[0101] For example, in a street scene, the image features output by multiple local feature extraction structures and the first image features include pixel features of targets such as pedestrians or vehicles. After the computer device performs target detection based on the image features output by multiple local feature extraction structures and the first image features, the image processing results obtained are the pixel features of pedestrians or vehicles that may exist in each image feature, and the size and position of the coverage box of the pixel features of pedestrians or vehicles in each image feature, thereby realizing the detection of vehicles or pedestrians.
[0102] For example, please refer to Figure 5, which is a structural diagram of a target detection task provided by an embodiment of the present application. As shown in Figure 5, the original image includes a target to be detected, and the feature extraction model includes a first local feature structure, a second local feature extraction structure and a ViT structure. Through the above steps 101-102, the third image feature output by the first local feature structure, the second image feature output by the second local feature extraction structure and the first image feature output by the ViT structure can be obtained. Then, target detection is performed on the third image feature, the second image feature and the first image feature respectively, and the target that may exist at each pixel position included in each image feature and the size and position of the target's coverage frame are obtained to achieve the target detection task. That is, as shown in Figure 5, each image feature of different scales can be used for target detection. For example, the corresponding low-resolution image features are used to detect large targets, and the high-resolution image features are used to detect small targets. In this way, the accuracy of the detection of the size and position of the target and the target's coverage frame can be further improved, and the accuracy of target detection can be further improved.
[0103] For example, after performing target detection, the detection results can be further processed to further improve target detection accuracy. For example, since detection results at different scales may be repeated or overlapped, a non-maximum suppression strategy can be used to perform non-maximum suppression based on the confidence level of the predicted target, filtering out some duplicate predictions and further improving target detection accuracy.
[0104] In an embodiment of the present application, the original image of the feature to be extracted is first obtained, and then the original image is input into the feature extraction model for feature extraction, and the first image feature of the original image is output. Among them, the feature extraction model includes a plurality of local feature extraction structures and a ViT structure connected in sequence, and the ViT structure is used to determine the first image feature based on the second image feature and the third image feature. The second image feature is the image feature output by the upper layer structure of the ViT structure, and the third image feature is the image feature output by the target local feature extraction structure among the multiple local feature extraction structures. The target local feature extraction structure includes one or more of the multiple local feature extraction structures except the upper layer structure of the ViT structure. The third image feature is a local feature sensitive to pixel spatial position information. Since the third image feature is a local feature sensitive to pixel spatial position information, the first image feature determined by the ViT structure based on the third image feature also has a strong pixel spatial position sensitivity, that is, the ViT structure uses the local feature sensitive to pixel spatial position information to enhance the sensitivity of the highly abstract global feature to the spatial position, which can improve the applicability of the ViT structure and expand the application scenario of the ViT structure, that is, expand the application scenario of the feature extraction model.
[0105] Please refer to Figure 6, which is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. As shown in Figure 6, the computer device includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps of the image feature extraction method in the above embodiment are implemented.
[0106] The computer device may be the computer device in the embodiment of FIG1 above. The computer device may be a near-eye display device, or a desktop computer, a portable computer, a network server, a PDA, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiment of the present application does not limit the type of computer device. It will be understood by those skilled in the art that FIG6 is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, etc.
[0107] The processor 601 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0108] In some embodiments, the memory 602 may be an on-chip memory or an off-chip memory of a computer device, such as a cache memory, SRAM (Static Random-Access Memory), DRAM (Dynamic Static Random-Access Memory), or a floppy disk. In other embodiments, the memory 602 may be a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like equipped on the computer device. Furthermore, the memory 602 may include both an on-chip memory of the computer device, an off-chip memory internal storage unit, and an external storage device. The memory 602 is used to store an operating system, application programs, a boot loader, data, and other programs. The memory 602 may also be used to temporarily store data that has been output or is about to be output.
[0109] An embodiment of the present application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.
[0110] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0111] An embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the steps in the above-mentioned various method embodiments.
[0112] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to the camera / terminal device, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, a non-transitory storage medium.
Claims
1. A method for extracting image features, characterized in that: The method comprises: Obtain the original image of the feature to be extracted; Inputting the original image into a feature extraction model to perform feature extraction, and outputting a first image feature of the original image; Among them, the feature extraction model includes multiple local feature extraction structures and a ViT structure connected in sequence; the ViT structure is used to determine the first image feature based on the second image feature and the third image feature, the second image feature is the image feature output by the upper layer structure of the ViT structure, the third image feature is the image feature output by the target local feature extraction structure among the multiple local feature extraction structures, the target local feature extraction structure includes one or more of the multiple local feature extraction structures except the upper layer structure of the ViT structure, and the third image feature is a local feature sensitive to pixel spatial position information.
2. The method according to claim 1, characterized in that The step of inputting the original image into a feature extraction model to extract features and outputting a first image feature of the original image includes: Inputting the original image into the feature extraction model, and performing feature extraction through each of the multiple local feature extraction structures in turn; The first image feature is determined according to the second image feature and the third image feature through the ViT structure.
3. The method according to claim 2, characterized in that There is a pixel space position mapping relationship between each pixel feature in the second image feature and Y pixel features in the third image feature, where Y is a positive integer; The determining the first image feature according to the second image feature and the third image feature through the ViT structure includes: Through the ViT structure, according to the pixel spatial position mapping relationship, each pixel feature in the second image feature is respectively fused with Y pixel features in the third image feature to obtain a first attention matrix, where the first attention matrix is used to describe the degree of attention of the ViT structure to each pixel feature in the second image feature based on the third image feature; Through the ViT structure, the first image feature is determined according to the first attention matrix, the second image feature and the third image feature.
4. The method according to claim 3, characterized in that The method of using the ViT structure to fuse each pixel feature in the second image feature with Y pixel features in the third image feature according to the pixel spatial position mapping relationship to obtain a first attention matrix includes: Determine a query vector group according to the second image feature through the ViT structure, wherein the query vector group is used to indicate each pixel feature to be queried in the second image feature; Determine, by means of the ViT structure, a first key position vector group according to the third image feature, wherein the first key position vector group is used to indicate relevant information of each pixel feature in the third image feature; By using the ViT structure, according to the pixel spatial position mapping relationship, each pixel feature included in the query vector group is aligned with the Y pixel features included in the first key position vector group; Through the ViT structure, feature fusion is performed according to the aligned query vector group and the aligned first key position vector group to obtain the first attention matrix.
5. The method according to claim 4, characterized in that The step of aligning each pixel feature included in the query vector group with the Y pixel features included in the first key vector group respectively according to the pixel spatial position mapping relationship through the ViT structure includes: By using the ViT structure, all pixel features included in the first key position vector group are rearranged in units of Y pixel features, and the rearranged first key position vector group includes Z pixel unit features, each of the Z pixel unit features includes Y pixel features, and Z is a positive integer; Through the ViT structure, the vector dimension of the query vector group is adjusted according to the vector dimension of the rearranged first key vector group, and the vector dimension of the adjusted query vector group is the same as the vector dimension of the rearranged first key vector group.
6. The method according to claim 4, characterized in that The determining the first image feature according to the first attention matrix, the second image feature and the third image feature through the ViT structure includes: Determine a first output feature vector group according to the third image feature through the ViT structure; Determine, by means of the ViT structure, a second key vector group and a second output feature vector group according to the second image feature, wherein the second key vector group is used to indicate relevant information of each pixel feature in the second image feature; By using the ViT structure, according to the query vector group and the second key vector group, determine A second attention matrix, where the second attention matrix is used to describe the degree of attention paid by the ViT structure to each pixel feature in the second image feature based on the second image feature; Through the ViT structure, the first image feature is determined according to the first attention matrix, the second attention matrix, the first output feature vector group and the second output feature vector group.
7. The method according to any one of claims 1 to 6, characterized in that: There are multiple ViT structures, and the multiple ViT structures correspond to the multiple local feature extraction structures one by one; the input of each ViT structure in the multiple ViT structures includes the output of the upper layer structure of each ViT structure and the output of the local feature extraction structure corresponding to each ViT structure, and the local feature extraction structure corresponding to each ViT structure is different from the upper layer structure of each ViT structure; The step of inputting the original image into a feature extraction model to extract features and outputting a first image feature of the original image includes: Inputting the original image into the feature extraction model, and performing feature extraction through each of the multiple local feature extraction structures in turn; Feature extraction is performed in turn through each of the multiple ViT structures based on the feature image output by the upper layer structure of each ViT structure and the image features extracted by the local feature extraction structure corresponding to each ViT structure to determine the first image feature.
8. The method according to any one of claims 1 to 6, characterized in that: The target local feature extraction structure among the multiple local feature extraction structures is a convolution structure.
9. The method according to claim 8, characterized in that The convolution structure includes multiple convolution modules, and the input of the ViT structure includes the image features output by the upper layer structure of the ViT structure, and the image features output by the first convolution module and the image features output by the last convolution module in the target local feature extraction structure.
10. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by the processor.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.
Citation Information
Patent Citations
Image processing method and device and medium
CN115661609A
CNN-Transform-based parallel fusion method
CN116051945A
Remote sensing image semantic segmentation method combining Unet and Transform
CN116091929A
Expression recognition method based on local features and global features
CN116645716A
Display device and method for manufacturing the same
KR1020240149484A
