Mixed image feature extraction method and device

By acquiring and splicing multiple image features and aggregating attention model, we can generate mixed image features with higher adaptability and mobility, and solve the problem of singleness and poor adaptability of feature extraction in the prior art, and improve computer vision Efficiency and effectiveness of tasks.

CN118918340APending Publication Date: 2024-11-08GUANGZHOU AUTOMOBILE GROUP CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310505848.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing image feature extraction method is single, resulting in the lack of diversity in the extracted features, poor adaptability, inability to effectively migrate to other tasks, and inefficient model training and inference.

Method used

The hybrid image feature extraction method is adopted to obtain unsupervised features, convolutional features, multi-layer perceptron features and self-attention features, and splice them in the channel dimension, and use the attention model to aggregate the initial mixed features to generate more diverse ones. mixed image features of sex.

Benefits of technology

It improves the adaptability and migration of image features, simplifies the network structure design of downstream tasks, improves the training and inference speed of model, and achieves better computer vision tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918340A_ABST
    Figure CN118918340A_ABST
Patent Text Reader

Abstract

The invention discloses a mixed image feature extraction method and device, and the method comprises the steps: obtaining unsupervised features of a target image, splicing the obtained unsupervised features of the target image in a channel dimension, and obtaining an unsupervised feature map; respectively obtaining a convolution feature, a multi-layer perceptron feature and a self-attention feature of the target image to obtain a convolution feature map, a multi-layer perceptron feature map and a self-attention feature map; splicing the unsupervised feature map, the convolutional feature map, the multi-layer perceptron feature map and the self-attention feature map in a channel dimension to obtain an initial mixed image feature map; and aggregating the initial mixed image feature map by using an attention model to obtain a final mixed image feature map. According to the method, the image features are extracted in multiple modes, and the image features extracted in different modes are aggregated by using an attention mechanism to obtain the mixed image features, so that the method has better adaptability and mobility for various types of downstream computer vision tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a hybrid image feature extraction method and device. Background Art

[0002] In recent years, with the increase in computing power and the continuous development of deep learning, computer vision technology has also made great progress. In almost all computer vision tasks, such as recognition and classification, target detection, image segmentation, representation learning, image and video generation, and the combination of vision and text, there is a necessary process: feature extraction. Among the common feature extraction methods, most of them are based on a single method to extract features. For example, in a deep convolutional neural network, the model only relies on the convolution layer to extract the deep convolution features of the image. In a deep multi-layer perceptron network, the model only relies on the fully connected layer to extract the multi-layer perceptron MLP (Multilayer Perceptron) features of the image.

[0003] However, the above-mentioned image feature extraction methods often only focus on the effects of specific tasks in specific fields, and the extracted image features lack diversity. For example, the traditional image feature extraction operator Histogram of Oriented Gradient HOG (Histogram of Oriented Gradient) only focuses on the image's directional gradient histogram, and the deep convolutional neural network only uses the convolution module to extract the image's convolution features. The image features extracted in this way are mostly only applicable to the current task, and have poor adaptability to other task types outside of this task, and cannot be transferred and used. In addition, the image features obtained in a single form require a more complex network structure to learn these feature expressions, which leads to low efficiency in model training and reasoning. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a method and device for extracting mixed image features, so that the obtained mixed image features have better adaptability and migration.

[0005] In order to solve the above technical problems, the present invention provides a hybrid image feature extraction method, comprising:

[0006] Obtain unsupervised features of the target image, and concatenate the obtained unsupervised features of the target image in the channel dimension to obtain an unsupervised feature map;

[0007] Respectively acquiring convolution features, multi-layer perceptron features, and self-attention features of the target image to obtain a convolution feature map, a multi-layer perceptron feature map, and a self-attention feature map;

[0008] The unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are concatenated in the channel dimension to obtain an initial mixed image feature map;

[0009] The initial mixed image feature map is aggregated using an attention model to obtain a final mixed image feature map.

[0010] Preferably, the step of acquiring the unsupervised features of the target image and splicing the acquired unsupervised features of the target image in the channel dimension to obtain an unsupervised feature map specifically includes:

[0011] Respectively obtain the local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image;

[0012] The acquired local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image are spliced ​​in the channel dimension to obtain the unsupervised feature map.

[0013] Preferably, obtaining the convolutional features of the target image specifically includes: selecting a deep convolutional neural network, inputting the target image into the deep convolutional neural network, extracting the convolutional features of the target image through multiple convolutional modules during forward propagation, and obtaining a convolutional feature map;

[0014] Acquiring the multi-layer perceptron features of the target image specifically includes: selecting a deep multi-layer perceptron neural network, inputting the target image into the deep multi-layer perceptron neural network, acquiring the multi-layer perceptron features of the target image through multiple multi-layer perceptron modules during forward propagation, and obtaining a multi-layer perceptron feature map;

[0015] Acquiring the self-attention features of the target image specifically includes: selecting a self-attention neural network model, inputting the target image into the self-attention neural network model, extracting the self-attention features of the target image through multiple self-attention modules during the forward propagation process, and obtaining a self-attention feature map.

[0016] Preferably, the initial mixed image feature map, the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are all of equal height and width.

[0017] Preferably, the using of the attention model to aggregate the initial mixed image feature map to obtain the final mixed image feature map specifically includes:

[0018] The channel attention SE structure is adopted to perform global average pooling on the initial mixed image feature map in the channel dimension, and the channel attention weight mask is obtained through channel feature learning to output the final mixed image feature map.

[0019] The embodiment of the present invention further provides a mixed image feature extraction device, comprising:

[0020] A first acquisition module is used to acquire unsupervised features of a target image, and to concatenate the acquired unsupervised features of the target image in a channel dimension to obtain an unsupervised feature map;

[0021] A second acquisition module is used to respectively acquire the convolution features, multi-layer perceptron features and self-attention features of the target image to obtain a convolution feature map, a multi-layer perceptron feature map and a self-attention feature map;

[0022] A third acquisition module is used to splice the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map in the channel dimension to obtain an initial mixed image feature map;

[0023] The output module is used to aggregate the initial mixed image feature map using an attention model to obtain a final mixed image feature map.

[0024] Preferably, the first acquisition module is specifically used for:

[0025] Respectively obtain the local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image;

[0026] The acquired local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image are spliced ​​in the channel dimension to obtain the unsupervised feature map.

[0027] Preferably, the second acquisition module acquires the convolutional features of the target image specifically comprising: selecting a deep convolutional neural network, inputting the target image into the deep convolutional neural network, extracting the convolutional features of the target image through multiple convolutional modules during forward propagation, and obtaining a convolutional feature map;

[0028] The second acquisition module acquires the multi-layer perceptron features of the target image specifically comprising: selecting a deep multi-layer perceptron neural network, inputting the target image into the deep multi-layer perceptron neural network, acquiring the multi-layer perceptron features of the target image through multiple multi-layer perceptron modules in a forward propagation process, and obtaining a multi-layer perceptron feature map;

[0029] The second acquisition module acquires the self-attention features of the target image specifically by: selecting a self-attention neural network model, inputting the target image into the self-attention neural network model, extracting the self-attention features of the target image through multiple self-attention modules during the forward propagation process, and obtaining a self-attention feature map.

[0030] Preferably, the initial mixed image feature map, the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are all of equal height and width.

[0031] Preferably, the output module is specifically used to: adopt a channel attention SE structure, perform global average pooling on the initial mixed image feature map in the channel dimension, obtain a channel attention weight mask through channel feature learning, and output the final mixed image feature map.

[0032] The implementation of the present invention has the following beneficial effects: compared with a single form of feature extraction, the present invention extracts image features by using multiple methods, and uses an attention mechanism to aggregate image features extracted by different methods, thereby obtaining more diverse mixed image features, and having better adaptability and transferability to various types of downstream computer vision tasks (such as recognition and classification, target detection, etc.), thereby simplifying the network structure design of downstream tasks, improving the training and reasoning speed of the model, and achieving better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 It is a flow chart of a hybrid image feature extraction method according to an embodiment of the present invention.

[0035] Figure 2 It is a specific flow chart of a hybrid image feature extraction method according to an embodiment of the present invention.

[0036] Figure 3 Schematic diagram of the structure of the image feature extraction network in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following descriptions of the embodiments refer to the accompanying drawings to illustrate specific embodiments in which the present invention may be implemented.

[0038] As mentioned above, most of the existing image feature extraction methods use a single form of feature extraction method, which makes the extracted image features lack diversity. This not only requires the design of a more complex network structure to learn these feature expressions, reducing the training and reasoning speed of the model, but also makes the extracted image features less adaptable and unable to be fully migrated in different downstream tasks. The attention mechanism originates from the study of human vision. In cognitive science, humans usually only pay attention to the part of all information that they are interested in, while ignoring other uninteresting information. This mechanism is called the attention mechanism. The attention mechanism mainly includes two aspects: one is to determine the part of the input that needs to be paid attention to, and the other is to allocate limited information processing resources to the important part. Due to the above characteristics of the attention mechanism, it is also widely used in neural networks. When the visual features extracted by multiple methods are mixed, the attention mechanism can help the downstream network learn the most important part of all features, thereby "summarizing" various features, making the feature extraction more effective. Based on the above considerations, the present invention uses the attention mechanism to aggregate the image features extracted by different methods to obtain mixed image features that can be used for downstream computer vision tasks.

[0039] Therefore, please refer to Figure 1 As shown, the first embodiment of the present invention provides a mixed image feature extraction method, including:

[0040] Obtain unsupervised features of the target image, and concatenate the obtained unsupervised features of the target image in the channel dimension to obtain an unsupervised feature map;

[0041] Respectively acquiring convolution features, multi-layer perceptron features, and self-attention features of the target image to obtain a convolution feature map, a multi-layer perceptron feature map, and a self-attention feature map;

[0042] The unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are concatenated in the channel dimension to obtain an initial mixed image feature map;

[0043] The initial mixed image feature map is aggregated using an attention model to obtain a final mixed image feature map.

[0044] Specifically, please combine Figure 2 , Figure 3As shown, the present invention first provides a variety of ways to obtain unsupervised features of an image. It can be understood that unsupervised features refer to image features obtained by traditional image processing algorithms, including local binary pattern LBP (Local Binary Pattern) features, scale-invariant feature transform SIFT (Scale-Invariant Feature Transform) features, HOG features, etc.

[0045] The process of obtaining the unsupervised features of the target image specifically includes:

[0046] (11) Obtaining LBP features

[0047] The target image to be extracted is converted into a grayscale mode, and the LBP feature description of the target image is obtained by the LBP algorithm. In a specific embodiment, the target image can be first adjusted to a size of 112×112, so that the LBP algorithm can obtain a 1×112×112 LBP feature descriptor (where the channel dimension is 1). The size of the sliding window is generally set to 3×3, and the corresponding number of sampling points is its 8 neighborhood points.

[0048] (12) Obtain SIFT features

[0049] The target image to be extracted features is converted into a grayscale mode, and the SIFT feature description of the target image is obtained by the SIFT algorithm. In a specific embodiment, the SIFT algorithm will obtain n feature descriptors with a dimension of 128. In order to keep consistent with the dimensions of the acquired LBP feature descriptor and HOG feature descriptor, the last 16 dimensions of the SIFT feature descriptor are discarded, that is, it becomes a 112-dimensional feature descriptor, and n is fixed to 112, that is, the number of feature points, and the redundant feature points are discarded, and the insufficient ones are padded by 0 vectors, thereby obtaining a SIFT feature descriptor of size 1×112×112 (where the channel dimension is 1).

[0050] (13) Obtaining HOG features

[0051] The target image to be extracted features is converted into grayscale mode, input into the HOG algorithm, and the HOG feature description of the image is obtained. In a specific embodiment, the image to be calculated is first adjusted to a size of 112×112, and a HOG feature descriptor of a size of 1×112×112 is obtained by the HOG algorithm (wherein the channel dimension is 1). The number of gradient direction intervals, the size of the cell, and the number of cells contained in each block of the HOG algorithm are generally set based on empirical values. As an example, in an embodiment of the present invention, the number of gradient direction intervals is set to 6, the cell size is 20×20, and the number of cells contained in the block is 2×2.

[0052] The unsupervised features of the target image obtained by the above three traditional algorithms are spliced ​​in the channel dimension to obtain an unsupervised feature map F1 of size c1×h×w, where c1 is the channel dimension of the unsupervised feature map F1, h is the height of the unsupervised feature map F1, and w is the width of the unsupervised feature map F1. In a specific embodiment, the LBP feature descriptor, the SIFT feature descriptor, and the HOG feature descriptor are spliced ​​to obtain an unsupervised feature map F1 of size 3×112×112 (c1, h, w are 3, 112, 112, respectively). It can be understood that splicing in the channel dimension is to superimpose the feature map on the channel dimension.

[0053] This embodiment introduces three traditional algorithms as examples when obtaining the unsupervised features of the current image. In practice, more traditional image feature extraction algorithms can be selected to obtain the unsupervised features of the target image. In addition, a certain unsupervised feature can be dynamically added or deleted according to the specific performance of the downstream computer vision task.

[0054] In addition to the unsupervised features of the target image mentioned above, this embodiment also obtains the convolution features, multi-layer perceptron MLP features and self-attention (Self-Attention) features of the target image respectively. It can be understood that the convolution features refer to the image depth features obtained by the deep convolutional neural network model, and the deep convolutional neural network can be a network structure such as VGG, ResNet, DenseNet, EfficientNet, etc. The MLP feature is the image depth feature extracted by the multi-layer perceptron model, and the multi-layer perceptron network can adopt a deep multi-layer perceptron network structure such as ResMLP, RepMLP, gMLP, MLP-Mixer, CycleMLP, Hire-MLP, etc. The self-attention feature is the image depth feature generated by the deep neural network composed of the self-attention module, and its model structure generally uses ViT, DETR, Swin-transformer, etc.

[0055] Specifically:

[0056] (21) Obtaining convolution features

[0057] A deep convolutional neural network is selected, and the target image to be extracted is input into the deep convolutional neural network. In the forward propagation process, the convolutional features of the target image are extracted through multiple convolutional modules to obtain a convolutional feature map F2 of size c2×h×w, where c2, h, and w are the number of channels, height, and width of the feature map F2, respectively. The structure of the deep convolutional neural network includes a basic convolutional layer, a pooling layer, and a batch normalization layer. In a specific embodiment, the target image to be extracted is adjusted to a size of 224×224, input into the convolutional neural network model VGG, and the output of the fourth convolutional layer of VGG is selected as the feature map F2; in order to reduce the amount of calculation, compared with the original VGG, this embodiment adjusts the number of convolution kernels of the previous layer to 64, so the number of channels, height, and width of the output feature map F2 are 64, 112, and 112, respectively.

[0058] (22) Obtaining Multilayer Perceptron Features

[0059] The principle of obtaining the features of the multi-layer perceptron is to first divide the input image into S patches, then flatten each patch into a one-dimensional vector with a dimension of p, and concatenate S such vectors together to form a tensor with a dimension of (S, p). Then, the tensor is mapped to a dimension of (S, C) through linear mapping, and sent to the MLP module in the neural network to obtain the MLP features of the image and use it for the current task.

[0060] Specifically, in this embodiment, a deep multi-layer perceptron neural network is selected, and the target image to be feature extracted is input into the deep multi-layer perceptron neural network. In the forward propagation process, the MLP features of the target image are obtained through multiple multi-layer perceptron modules, and the size is reshaped into an MLP feature map F3 of c3×h×w, where c3 is the number of channels of the feature map F3, and h and w are the height and width of the feature map F3. In a specific embodiment, the target image to be feature extracted is adjusted to a size of 224×224, and input into a multi-layer perceptron neural network model ResMLP with a depth of 12. The output of the fourth MLP module in ResMLP is selected, that is, the size is 196×512, and then it is reconstructed into a dimension of 8×112×112, which is the MLP feature map F3 of the current image.

[0061] (23) Obtaining self-attention features

[0062] The self-attention neural network model is used to extract the self-attention features of the image. In self-attention neural network models such as ViT and Swin-transformer, the processing of the input image is the same as that of the deep multi-layer perceptron neural network, that is, the final (S, C) dimensional tensor is sent to the encoder module in the neural network to extract the self-attention features of the image.

[0063] Specifically, in this embodiment, a self-attention neural network model is selected, and the target image to be extracted is input into the self-attention neural network model. In the forward propagation process, the self-attention features of the target image are extracted through multiple self-attention modules to obtain a self-attention feature map F4 of size c4×h×w, where c4 is the number of channels of the feature map F4, and h and w are the height and width of the feature map. In a specific embodiment, the size of the target image to be extracted is adjusted to 224×224, and the input is a deep self-attention neural network ViT-B / 16 with a depth of 12. The output feature map of the fourth encoder module in ViT-B / 16 is selected, and the size is 196×768. It is reconstructed into a size of 12×112×112, which is the self-attention feature map F4 of the current image.

[0064] It should be noted that the selection of the deep convolutional neural network, deep multi-layer perceptron network and deep self-attention neural network involved in the above process and the specific layer of the model output used as the image feature can be determined in combination with the specific downstream computer vision task. In different computer vision tasks, different models and outputs of different layers will have different degrees of impact on the performance of downstream visual tasks. In an embodiment of the present invention, the selected neural network model and the output of the corresponding layer are combined with the downstream classification task, so the selected models are all classification models with good performance in their respective fields, and the output of the relatively shallow 4th layer is selected as the image feature.

[0065] Based on the features of the target image, this embodiment further uses an attention module to aggregate them to obtain mixed image features that can be used for downstream computer vision tasks. The specific process is as follows:

[0066] The unsupervised feature map F1, convolution feature map F2, multi-layer perceptron feature map F3 and self-attention feature map F4 obtained in the above manner are concatenated in the channel dimension to obtain the initial mixed image feature map F0;

[0067] The size of the initial mixed image feature map F0 is c0×h×w, where c0, h, and w are the number of channels, height, and width of the initial mixed image feature map F0, respectively. It should be noted that the height h and width w of the initial mixed image feature map F0 are the same as those of the aforementioned feature maps F1, F2, F3, and F4. In a specific embodiment, the initial mixed image feature map F0 is obtained by splicing the above-mentioned unsupervised feature map F1, convolutional feature map F2, MLP feature map F3, and self-attention feature map F4, and its number of channels, height, and width are 87, 112, and 112, respectively.

[0068] The initial mixed image feature map F0 is aggregated using the attention model to obtain the final mixed image feature map F, which has a size of c×h×w, where c, h, and w are the number of channels, height, and width of the final mixed image feature map F, respectively. The structure of the attention model used in this embodiment includes a basic pooling layer and a linear layer. In a specific embodiment, the attention model adopts a channel attention SE (Squeeze Excitation) structure, that is, global average pooling is performed in the channel dimension, and the channel attention weight mask is obtained by channel feature learning, and the final mixed image feature map F is output, which has a size of 87×112×112. Of course, the height and width of the final mixed image feature map F can be consistent with the aforementioned feature maps F1, F2, F3, and F4, and can also be selected in combination with downstream computer vision tasks and network design. In addition, the attention model used for the aggregation of the initial mixed image features can also be selected on a preferential basis, and different attention models will have different aggregation effects on mixed features, so it is also necessary to select in combination with downstream computer vision tasks and the final effect of the model. For example, for recognition and classification tasks, different attention models can be used: such as the channel attention SE mentioned in this embodiment; there are also some existing ones such as spatial attention and the combination of channel attention and spatial attention (such as CBAM, etc.) to aggregate the final image features, and compare the final effects of the models (some quantitative indicators can be calculated). The attention model that has the best final effect after aggregating the image features is selected, and the same is true for target detection tasks.

[0069] From the above, it can be seen that the hybrid image feature extraction method of this embodiment can be used in a variety of computer vision tasks, such as recognition and classification, target detection, image segmentation, representation learning, image and video generation, vision and text combination, etc. The hybrid image features extracted by the hybrid image feature extraction method of this embodiment can be used as the input of the network used by downstream computer vision tasks.

[0070] Corresponding to the mixed image feature extraction method described in the first embodiment of the present invention, the embodiment of the present invention provides a mixed image feature extraction device, including:

[0071] A first acquisition module is used to acquire unsupervised features of a target image, and to concatenate the acquired unsupervised features of the target image in a channel dimension to obtain an unsupervised feature map;

[0072] A second acquisition module is used to respectively acquire the convolution features, multi-layer perceptron features and self-attention features of the target image to obtain a convolution feature map, a multi-layer perceptron feature map and a self-attention feature map;

[0073] A third acquisition module is used to splice the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map in the channel dimension to obtain an initial mixed image feature map;

[0074] The output module is used to aggregate the initial mixed image feature map using an attention model to obtain a final mixed image feature map.

[0075] Preferably, the first acquisition module is specifically used for:

[0076] Respectively obtain the local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image;

[0077] The acquired local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image are spliced ​​in the channel dimension to obtain the unsupervised feature map.

[0078] Preferably, the second acquisition module acquires the convolutional features of the target image specifically comprising: selecting a deep convolutional neural network, inputting the target image into the deep convolutional neural network, extracting the convolutional features of the target image through multiple convolutional modules during forward propagation, and obtaining a convolutional feature map;

[0079] The second acquisition module acquires the multi-layer perceptron features of the target image specifically comprising: selecting a deep multi-layer perceptron neural network, inputting the target image into the deep multi-layer perceptron neural network, acquiring the multi-layer perceptron features of the target image through multiple multi-layer perceptron modules in a forward propagation process, and obtaining a multi-layer perceptron feature map;

[0080] The second acquisition module acquires the self-attention features of the target image specifically by: selecting a self-attention neural network model, inputting the target image into the self-attention neural network model, extracting the self-attention features of the target image through multiple self-attention modules during the forward propagation process, and obtaining a self-attention feature map.

[0081] Preferably, the initial mixed image feature map, the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are all of equal height and width.

[0082] Preferably, the output module is specifically used to: adopt a channel attention SE structure, perform global average pooling on the initial mixed image feature map in the channel dimension, obtain a channel attention weight mask through channel feature learning, and output the final mixed image feature map.

[0083] For the working principle and process of this embodiment, please refer to the description of the first embodiment of the present invention, which will not be repeated here.

[0084] From the above description, it can be seen that compared with the prior art, the beneficial effect of the present invention is that compared with a single form of feature extraction, the present invention extracts image features by using multiple methods, and uses an attention mechanism to aggregate image features extracted by different methods, thereby obtaining more diverse mixed image features, and having better adaptability and transferability to various types of downstream computer vision tasks (such as recognition and classification, target detection, etc.), thereby simplifying the network structure design of downstream tasks, improving the training and reasoning speed of the model, and achieving better results.

[0085] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A hybrid image feature extraction method, characterized in that: include: Obtain unsupervised features of the target image, and concatenate the obtained unsupervised features of the target image in the channel dimension to obtain an unsupervised feature map; Respectively acquiring convolution features, multi-layer perceptron features, and self-attention features of the target image to obtain a convolution feature map, a multi-layer perceptron feature map, and a self-attention feature map; The unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are concatenated in the channel dimension to obtain an initial mixed image feature map; The initial mixed image feature map is aggregated using an attention model to obtain a final mixed image feature map.

2. The hybrid image feature extraction method according to claim 1, characterized in that: The step of acquiring the unsupervised features of the target image and concatenating the acquired unsupervised features of the target image in the channel dimension to obtain an unsupervised feature map specifically includes: Respectively obtain the local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image; The acquired local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image are spliced ​​in the channel dimension to obtain the unsupervised feature map.

3. The hybrid image feature extraction method according to claim 2, characterized in that: Acquiring the convolutional features of the target image specifically includes: selecting a deep convolutional neural network, inputting the target image into the deep convolutional neural network, extracting the convolutional features of the target image through multiple convolutional modules during forward propagation, and obtaining a convolutional feature map; Acquiring the multi-layer perceptron features of the target image specifically includes: selecting a deep multi-layer perceptron neural network, inputting the target image into the deep multi-layer perceptron neural network, acquiring the multi-layer perceptron features of the target image through multiple multi-layer perceptron modules during forward propagation, and obtaining a multi-layer perceptron feature map; Acquiring the self-attention features of the target image specifically includes: selecting a self-attention neural network model, inputting the target image into the self-attention neural network model, extracting the self-attention features of the target image through multiple self-attention modules during the forward propagation process, and obtaining a self-attention feature map.

4. The hybrid image feature extraction method according to claim 3, characterized in that: The initial mixed image feature map, the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are all equal in height and width.

5. The hybrid image feature extraction method according to claim 1, characterized in that: The using of the attention model to aggregate the initial mixed image feature map to obtain the final mixed image feature map specifically includes: The channel attention SE structure is adopted to perform global average pooling on the initial mixed image feature map in the channel dimension, and the channel attention weight mask is obtained through channel feature learning to output the final mixed image feature map.

6. A hybrid image feature extraction device, characterized in that: include: A first acquisition module is used to acquire unsupervised features of a target image, and to concatenate the acquired unsupervised features of the target image in a channel dimension to obtain an unsupervised feature map; A second acquisition module is used to respectively acquire the convolution features, multi-layer perceptron features and self-attention features of the target image to obtain a convolution feature map, a multi-layer perceptron feature map and a self-attention feature map; A third acquisition module is used to splice the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map in the channel dimension to obtain an initial mixed image feature map; The output module is used to aggregate the initial mixed image feature map using an attention model to obtain a final mixed image feature map.

7. The hybrid image feature extraction device according to claim 6, characterized in that: The first acquisition module is specifically used for: Respectively obtain the local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image; The acquired local binary pattern features, scale-invariant feature transformation features and directional gradient histogram features of the target image are spliced ​​in the channel dimension to obtain the unsupervised feature map.

8. The hybrid image feature extraction device according to claim 7, characterized in that: The second acquisition module acquires the convolutional features of the target image specifically including: selecting a deep convolutional neural network, inputting the target image into the deep convolutional neural network, extracting the convolutional features of the target image through multiple convolutional modules during the forward propagation process, and obtaining a convolutional feature map; The second acquisition module acquires the multi-layer perceptron features of the target image specifically comprising: selecting a deep multi-layer perceptron neural network, inputting the target image into the deep multi-layer perceptron neural network, acquiring the multi-layer perceptron features of the target image through multiple multi-layer perceptron modules in a forward propagation process, and obtaining a multi-layer perceptron feature map; The second acquisition module acquires the self-attention features of the target image specifically by: selecting a self-attention neural network model, inputting the target image into the self-attention neural network model, extracting the self-attention features of the target image through multiple self-attention modules during the forward propagation process, and obtaining a self-attention feature map.

9. The hybrid image feature extraction device according to claim 8, characterized in that: The initial mixed image feature map, the unsupervised feature map, the convolution feature map, the multi-layer perceptron feature map and the self-attention feature map are all equal in height and width.

10. The hybrid image feature extraction device according to claim 1, characterized in that: The output module is specifically used to: adopt a channel attention SE structure to perform global average pooling on the initial mixed image feature map in the channel dimension, obtain a channel attention weight mask through channel feature learning, and output the final mixed image feature map.

Citation Information

Patent Citations

  • A self-adaptive weight deep learning target classification method based on feature fusion

    CN109886357A

  • Power equipment identification method and system based on infrared image

    CN112287919A