An image feature extraction method, apparatus, storage medium, and image classification method

By using a fill-free downsampling submodule and a feature extraction module in the image feature extraction method, the problems of high computational cost and insufficient feature extraction effectiveness in the prior art are solved, and efficient feature extraction on edge devices is achieved.

CN117934862BActive Publication Date: 2026-11-10JIANGNAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410055055.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-01-12
Publication Date
2026-11-10
Estimated Expiration
2044-01-12

AI Technical Summary

Technical Problem

Existing image feature extraction methods cannot reduce the computational load of the model while ensuring the effectiveness of feature extraction, making it difficult to deploy the model on resource-constrained edge devices.

Method used

The module employs a padding-free downsampling submodule and a feature extraction module, including a padding-free convolution unit, a padding-free max pooling unit, and a feature fusion unit. It extracts key features from the input feature map through padding-free convolution and max pooling, thereby reducing the feature map size and computational cost.

Benefits of technology

It effectively reduces the computational load and cost of the model, while improving the effectiveness and accuracy of feature extraction, making it suitable for resource-constrained edge devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117934862B_ABST
    Figure CN117934862B_ABST
Patent Text Reader

Abstract

This invention relates to an image feature extraction method, apparatus, storage medium, and image classification method, belonging to the field of image processing technology. The method involves obtaining an image feature extraction model, which includes a stem module and multiple feature extraction modules. Each feature extraction module includes a padding-free downsampling submodule and a feature extraction convolution submodule. Each padding-free downsampling submodule includes a padding-free convolution unit, a padding-free max pooling unit, and a feature fusion unit. Image samples are input into the stem module, which outputs an initial feature map. The initial feature map is then input into the padding-free downsampling submodule of the first feature extraction module, which outputs a downsampled feature map. The downsampled feature map is then input into the feature extraction convolution submodule of the first feature extraction module, which is used as the input to the next feature extraction module. The output of the feature extraction convolution submodule of the last feature extraction module is used as the target feature map.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image feature extraction method, apparatus, computer-readable storage medium, and image classification method. Background Technology

[0002] In recent years, machine learning (ML) and deep learning (DL) have made significant progress in the field of computer vision. Starting with the first deep convolutional neural network, AlexNet, deeper networks with increasingly more parameters and computational costs have been proposed, such as VGGNet, Inception, and ResNet. These models are widely used for image feature extraction tasks. To improve the effectiveness of feature extraction, convolutional neural network models are becoming increasingly complex, with more and more computational costs and parameters, which in turn increases the demand on hardware resources, making it difficult to deploy these models on resource-constrained edge devices.

[0003] Existing technologies have a number of methods for compressing deep neural network models to reduce computational cost and parameter count without compromising image feature extraction accuracy. Examples include pruning, quantization, and knowledge distillation. However, the compression effectiveness of these methods depends on the original model; when the original model is too large, these methods are not effective in reducing computational cost. Therefore, existing technologies explore ways to obtain features similar to traditional large models with less computation and fewer parameters by focusing on model architecture. For example, using depthwise separable convolutions instead of ordinary convolutions and performing channel shuffling on grouped convolutions can minimize computational cost while maintaining image feature extraction accuracy. While depthwise convolutions reduce computational cost compared to ordinary convolutions, they still increase the number of parameters. Furthermore, oversimplifying the model may reduce the effectiveness of feature extraction; for example, in high-precision classification scenarios, depthwise convolutions cannot achieve the same classification accuracy as ordinary convolutions.

[0004] In summary, existing image feature extraction methods cannot simultaneously reduce the computational cost of the model and ensure the effectiveness of feature extraction. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem that existing image feature extraction methods cannot reduce the computational load of the model and improve the feature extraction efficiency while ensuring the effectiveness of feature extraction.

[0006] To address the aforementioned technical problems, this invention provides an image feature extraction method, comprising:

[0007] A pre-trained image feature extraction model is obtained. The image feature extraction model includes a STEM module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit.

[0008] The image sample is input into the stem module for initial feature extraction, and an initial feature map is output.

[0009] The initial feature map is input into the no-padding downsampling submodule of the first feature extraction module for downsampling, which specifically includes:

[0010] The initial feature map is input into the padding-free convolution unit in the padding-free downsampling submodule for convolution, and the first feature map is output.

[0011] The initial feature map is input into the no-fill max pooling unit in the no-fill downsampling submodule for max pooling, and a second feature map is output; wherein, the size of the first feature map and the size of the second feature map are the same.

[0012] The first feature map and the second feature map are input into the feature fusion unit in the no-fill downsampling submodule for point-by-point addition, and the downsampled feature map is output.

[0013] The downsampled feature map is input into the feature extraction convolution submodule in the first feature extraction module for deep feature extraction, and the feature map after deep feature extraction is used as the input of the next feature extraction module.

[0014] The output of the feature extraction convolution submodule in the last feature extraction module is used as the target feature map of the image sample.

[0015] In one embodiment of the present invention, the stem module includes a regular convolution submodule, a DW convolution submodule, a max pooling submodule, and a channel fusion submodule.

[0016] In one embodiment of the present invention, the step of inputting image samples into the stem module for initial feature extraction and outputting an initial feature map includes:

[0017] The image sample is input into the ordinary convolution submodule for ordinary convolution, and the third feature map is output.

[0018] The third feature map is input into the DW convolution submodule for DW convolution, and the fourth feature map is output.

[0019] The third feature map is input into the max pooling submodule for max pooling, and the fifth feature map is output; wherein, the size of the fourth feature map and the size of the fifth feature map are the same;

[0020] The fourth and fifth feature maps are input into the channel fusion submodule for pointwise convolution to output the initial feature map.

[0021] In one embodiment of the present invention, after outputting the initial feature map, the method further includes: performing feature extraction on the initial feature map, outputting a target initial feature map, and using the target initial feature map as the input of the first feature extraction module.

[0022] In one embodiment of the present invention, the training process of the image feature extraction model includes:

[0023] Obtain image samples from the training set;

[0024] The image samples are input into the stem module, which outputs an initial feature map.

[0025] The initial feature map is input into multiple feature extraction modules with different kernel sizes that are sequentially connected along the forward propagation direction, and an image feature extraction loss function is constructed based on the output of the last feature extraction module.

[0026] The STEM module and multiple feature extraction modules with different convolutional kernel sizes are iteratively trained using image samples from the training set until the value of the image feature extraction loss function is minimized, thus obtaining the trained STEM module and multiple feature extraction modules with different convolutional kernel sizes.

[0027] Based on the trained stem module and multiple feature extraction modules with different convolutional kernel sizes, a trained image feature extraction model is obtained.

[0028] In one embodiment of the present invention, the image feature extraction loss function is:

[0029]

[0030] Where L represents the image feature extraction loss function, N represents the number of image samples, and y ic Let y be the sign function, and when the true class of the i-th image sample is c, y ic =1, otherwise, y ic =0, p ic Let M be the predicted probability that the i-th image sample belongs to category c, and M be the number of categories.

[0031] The present invention also provides an image feature extraction device, comprising:

[0032] The model acquisition module is used to acquire a pre-trained image feature extraction model. The image feature extraction model includes a stem module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit.

[0033] The initial feature map acquisition module is used to input image samples into the stem module for initial feature extraction and output an initial feature map.

[0034] The downsampling feature map acquisition module is used to input the initial feature map into the padding-free downsampling submodule in the first feature extraction module for downsampling, and specifically includes:

[0035] The first feature map acquisition submodule is used to input the initial feature map into the no-padding convolution unit in the no-padding downsampling submodule for convolution and output the first feature map.

[0036] The second feature map acquisition submodule is used to input the initial feature map into the no-fill max pooling unit in the no-fill downsampling submodule for max pooling and output the second feature map.

[0037] Wherein, the size of the first feature map and the size of the second feature map are the same;

[0038] The feature fusion submodule is used to input the first feature map and the second feature map into the feature fusion unit in the no-fill downsampling submodule for point-by-point addition and output the downsampled feature map.

[0039] The deep feature extraction module is used to input the downsampled feature map into the feature extraction convolution submodule in the first feature extraction module for deep feature extraction, and use the feature map after deep feature extraction as the input of the next feature extraction module;

[0040] The target feature map acquisition module is used to take the output of the feature extraction convolution submodule in the last feature extraction module as the target feature map of the image sample.

[0041] In one embodiment of the present invention, the initial feature map acquisition module includes:

[0042] The third feature map acquisition submodule is used to input the image sample into the ordinary convolution submodule for ordinary convolution and output the third feature map;

[0043] The fourth feature map acquisition submodule is used to input the third feature map into the DW convolution submodule to perform DW convolution and output the fourth feature map;

[0044] The fifth feature map acquisition submodule is used to input the third feature map into the max pooling submodule for max pooling and output the fifth feature map; wherein, the size of the fourth feature map is the same as the size of the fifth feature map;

[0045] The channel fusion submodule is used to input the fourth feature map and the fifth feature map into the channel fusion submodule for pointwise convolution and output the initial feature map.

[0046] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described image feature extraction method.

[0047] The present invention also provides an image classification method, comprising:

[0048] The target feature map of the image sample is extracted using the image feature extraction method described above;

[0049] The target feature map is input into a fully connected layer, and the classification result of the image sample is output.

[0050] The image feature extraction method provided by this invention incorporates a padding-free downsampling submodule in each feature extraction module. A padding-free convolutional unit is used to convolve the input feature map, outputting a first feature map smaller than the input feature map. A padding-free max pooling unit is then used to extract key features from the input feature map, resulting in a second feature map with the same size as the first feature map. The first and second feature maps are then summed point-by-point to obtain a downsampled feature map containing both global and key features. This application uses padding-free downsampling to downsample the input feature map, eliminating the need for padding and avoiding unnecessary computation. This ensures that the output feature map size in each downsampling stage is smaller than twice the size of a traditional downsampled feature map, reducing not only the computational load of the downsampling stage itself but also subsequent feature extraction stages, effectively reducing the computational load of the model and lowering computational costs. Furthermore, since no padding is applied to the input feature map during the downsampling stage, invalid features are not introduced, ensuring that each output downsampled feature map only contains global and key features relevant to the input feature map, thereby improving the effectiveness of feature extraction. Attached Figure Description

[0051] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein...

[0052] Figure 1 Flowchart of the image feature extraction method provided in this application;

[0053] Figure 2 This is a schematic diagram comparing the receptive fields of the fill-free downsampling method and the fill-in downsampling method provided in this application; wherein, Figure 2 (a) in the diagram is a schematic diagram of the receptive field in the no-fill downsampling method. Figure 2 (b) in the diagram is a schematic diagram of the receptive field with a filled downsampling method;

[0054] Figure 3 This is a schematic diagram illustrating the principle of the fill-free downsampling method provided in this application; wherein, Figure 3 (a) in the diagram is a flowchart of downsampling without padding. Figure 3 (b) in the diagram is a schematic of the model without padding downsampling;

[0055] Figure 4 This is a schematic diagram illustrating the principle of initial feature map extraction provided in this application; wherein, Figure 4 (a) in the diagram is the flowchart for the initial feature map extraction. Figure 4 (b) in the diagram is a schematic diagram of the initial feature map extraction model;

[0056] Figure 5 Schematic diagrams of the image feature extraction model provided in this application and existing image feature extraction models; wherein, Figure 5 (a) in the figure is a schematic diagram of the image feature extraction model structure provided in this application. Figure 5 (b) is a schematic diagram of the image feature extraction model structure in the prior art;

[0057] Figure 6 The diagram illustrates the model accuracy and computational cost of applying non-filling downsampling and the stem module to different models, as provided in this application. Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0059] Existing technologies mainly address the problem of high computational cost in models from two directions: model compression and knowledge distillation. Pruning primarily involves parameter trimming, which involves finding the smallest subset of parameters that does not affect accuracy and iteratively trimming parameters with the lowest "significance" scores using a pre-trained network that has reached a reasonable quality. Quantization mainly involves the quantization of weights and activations, reducing the storage and computational bits of weights and activations in the network to decrease the model's space usage and computational cost. Knowledge distillation mainly involves transferring knowledge from a large model to a smaller model, improving the accuracy of the smaller model without changing its structure. Secondly, there is the model architecture design, which can be broadly categorized into CNN-based model architecture design, Transformer-based model architecture design, and hybrid CNN / Transformer model architecture design. Among these, CNN-based model architecture design is the oldest. Due to CNN's inherent inductive bias and efficient local information extraction, coupled with significant optimizations and accelerations to traditional convolutional frameworks, it still dominates the lightweight model field. Transformer-based models have demonstrated excellent performance in the large model domain, and researchers have attempted to refactor these large models into lightweight architectures. However, due to the complexity of multi-head self-attention mechanisms, further exploration is needed to further reduce the weight of Transformers. Models combining CNN and Transformer integrate the advantages of both models, namely efficient feature extraction and global receptive field.

[0060] It can be seen that existing technologies, when designing efficient lightweight network models, mostly focus on efficient feature extraction, neglecting the optimization of the downsampling stage. An ideal downsampling operation can not only reduce the size of the feature map to save the computational cost of the model, but also minimize the loss of feature information to ensure model accuracy. Therefore, the quality of a model is inseparable from the downsampling stage. Based on this, this application designs the downsampling stage of the model to reduce the computational cost of the model while ensuring model accuracy.

[0061] Please see Figure 1 , Figure 1 The diagram shown is a flowchart of the image feature extraction method provided in this application. The method specifically includes:

[0062] S10: Obtain a pre-trained image feature extraction model. The image feature extraction model includes a stem module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit.

[0063] S20: Input the image sample into the stem module for initial feature extraction and output the initial feature map;

[0064] S30: Input the initial feature map into the padding-free downsampling submodule of the first feature extraction module for downsampling, which specifically includes:

[0065] S300: Input the initial feature map into the padding-free convolutional module in the padding-free downsampling submodule.

[0066] The first feature map is output by performing convolution on the element.

[0067] S301: Input the initial feature map into the no-padding max pooling unit in the no-padding downsampling submodule for max pooling, and output the second feature map; wherein, the size of the first feature map and the second feature map are...

[0068] The two feature maps are the same size;

[0069] S302: Input the first feature map and the second feature map into the feature fusion unit in the no-fill downsampling submodule for point-by-point addition, and output the downsampled feature map;

[0070] S40: Input the downsampled feature map into the feature extraction convolution submodule in the first feature extraction module for deep feature extraction, and use the feature map after deep feature extraction as the input of the next feature extraction module;

[0071] S50: Use the output of the feature extraction convolution submodule in the last feature extraction module as the target feature map of the image sample.

[0072] Specifically, in the embodiments of this application, each feature extraction module may include one or more feature extraction convolutional sub-modules, and the number of feature extraction convolutional sub-modules in different feature extraction modules may be the same or different, and this application does not limit this.

[0073] In existing technologies, zero-padding downsampling is commonly used to downsample the feature map when extracting image features in order to enhance the extraction of boundary information. This involves padding the boundaries of the input feature map. However, after several padding and downsampling operations, only a very small portion of the pixels covered by the edge convolution kernel come from the original feature map, while the rest are invalid features brought about by zero padding. This not only increases the amount of unnecessary computation but also results in invalid features in the final image features, affecting the effectiveness of feature extraction and consequently the accuracy of subsequent image classification results based on image features.

[0074] Based on this, the image feature extraction method provided in this application designs a padding-free downsampling submodule in each feature extraction module. A padding-free convolution unit is used to convolve the input feature map, outputting a first feature map with a size smaller than twice the size of a traditional downsampled feature map. A padding-free max pooling unit is then used to extract key features from the input feature map, resulting in a second feature map with the same size as the first feature map. The first and second feature maps are then added bit-by-bit to obtain a downsampled feature map containing both global and key features. This application uses padding-free downsampling to downsample the input feature map, eliminating the need for padding and avoiding unnecessary computation. This ensures that the output feature map size in each downsampling stage is smaller than the input feature map size, reducing not only the computational cost of the downsampling stage itself but also subsequent feature extraction stages, effectively reducing the computational load of the model and lowering computational costs. Furthermore, since no padding is applied to the input feature map during the downsampling stage, invalid features are not introduced, ensuring that each output downsampled feature map only contains global and key features relevant to the input feature map, thereby improving the effectiveness of feature extraction.

[0075] There are two main types of downsampling: one is average pooling with kernel_size=2 and stride=2, and the other is convolution with kernel_size=3, stride=2, and padding=1. Both of these downsampling methods can achieve a 2x downsampling of the feature map. When performing downsampling, the input feature map is usually... Figure Four In previous methods, zero-padding was used to ensure consistent feature map sizes before and after convolution, thereby constructing a structured model and enhancing boundary information extraction. However, zero-padding feature points themselves do not contain image features, so calculating these zero-padding points is meaningless. In contrast, this application uses no padding in the downsampling stage, which not only reduces the size of the feature map but also ensures that using the downsampled feature map obtained by the no-padding downsampling submodule provided in this application for subsequent feature extraction does not affect the model accuracy.

[0076] To verify that the padding-free downsampling method designed in this application can reduce computational cost while ensuring the effectiveness of feature extraction, embodiments of this application also provide a comparison between padding-free downsampling and existing padding downsampling methods in terms of receptive field, feature map size, and number of convolutional computations:

[0077] For padded downsampling, if the size of the input feature map is n*n and padding = 1, then the proportion of padded feature points in the entire image is:

[0078]

[0079] It can be seen that when the size n of the feature map is relatively small, the proportion of filled feature points in the entire feature map is larger, resulting in a larger proportion of invalid feature points in the feature map. In addition, the network model fills the feature map boundary with each convolution. After several fillings and convolutions, only a very small part of the pixels covered by the edge convolution kernel comes from the actual input, and the rest are zero-filled. This results in invalid features being included in the feature map obtained by each downsampling, thereby reducing the effectiveness of feature extraction.

[0080] For example, such as Figure 2 The diagram shown is a comparison of the receptive fields of the fill-free downsampling method and the fill-based downsampling method provided in the embodiments of this application; wherein, Figure 2 (a) in the diagram is a schematic diagram of the receptive field in the no-fill downsampling method. Figure 2 (b) in the diagram is a schematic diagram of the receptive field with a filled downsampling method.

[0081] Taking 3x3 convolution as an example, from Figure 2 As shown in (a), after a normal convolution operation, one point in the feature map fuses 9 feature points. The receptive field after two convolutions is 25, and after three convolutions it is 49. The receptive field after n convolutions is:

[0082] Receptive Field = (3 + 2(n-1)) 2 ,

[0083] like Figure 2 As shown in (b), when padding = 1, the effective receptive field of the boundary feature points is 4 after one padding, 9 after two padding convolutions, and 16 after three padding convolutions. The effective receptive field after n padding convolutions is:

[0084] Receptive Field=(3+2(n-1)-n) 2 ,

[0085] It can be seen that after multiple feature extractions, the actual receptive field of the convolution with padding is much smaller than that of the ideal convolution, and very few effective features can be extracted.

[0086] Additionally, the feature map size obtained after downsampling with padding is:

[0087]

[0088] The feature map size obtained after padding-free downsampling is:

[0089]

[0090] It can be seen that the feature map size obtained by using no-padding downsampling is significantly smaller than the feature map size obtained by padding downsampling. Therefore, using no-padding downsampling can reduce the computational cost of the model.

[0091] In addition, the padding-free downsampling method used in this application can reduce the number of convolution calculations: if the convolution stride is 1, the number of convolutions required using the padding downsampling method is:

[0092]

[0093] The number of convolutions required to use the padding-free downsampling method provided in this application is:

[0094]

[0095] Assuming the model comprises four stages (2-3-6-3), i.e., four feature extraction modules, with the first module containing two convolutional sub-modules, the second containing three, the third containing six, and the fourth containing three, the reduction in convolutions by the padding-free downsampling method compared to the padding-based downsampling method provided in this application is:

[0096]

[0097] The ratio of the reduced number of convolutions to the number of convolutions required by the padded downsampling method is:

[0098]

[0099] It can be seen that the padding-free downsampling method provided in this application reduces computational costs more significantly on small-resolution datasets.

[0100] Furthermore, in recent years, lightweight mobile models have focused not only on FLOPs but also on actual training and inference speeds. This is because FLOPs are only a basic metric; inference speed is related not only to FLOPs but also to factors such as tensor dimension changes, inference library optimization, and memory usage. Moreover, reducing model activations has a stronger correlation with model speed than FLOPs. However, existing lightweight models struggle to strike a good balance between speed and accuracy. Many theoretically efficient operations that reduce computation are not fast in practical applications. The image feature extraction method provided in this application uses padding-free downsampling to fundamentally reduce the size of the feature map, effectively reducing computation and improving model speed and feature extraction efficiency.

[0101] Furthermore, the features obtained after feature extraction from image samples are widely used in fields such as image classification, image retrieval, and image-text matching. Therefore, the effectiveness of image feature extraction is also crucial. To address this, this application designs a padding-free max pooling unit in the padding-free downsampling submodule to further combine the information of the largest feature points.

[0102] Specifically, such as Figure 3 The diagram shown is a schematic diagram of the principle of the padding-free downsampling method provided in this application. After performing padding-free convolution and padding-free max pooling on the input feature map to obtain the first feature map and the second feature map, the first feature map and the second feature map are added bit by bit to obtain the downsampled feature map.

[0103] As a specific example of this application, the kernel size in the unfilled convolutional unit is 3*3 with a stride of 2, and the kernel size in the unfilled max pooling unit is 3*3 with a stride of 2.

[0104] Since the first feature map and the second feature map are the same size, this application adds the two feature maps instead of merging the channel dimensions to further reduce the computational cost of the model.

[0105] Because the size of the feature map after padding downsampling is The size of the feature map after unpadded downsampling is As the feature map becomes smaller, it contains less information. Therefore, using unfilled downsampling will cause a slight decrease in accuracy. To address this, in some embodiments of this application, a hybrid stem module is used to replace the stem module in the model. That is, max pooling is added to the stem module to highlight key features, so that the feature map after the size is reduced still contains the key information of the original image sample, thereby improving the effectiveness of feature extraction.

[0106] Specifically, in some embodiments of this application, the stem module includes a regular convolution submodule, a DW convolution submodule, a max pooling submodule, and a channel fusion submodule.

[0107] Further, in step S20, the image samples are input to the stem module for initial feature extraction, and the output initial feature map includes:

[0108] S200: Input the image sample into the ordinary convolution submodule for ordinary convolution, and output the third feature map;

[0109] S201: Input the third feature map into the DW convolution submodule to perform DW convolution, and output the fourth feature map;

[0110] S202: Input the third feature map into the max pooling submodule for max pooling and output the fifth feature map; wherein, the size of the fourth feature map and the size of the fifth feature map are the same;

[0111] S203: Input the fourth and fifth feature maps into the channel fusion submodule for pointwise convolution and output the initial feature map.

[0112] like Figure 4 The diagram shown is a schematic of the initial feature map extraction principle provided in this application. As a specific example of this application, the kernel size of the ordinary convolutional submodule is 3*3, the stride is 1, and the padding is 1.

[0113] Optionally, in some embodiments of this application, after step S20, feature extraction is performed on the initial feature map to output a target initial feature map, and the target initial feature map is used as the input of the first feature extraction module.

[0114] This application also provides a training process for an image feature extraction model, which specifically includes:

[0115] Step 1: Obtain image samples from the training set;

[0116] Step 2: Input the image samples into the stem module and output the initial feature map;

[0117] Step 3: Input the initial feature map into multiple feature extraction modules with different kernel sizes that are sequentially connected along the forward propagation direction, and construct the image feature extraction loss function based on the output of the last feature extraction module;

[0118] Specifically, the image feature extraction loss function is:

[0119]

[0120] Where L represents the image feature extraction loss function, N represents the number of image samples, and y ic Let y be the sign function, and when the true class of the i-th image sample is c, y ic =1, otherwise, y ic =0, p ic Let M be the predicted probability that the i-th image sample belongs to category c, and M be the number of categories.

[0121] Step 4: Iteratively train the STEM module and multiple feature extraction modules with different convolutional kernel sizes using image samples from the training set until the image feature extraction loss function is minimized, thus obtaining the trained STEM module and multiple feature extraction modules with different convolutional kernel sizes;

[0122] Step 5: Based on the trained STEM module and multiple feature extraction modules with different convolutional kernel sizes, a fully trained image feature extraction model is obtained.

[0123] like Figure 5The diagram shows the structural schematics of the image feature extraction model provided in this application and the image feature extraction model in the prior art. Figure 5 (a) in the figure is a schematic diagram of the image feature extraction model structure provided in this application. Figure 5 (b) in the figure is a schematic diagram of the structure of an image feature extraction model in the prior art.

[0124] Furthermore, in order to verify the effectiveness of the image feature extraction model provided in this application, embodiments of this application replace the downsampling module and stem module in various deep learning networks with the unfilled downsampling submodule and stem module in the model, and experiments were conducted on several visual benchmark datasets.

[0125] The benchmark dataset includes:

[0126] The CIFAR-100 dataset contains 100 categories, with 600 color images per category, including 500 training images and 100 test images per category, and each image is 32*32 pixels in size. The CIFAR dataset is currently one of the standard datasets for machine vision. For fairness, this application uses standard data augmentation of the CIFAR dataset, i.e., zero-padding the image edges with 4 pixels and randomly cropping to produce 32*32 images, while also using random horizontal flipping.

[0127] The ImageNet dataset contains 1.3 million images for training and 50,000 images for testing, covering 1,000 categories. In this embodiment, standard data augmentation is used to resize the images to 256*256 and then randomly crop them to 224*224.

[0128] The Stanford Dogs dataset is a fine-grained dataset containing images of 120 dog breeds from around the world. It is a subset of the ImageNet dataset and contains a total of 20,580 images. The number of images in each category varies, but is always more than 150. In this embodiment, the images are adjusted to 256*256 and randomly cropped to 224*224. Data augmentation uses random horizontal flipping and ColorJitter transformation.

[0129] For the CIFAR-100 dataset, this application example sets two sets of hyperparameters: one set uses the SGD optimizer with a momentum of 0.9 and a weight decay of 10. -4 The initial learning rate was 0.1, and the training lasted for 200 epochs. The learning rate decayed by a factor of 0.1 at epochs 100, 150, and 180. The batch size was fixed at 128. The second set of hyperparameters used the SGD optimizer with a momentum of 0.9 and a weight decay of 10. -4The initial learning rate was 0.03, and the training lasted for 300 epochs. The learning rate used a cosine annealing decay strategy, and the bitch_size was fixed at 64.

[0130] For the ImageNet dataset, the SGD optimizer was used with a momentum of 0.9 and a weight decay of 10. -4 The initial learning rate was 0.1, and the training was conducted for 100 rounds, with the rate decreasing by 0.1 at rounds 30, 60, and 90.

[0131] For the Stanford Dogs dataset, the SGD optimizer was used with a momentum of 0.9 and a weight decay of 10. -4 The initial learning rate is 0.1, which decays linearly in each epoch for a total of 100 epochs, and the bitch_size is fixed at 64.

[0132] Based on the above dataset and hyperparameter settings, experiments were first conducted on MobileNetv3_large, MobileNetv3_large_bsconvs, ResNet-20, ResNet-110 (BSConv-U), WideResNet-40-3, and WideResNet-40-8 (BSConv-U), that is, the downsampling module and stem layer in the original model were replaced with the padding-free downsampling submodule and stem module provided in this application.

[0133] Specifically, the settings of the stem layer differ in different models. For example, MobileNetv3 uses the HSwish activation function in its stem layer, so the HSwish activation function is also used when replacing it with the stem module provided in this application. WideResNet's stem layer performs feature extraction in the order of normalization, activation, and convolution, so the same order is followed when replacing it with the stem module provided in this application. ResNet series models extensively use residual connections (skip-connection), so parameter connections are used when replacing it with the stem module provided in this application, i.e., the output of the first 3*3 convolution is connected to the output of the stem module to minimize the influence of other factors.

[0134] Unlike networks such as MobileNetv3, ResNet and WideResNet use skip-connections throughout the network, including in the downsampling stage. This involves changes in feature map size and channel dimensions. Since the original network uses 1x1 convolutions, but when using the padding-free downsampling provided in this application, the feature maps are not in a 2x downsampling relationship, and cannot be implemented using 1x1 convolutions. Therefore, in order to preserve the original model as much as possible, the embodiments of this application maintain skip connections in the downsampling stage. Specifically, for WideResNet-40-3, the embodiments of this application use pointwise addition of ordinary convolution + max pooling to adjust the feature map size to achieve skip connections. For ResNet series models, depthwise convolution + pointwise convolution is used to achieve skip connections.

[0135] Table 1 shows the model parameters, computational cost, and accuracy results obtained from experiments using the first set of hyperparameters on CIFAR-100:

[0136] Table 1

[0137]

[0138] As shown in Table 1, replacing the downsampling and stem layers in the original model with the padding-free downsampling and stem modules provided in this application can effectively reduce the computational cost of the model while maintaining accuracy. For ResNet series models, the number of parameters increases slightly, while the computational cost decreases by about 10%. For MobileNet series models, the number of parameters remains almost unchanged, while the computational cost decreases by about 20%. The increase in parameters for ResNet series models is greater than that for MobileNet series models because ResNet series models also use skip connections in the downsampling stage. To maintain the integrity of the original model architecture, the embodiments in this application also use skip connections. Therefore, depthwise convolution and pointwise convolution are needed to change the feature map size and channel dimension, all of which require more parameters and computational cost.

[0139] Table 2 shows the model parameters, computational cost, and accuracy results obtained by using the second set of hyperparameters on a more advanced lightweight model with the fill-free downsampling and stem module provided in this application:

[0140] Table 2

[0141]

[0142]

[0143] As shown in Table 2, after sufficient convergence, the lightweight model using the padding-free downsampling and stem module provided in this application exhibits higher gains than the classic model. The number of parameters remains almost unchanged, while the computational cost is reduced by approximately 20%, and the accuracy slightly increases. This is because current mainstream lightweight models, in order to balance accuracy and model size, typically perform extensive feature extraction in the later stages of the network, i.e., on low-resolution feature maps, resulting in the computational bottleneck being concentrated in the later stages. The padding-free downsampling provided in this application further reduces the size of the feature maps in the next stage, thus reducing further computational cost. However, classic networks generally use the same number of convolutional layers for feature extraction in each stage, ensuring that the feature map size is the same in each stage, which implicitly reduces the model's gain.

[0144] Notably, the MobileNetV3-large(x1.0) in Table 2 and the MobileNetV3-large(x1.0) in Table 1 are not exactly the same model. The MobileNetV3-large(x1.0) in Table 1 uses a lightweight classifier, which means that only one fully connected layer is used in the classification stage. The MobileNetV3-large(x1.0) in Table 2 is the original model without any modifications, which contains two fully connected layers.

[0145] Table 3 shows the model parameters, computational cost, and accuracy results obtained from experiments conducted on Stanford Dogs:

[0146] Table 3

[0147]

[0148] Table 4 shows the model parameters, computational cost, and accuracy results obtained from experiments on ImageNet:

[0149] Table 4

[0150]

[0151]

[0152] As can be seen from Tables 3 and 4, even for standard resolution images, the padding-free downsampling and stem module provided in this application can still effectively reduce the computational load while improving the model's accuracy. This indicates that the padding-free downsampling and stem module provided in this application is not only suitable for low-resolution images but also effective for standard resolution images. Furthermore, since a 224*224 image already has sufficient information to extract, discarding the padding of edge pixels results in less information loss, which is beneficial for model training and fitting.

[0153] Since the current field of lightweight models focuses not only on computational FLOPs and parameter counts, but also more directly on inference speed, this application also tests the contributions of padding-free downsampling and the stem module to improving model speed. Because lightweight models are generally used for CPU-based inference, this application also tests the inference speed on high-performance PCs and lower-performance mobile CPUs.

[0154] Specifically, the PC CPU is an AMD Ryzen 5 5600H. All experiments on all models in this embodiment are based on PyTorch, and the inference framework is also PyTorch, version 1.11.0. The mobile CPU is a MediaTek Dimensity 1000+, and the inference framework is PyTorch_Mobile. It can directly export PyTorch models as .ptl models using the torch.jit toolkit and run them on Android applications. Different mobile deep learning inference libraries have their own advantages and disadvantages; no single inference library is superior to others in all aspects. Therefore, this embodiment only verifies model speed under the PyTorch_Mobile framework. No quantization techniques are used to simplify or optimize the model during deployment. The inference model on the mobile device completely maintains the original model architecture; therefore, accuracy is lossless, and speed is consistent.

[0155] Table 5 shows the model inference speed obtained by applying the padding-free downsampling and stem module provided in this application to different models:

[0156] Table 5

[0157]

[0158]

[0159] As shown in Table 5, when using the padding-free downsampling and stem module provided in this application, the inference speed of all models on high-performance CPUs did not improve, but rather slightly decreased; the lightweight model, however, saw the expected improvement in inference speed on low-performance CPUs. This is because the padding-free downsampling provided in this application requires max pooling and bitwise addition, and the stem module also requires max pooling, along with channel-level addition. These operations are not within the scope of the floating-point operation count (FLOPs). For high-performance CPUs, the FLOPs are high, and although the strategy provided in this application reduces computation, the advantage of reducing computation is weakened, and the above operations incur additional time consumption, thus slightly reducing the speed. For low-performance CPUs, the FLOPs are low, and the advantage of reducing computation is reflected; even with additional time consumption, the overall speed is still improved.

[0160] like Figure 6 The diagram shows the model accuracy and computational cost of applying non-filling downsampling and the stem module to different models, as provided in this application.

[0161] To verify the effectiveness of padding-free downsampling and the stem module, this application also used a second set of hyperparameters and conducted ablation experiments on MobileNetv3, ParC_Mobilenet_v2, GhostNet, and ShuffleNetV2 to explore the specific gains of padding-free downsampling and the stem module.

[0162] Table 6 shows the experimental results of ablation experiments performed on CIFAR-100 using the second set of hyperparameters:

[0163] Table 6

[0164]

[0165] As shown in Table 6, directly using padding-free downsampling on some models slightly affects accuracy. Replacing the stem layer can compensate for this problem, resulting in consistent or slightly improved accuracy. Performance varies across different models. For example, Parc_MobileNet_v2 actually improves accuracy when using only padding-free downsampling because Parc_MobileNet_v2 does not suffer from insufficient edge information extraction; it can still extract edge information sufficiently even without padding. However, MobileNetv3's accuracy slightly decreases after adding the stem module because MobileNetv3 uses the optimal architecture derived through neural network architecture search, and replacing the module disrupts its original optimal architecture. Nevertheless, overall, the padding-free downsampling and stem module provided in this application are effective.

[0166] Based on the above image feature extraction method, this application embodiment also provides an image feature extraction device, which specifically includes:

[0167] The model acquisition module is used to acquire a pre-trained image feature extraction model. The image feature extraction model includes a stem module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit.

[0168] The initial feature map acquisition module is used to input image samples into the stem module for initial feature extraction and output the initial feature map.

[0169] The downsampling feature map acquisition module is used to input the initial feature map into the padding-free downsampling submodule in the first feature extraction module for downsampling. Specifically, it includes:

[0170] The first feature map acquisition submodule is used to input the initial feature map into the padding-free convolution unit in the padding-free downsampling submodule for convolution and output the first feature map.

[0171] The second feature map acquisition submodule is used to input the initial feature map into the no-padding max pooling unit in the no-padding downsampling submodule for max pooling, and output the second feature map; wherein...

[0172] The size of the first feature map is the same as the size of the second feature map;

[0173] The feature fusion submodule is used to input the first feature map and the second feature map into the feature fusion unit in the no-fill downsampling submodule for point-by-point addition and output the downsampled feature map;

[0174] The deep feature extraction module is used to input the downsampled feature map into the feature extraction convolution submodule in the first feature extraction module for deep feature extraction, and use the feature map after deep feature extraction as the input of the next feature extraction module;

[0175] The target feature map acquisition module is used to take the output of the feature extraction convolution submodule in the last feature extraction module as the target feature map of the image sample.

[0176] Furthermore, in some embodiments of this application, the initial feature map acquisition module includes:

[0177] The third feature map acquisition submodule is used to input image samples into the ordinary convolution submodule for ordinary convolution and output the third feature map.

[0178] The fourth feature map acquisition submodule is used to input the third feature map into the DW convolution submodule for DW convolution and output the fourth feature map.

[0179] The fifth feature map acquisition submodule is used to input the third feature map into the max pooling submodule for max pooling and output the fifth feature map; wherein, the size of the fourth feature map and the size of the fifth feature map are the same;

[0180] The channel fusion submodule is used to input the fourth and fifth feature maps into the channel fusion submodule for pointwise convolution and output the initial feature map.

[0181] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the image feature extraction method described above.

[0182] This application also provides an image classification method based on the above-described image feature extraction method, which specifically includes:

[0183] The target feature map of the image sample is extracted using the image feature extraction method described above;

[0184] The target feature map is input into a fully connected layer, and the classification result of the image sample is output.

[0185] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure One One or more processes and / or boxes Figure One A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure One One or more processes and / or boxes Figure One The function specified in one or more boxes.

[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure One One or more processes and / or boxes Figure One The steps of the function specified in one or more boxes.

[0189] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An image feature extraction method, characterized in that, include: A pre-trained image feature extraction model is obtained. The image feature extraction model includes a STEM module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit. Image samples are input into the stem module for initial feature extraction and an initial feature map is output; wherein, the stem module includes a regular convolution submodule, a DW convolution submodule, a max pooling submodule and a channel fusion submodule; The initial feature map is input into the no-padding downsampling submodule of the first feature extraction module for downsampling, which specifically includes: The initial feature map is input into the padding-free convolution unit in the padding-free downsampling submodule for convolution, and the first feature map is output. The initial feature map is input into the no-fill max pooling unit in the no-fill downsampling submodule for max pooling, and a second feature map is output; wherein, the size of the first feature map and the size of the second feature map are the same. The first feature map and the second feature map are input into the feature fusion unit in the no-fill downsampling submodule for point-by-point addition, and the downsampled feature map is output. The downsampled feature map is input into the feature extraction convolution submodule in the first feature extraction module for deep feature extraction, and the feature map after deep feature extraction is used as the input of the next feature extraction module. The output of the feature extraction convolution submodule in the last feature extraction module is used as the target feature map of the image sample.

2. The image feature extraction method according to claim 1, characterized in that, The step of inputting image samples into the stem module for initial feature extraction and outputting an initial feature map includes: The image sample is input into the ordinary convolution submodule for ordinary convolution, and the third feature map is output. The third feature map is input into the DW convolution submodule for DW convolution, and the fourth feature map is output. The third feature map is input into the max pooling submodule for max pooling, and the fifth feature map is output; wherein, the size of the fourth feature map and the size of the fifth feature map are the same; The fourth and fifth feature maps are input into the channel fusion submodule for pointwise convolution to output the initial feature map.

3. The image feature extraction method according to claim 1, characterized in that, After outputting the initial feature map, the method further includes: performing feature extraction on the initial feature map, outputting a target initial feature map, and using the target initial feature map as the input of the first feature extraction module.

4. The image feature extraction method according to claim 1, characterized in that, The training process of an image feature extraction model includes: Obtain image samples from the training set; The image samples are input into the stem module, which outputs an initial feature map. The initial feature map is input into multiple feature extraction modules with different kernel sizes that are sequentially connected along the forward propagation direction, and an image feature extraction loss function is constructed based on the output of the last feature extraction module. The STEM module and multiple feature extraction modules with different convolutional kernel sizes are iteratively trained using image samples from the training set until the value of the image feature extraction loss function is minimized, thus obtaining the trained STEM module and multiple feature extraction modules with different convolutional kernel sizes. Based on the trained stem module and multiple feature extraction modules with different convolutional kernel sizes, a trained image feature extraction model is obtained.

5. The image feature extraction method according to claim 4, characterized in that, The image feature extraction loss function is: , in, This represents the image feature extraction loss function. Indicates the number of image samples. For the sign function, when the first The true category of each image sample is hour, ,on the contrary, , For the first The image samples belong to the category The predicted probability, This represents the number of categories.

6. An image feature extraction device, characterized in that, To implement the image feature extraction method of claim 1, the method includes: The model acquisition module is used to acquire a pre-trained image feature extraction model. The image feature extraction model includes a stem module and multiple feature extraction modules with different kernel sizes connected in series along the forward propagation direction. Each feature extraction module includes a no-padding downsampling submodule and a feature extraction convolution submodule. Each no-padding downsampling submodule includes a no-padding convolution unit, a no-padding max pooling unit, and a feature fusion unit. The initial feature map acquisition module is used to input image samples into the stem module for initial feature extraction and output an initial feature map. The downsampling feature map acquisition module is used to input the initial feature map into the padding-free downsampling submodule in the first feature extraction module for downsampling, and specifically includes: The first feature map acquisition submodule is used to input the initial feature map into the no-padding convolution unit in the no-padding downsampling submodule for convolution and output the first feature map. The second feature map acquisition submodule is used to input the initial feature map into the no-fill max pooling unit in the no-fill downsampling submodule for max pooling and output the second feature map; wherein, the size of the first feature map and the size of the second feature map are the same. The feature fusion submodule is used to input the first feature map and the second feature map into the feature fusion unit in the no-fill downsampling submodule for point-by-point addition and output the downsampled feature map. The deep feature extraction module is used to input the downsampled feature map into the feature extraction convolution submodule in the first feature extraction module for deep feature extraction, and use the feature map after deep feature extraction as the input of the next feature extraction module; The target feature map acquisition module is used to take the output of the feature extraction convolution submodule in the last feature extraction module as the target feature map of the image sample.

7. The image feature extraction device according to claim 6, characterized in that, The initial feature map acquisition module includes: The third feature map acquisition submodule is used to input the image sample into the ordinary convolution submodule for ordinary convolution and output the third feature map; The fourth feature map acquisition submodule is used to input the third feature map into the DW convolution submodule to perform DW convolution and output the fourth feature map; The fifth feature map acquisition submodule is used to input the third feature map into the max pooling submodule for max pooling and output the fifth feature map; wherein, the size of the fourth feature map is the same as the size of the fifth feature map; The channel fusion submodule is used to input the fourth feature map and the fifth feature map into the channel fusion submodule for pointwise convolution and output the initial feature map.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image feature extraction method according to any one of claims 1-5.

9. An image classification method, characterized in that, include: The target feature map of the image sample is extracted using the image feature extraction method as described in any one of claims 1-5; The target feature map is input into a fully connected layer, and the classification result of the image sample is output.