Image processing method based on multi-scale multi-stage residual attention network

Abstract

The invention provides an image processing method based on a multi-scale and multi-stage residual attention network, and the method comprises the following steps: constructing a multi-scale and multi-stage residual attention network model in advance, the multi-scale and multi-stage residual attention network model comprising a preprocessing unit, a fusion residual attention unit and a post-processing unit; obtaining a to-be-processed finger vein image, and performing linear feature extraction and filtering on the to-be-processed finger vein image by using the preprocessing unit to obtain a target feature of a finger vein; fusing the target features of the finger veins by using the fusion residual attention module to obtain fusion features of the finger veins; and the post-processing unit is used for processing the finger vein fusion features, so that a feature map is mapped to an interval from 0 to 1, and a result with the maximum weight is output. Therefore, various types of image features can be effectively extracted, and both accuracy and speed are considered.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image processing method based on a multi-scale and multi-stage residual attention network. Background Art

[0002] Finger vein image processing is an important branch of computer vision and an important research direction in the field of computer vision. It aims to analyze, process and understand finger vein images so that computers can understand and identify finger vein features.

[0003] At present, the field of finger vein image processing faces some challenges and key problems, mainly including:

[0004] (1) Image classification and recognition: There are a large number of categories and variations in finger vein images, such as different objects, different angles, and different lighting conditions. Therefore, effective image classification and recognition algorithms need to have robustness and generalization capabilities for different categories;

[0005] (2) Object detection and localization: Accurately detecting and localizing objects in finger vein images is a challenging task, which involves modeling the changes in the shape, scale, posture, and background of the object to achieve accurate object detection and localization;

[0006] (3) Image segmentation: Image segmentation is the process of dividing an image into regions with semantic meaning. Objects in natural images usually have complex shapes and textures, and they may overlap, occlude, or be similar to the background, which increases the challenge of image segmentation;

[0007] (4) Image denoising and enhancement: Noise and distortion often exist in finger vein images, which may come from image acquisition equipment, compression algorithms or other environmental factors. Therefore, image denoising and enhancement are important issues in natural image processing, aiming to improve image quality and clarity.

[0008] (5) Feature extraction and representation: Objects and scenes in finger vein images usually have rich features and structures. Therefore, effective feature extraction and representation methods are crucial for finger vein image processing. This includes techniques such as local feature extraction, context modeling, and multi-scale analysis.

[0009] Although deep learning and other technologies have been applied in the field of finger vein image processing, many recognition methods are difficult to perform well on small-scale finger vein datasets due to their overly complex network structures. Therefore, how to balance the relationship between speed and accuracy is a problem worth studying.

[0010] In order to solve the above problems, people have been seeking an ideal technical solution. Summary of the invention

[0011] The purpose of the present invention is to address the deficiencies of the prior art and to provide an image processing method based on a multi-scale and multi-stage residual attention network.

[0012] In order to achieve the above object, the first aspect of the present invention provides an image processing method based on a multi-scale multi-stage residual attention network, the method comprising the following steps:

[0013] Pre-constructing a multi-scale multi-stage residual attention network model, wherein the multi-scale multi-stage residual attention network model includes a pre-processing unit, a fusion residual attention unit and a post-processing unit;

[0014] Acquire a finger vein image to be processed, and use the preprocessing unit to extract and filter linear features of the finger vein image to be processed to obtain target features of the finger vein;

[0015] Using the fusion residual attention module to fuse the target features of the finger vein, a fusion feature of the finger vein is obtained;

[0016] The post-processing unit is used to process the finger vein fusion feature, so that the feature map is mapped to the interval of 0 to 1, and the result with the maximum weight is output.

[0017] In order to achieve the above-mentioned objectives, the second aspect of the present invention provides a lightweight image processing device based on a multi-scale multi-stage residual attention network, comprising: a memory and a processor coupled to the memory, the processor being configured to execute the program of the above-mentioned image processing method based on the multi-scale multi-stage residual attention network based on instructions stored in the memory.

[0018] In order to achieve the above-mentioned objectives, the third aspect of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned image processing method based on a multi-scale multi-stage residual attention network is implemented.

[0019] The beneficial effects of the present invention are:

[0020] In order to adapt to the characteristics of low resolution, grayscale pixels and linear patterns of natural images, the present invention introduces a residual attention mechanism and proposes a multi-scale and multi-stage residual attention network model. By fusing the two paths in the residual attention unit, the image features extracted at different learning stages are integrated, which can effectively extract various types of image features and combine image features of different dimensions, further improving the model performance of extracting high-level abstract features. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the overall framework diagram of the multi-scale and multi-stage residual attention network model of the present invention;

[0022] Figure 2 It is a schematic diagram of the topological structure of the pre-processing unit of the present invention;

[0023] Figure 3 It is a schematic diagram of the topological structure of the main feature extraction module MVP of the present invention that integrates the residual attention unit;

[0024] Figure 4 Schematic diagram of the topological structure of the guided attention module GAP of the present invention that integrates the residual attention unit;

[0025] Figure 5 It is a schematic diagram of the topological structure of the inception module of the post-processing unit of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described in detail below through specific implementation methods.

[0027] Example 1

[0028] This embodiment provides a specific implementation method of an image processing method based on a multi-scale and multi-stage residual attention network.

[0029] As attached Figure 1 As shown, the image processing method based on the multi-scale multi-stage residual attention network includes the following steps:

[0030] Pre-constructing a multi-scale multi-stage residual attention network model, wherein the multi-scale multi-stage residual attention network model includes a preprocessing unit, a fused residual attention unit FRAB, and a post-processing unit;

[0031] Acquire a finger vein image to be processed, and use the preprocessing unit to extract and filter linear features of the finger vein image to be processed to obtain target features of the finger vein;

[0032] Using the fusion residual attention module to fuse the target features of the finger vein, a fusion feature of the finger vein is obtained;

[0033] The post-processing unit is used to process the finger vein fusion feature, so that the feature map is mapped to the interval from 0 to 1, and the result with the maximum weight is output.

[0034] It should be noted that the multi-scale multi-stage residual attention network model has fewer trainable parameters and a small size, which shortens the model recognition time and does not affect the model performance; in addition, the newly added multi-scale feature extraction structure and multi-stage feature learning method can transform the network from the previous single linear feature learning to multi-scale vein feature learning, so that the model learns the features of multiple receptive fields across layers, further improving the accuracy of finger vein recognition.

[0035] In some embodiments, the preprocessing unit includes a convolution layer, a maximum pooling layer, and a residual unit connected in sequence, wherein the residual unit includes a normalization layer, an activation function, and a 1x1 convolution three-layer cascade input and output to construct a residual connection, such as Figure 2 As shown;

[0036] The convolution layer of the preprocessing unit is used to extract the shallow linear features of the finger vein image to be processed; the maximum pooling layer of the preprocessing unit is used to filter the extracted shallow linear features; the residual block of the preprocessing unit is used to map the filtered shallow linear features into a nonlinear pattern;

[0037] It can be understood that the preprocessing unit roughly extracts the main texture features of the finger vein image to be processed and reduces the dimension of the features; the pre-activated residual block Residual Unit performs nonlinear activation on the feature map and reduces the feature variance through normalization.

[0038] In some embodiments, the fused residual attention unit includes a main feature extraction module MVP, a guided attention module GAP and a multi-stage residual attention connection module MRAC. The input end of the main feature extraction module MVP and the input end of the guided attention module GAP are respectively connected to the output end of the preprocessing unit through a residual block; between the output end of the guided attention module GAP and the input end of the multi-stage residual attention connection module MRAC, there are provided a convolutional layer I, a convolutional layer II and a Softmax function module connected in sequence; the output end of the guided attention module GAP is connected to the input end of the multi-stage residual attention connection module MRAC; the multi-stage residual attention connection module MRAC is also respectively connected to the residual block of the fused residual attention unit and the residual block of the preprocessing unit.

[0039] It can be understood that the fused residual attention unit is a dual-path structure consisting of a main feature path and a guided attention path. The main feature path uses a multi-branch residual structure to extract finger vein features at multiple scales, and the guided attention path uses an "hourglass network" to generate a weight map to guide the weight setting of the corresponding position in the feature map.

[0040] As attached Figure 3 As shown in the figure, the overall structure of the main feature path is a separate initialization-residual network unit (DIRU). Compared with the residual unit, the DIRU uses filters of different fields to extract image features at multiple scales and can capture image details at different levels. Each subpath passes through the convolution layer, normalization (BN) layer, and activation function (RELU) layer in sequence. The convolution layer extracts the linear features of the image, while the BN layer calculates the mean and variance of each small batch of feature maps on the same channel for normalization. By reducing the distribution difference between different sample features in each small batch, the distribution of samples in the feature space is more uniform, avoiding the gradient vanishing or gradient exploding phenomenon during the back propagation process, accelerating the convergence speed of the model, reducing the training error, and thus improving the model recognition ability. The processing process of the batch normalization layer can be described by the following formula:

[0041]

[0042]

[0043]

[0044] Where E[x] and V[x] are μ(B) and σ respectively over all training mini-batches. 2 (B) average; μ(B) and σ 2 (B) represents the mean and variance of each mini-batch B containing m images I, ∈ is an empirical constant to prevent calculation errors caused by variance being 0; β is a feature shift factor, and γ is a feature scaling factor. After the normalization operation, the feature distribution will be too similar, and the feature shift factor β and the feature scaling factor γ will help the network avoid the performance degradation caused by this phenomenon. β and γ are two hyperparameters, and their values ​​will be gradually adjusted during the network training process until they converge to the optimal value.

[0045] As attached Figure 4 As shown, the attention-guided path adopts a symmetrical sampling structure to extract image features at each scale and combine them;

[0046] The upper part of the hourglass module implements downsampling, sampling the input features to the lowest resolution (such as 4×4 pixels). This bottom-up extraction method collects finger vein features of different scales and compresses the size of the feature map from 32×32 to 4×4. Since the extraction structure consists of multiple filters of different sizes, finger vein features of different scales can be extracted from the entire image; therefore, while reducing the feature dimension, the feature still contains the global information of the finger vein image to be processed;

[0047] The lower part of GAP performs the opposite operation (upsampling) on ​​the features. Its top-down structure also includes convolutional layers and maximum pooling layers, which are responsible for continuously reconstructing feature units until the size of the output feature map is the same as the input size (upsampling from 4×4 to 32×32).

[0048] Between the upper and lower parts of the path, a skip connection is used for cross-layer connection of the feature information. Between the upper and lower parts of the path, a skip connection is used for cross-layer connection of the finger vein feature information.

[0049] It should be noted that the two convolutional layers and one Sigmoid layer in the fused residual attention unit generate pixelated attention weights (attention weights), further filter the features and map the output results to the range of 0 to 1 as shown in the following formula:

[0050]

[0051]

[0052] Among them, f m (i,j) represents the input feature at position (i, j) on the mth channel in the symmetric sampling structure. After downsampling operation DS(·) and upsampling operation US(·), the output feature z at the corresponding position can be obtained. m (i,j) .f m GAP (i, j) represents the GAP output feature at position (i, j) on the mth channel after two layers of convolution kernels and one layer of sigmoid.

[0053] It can be understood that the multi-stage residual attention connection module MRAC uses branch modules and multi-stage residual attention connection modules to perform nonlinear activation on the feature map and reduce the feature variance through normalization;

[0054] The branch module is to converge the output of the main feature path and the attention-guided path by calculating the elementary product of the two features, with two paths having the same output dimension (such as 32×32×32). Each value (from 0 to 1) generated by the attention-guided path can be regarded as the weight of the feature at each corresponding position in the main feature path, as described by the following formula:

[0055]

[0056] Among them, F fuse It refers to the aggregated features obtained by aggregating the two types of features, MVP and GAP, by pixel-by-pixel multiplication; W GAP is the guide path feature (the feature map that guides the attention module GAP), for W GAP Each element in, for example, f i Gap (x, y), i∈(1, m), the feature value of the position (x, y) in the i-th channel is used as an attention weight to activate the feature value of the corresponding position on the MVP output feature map (such as f i MVP (x, y)); X MVP It is a set of main feature path outputs containing the entire features of each image. The larger the weight, the greater the feature response at the corresponding position;

[0057] In order to balance the features of each part, the multi-level residual attention connection module uses three feature factors a1, a2, and a3 to control three different feature responses, converting the original input feature x in , pre-activation feature x pa , MVP feature M(x), do a whole direct mapping; when the GAP feature G(x) is close to 0, O(x) still contains enough features to prevent the reduction of effective information. The final feature mapping function is as follows:

[0058] O(x)=aT×xin+a2×x pa +a3×(1+G(x))M(x)

[0059] Among them, after multiple experiments, a1, a2, and a3 are determined to be 0.5, 0.3, 0.2, and x in is the original input feature of the module, in x in After reaching the residual unit, a pre-activated feature map can be obtained, using x pa express.

[0060] In some embodiments, the post-processing unit includes a residual block I, a residual block II, an inception module, a maximum pooling layer, a residual block III, an average pooling layer I, an average pooling layer II and a Softmax function module connected in sequence; wherein the topological structure diagram of the inception module is as shown in the attached Figure 5 As shown;

[0061] It can be understood that when reaching the flattening layer, the post-processing unit flattens the two-dimensional feature map into a one-dimensional feature vector, and then the feature vector is mapped into a probability value in the interval [0,1] under the action of the Softmax function, and finally the output layer outputs the probability distribution of N samples in C categories.

[0062] Example 2

[0063] The present embodiment provides a lightweight image processing device based on a multi-scale multi-stage residual attention network, which includes: a memory and a processor coupled to the memory, and the processor is configured to execute the program of the image processing method based on the multi-scale multi-stage residual attention network described in Example 1 based on instructions stored in the memory.

[0064] The memory may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0065] The device may also include an input / output interface, a network interface, a storage interface, etc. These interfaces, the memory, and the processor may be connected, for example, via a bus. The input / output interface provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen, the network interface provides a connection interface for various networked devices, and the storage interface provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0066] The experimental platform of this embodiment is Ubuntu system, the graphics card is Nvidia RTX 3090Ti, the video memory is 24G, and the processor is Intel(R) Xeon(R) Silver CPU E5-2678 v3@2.50GHz. The experimental model is implemented based on the pytorch framework.

[0067] In order to obtain the best recognition results, this example uses the ROI version of the finger vein images of the four datasets (on the HKPU dataset, the original images are used for model training). The input layer of the model receives 64×64 pixel images; therefore, all images are normalized to the required resolution before being input into the model;

[0068] In the experiment, each finger is identified as a separate class. Therefore, SDUMLA has 636 classes, FV-USM 492 classes, MMCBNU 600 classes, and UTFVP 360 classes, and the number of available samples for each class varies from dataset to dataset. For SDUMLA, the first five images are selected as training data, and the remaining images are used as test data, so that the model can learn as many vein features as possible given the limited amount of data for each class. Similarly, for FVUSM, 8 images from two stages are used for training, and 4 images are used for testing. For MMCBNU, 60% are used for training and 40% for testing. For UTFVP, since there are only 4 image samples per class, this embodiment applies data augmentation techniques (brightness transformation and Gaussian noise) to expand the number of images and avoid overfitting the model on the training set. After data augmentation, each class has a total of 14 images for training and 2 images for testing. Specifically, in the dataset of HKPU, the first 105 subjects were collected on two stages, but the last 51 subjects were collected on only one stage. In order to keep the data balanced, this example selects samples from the first 105 subjects for model training. Therefore, HKPU has 210 classes, and this example selects 4 images from each stage for training and the last 2 images for testing.

[0069] In terms of model training, the batch size is set according to the number of samples in each class. For SDUMLA, the batch size is 5. The batch sizes on the other four datasets are set as follows: FV-USM and HKPU are set to 8, MMCBNU is set to 6, and UTFVP is set to 14. In order to improve the accuracy of the model and ensure that it can converge with minimal loss, 500 rounds of network training were performed on the data. ADAM was selected as the optimizer for network training, and the initial learning rate (LR) value was 0.0001. During the training process, the learning rate is reduced by 10 times for every 20% of the total epochs completed. This dynamic adjustment helps keep the gradient moving steadily downward. Before the model converges, the accuracy and loss of each epoch will not change excessively.

[0070] In order to verify the performance advantage of the proposed network, this embodiment compares the proposed network with some SOTA models on 5 public data sets, such as ResNet50, FVCNN, ViT, etc., as shown in the following table:

[0071]

[0072] It can be clearly seen from the results in the table that, compared with other networks, this embodiment has a huge advantage in recognition accuracy. At the same time, in terms of the number of model parameters, the model of this embodiment is very close to that of the lightweight network.

[0073] Example 3

[0074] This embodiment provides a non-transitory computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method based on the multi-scale multi-stage residual attention network described in Example 1 is implemented.

[0075] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer non-transient readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0076] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0077] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. An image processing method based on a multi-scale multi-stage residual attention network, characterized in that: The following steps are involved: Pre-constructing a multi-scale multi-stage residual attention network model, wherein the multi-scale multi-stage residual attention network model includes a pre-processing unit, a fusion residual attention unit and a post-processing unit; Acquire a finger vein image to be processed, and use the preprocessing unit to extract and filter linear features of the finger vein image to be processed to obtain target features of the finger vein; Using the fusion residual attention module to fuse the target features of the finger vein, a fusion feature of the finger vein is obtained; The post-processing unit is used to process the finger vein fusion feature and output a result with the maximum weight.

2. The image processing method based on the multi-scale multi-stage residual attention network according to claim 1, characterized in that: The preprocessing unit includes a convolution layer, a maximum pooling layer and a residual block connected in sequence. The convolution layer of the preprocessing unit is used to extract the shallow linear features of the finger vein image to be processed, the maximum pooling layer of the preprocessing unit filters the extracted shallow linear features, and the residual block of the preprocessing unit is used to map the filtered shallow linear features into a nonlinear pattern.

3. The image processing method based on the multi-scale multi-stage residual attention network according to claim 1, characterized in that: The post-processing unit includes a residual block I, a residual block II, an inception module, a maximum pooling layer, a residual block III, an average pooling layer I, an average pooling layer II and a Softmax function module which are connected in sequence.

4. The image processing method based on multi-scale multi-stage residual attention network according to claim 1, characterized in that: The fusion residual attention unit includes a main feature extraction module MVP, a guided attention module GAP and a multi-stage residual attention connection module MRAC. The input end of the main feature extraction module MVP and the input end of the attention guidance module GAP are respectively connected to the output end of the preprocessing unit through a residual block; Between the output end of the guided attention module GAP and the input end of the multi-stage residual attention connection module MRAC, there are sequentially connected convolution layer I, convolution layer II and Softmax function module; The output end of the guided attention module GAP is connected to the input end of the multi-stage residual attention connection module MRAC; The multi-stage residual attention connection module MRAC is also connected to the residual block of the fused residual attention unit and the residual block of the preprocessing unit respectively.

5. A lightweight image processing device based on a multi-scale multi-stage residual attention network, comprising: A memory and a processor coupled to the memory, wherein the processor is configured to execute a program of an image processing method based on a multi-scale and multi-stage residual attention network as described in any one of claims 1 to 4 based on instructions stored in the memory.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method based on a multi-scale and multi-stage residual attention network as described in any one of claims 1 to 4.

Images