Image fusion recognition method, device, medium and equipment based on ultra-lightweight network

Abstract

The application relates to the technical field of biometric image recognition, and discloses an image fusion recognition method, device, medium and equipment based on an ultralight network, the method comprising the following steps: adopting an improved Co3Net network module to perform feature extraction on inputted multi-modal images to obtain multi-scale features; adopting an ultralight network module to perform feature extraction on the inputted multi-scale features to obtain global features; adopting a self-attention mechanism module to draw a self-attention graph according to the inputted global features; adopting a forward propagation unit to perform forward propagation on the self-attention graph to obtain final forward propagation output; and adopting a classifier to obtain a multi-modal image classification label according to the final forward propagation output. The application can reduce the calculation complexity and parameter quantity of a palm print and palm vein recognition method, and solves the problem that the method is difficult to efficiently run on resource-limited embedded devices and mobile terminals.

Description

Technical Field

[0001] This invention relates to the fields of image processing and biometrics, and in particular to an image fusion recognition method, apparatus, medium and device based on an ultralight network. Background Technology

[0002] Currently, biometric identification technology is widely used as a secure and reliable means of identity authentication. Palmprint and palm vein fusion recognition is receiving increasing attention due to its unique advantages. Palmprints and palm veins, as inherent physiological characteristics of the human body, possess uniqueness, anti-counterfeiting features, and stability. Palm vein patterns are hidden deep within the skin tissue, making them difficult to copy and forge; palmprint features are rich, contain a large amount of information, and are easy to collect. The combination of palmprint and palm vein biometric features can effectively improve the security and accuracy of the identification system.

[0003] With the development of deep learning technology, significant progress has been made in palmprint and palm vein recognition methods based on deep neural networks. Deep learning models, especially convolutional neural networks (CNNs), can automatically learn and extract optimal features from data, avoiding the limitations of manually designed features in traditional methods. Researchers have proposed various deep learning-based recognition methods, such as palmprint recognition methods based on improved convolutional neural networks and feature extraction networks incorporating attention mechanisms. These methods have achieved good recognition accuracy, but their computational complexity and number of parameters are large, making them difficult to deploy in practical applications, especially on resource-constrained embedded devices or mobile terminals.

[0004] To address the high computational complexity of deep learning models, researchers have conducted studies on model lightweighting. Current mainstream lightweighting methods include network pruning, knowledge distillation, and low-rank decomposition. These methods have reduced the computational complexity and number of parameters in deep learning models to some extent. However, the application of existing lightweighting methods in palmprint and palm vein recognition still faces some challenges. First, while reducing the number of model parameters and computational complexity, lightweighting methods may lead to a decrease in feature extraction capabilities, especially when processing multispectral images, potentially affecting recognition accuracy and robustness. Second, existing lightweighting methods have limited research on multimodal information fusion, failing to fully utilize the complementarity of multimodal data and limiting the model's recognition capabilities.

[0005] Therefore, how to construct ultra-lightweight networks to achieve multimodal image fusion recognition, and improve the model's recognition ability and robustness, making it suitable for resource-constrained embedded devices and mobile terminals, has become a technical problem that needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing palm print and palm vein recognition methods in terms of computational complexity and number of parameters, particularly the difficulty in efficient operation on resource-constrained embedded devices and mobile terminals. This invention provides an image fusion recognition method, device, medium, and apparatus based on ultra-lightweight networks. Addressing the real-time recognition needs in practical application scenarios, this invention significantly reduces the computational complexity and number of parameters of the model by combining depthwise separable convolution, channel pruning, and self-attention mechanisms, while maintaining high-precision recognition performance. It is suitable for resource-constrained embedded devices and mobile terminals.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] According to one aspect of the present invention, an image fusion and recognition device based on an ultra-lightweight network is provided, comprising: an improved Co3Net network module, an ultra-lightweight network module, a self-attention mechanism module, a forward propagation unit, and a classifier; wherein,

[0009] The improved Co3Net network module is configured to extract features from the input multimodal image to obtain multi-scale features;

[0010] The ultra-lightweight network module is configured to extract features from the multi-scale features of the input to obtain global features;

[0011] The self-attention mechanism module is configured to draw a self-attention map based on the global features of the input;

[0012] The forward propagation unit is configured to propagate the self-attention map forward to obtain the final forward propagation output;

[0013] The classifier is configured to obtain multimodal image classification labels based on the final forward propagation output;

[0014] The improved Co3Net network module is obtained by pruning a baseline Co3Net network model trained using Co3Net network on multimodal images, wherein the standard convolutional layers are replaced with a combination of sequentially connected depthwise separable convolutional layers and point convolutional layers.

[0015] According to one embodiment of the present invention, the depth-separable convolutional layer is configured to extract features independently using a convolutional kernel for each input channel;

[0016] The point convolutional layer is configured to perform point-wise convolution on the features of each input channel and use a 1×1 convolutional kernel to integrate the information between channels.

[0017] According to one embodiment of the present invention, the ultralight network module includes m standard convolutional layers, n sets of sequentially connected combinations of depthwise separable convolutional layers and point convolutional layers, batch normalization layers, activation layers and pooling layers; m takes the value of 1-3 and n takes the value of 3-5.

[0018] According to an embodiment of the present invention, the improved Co3Net network module includes an input layer and multiple parallel competing channel units. Each competing channel unit is obtained by pruning the corresponding competing channel unit according to the baseline Co3Net network model. Specifically, the pruning method is as follows: the L1 norm value of each competing channel unit is calculated as an importance index, and pruning is performed according to the importance of each competing channel unit.

[0019] According to one embodiment of the present invention, the self-attention mechanism module divides the input global features into multiple heads for parallel computation, captures remote dependencies and global context information, obtains multiple attention maps, concatenates the multiple attention maps and maps them back to the original feature space through a projection matrix to obtain a self-attention map.

[0020] On the other hand, the present invention also provides an image fusion recognition method based on an ultra-lightweight network according to the above-described device, comprising the following steps:

[0021] An improved Co3Net network module is used to extract features from the input multimodal image to obtain multi-scale features;

[0022] An ultra-lightweight network module is used to extract features from the input multi-scale features to obtain global features;

[0023] A self-attention mechanism module is used to generate a self-attention map based on the global features of the input;

[0024] The self-attention map is forward-propagated using a forward propagation unit to obtain the final forward propagation output;

[0025] A classifier is used to obtain multimodal image classification labels based on the final forward propagation output.

[0026] According to an embodiment of the present invention, the improved Co3Net network module is obtained through the following steps:

[0027] A baseline Co3net network model is obtained by training the Co3net network using multimodal images. The baseline Co3net network model includes multiple parallel competing channel units.

[0028] The standard convolution operation of the baseline Co3net network model is replaced with depthwise separable convolution and pointwise convolution.

[0029] Based on the L1 norm, channel importance analysis is performed on the competing channel units of the baseline Co3net network model, and each competing channel unit is pruned according to its channel importance.

[0030] According to one embodiment of the present invention, the method further includes: after pruning each competing channel unit according to channel importance, fine-tuning the baseline Co3net network model to obtain an improved Co3Net network module.

[0031] On the other hand, a computer storage medium is also provided, wherein instructions are stored in the computer storage medium, and when the instructions are executed, the above-described image fusion recognition method based on ultra-lightweight networks is implemented.

[0032] On the other hand, a computing device is also provided, including a processor and a communication interface coupled to the processor; the processor is used to run computer programs or instructions to implement the above-described image fusion recognition method based on ultra-lightweight networks.

[0033] The image fusion and recognition method using an ultra-lightweight network described in this invention can achieve palm print and palm vein fusion and recognition, and has the following advantages compared with existing technologies:

[0034] 1. The lightweight network on which this invention is based has a significant lightweight effect, reducing the number of model parameters from 150.4M to 26.63M, a reduction of 82.3%, while maintaining a recognition accuracy of 99.50% on the Tongji dataset;

[0035] 2. The lightweight network upon which this invention is based can significantly reduce computational complexity and the number of parameters, thereby significantly improving inference speed and making it more suitable for deployment on mobile terminals and embedded devices.

[0036] 3. The present invention has a good feature fusion effect. The improved feature fusion module can make full use of the complementary information of palm prints and palm veins, and has better model recognition ability and robustness. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0038] Figure 1 This is a schematic diagram of an image fusion and recognition device based on an ultralight network according to an exemplary embodiment of the present invention.

[0039] Figure 2 This is a flowchart of an image fusion and recognition method based on an ultralight network according to an exemplary embodiment of the present invention.

[0040] Figure 3 This is a graph showing the change in recognition accuracy and parameter quantity during the improved benchmark model process according to an exemplary embodiment of the present invention.

[0041] Figure 4 This is a comparison chart of the recognition accuracy and model parameter quantity of different lightweight methods obtained on Tongji according to an exemplary embodiment of the present invention.

[0042] Figure 5 This is a comparison chart of the recognition accuracy and model parameter count of different lightweight methods obtained on CASIA according to an exemplary embodiment of the present invention. Detailed Implementation

[0043] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0044] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0045] In this invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one" or similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, "at least one of a, b, or c" can represent: a, b, c, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0046] This invention provides an image fusion recognition method, apparatus, medium, and device based on ultra-lightweight networks. The method significantly reduces the computational complexity and number of parameters of the model by combining depthwise separable convolution, channel pruning, and self-attention mechanisms, while maintaining high-precision recognition performance. Furthermore, the deep fusion of palmprint and palm vein image information during the feature extraction stage further improves the model's recognition ability and robustness, making it suitable for resource-constrained embedded devices and mobile terminals.

[0047] like Figure 1 As shown, the image fusion and recognition device based on ultra-lightweight networks includes: an improved Co3Net network module, an ultra-lightweight network module, a self-attention mechanism module, a forward propagation unit, and a classifier;

[0048] The improved Co3Net network module is configured to extract features from the input multimodal image to obtain multi-scale features;

[0049] The ultra-lightweight network module is configured to extract features from the multi-scale features of the input to obtain global features;

[0050] The self-attention mechanism module is configured to draw a self-attention map based on the global features of the input;

[0051] The forward propagation unit is configured to propagate the self-attention map forward to obtain the final forward propagation output;

[0052] The classifier is configured to obtain multimodal image classification labels based on the final forward propagation output;

[0053] The improved Co3Net network module is obtained by pruning a baseline Co3Net network model trained using Co3Net on multimodal images. The standard convolutional layers are replaced with a combination of sequentially connected depthwise separable convolutional layers and point convolutional layers. The depthwise separable convolutional layers are configured to extract features independently using convolutional kernels for each input channel; the point convolutional layers are configured to perform pointwise convolution on the features of each input channel, using 1×1 convolutional kernels to integrate information between channels.

[0054] The expression for the depth-separable convolutional layer is:

[0055] y = DepthwiseConv(x) (1)

[0056] Where x is the input feature map and y is the output of the depthwise separable convolutional layer;

[0057] The expression for the point convolutional layer is:

[0058] y′=PointwiseConv(y) (2)

[0059] Where y is the output of the depthwise separable convolutional layer, and y' is the output of the point convolutional layer;

[0060] Each depthwise separable convolutional layer is followed by a point convolutional layer, which has the advantage of learning spatial features without introducing too much computational complexity.

[0061] The ultra-lightweight network module includes m standard convolutional layers, n sets of sequentially connected combinations of depthwise separable convolutional layers and point convolutional layers, batch normalization layers, activation layers, and pooling layers; m takes values ​​of 1-3, and n takes values ​​of 3-5. The standard convolutional layers use 3×3 convolutional kernels, combined with batch normalization and the SiLU activation function.

[0062] The self-attention mechanism module divides the input global features into multiple heads for parallel computation, captures remote dependencies and global context information, obtains multiple attention maps, concatenates the multiple attention maps and maps them back to the original feature space through a projection matrix to obtain a self-attention map.

[0063] The self-attention mechanism module divides the input into multiple heads for parallel computation. Each head computes a different attention map, including a multi-head self-attention mechanism to capture remote dependencies and global context information, comprising the following steps:

[0064] 1) Perform a linear transformation on the input to obtain the query Q, key K, and value V;

[0065] Q,K,V=Linear(X) (3)

[0066] Where X is the input to the self-attention mechanism module;

[0067] 2) Calculate the attention score Attn(Q,K) between query Q and key K:

[0068]

[0069] 3) Perform a weighted summation on the value V:

[0070] Output = Attn(Q,K)V (5)

[0071] 4) Concatenate the attention maps and map them back to the original feature space using a projection matrix:

[0072] MultiHead(Q,K,V)=Concat(head1,…,headh)WO (6)

[0073] Where h is the number of attention heads, and WO is the learnable projection matrix; h can be 4-16.

[0074] The output of the self-attention mechanism module is forward-propagated, and then mapped through an adaptive average pooling layer and a fully connected layer to obtain the classification result, which is expressed as follows:

[0075] Y=Head(ReLU(FC1(AdaptiveAvgPool(x)))) (7)

[0076] Where Y represents the classification result.

[0077] The improved Co3Net network module includes an input layer and multiple parallel competing channel units. Each competing channel unit is obtained by pruning the corresponding competing channel unit of the baseline Co3Net network model. Specifically, the pruning method involves calculating the L1 norm value of each competing channel unit as an importance index and pruning according to the importance of each competing channel unit. Furthermore, the pruned network model can be fine-tuned and optimized.

[0078] like Figure 2 As shown, the image fusion and recognition method based on ultra-lightweight networks is characterized by the following steps:

[0079] Step S1: Use the improved Co3Net network module to extract features from the input multimodal image to obtain multi-scale features;

[0080] Step S2: Use an ultra-lightweight network module to extract features from the input multi-scale features to obtain global features;

[0081] Step S3: Use the self-attention mechanism module to draw a self-attention map based on the input global features;

[0082] Step S4: Use a forward propagation unit to propagate the self-attention map forward to obtain the final forward propagation output;

[0083] Step S5: Use a classifier to obtain multimodal image classification labels based on the final forward propagation output.

[0084] The improved Co3Net network module is obtained through the following steps:

[0085] A baseline Co3net network model is obtained by training the Co3net network using multimodal images. The baseline Co3net network model includes multiple parallel competing channel units.

[0086] The standard convolution operation of the baseline Co3net network model is replaced with depthwise separable convolution and pointwise convolution.

[0087] Based on the L1 norm, channel importance analysis is performed on the competing channel units of the baseline Co3net network model, and each competing channel unit is pruned according to its channel importance.

[0088] The method further includes: after pruning each competing channel unit according to channel importance, fine-tuning the baseline Co3net network model to obtain an improved Co3Net network module.

[0089] The multimodal images include homologous palm print and palm vein images, or homologous fundus and ocular vein images.

[0090] Taking homologous palm print and palm vein images as an example, the palm print and palm vein fusion recognition method based on a lightweight network includes the following steps:

[0091] S10. Input the homologous palm print and palm vein images into a standard convolutional layer of the improved Co3Net network module to obtain the shallow palm print features and the shallow palm vein features.

[0092] S20: The palm print and palm vein superficial features obtained in S10 are spliced ​​together to obtain the spliced ​​superficial features.

[0093] S30. Input the spliced ​​shallow features into the depth-separable convolutional layer and pointwise convolutional layer of the improved Co3Net network module to obtain multi-scale features;

[0094] S40. Input multi-scale features into the ultra-lightweight network module, and extract global features from the concatenated shallow features through a series of convolutional layers, batch normalization, activation functions and pooling layers; the series of convolutional layers includes standard convolutional layers, depthwise separable convolutional layers and point convolutional layers; wherein, the standard convolutional layer uses a 3×3 convolutional kernel, combined with batch normalization and SiLU activation function;

[0095] S50. Divide the global feature input into multiple heads for parallel computation, capture long-range dependencies and global context information, obtain multiple attention maps and concatenate them, specifically including the following sub-steps:

[0096] S501: Multiple attention heads perform a linear transformation on the input in parallel to obtain query Q, key K and value V;

[0097] S502. Calculate the attention score between multiple attention head queries Q and key K to obtain multiple attention scores;

[0098] S503. Weighted summation of value V based on attention score to obtain multiple attention maps;

[0099] S504. Concatenate multiple attention maps and map them back to the original feature space using a projection matrix to obtain a self-attention map;

[0100] S60. Perform forward propagation on the self-attention map to obtain the final forward propagation output, and then map it through an adaptive average pooling layer and a fully connected layer to obtain the classification result.

[0101] Example 1: Training a baseline Co3Net network model

[0102] In this embodiment, the Tongji palm print and palm vein dataset and the CASIA multispectral palm print dataset are used to train a deep learning model to obtain a benchmark Co3Net network model. The relevant information of the dataset is shown in Table 1.

[0103] Table 1: Palmprint and Palmar Vein Dataset

[0104]

[0105] The Tongji palm print and palm vein dataset comprises 24,000 palm images from 300 individuals collected using a non-contact method. Each palm was captured twice, with 10 images captured each time, and the average interval between the two captures was 61 days. The image size was 800×600 pixels.

[0106] The CASIA multispectral palmprint dataset comprises 7,200 palm images of 100 individuals captured using a self-designed multispectral imaging device. For each hand, two sets of palm images were captured, with an interval of over one month between the two acquisitions. Three samples were acquired each time, and these images were collected under six different electromagnetic spectra, corresponding to illuminator wavelengths of 460nm, 630nm, 700nm, 850nm, 940nm, and white light, with an image size of 768×576.

[0107] In this embodiment, the two datasets mentioned above are used. As shown in Table 2, for the Tongji dataset, all palm print and vein images were used. For the CASIA dataset, palm print images were taken at 460nm and 630nm, and palm vein images were taken at 850nm and 940nm. This embodiment uses the Tongji and CASIA palm print and palm vein databases for experimental verification.

[0108] Table 2: Palmprint and palm vein dataset used in the experiment

[0109]

[0110] A baseline Co3Net network model was obtained by training the Co3Net network using a palm print and palm vein dataset, including the following steps:

[0111] S101: In the data preprocessing stage, all images are first scaled to 224×224 and standardized, with a mean of 0.5 and a standard deviation of 0.5.

[0112] S102: To ensure the generalization ability of the model, data augmentation methods such as random horizontal flipping, random rotation, and random brightness adjustment (±0.2) are used to expand the training samples; the dataset is randomly divided into training set and test set in an 8:2 ratio;

[0113] S103: In the model analysis phase, the baseline Co3net network structure is first analyzed in depth. The input layer of the Co3net network receives a 224×224×6 palm print and palm vein fusion image, and extracts local texture features through the first Gabor convolutional layer (Kernel_size=3×3, stride=2). This layer contains 1728 learnable parameters.

[0114] S104: Subsequently, multi-scale feature extraction is performed through three parallel competing channel units (CB1-CB3). Each CB module includes Gabor convolution, self-attention mechanism and soft-argmax sub-unit; the number of output channels of the three CB competing channel units are 128, 256 and 512 respectively.

[0115] S105: After performing multi-scale feature extraction, feature fusion and mapping are performed through two fully connected layers. The first fully connected layer maps the 14×14×512 feature map to a 512-dimensional vector space, and the second fully connected layer further reduces the features to 128 dimensions. The total number of parameters of the entire network is approximately 150.4M.

[0116] By following the steps above, a baseline Co3Net network model can be obtained by training the Co3Net network using the palm print and palm vein dataset.

[0117] Example 2: An improved Co3Net network module is obtained based on the baseline Co3Net network model.

[0118] S201: To reduce model complexity, depthwise separable convolution and pointwise convolution are first introduced to replace the standard convolution operation in the baseline Co3Net network model;

[0119] In practice, each standard convolution in the Co3Net network is decomposed into two steps: depthwise convolution and pointwise convolution.

[0120] In the depthwise convolution stage, a 3×3 convolution kernel is used independently for feature extraction in each input channel; in the pointwise convolution stage, a 1×1 convolution kernel is used to integrate the information between channels. This improvement reduces the number of parameters to 123.43M, but due to the change in convolution operations, the model accuracy drops slightly to 94.21%.

[0121] S202: Channel importance analysis and pruning of the network based on L1 norm;

[0122] First, the L1 norm value of each CB channel is calculated as an importance index. 30% of the least important channels are pruned from the CB1 module, 40% from the CB2 module, and 50% from the CB3 module. After pruning, the network is fine-tuned and trained. At this time, the number of network parameters is reduced to 19.75M, but the accuracy also drops to 73.21%.

[0123] Example 3: Add UltraLightNet network and attention mechanism.

[0124] S301: To improve model performance, the UltraLightNet lightweight feature extraction network was designed and added after the improved network obtained in Example 2 above. This UltraLightNet network includes a standard convolutional layer (3×3, 32 channels) and four groups of depthwise separable convolutional blocks; each group of convolutions contains a 3×3 depthwise convolutional layer and a 1×1 pointwise convolutional layer, with channel numbers of 64, 128, 256, and 256 respectively.

[0125] S302: Each convolutional layer is followed by BatchNorm, SiLU activation function and max pooling layer; this design significantly reduces the computational cost while maintaining the receptive field, reducing the number of parameters to 22.43M and improving the accuracy to 97.37%.

[0126] S303: Finally, a multi-head self-attention mechanism is introduced to enhance feature representation. The feature map output by UltraLightNet is flattened into a sequence and linearly transformed to obtain the query, key, and value matrix.

[0127] S304: Employs an 8-head parallel attention computing mechanism, where each head independently calculates attention weights and learns different feature representations;

[0128] S305: Finally, the multi-head outputs are spliced ​​together and linearly projected to obtain the enhanced features.

[0129] After incorporating the attention mechanism, the final model has 26.63M parameters and achieves an accuracy of 99.50% on the Tongji dataset, with an equal error rate (EER) of only 0.57%. The results are shown in Table 3. Compared with the baseline model, the model parameter size has been significantly reduced while maintaining similar accuracy.

[0130] Table 3: Changes in recognition performance and parameter quantity at each stage compared to the baseline model.

[0131]

[0132]

[0133] like Figure 3 As shown, a diagram illustrating the changes in recognition performance and parameter count at each stage compared to the baseline model is presented. It can be seen that although the model parameters are significantly reduced after pruning, the accuracy also drops significantly. By employing an ultra-lightweight network and attention mechanism, the model size remains relatively unchanged, but the accuracy reaches a level similar to the baseline model, achieving a lightweight transformation of the model.

[0134] Example 4:

[0135] This example includes a performance comparison of the method of this invention with other lightweight modeling methods on two datasets.

[0136] S401: Dataset Partitioning. This example uses two datasets, Tongji and CASIA, to train a deep learning network model. For the Tongji dataset, images collected in the first stage are used for the training stage, and images collected in the second stage are used for the testing stage. For the CASIA dataset, 460nm and 850nm are used for training, and the other two modalities are used for testing.

[0137] S402: Image preprocessing. Image size is uniformly scaled to 224×224, and data augmentation is performed through random flipping, rotation, and brightness adjustment.

[0138] S403: Conduct a quantization comparison experiment. First, perform statistical analysis on the parameter distribution of the benchmark Co3net model, recording the numerical range and distribution characteristics of the parameters at each layer. Determine the optimal quantization range by calculating the maximum and minimum values ​​for each layer.

[0139] S404: Employs a symmetric linear quantization scheme to quantize a 32-bit floating-point parameter x into an 8-bit fixed-point number q according to the following formula:

[0140] q = round(x / scale), where scale is the quantization step size, which is determined by the dynamic range of the parameter;

[0141] S405: Design a quantization-aware training strategy, employing a straight-through estimator to handle gradient propagation during quantization operations. Specifically, the forward propagation uses the quantization parameters, while the backward propagation uses continuous values.

[0142] S406: Each layer of the network was quantized separately. After quantization, the number of model parameters was reduced from 150.4M to 28.5M. The accuracy rate was 90.80% on the Tongji dataset and 93.76% on the CASIA dataset.

[0143] S407: Conduct a low-rank decomposition experiment to reconstruct the weight matrix W of each convolutional layer in the benchmark Co3net network and expand it into a two-dimensional matrix form.

[0144] S408: Perform singular value decomposition (SVD) on the expanded weight matrix, W = USV T , where U and V are orthogonal matrices, and S is a diagonal singular value matrix;

[0145] S409: By setting a threshold to retain the k largest singular values ​​and their corresponding singular vectors, a large convolutional layer is decomposed into two smaller convolutional layers. The weights of the first convolutional layer are US' and the weights of the second convolutional layer are V. T , where S' is the truncated diagonal matrix;

[0146] S410: Fine-tune the decomposed network structure using backpropagation optimization with the original dataset. A small learning rate is used during training to preserve the structural characteristics of the decomposed network.

[0147] S411: The final model parameter count was reduced to 40.1M, achieving 92.60% and 91.88% accuracy on the Tongji and CASIA datasets, respectively;

[0148] S412: Conduct knowledge distillation experiments, select the benchmark Co3net as the teacher network, and design a student network structure with fewer parameters. The complexity of the student network is reduced by decreasing the number of convolutional layers and channels.

[0149] S413: Define the knowledge distillation loss function: L=α*LKD+(1-α)*LCE, where LKD is the distillation loss, LCE is the cross-entropy loss, and α is the balance factor;

[0150] S414: Design a soft tag delivery mechanism that uses a temperature coefficient T to soften the output of the teacher network: qi = Softmax(zi / T), where zi is logits and the temperature coefficient T is set to 4;

[0151] S415: The student online training process is divided into two stages. The first stage uses only the teacher's knowledge for pre-training, and the second stage combines hard and soft labels for joint optimization.

[0152] S416: The model with 67.0M parameters after knowledge distillation achieved 95.20% and 96.21% accuracy on the two datasets, respectively.

[0153] Table 4: Comparison results with other methods

[0154]

[0155] As shown in Table 4, the performance test results demonstrate that the method of this invention maintains high recognition accuracy while significantly reducing the number of parameters, achieving 99.50% accuracy and 0.57% equal error rate on the Tongji dataset; and 98.75% accuracy and 0.49% equal error rate on the CASIA dataset. Figure 4 As shown, a comparison chart of the recognition accuracy and model parameter count of different lightweight methods obtained on Tongji is presented. Figure 5 As shown, a comparison chart of the recognition accuracy and model parameter count of different lightweight methods obtained on CASIA is presented. The above results demonstrate that the method of this invention is significantly superior to other lightweight methods.

[0156] Furthermore, an exemplary embodiment of the present invention may also provide a computer-readable storage medium storing a computer program. This computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform an image fusion recognition method based on an ultra-lightweight network according to an exemplary embodiment of the present invention. This computer-readable recording medium is any data storage device capable of storing data readable by a computer system. Examples of computer-readable recording media include: read-only memory, random access memory, read-only optical disk, magnetic tape, floppy disk, optical data storage device, and carrier waves (such as data transmission via the Internet through wired or wireless transmission paths).

[0157] Furthermore, an exemplary embodiment of the present invention may also provide a computing device. The computing device includes a processor and a memory. The memory stores a computer program. The computer program is executed by the processor, causing the processor to execute a computer program for an image fusion recognition method based on an ultralight network according to an exemplary embodiment of the present invention.

[0158] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and other materials. In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several of the functions listed in the specification. While certain measures are described in different embodiments, this does not mean that these measures cannot be combined to produce good results.

[0159] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the invention and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. An image fusion and recognition device based on an ultra-lightweight network, characterized in that, include: Improved Co3Net network module, ultra-lightweight network module, self-attention mechanism module, forward propagation unit and classifier; The improved Co3Net network module is configured to extract features from the input multimodal image to obtain multi-scale features; The ultra-lightweight network module is configured to extract features from the multi-scale features of the input to obtain global features; The ultra-lightweight network module includes m standard convolutional layers, n sets of sequentially connected combinations of depthwise separable convolutional layers and point convolutional layers, batch normalization layers, activation layers, and pooling layers; m takes the value of 1-3, and n takes the value of 3-5; The self-attention mechanism module is configured to draw a self-attention map based on the global features of the input; The forward propagation unit is configured to propagate the self-attention map forward to obtain the final forward propagation output; The classifier is configured to obtain multimodal image classification labels based on the final forward propagation output; The improved Co3Net network module includes an input layer and multiple parallel competing channel units. It is obtained by pruning a baseline Co3Net network model trained using Co3Net network on multimodal images. Each competing channel unit is pruned based on the corresponding competing channel unit of the baseline Co3Net network model. Specifically, the pruning method is as follows: the L1 norm value of each competing channel unit is calculated as an importance index, and pruning is performed according to the importance of each competing channel unit. The standard convolutional layer is replaced with a combination of sequentially connected depthwise separable convolutional layers and point convolutional layers. The depthwise separable convolutional layer is configured to independently use a 3×3 convolutional kernel for feature extraction on each input channel. The point convolutional layer is configured to perform pointwise convolution on the features of each input channel and use a 1×1 convolutional kernel to integrate the information between channels.

2. The image fusion and recognition device based on ultra-lightweight networks according to claim 1, characterized in that, The self-attention mechanism module divides the input global features into multiple heads for parallel computation, captures remote dependencies and global context information, obtains multiple attention maps, concatenates the multiple attention maps and maps them back to the original feature space through a projection matrix to obtain a self-attention map.

3. An image fusion and recognition method based on ultra-lightweight networks, implemented using the apparatus described in any one of claims 1 to 2, characterized in that, Includes the following steps: An improved Co3Net network module is used to extract features from the input multimodal image to obtain multi-scale features; An ultra-lightweight network module is used to extract features from the input multi-scale features to obtain global features; A self-attention mechanism module is used to generate a self-attention map based on the global features of the input; The self-attention map is forward-propagated using a forward propagation unit to obtain the final forward propagation output; A classifier is used to obtain multimodal image classification labels based on the final forward propagation output.

4. The image fusion and recognition method based on ultra-lightweight networks according to claim 3, characterized in that, The improved Co3Net network module is obtained through the following steps: A baseline Co3net network model is obtained by training the Co3net network using multimodal images. The baseline Co3net network model includes multiple parallel competing channel units. The standard convolution operation of the baseline Co3net network model is replaced with depthwise separable convolution and pointwise convolution. Based on the L1 norm, channel importance analysis is performed on the competing channel units of the baseline Co3net network model, and each competing channel unit is pruned according to its channel importance.

5. The image fusion and recognition method based on ultra-lightweight networks according to claim 4, characterized in that, The method further includes: after pruning each competing channel unit according to channel importance, fine-tuning the baseline Co3net network model to obtain an improved Co3Net network module.

6. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed, implement the image fusion and recognition method based on an ultralight network as described in any one of claims 3 to 5.

7. A computing device, characterized in that, It includes a processor and a communication interface coupled to the processor; the processor is used to run computer programs or instructions to implement the image fusion recognition method based on ultralight networks as described in any one of claims 3 to 5.

Images