Automatic insect classification method and system based on multi-view image fusion

Abstract

The invention provides an automatic insect classification method and system based on multi-view image fusion, and relates to the technical field of computer vision, and the method comprises the steps: obtaining three types of images of a front image, a side image and a bottom image of an insect sample, and carrying out the size unification, background removal and image enhancement preprocessing of the three types of images; a single insect three-view image is obtained; respectively extracting deep semantic features of a front image, a side image and a bottom image in the single insect three-view image through three paths of independent image feature extraction networks; fusing deep semantic features of the image, inputting the fused comprehensive features into a classification layer, and outputting an insect category label through a Softmax function to obtain a multi-view insect image data set; supervised training is carried out based on a multi-view insect image data set with labels, and model parameters are optimized by adopting a cross entropy loss function. According to the invention, comprehensive discrimination can be realized, and the performance and stability of the model in complex backgrounds and multi-class insect identification tasks are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an automatic insect classification method and system based on multi-view image fusion. Background Art

[0002] With the rapid development of computer vision and deep learning technologies, automated biological classification methods based on image recognition have gained widespread application in agriculture, ecological monitoring, and biodiversity research. As the most diverse biological group in nature, the rapid and accurate classification and identification of insects is crucial for early warning of pests and diseases, assessing ecosystem health, and discovering new species.

[0003] Traditional insect classification relies primarily on manual observation of insect anatomy and structural features, such as antennae, legs, wings, and dorsal placodes, combined with taxonomic knowledge. These methods not only rely on specialized knowledge but also suffer from low recognition efficiency and high subjectivity, making them difficult to adapt to large-scale, rapid identification scenarios. In recent years, deep learning-based image recognition methods have been increasingly applied to insect classification. These methods utilize models such as convolutional neural networks (CNNs) to automatically learn and classify insect images, significantly improving recognition efficiency and accuracy. Existing image recognition-based insect classification methods often use single-view images for training and recognition. The mainstream approach uses frontal (overhead) images of insects as input, constructing convolutional neural networks (CNNs) or transfer learning models (such as ResNet, Inception, and VGG) to extract texture, shape, and color features from the images, thereby automatically classifying insect species. Some research has also introduced lightweight models suitable for deployment on mobile devices or embedded insect monitoring devices.

[0004] In addition, some studies have attempted to use data augmentation techniques (such as rotation, scaling, flipping, etc.) to expand the data diversity of single-view images to improve the robustness of the model. However, such enhancement operations only perform transformations within the image plane and cannot fundamentally increase the structural information of insects at different angles, making it difficult to effectively represent key classification features such as abdominal patterns, foot features, and wing vein distribution. A small number of studies involve three-dimensional reconstruction or multi-view imaging, but most of them remain at the image acquisition or morphological visualization stage and fail to effectively integrate multi-view information into the classification model. Currently, there is no unified model framework for joint modeling and feature fusion of insect front, side, and bottom images, resulting in insufficient information utilization and limiting further improvement in classification performance.

[0005] However, most existing image recognition methods are based on images from a single perspective, typically a frontal aerial view of the insect, lacking important structural information such as the side and bottom views. Such single-perspective images often fail to fully display the three-dimensional morphological characteristics of insects, especially since certain key classification structures (such as thorax and abdomen structure, leg morphology, wing vein characteristics, etc.) are more obvious from the side or bottom view. Therefore, image recognition methods based on a single perspective often suffer from insufficient recognition accuracy and poor generalization ability when faced with subtle differences between different species, making it difficult to meet the high-precision recognition requirements of practical applications. This is especially true for insect species or subfamilies with similar structures, which are prone to misjudgment, poor generalization ability, and limited classification accuracy.

[0006] In order to improve the performance of the automatic insect classification system, there is an urgent need for an efficient recognition method that can integrate multi-view image information and comprehensively extract insect morphological features. Summary of the Invention

[0007] In view of this, in order to overcome the problems of single perspective, incomplete feature extraction and low recognition accuracy in the existing insect classification methods, the purpose of the present invention is to propose an automatic insect classification method and system based on multi-perspective image fusion. By obtaining the front view, side view and bottom view of the insect, a multi-channel deep learning structure is used to extract image features under different perspectives respectively, and comprehensive discrimination is achieved through the feature fusion mechanism, thereby improving the performance and stability of the model in complex backgrounds and multi-type insect recognition tasks.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] Based on the above objectives, in a first aspect, the present invention provides a method for automatic insect classification based on multi-view image fusion, comprising the following steps:

[0010] Image acquisition and preprocessing: Acquire three types of images of the insect sample: front view, side view, and bottom view. These three types of images are preprocessed by size unification, background removal, and image enhancement to obtain three-view images of a single insect.

[0011] Multi-view feature extraction: Three independent image feature extraction networks are used to extract deep semantic features from the front, side, and bottom views of a single insect from three different perspectives.

[0012] Cross-view feature fusion: The deep semantic features of the image are fused and the fused comprehensive features are input into the Softmax classification layer. The Softmax function outputs the insect category label to obtain a multi-view insect image dataset.

[0013] Model training optimization: Supervised training is performed based on a labeled multi-view insect image dataset, and the cross-entropy loss function is used to optimize the model parameters.

[0014] As a further solution of the present invention, the cross-view feature fusion includes the following steps:

[0015] The weight coefficient of each perspective feature is calculated through the self-attention mechanism, and the weighted sum of the three-way deep semantic features is performed;

[0016] The weight coefficient calculation of the self-attention mechanism includes: performing global average pooling on the three-way deep semantic features of the three-view image of a single insect, inputting the pooling result into a multi-layer perceptron to generate a weight vector, and then using it as the fusion weight after Softmax normalization.

[0017] As a further solution of the present invention, the cross-view feature fusion adopts a cross-attention mechanism, specifically including:

[0018] Use any two perspective features as query vectors and key-value vectors to perform cross-attention calculations to generate correlation features between perspectives;

[0019] The three sets of cross-attention output features are concatenated and used as fusion features after dimensionality reduction through a fully connected layer.

[0020] As a further solution of the present invention, the three types of images are subjected to size unification, background removal, and image enhancement preprocessing, including the following steps:

[0021] The main insect area is extracted through background segmentation algorithm, and histogram equalization is used to enhance texture details;

[0022] The three types of view images are standardized and aligned to ensure that the insect posture remains spatially consistent in the front, side, and bottom views.

[0023] As a further solution of the present invention, when three independent image feature extraction networks are used, the three deep neural networks share some bottom convolutional layer parameters and adopt an independent branch structure in the high-level network.

[0024] As a further embodiment of the present invention, the method further comprises:

[0025] An interpretability analysis module is introduced before the output of the classification layer to generate attention heatmaps through class activation mapping (CAM) or gradient-weighted class activation mapping (Grad-CAM);

[0026] The heat map is superimposed on the original image to visualize the degree of attention the model pays to the insect's legs, wing veins, or abdominal segments.

[0027] As a further aspect of the present invention, a dynamic data augmentation strategy is adopted during supervised training, including:

[0028] Applying random rotation, scaling or color perturbations independently to single-view images in a multi-view image set;

[0029] Enforce the consistency of enhancement operations on three-view images of the same sample.

[0030] As a further solution of the present invention, the output results of the classification layer include three-level classification labels of insects: subfamily, genus, and species, and a hierarchical loss function is used to jointly optimize the multi-level classification accuracy.

[0031] In a second aspect, the present invention provides an automatic insect classification system based on multi-view image fusion, comprising the following components:

[0032] Multi-view image acquisition and preprocessing module: used to obtain three types of images of insect samples: front view, side view, and bottom view; including an image normalization unit for size unification, background removal, and image enhancement preprocessing of the three types of images, and output three-view images of a single insect;

[0033] A multi-branch feature extraction module comprises three independent image feature extraction networks, each connected to the preprocessing module, for extracting deep semantic features of the front view, side view, and bottom view; each feature extraction network is a convolutional neural network (CNN) or a visual transformer (ViT);

[0034] Cross-view feature fusion module: connected to the multi-branch feature extraction module, used to fuse the three-way deep semantic features. The fusion method includes at least one of the following:

[0035] After feature concatenation, the dimension is reduced through the fully connected layer;

[0036] Perform weighted summation of each view feature based on the self-attention mechanism;

[0037] Generate correlation features between views through the cross-attention mechanism;

[0038] Classification decision module: connected to the feature fusion module, including a Softmax classification layer for converting the fused comprehensive features into insect category labels;

[0039] Model training and optimization module: configured to be based on a labeled multi-view insect image dataset, using a cross-entropy loss function to perform end-to-end supervised training on the feature extraction module, feature fusion module, and classification decision module to optimize network parameters.

[0040] As a further solution of the present invention, the self-attention mechanism in the cross-view feature fusion module includes:

[0041] The global average pooling unit is used to compress the three-way deep semantic features and generate the global feature vector of each viewpoint;

[0042] A multi-layer perceptron (MLP) receives the global feature vector and outputs a weight coefficient;

[0043] The Softmax normalization unit normalizes the weight coefficients and then performs weighted summation to generate fusion features.

[0044] As a further solution of the present invention, the cross-attention mechanism in the cross-view feature fusion module includes:

[0045] Cross-attention calculation unit, used to take any two-way view features as query vector and key vector, and generate associated features through multi-head attention mechanism;

[0046] Feature splicing unit, which splices the three sets of cross-attention output features;

[0047] The fully connected dimensionality reduction unit reduces the dimension of the concatenated features and outputs the fused features.

[0048] As a further solution of the present invention, the image normalization unit includes:

[0049] The background segmentation submodule uses semantic segmentation algorithm to extract the main insect area;

[0050] The spatial alignment submodule aligns the three-view images through affine transformation to ensure that the insect posture remains spatially consistent in the front, side, and bottom views.

[0051] The texture enhancement submodule uses histogram equalization to enhance the texture details of wing veins and feet in the image.

[0052] As a further solution of the present invention, in the multi-branch feature extraction module, three independent image feature extraction networks share the underlying convolution kernel parameters and adopt an independent branch structure in the high-level network.

[0053] As a further embodiment of the present invention, the system further comprises:

[0054] The explainability analysis module is embedded before the classification decision module and is configured to generate attention heatmaps through class activation mapping (CAM) or gradient-weighted class activation mapping (Grad-CAM);

[0055] The visualization output unit overlays the heat map with the original image and marks the model's focus areas on the insect's legs, wing veins, or abdominal segments.

[0056] As a further solution of the present invention, the model training and optimization module includes a dynamic data enhancement unit configured as follows:

[0057] Random rotation, translation, or color perturbations are synchronously applied to the three-view images of the same insect sample to maintain the consistency of the data augmentation operation;

[0058] Noise interference or local occlusion enhancement is applied independently to single-view images.

[0059] As a further solution of the present invention, the classification decision module supports multi-level classification, and the output labels include three-level classification results of insect subfamily, genus, and species, and a hierarchical loss function is used to jointly optimize the multi-level classification accuracy.

[0060] As a further solution of the present invention, the system is deployed in an embedded insect monitoring device, including:

[0061] A multi-view imaging device integrates a top-down camera, a side-view camera, and a bottom-view camera to simultaneously capture insect images from three perspectives.

[0062] The edge computing unit has the multi-branch feature extraction module, feature fusion module and classification decision module built in to achieve real-time classification and result feedback.

[0063] Compared with the existing technology, the automatic insect classification method and system based on multi-view image fusion proposed in the present invention has the following beneficial effects:

[0064] 1. By integrating multi-perspective information, feature expression is more comprehensive, multi-perspective information complementarity is achieved, and feature integrity is improved.

[0065] The present invention collects three-perspective images of insects from the front, side, and bottom, and combines them with a multi-branch feature extraction network to comprehensively cover the three-dimensional morphological information of insects, significantly improving the model's ability to understand the three-dimensional structural information of insects, avoiding the information loss problem caused by single-perspective images, and solving the problem of key feature loss caused by angle limitations of single-perspective images. It can accurately classify insect species with similar morphology, supports fine-grained classification tasks, and is suitable for high-precision scenarios such as biodiversity research.

[0066] 2. The cross-perspective fusion mechanism enhances the model's discrimination ability, realizes multi-level classification and hierarchical loss optimization, and improves classification accuracy and robustness.

[0067] The present invention highlights key perspectives through weight distribution, suppresses noise interference, and improves model robustness; it can capture implicit associations between perspectives and enhance the ability to model complex morphological differences; it embeds a class activation mapping module to generate a heat map visualization model focus area, intuitively displaying the morphological basis on which classification decisions depend, and is highly consistent with insect taxonomy knowledge; it supports three-level classification label output of subfamily, genus, and species, and adopts a hierarchical loss function to jointly optimize multi-level classification results. Multi-perspective fusion enhances the model's ability to distinguish fine-grained morphological differences, and is particularly suitable for subtle identification at the subfamily and genus levels, with stronger adaptability and lower misjudgment rate.

[0068] 3. The model is highly interpretable, easy to verify by experts, adaptable to various application scenarios, and has wide promotion value.

[0069] The present invention visualizes the model's focus area through the attention mechanism or class activation map technology, making the recognition process more transparent and helping researchers understand the structural basis of insect classification. It can be widely used in the fields of automatic agricultural pest and disease monitoring systems, ecological monitoring platforms, and biodiversity database construction, and has good practicality and promotion prospects.

[0070] These and other aspects of the present application will be more clearly understood in the following description of the embodiments. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the following briefly introduces the drawings required for the exemplary embodiments or related technical descriptions. The drawings are used to provide a further understanding of the present invention and constitute part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the drawings:

[0072] Figure 1 The figure is a flow chart of an automatic insect classification method based on multi-view image fusion according to an embodiment of the present invention.

[0073] Figure 2 The figure is a schematic diagram of the principle of an automatic insect classification method based on multi-view image fusion according to an embodiment of the present invention.

[0074] Figure 3 The present invention provides a flowchart of preprocessing in an automatic insect classification method based on multi-view image fusion according to an embodiment of the present invention.

[0075] Figure 4 This is a diagram of a three-way feature extraction network structure in an automatic insect classification method based on multi-view image fusion in an embodiment of the present invention.

[0076] Figure 5 The present invention provides a flowchart of cross-view feature fusion in an automatic insect classification method based on multi-view image fusion according to an embodiment of the present invention.

[0077] Figure 6 This is a schematic diagram of an interpretable heat map in an automatic insect classification method based on multi-view image fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0079] To make the purpose, technical solutions and advantages of the present invention more clearly understood, the following is a further detailed description of the embodiments of the present invention in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0080] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are intended to distinguish two non-identical entities or non-identical parameters with the same name. Therefore, "first" and "second" are used for convenience of expression only and should not be understood as limitations on the embodiments of the present invention. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, other steps or units inherent to a process, method, system, product, or device that includes a series of steps or units.

[0081] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0082] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, combined, or partially merged, so the actual execution order may vary depending on the actual situation.

[0083] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features therein may be combined with each other.

[0084] In order to overcome the problems of single perspective, incomplete feature extraction and low recognition accuracy in the existing insect classification methods, the present invention proposes an automatic insect classification method and system based on multi-perspective image fusion. By obtaining the front view, side view and bottom view of the insect, a multi-channel deep learning structure is used to extract image features from different perspectives respectively, and comprehensive discrimination is achieved through the feature fusion mechanism, thereby improving the performance and stability of the model in complex backgrounds and multi-type insect recognition tasks.

[0085] See also Figures 1 to 6As shown, an embodiment of the present invention provides an automatic insect classification method based on multi-view image fusion, which includes the following steps:

[0086] Step S10, image acquisition and preprocessing: obtain three types of images of the insect sample: front view, side view and bottom view, and perform size unification, background removal and image enhancement preprocessing on the three types of images to obtain three-view images of a single insect.

[0087] In this step, see Figure 3 As shown, the three types of images are preprocessed by size unification, background removal and image enhancement, including the following steps:

[0088] Step S101: extract the main insect area using a background segmentation algorithm, and enhance texture details using histogram equalization;

[0089] Step S102: Standardize and align the three types of perspective images to ensure that the insect posture maintains spatial consistency in the front, side, and bottom views.

[0090] The automatic insect classification method based on multi-perspective image fusion in this embodiment integrates multi-perspective information, provides more comprehensive feature expression, achieves multi-perspective information complementarity, and improves feature integrity. By collecting insect images from the front, side, and bottom views and combining them with a multi-branch feature extraction network, the method comprehensively covers the insect's three-dimensional morphological information. This significantly improves the model's ability to understand the insect's three-dimensional structural information, avoids the information loss problem caused by single-perspective images, and solves the problem of missing key features due to angle limitations in single-perspective images. It can accurately classify morphologically similar insect species, supports fine-grained classification tasks, and is suitable for high-precision scenarios such as biodiversity research.

[0091] Step S20, multi-view feature extraction: through three independent image feature extraction networks, extract the deep semantic features of the front view, side view and bottom view of the single insect three-view image respectively.

[0092] In this step, when three independent image feature extraction networks are used, the three deep neural networks share some underlying convolutional layer parameters and adopt an independent branch structure in the high-level network.

[0093] Step S30, cross-view feature fusion: fuse the deep semantic features of the image, input the fused comprehensive features into the Softmax classification layer, output the insect category label through the Softmax function, and obtain a multi-view insect image dataset.

[0094] In this step, see Figure 4 and Figure 5 As shown in Figure 2, cross-view feature fusion includes the following steps:

[0095] Step S301: Calculate the weight coefficient of each view feature through the self-attention mechanism and perform weighted summation on the three-way deep semantic features;

[0096] Step S302, calculating the weight coefficient of the self-attention mechanism includes: performing global average pooling on the three-way deep semantic features of the single insect three-view image, inputting the pooling result into the multi-layer perceptron to generate a weight vector, and using it as the fusion weight after Softmax normalization.

[0097] Among them, cross-view feature fusion adopts the cross-attention mechanism, which specifically includes:

[0098] Use any two perspective features as query vectors and key-value vectors to perform cross-attention calculations to generate correlation features between perspectives;

[0099] The three sets of cross-attention output features are concatenated and used as fusion features after dimensionality reduction through a fully connected layer.

[0100] In this embodiment, the output results of the classification layer include three-level classification labels of insects: subfamily, genus, and species, and a hierarchical loss function is used to jointly optimize the multi-level classification accuracy.

[0101] The embodiment of the present invention enhances the model's discrimination ability through a cross-perspective fusion mechanism, realizes multi-level classification and hierarchical loss optimization, and improves classification accuracy and robustness. The present invention highlights key perspectives through weight distribution, suppresses noise interference, and improves model robustness; it can capture implicit associations between perspectives and enhance the ability to model complex morphological differences; it embeds a class activation mapping module to generate a heat map visualization model focus area, intuitively displaying the morphological basis on which classification decisions depend, and is highly consistent with insect taxonomy knowledge; it supports the output of three-level classification labels at the subfamily, genus, and species level, and uses a hierarchical loss function to jointly optimize multi-level classification results. Multi-perspective fusion enhances the model's ability to discern fine-grained morphological differences, and is particularly suitable for subtle identification at the subfamily and genus levels, with greater adaptability and a lower misjudgment rate.

[0102] Step S40, model training optimization: supervised training is performed based on a labeled multi-view insect image dataset, and the model parameters are optimized using a cross entropy loss function.

[0103] In this step, a dynamic data enhancement strategy is used during supervised training, including:

[0104] Applying random rotation, scaling or color perturbations independently to single-view images in a multi-view image set;

[0105] Enforce the consistency of enhancement operations on three-view images of the same sample.

[0106] In some embodiments, the method further comprises:

[0107] An interpretability analysis module is introduced before the output of the classification layer. Attention heat maps are generated through class activation mapping (CAM) or gradient-weighted class activation mapping (Grad-CAM) to visualize the insect areas that the model focuses on, improve model interpretability, and assist researchers in understanding the classification basis.

[0108] The heat map is superimposed on the original image to visualize the degree of attention the model pays to the insect's legs, wing veins, or abdominal segments.

[0109] In the embodiments of the present invention, the model is highly interpretable, facilitates expert verification, and is adaptable to a variety of application scenarios, thus possessing broad application value. This invention utilizes an attention mechanism or class activation map technique to visualize the model's focus area, making the recognition process more transparent and helping researchers understand the structural basis of insect classification. The model can be widely applied in fields such as automated agricultural pest and disease monitoring systems, ecological monitoring platforms, and biodiversity database construction, demonstrating excellent practicality and potential for widespread adoption.

[0110] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0111] It should be understood that, although the above is described in a certain order, these steps are not necessarily performed in sequence according to the above order. Unless clearly stated herein, the execution of these steps does not have strict order restrictions, and these steps can be performed in other orders. Moreover, a part of the steps of the present embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0112] In a second aspect of the embodiments of the present invention, the present invention further provides an automatic insect classification system based on multi-view image fusion, comprising the following components:

[0113] Multi-view image acquisition and preprocessing module: used to obtain three types of images of insect samples: front view, side view, and bottom view; including an image normalization unit for size unification, background removal, and image enhancement preprocessing of the three types of images, and output three-view images of a single insect;

[0114] A multi-branch feature extraction module comprises three independent image feature extraction networks, each connected to the preprocessing module, for extracting deep semantic features of the front view, side view, and bottom view; each feature extraction network is a convolutional neural network (CNN) or a visual transformer (ViT);

[0115] Cross-view feature fusion module: connected to the multi-branch feature extraction module, used to fuse the three-way deep semantic features. The fusion method includes at least one of the following:

[0116] After feature concatenation, the dimension is reduced through the fully connected layer;

[0117] Perform weighted summation of each view feature based on the self-attention mechanism;

[0118] Generate correlation features between views through the cross-attention mechanism;

[0119] Classification decision module: connected to the feature fusion module, including a Softmax classification layer for converting the fused comprehensive features into insect category labels;

[0120] Model training and optimization module: configured to be based on a labeled multi-view insect image dataset, using a cross-entropy loss function to perform end-to-end supervised training on the feature extraction module, feature fusion module, and classification decision module to optimize network parameters.

[0121] In this embodiment, the self-attention mechanism in the cross-view feature fusion module includes:

[0122] The global average pooling unit is used to compress the three-way deep semantic features and generate the global feature vector of each viewpoint;

[0123] A multi-layer perceptron (MLP) receives the global feature vector and outputs a weight coefficient;

[0124] The Softmax normalization unit normalizes the weight coefficients and then performs weighted summation to generate fusion features.

[0125] In this embodiment, the cross-attention mechanism in the cross-view feature fusion module includes:

[0126] Cross-attention calculation unit, used to take any two-way view features as query vector and key-value vector, and generate related features through multi-head attention mechanism;

[0127] Feature splicing unit, which splices the three sets of cross-attention output features;

[0128] The fully connected dimensionality reduction unit reduces the dimension of the concatenated features and outputs the fused features.

[0129] In this embodiment, the image normalization unit includes:

[0130] The background segmentation submodule uses semantic segmentation algorithm to extract the main insect area;

[0131] The spatial alignment submodule aligns the three-view images through affine transformation to ensure that the insect posture remains spatially consistent in the front, side, and bottom views.

[0132] The texture enhancement submodule uses histogram equalization to enhance the texture details of wing veins and feet in the image.

[0133] In this embodiment, in the multi-branch feature extraction module, three independent image feature extraction networks share the underlying convolution kernel parameters and adopt an independent branch structure in the high-level network.

[0134] In some embodiments, the system further comprises:

[0135] The explainability analysis module is embedded before the classification decision module and is configured to generate attention heatmaps through class activation mapping (CAM) or gradient-weighted class activation mapping (Grad-CAM);

[0136] The visualization output unit overlays the heat map with the original image and marks the model's focus areas on the insect's legs, wing veins, or abdominal segments.

[0137] In this embodiment, the model training and optimization module includes a dynamic data enhancement unit configured as follows:

[0138] Random rotation, translation, or color perturbations are synchronously applied to the three-view images of the same insect sample to maintain the consistency of the data augmentation operation;

[0139] Noise interference or local occlusion enhancement is applied independently to single-view images.

[0140] In this embodiment, the classification decision module supports multi-level classification, and the output labels include three-level classification results of insect subfamily, genus, and species, and a hierarchical loss function is used to jointly optimize the multi-level classification accuracy.

[0141] In this embodiment, the system is deployed in an embedded insect monitoring device, including:

[0142] A multi-view imaging device integrates a top-down camera, a side-view camera, and a bottom-view camera to simultaneously capture insect images from three perspectives.

[0143] The edge computing unit has the multi-branch feature extraction module, feature fusion module and classification decision module built in to achieve real-time classification and result feedback.

[0144] Through the above detailed steps, the automatic insect classification system based on multi-view image fusion of the present invention is used to execute the steps of the automatic insect classification method based on multi-view image fusion in the above embodiment, which will not be repeated here.

[0145] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope of the embodiments disclosed in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any particular order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless expressly limited to the singular.

[0146] It should be understood that, as used herein, the singular form "a" or "an" is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, "and / or" refers to any and all possible combinations of one or more of the items listed in association. The serial numbers of the embodiments disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the disclosure of the embodiments of the present invention (including the claims) is limited to these examples. Within the spirit of the embodiments of the present invention, the technical features of the above embodiments or different embodiments may be combined, and there are many other variations of different aspects of the above embodiments of the present invention, which are not provided in detail for the sake of simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present invention should be included in the scope of protection of the embodiments of the present invention.

Claims

1. An automatic insect classification method based on multi-view image fusion, characterized in that: The method comprises the following steps: Image acquisition and preprocessing: Acquire three types of images of the insect sample: front view, side view, and bottom view. These three types of images are preprocessed by size unification, background removal, and image enhancement to obtain three-view images of a single insect. Multi-view feature extraction: Three independent image feature extraction networks are used to extract deep semantic features from the front, side, and bottom views of a single insect from three different perspectives. Cross-view feature fusion: The deep semantic features of the image are fused and the fused comprehensive features are input into the Softmax classification layer. The Softmax function outputs the insect category label to obtain a multi-view insect image dataset. Model training optimization: Supervised training is performed based on a labeled multi-view insect image dataset, and the cross-entropy loss function is used to optimize the model parameters.

2. The method for automatic insect classification based on multi-view image fusion according to claim 1, characterized in that: The cross-view feature fusion includes the following steps: The weight coefficient of each perspective feature is calculated through the self-attention mechanism, and the weighted sum of the three-way deep semantic features is performed; The weight coefficient calculation of the self-attention mechanism includes: performing global average pooling on the three-way deep semantic features of the three-view image of a single insect, inputting the pooling result into a multi-layer perceptron to generate a weight vector, and then using it as the fusion weight after Softmax normalization.

3. The method for automatic insect classification based on multi-view image fusion according to claim 2, characterized in that: The cross-view feature fusion adopts the cross-attention mechanism, which specifically includes: Use any two perspective features as query vectors and key-value vectors to perform cross-attention calculations to generate correlation features between perspectives; The three sets of cross-attention output features are concatenated and used as fusion features after dimensionality reduction through a fully connected layer.

4. The method for automatic insect classification based on multi-view image fusion according to claim 3, characterized in that: The three types of images are preprocessed by size unification, background removal, and image enhancement, including the following steps: The main insect area is extracted through background segmentation algorithm, and histogram equalization is used to enhance texture details; The three types of view images are standardized and aligned to ensure that the insect posture remains spatially consistent in the front, side, and bottom views.

5. The method for automatic insect classification based on multi-view image fusion according to claim 4, characterized in that: When using three independent image feature extraction networks, the three-way deep neural network shares some underlying convolutional layer parameters and adopts an independent branch structure in the high-level network.

6. The method for automatic insect classification based on multi-view image fusion according to claim 1, characterized in that: The method further comprises: An interpretability analysis module is introduced before the output of the classification layer to generate an attention heat map through class activation mapping or gradient-weighted class activation mapping; The heat map is superimposed on the original image to visualize the degree of attention the model pays to the insect's legs, wing veins, or abdominal segments.

7. The method for automatic insect classification based on multi-view image fusion according to claim 1, characterized in that: Dynamic data augmentation strategies are used during supervised training, including: Applying random rotation, scaling or color perturbations independently to single-view images in a multi-view image set; Enforce the consistency of enhancement operations on three-view images of the same sample.

8. The method for automatic insect classification based on multi-view image fusion according to claim 1, characterized in that: The output results of the classification layer include three-level classification labels of insects: subfamily, genus, and species, and a hierarchical loss function is used to jointly optimize the multi-level classification accuracy.

9. An automatic insect classification system based on multi-view image fusion, characterized in that: The system is used to perform the method for automatic insect classification based on multi-view image fusion according to any one of claims 1 to 8, comprising: Multi-view image acquisition and preprocessing module: used to obtain three types of images of insect samples: front view, side view, and bottom view; including an image normalization unit for size unification, background removal, and image enhancement preprocessing of the three types of images, and output three-view images of a single insect; Multi-branch feature extraction module: comprising three independent image feature extraction networks, respectively connected to the preprocessing module, for extracting deep semantic features of the front view, side view, and bottom view; Cross-view feature fusion module: connected to the multi-branch feature extraction module, used to fuse three-way deep semantic features; Classification decision module: connected to the feature fusion module, including a Softmax classification layer for converting the fused comprehensive features into insect category labels; Model training and optimization module: configured to be based on a labeled multi-view insect image dataset, using a cross-entropy loss function to perform end-to-end supervised training on the feature extraction module, feature fusion module, and classification decision module to optimize network parameters.

10. The automatic insect classification system based on multi-view image fusion according to claim 9, characterized in that: The cross-attention mechanism in the cross-view feature fusion module includes: Cross-attention calculation unit, used to take any two-way view features as query vector and key-value vector, and generate related features through multi-head attention mechanism; Feature splicing unit, which splices the three sets of cross-attention output features; The fully connected dimensionality reduction unit reduces the dimension of the concatenated features and outputs the fused features.

Images