Fine-grained image recognition method and device based on weakly supervised learning and readable medium

Abstract

The application discloses a fine-grained image recognition method and device based on weak supervision learning and readable medium, a VGG_Res-Inception model based on an attention mechanism is constructed and trained through two-step transfer learning, the VGG_Res-Inception model based on the attention mechanism comprises a pre-trained VGG16 model, a Res-Inception part and an attention mechanism part, the Res-Inception part comprises a fourth batch normalization layer, a plurality of Res-Inception modules, a third convolutional layer and a third batch normalization layer, the Res-Inception module comprises a first convolutional layer based on residual connection, a first batch normalization layer, an Inception-A unit, a second convolutional layer and a second batch normalization layer, the attention mechanism part comprises an attention mechanism module, a global average pooling layer, a full connection layer and a softmax layer, the two-step transfer learning process comprises transfer learning between a source domain and a transition domain and transfer learning between the transition domain and a target domain; a fine-grained image of a plant leaf disease degree is acquired and input into the trained VGG_Res-Inception model based on the attention mechanism, and a classification result is output. Stability and accuracy can be improved.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and computer vision, specifically to a fine-grained image recognition method, apparatus, and readable medium based on weakly supervised learning. Background Technology

[0002] Fine-grained image classification aims to distinguish subordinate categories with subtle visual differences, making it more challenging than traditional coarse-grained image classification. This is due to two main factors: firstly, the feature differences between images are smaller, resulting in more subtle discriminative features; secondly, the training dataset is limited, and images contain many uncertain factors, such as lighting differences and background interference. The key to fine-grained image classification is obtaining the most salient local difference features. Based on the different requirements of neural networks for supervised information in the training data, research algorithms for fine-grained image classification are mainly divided into two categories: weakly supervised algorithms and strongly supervised algorithms. Strongly supervised fine-grained classification algorithms rely on more manually labeled information (such as object bounding boxes and part annotations) in addition to providing image category labels. Weakly supervised learning aims to achieve fine-grained classification tasks solely based on category labels. Currently, most fine-grained image classification methods employ strong supervision, requiring additional finely labeled datasets in addition to category labels, consuming significant human and material resources and severely limiting the application of these algorithms in real-world scenarios. Therefore, designing a weakly supervised fine-grained image recognition algorithm that does not require additional manual annotation is of great research significance.

[0003] In the past, accurate classification required the expertise of domain specialists, leading to significantly increased costs. For example, in agricultural pest and disease identification, the characteristics of different lesions are very similar, and the differences in leaf features between different degrees of disease are even more difficult to distinguish, making it difficult for non-professionals to discern with the naked eye. By using deep learning models and weakly supervised learning methods, the accuracy of classification can be improved by focusing on the detection of subtle features in key local areas. In the industrial field, such as the defect detection of some precision components, microscopes and professional personnel are often required for identification. Weakly supervised learning methods can be used to detect subtle defects without complex labeled datasets. Summary of the Invention

[0004] In view of the aforementioned technical problems, the purpose of the embodiments of this application is to propose a fine-grained image recognition method, apparatus, and readable medium based on weakly supervised learning to solve the technical problems mentioned in the background section.

[0005] In a first aspect, embodiments of this application provide a fine-grained image recognition method based on weakly supervised learning, comprising the following steps:

[0006] A VGG_Res-Inception model based on an attention mechanism was constructed and trained through two steps of transfer learning to obtain a trained VGG_Res-Inception model based on an attention mechanism. The VGG_Res-Inception model based on an attention mechanism includes a pre-trained VGG16 model, a Res-Inception part, and an attention mechanism part. The Res-Inception part includes a fourth batch normalization layer, multiple Res-Inception modules, a third convolutional layer, and a third batch normalization layer. The Res-Inception module includes a first convolutional layer based on residual connections, a first batch normalization layer, an Inception-A unit, a second convolutional layer, and a second batch normalization layer. The attention mechanism part includes an attention mechanism module, a global average pooling layer, a fully connected layer, and a softmax layer. The two-step transfer learning training process includes transfer learning between the source domain and the transition domain, as well as transfer learning between the transition domain and the target domain, where the transition domain is a coarse-grained image dataset.

[0007] Acquire fine-grained images of plant leaf disease severity and input them into a trained VGG_Res-Inception model based on an attention mechanism to output classification results.

[0008] In some embodiments, fine-grained images of plant leaf disease severity undergo multiple feature extractions and feature fusions using a pre-trained VGG16 model and the Res-Inception part to obtain fused feature data; the fused feature data is then input into the attention mechanism part to extract fine-grained features and perform classification.

[0009] In some embodiments, the attention mechanism module is an SENet network. The SENet network introduces a residual-like structure, including a global average pooling layer, two fully connected layers and a sigmoid layer connected in sequence. The feature fusion data is input into the SENet network to obtain the global features of each channel of the feature map and to activate the global features. By obtaining the weights of different channels, the relationship between the channels is learned. Finally, the fine-grained features are obtained by multiplying by the original feature map.

[0010] In some embodiments, the Res-Inception part includes a first Res-Inception module, a second Res-Inception module, and a third Res-Inception module. The output of the first batch of normalization layers is input into the first Res-Inception module. The output of the first batch of normalization layers is combined with the output of the first Res-Inception module to perform feature fusion, resulting in first feature fusion data. The first feature fusion data is input into the second Res-Inception module. The first feature fusion data is combined with the output of the second Res-Inception module to perform feature fusion, resulting in second feature fusion data. The second feature fusion data is input into the third Res-Inception module. The second feature fusion data is combined with the output of the third Res-Inception module to perform feature fusion, resulting in third feature fusion data. The third feature fusion data passes through a second convolutional layer and a second batch of normalization layers to obtain fourth feature fusion data. The output of the first batch of normalization layers is combined with the fourth feature fusion data to perform feature fusion, resulting in fifth feature fusion data.

[0011] In some embodiments, the Inception-A unit includes a bottleneck structure network consisting of multiple convolutional layers with a kernel size of 1*1, convolutional layers with a kernel size of 3*3, and an average pooling layer with a kernel size of 1*1.

[0012] In some embodiments, the transfer learning between the source domain and the transition domain during the two-step transfer learning training process specifically includes:

[0013] The VGG16 model is pre-trained using the source domain to achieve convolutional layer parameter transfer, resulting in a pre-trained VGG16 model.

[0014] The weights and parameters of the pre-trained VGG16 model are fixed. Using the pre-trained VGG16 model as a feature extractor, the network parameters of the attention-based VGG_Res-Inception model are initialized through the transition domain, resulting in the initialized attention-based VGG_Res-Inception model.

[0015] In some embodiments, the transfer learning between the transition domain and the target domain during the two-step transfer learning training process specifically includes:

[0016] The initial attention-based VGG_Res-Inception model is fine-tuned based on the transition domain and the target domain to achieve feature transfer between the transition domain and the target domain, resulting in a trained attention-based VGG_Res-Inception model.

[0017] Secondly, embodiments of this application provide a fine-grained image recognition method based on weakly supervised learning, comprising:

[0018] The model building and training module is configured to build an attention-based VGG_Res-Inception model and train it through two steps of transfer learning to obtain a trained attention-based VGG_Res-Inception model. The attention-based VGG_Res-Inception model includes a pre-trained VGG16 model, a Res-Inception part, and an attention mechanism part. The Res-Inception part includes a fourth batch normalization layer, multiple Res-Inception modules, a third convolutional layer, and a third batch normalization layer. The Res-Inception module includes a first convolutional layer based on residual connections, a first batch normalization layer, an Inception-A unit, a second convolutional layer, and a second batch normalization layer. The attention mechanism part includes an attention mechanism module, a global average pooling layer, a fully connected layer, and a softmax layer. The two-step transfer learning training process includes transfer learning between the source domain and the transition domain, as well as transfer learning between the transition domain and the target domain, where the transition domain is a coarse-grained image dataset.

[0019] The output module is configured to acquire fine-grained images of the degree of plant leaf disease and input them into a trained VGG_Res-Inception model based on an attention mechanism to output classification results.

[0020] Thirdly, embodiments of this application provide an electronic device including one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method described in any implementation of the first aspect.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) The present invention can achieve fine-grained image classification by using only coarse-grained labels.

[0024] (2) Compared with traditional transfer learning methods, the present invention has greatly improved recognition accuracy and model operation stability, and the accuracy has also been greatly improved.

[0025] (3) This invention reduces the problems of overfitting and “negative transfer” in the training process of the model through a two-step transfer learning training method, and the visualization region localization based on the SENet network attention mechanism is more accurate. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is an exemplary device architecture diagram in which an embodiment of this application can be applied;

[0028] Figure 2 This is a flowchart illustrating a fine-grained image recognition method based on weakly supervised learning, as an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the VGG_Res-Inception model, which is an attention mechanism-based fine-grained image recognition method based on weakly supervised learning, according to an embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram illustrating the pre-training process of the VGG16 model, a fine-grained image recognition method based on weakly supervised learning, as an embodiment of the present invention.

[0031] Figure 5 This is a schematic diagram of the Inception-A module of the fine-grained image recognition method based on weakly supervised learning, as described in an embodiment of the present invention.

[0032] Figure 6 This is a schematic diagram of the bottleneck structure of a fine-grained image recognition method based on weakly supervised learning, according to an embodiment of the present invention.

[0033] Figure 7 This is a schematic diagram of the Res-Inception module of the fine-grained image recognition method based on weakly supervised learning, as described in an embodiment of the present invention.

[0034] Figure 8 The diagram shows the SENet network structure with added fusion features in a fine-grained image recognition method based on weakly supervised learning, as described in an embodiment of the present invention.

[0035] Figure 9 This is a schematic diagram of the two-step transfer learning training of a fine-grained image recognition method based on weakly supervised learning, according to an embodiment of the present invention.

[0036] Figure 10This is a schematic diagram of a fine-grained image recognition device based on weakly supervised learning, according to an embodiment of the present invention.

[0037] Figure 11 This is a schematic diagram of the structure of a computer device suitable for implementing the electronic device of the present application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0039] Figure 1 An exemplary device architecture 100 is shown, which can be applied to the fine-grained image recognition method or the fine-grained image recognition device based on weakly supervised learning according to the embodiments of this application.

[0040] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0041] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, and 103.

[0042] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., software programs or software modules used to provide distributed services) or as a single software program or software module. No specific limitations are imposed here.

[0043] Server 105 can be a server that provides various services, such as a background data processing server that processes files or data uploaded by terminal devices 101, 102, and 103. The background data processing server can process the acquired files or data and generate processing results.

[0044] It should be noted that the fine-grained image recognition method based on weakly supervised learning provided in this application embodiment can be executed by server 105 or by terminal devices 101, 102, and 103. Correspondingly, the fine-grained image recognition device based on weakly supervised learning can be set in server 105 or in terminal devices 101, 102, and 103.

[0045] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. If the data being processed does not need to be retrieved remotely, the above architecture may not include a network, requiring only servers or terminal devices.

[0046] Figure 2 The illustration shows an embodiment of the fine-grained image recognition method based on weakly supervised learning provided by this application, comprising the following steps:

[0047] S1. Construct an attention-based VGG_Res-Inception model and train it through two steps of transfer learning to obtain a trained attention-based VGG_Res-Inception model. The attention-based VGG_Res-Inception model includes a pre-trained VGG16 model, a Res-Inception part, and an attention mechanism part. The Res-Inception part includes a fourth batch normalization layer, multiple Res-Inception modules, a third convolutional layer, and a third batch normalization layer. The Res-Inception module includes a first convolutional layer based on residual connections, a first batch normalization layer, an Inception-A unit, a second convolutional layer, and a second batch normalization layer. The attention mechanism part includes an attention mechanism module, a global average pooling layer, a fully connected layer, and a softmax layer. The two-step transfer learning training process includes transfer learning between the source domain and the transition domain, as well as transfer learning between the transition domain and the target domain, where the transition domain is a coarse-grained image dataset.

[0048] In a specific embodiment, to extract valuable information from images within the network model, this invention proposes a VGG_Res-Inception model based on an attention mechanism, which consists of three parts. The first part is a pre-trained VGG16 model, serving as the basic feature extractor. The second part is the Res-Inception component, which leverages the excellent structural features of ResNet and Inception modules to extract high-dimensional features, obtaining multi-scale feature fusion data. The third part is the attention mechanism, employing the typical channel-level attention mechanism SENet. These three parts construct a network structure with strong capabilities for extracting fine-grained features.

[0049] Specifically, the following section will detail the process of building the VGG_Res-Inception model based on the attention mechanism, and its specific model structure is as follows: Figure 3 As shown. First, a pre-trained VGG16 model is obtained using a dynamic fine-tuning method. The dynamic fine-tuning process of the pre-trained VGG16 model is as follows: Figure 4 As shown, the convolutional layers of the VGG16 model are first frozen, the fully connected layer network structure is adjusted, and the parameters are fine-tuned. Before fine-tuning, it is expected that the model will stabilize and no longer decrease. Then, the parameters of the higher-level convolutional layers are released, and the higher-level convolutional layers and fully connected layers are fine-tuned. During the pre-training process, the convolutional neural network has a forward propagation and backward parameter tuning process. When the cost function of the network structure is at its minimum during the entire training process of the neural network, the randomly initialized network parameters are adjusted, and the pre-trained neural network model is obtained. In the specific pre-training process of the VGG16 network model, it is assumed that the source domain contains... There are training samples, and a single input sample is . ,in express 3D input vector, Labels representing samples; Indicates the VGG-16's If a layer is defined as follows, then the input feature vector of that layer is represented as... The output feature vector of this layer is represented as and the corresponding weights of that layer. and bias value The forward propagation of a VGG16 convolutional network can then be represented as:

[0050] ;

[0051] This function This represents the activation function; the activation function used in VGG16 is the ReLU function.

[0052] The overall cost function of the network model is expressed as:

[0053] ;

[0054] in, This represents the output value when the network performs forward propagation. The total number of layers in the network. Indicates the first The number of nodes in a layered neural network.

[0055] Batch gradient descent is used to adjust parameters to find the minimum of the overall cost function. The updated parameter expression is shown below:

[0056] ;

[0057] ;

[0058] The learning rate, represented by the cost function, is used to complete the pre-training process for the source domain after multiple iterations and updates, until the cost function value is minimized. In this application, the ImageNet dataset is used as the source domain. This dataset is rich in features, and through continuous iterative training, its parameters can effectively extract various features from images. Therefore, this application transfers the pre-trained model parameters to the target domain, achieving model transfer.

[0059] A fourth batch normalization layer is added after the pre-trained VGG16 model, followed by three Res-Inception modules. Because it possesses the residual structure of ResNet and the parallel structure of Inception, it can extract plant leaf lesion features more meticulously. After multiple feature extractions and fusions, the extracted features are input into the SENet network structure. In the embodiments of this application, to ensure the SENet network is compatible with the overall structure of the VGG_Res-Inception model based on the attention mechanism proposed in this invention, the input to SENet no longer utilizes the pre-trained Inception structure, but instead uses the fused feature data as the SENet data input. SENet can improve features useful to the network while suppressing less useful features based on their importance, achieving the redirection of fused features. Then, global average pooling is added after SENet. Global average pooling, like fully connected layers, extracts global information while significantly reducing the number of parameters and computational cost, and has better interpretability, greatly aiding in the subsequent addition of class activation maps.

[0060] In a specific embodiment, the Res-Inception part includes three Res-Inception modules. The basic components of each Res-Inception module are described below. Inception-v4 is a typical multi-scale convolutional kernel neural network, composed of numerous multi-scale convolutional kernel modules. After extensive network parameter adjustments, Inception-v4 is widely recognized as a high-performing GoogLeNet convolutional neural network. Therefore, this application borrows the Inception-A module as the multi-scale convolutional kernel for the Res-Inception module. Figure 5 As shown, the Inception-A unit consists of a bottleneck structure network composed of multiple convolutional layers with a kernel size of 1*1, convolutional layers with a kernel size of 3*3, and an average pooling layer with a kernel size of 1*1. It mainly includes two types of convolutional kernels: 1*1 and 3*3. The 1*1 convolutional kernel is used to construct the bottleneck structure to reduce computational cost.

[0061] To address the structural characteristics of the residual structure and Inception-A mentioned above, a Res-Inception module was designed. Multiple Res-Inception modules are cascaded together, and the connection between these module groups is achieved through the ResNet residual skip structure. The Res-Inception module is introduced first, and it initially borrows the bottleneck structure proposed by GoogLeNet. The purpose of the bottleneck structure is to reduce the computational cost of convolutional layers; that is, before computing large convolutional layers, a bottleneck is first constructed. Convolutions are used to compress the number of channels in the input feature map of the convolutional layer, thereby reducing the computational cost; after the large convolutional layer completes the computation, further calculations will be performed as needed. The convolutional layer restores the number of channels in the output feature map. The following diagram illustrates the bottleneck structure. Figure 6 As shown, where .

[0062] Figure 7The diagram shown illustrates the structure of the Res-Inception module. As illustrated, a 1x1 convolutional layer is first used. The primary purpose is to ensure the feature map size output from the last convolutional layer of the pre-trained VGG16 model is 512. To match the number of input feature maps in Inception-A, the input data first passes through a 1x1 convolutional layer with 384 kernels, minimizing modifications to the Inception-A hyperparameters. After the first convolutional layer, a batch normalization layer is added. This is because complex machine learning systems are prone to internal covariate shifts during training. Batch normalization not only prevents these shifts but also makes the neural network more stable during training, less sensitive to initial values, and allows for a larger learning rate to accelerate convergence. Finally, the Inception-A module is connected. Because the Inception structure uses convolutional kernels of different sizes, it is effective in detecting leaf lesions of varying sizes. Subsequently, a second 1x1 convolutional layer is added. To restore the number of feature maps before input, this layer restores the number of feature maps. The number of convolutional kernels is set to 512, thus realizing the bottleneck structure of the model backbone. Furthermore, inspired by residual structures, to avoid "model degradation" and overfitting during the overall model construction process, residual connection structures are added to the entire model structure. The above operations complete the construction of a Res-Inception module.

[0063] In a specific embodiment, the Res-Inception part includes a first Res-Inception module, a second Res-Inception module, and a third Res-Inception module. The output of the first batch of normalization layers is input into the first Res-Inception module. The output of the first batch of normalization layers is combined with the output of the first Res-Inception module to perform feature fusion, resulting in first feature fusion data. The first feature fusion data is input into the second Res-Inception module. The first feature fusion data is combined with the output of the second Res-Inception module to perform feature fusion, resulting in second feature fusion data. The second feature fusion data is input into the third Res-Inception module. The second feature fusion data is combined with the output of the third Res-Inception module to perform feature fusion, resulting in third feature fusion data. The third feature fusion data passes through a second convolutional layer and a second batch of normalization layers to obtain fourth feature fusion data. The output of the first batch of normalization layers is combined with the fourth feature fusion data to perform feature fusion, resulting in fifth feature fusion data.

[0064] The computation of the attention mechanism can be divided into two steps: the first step is to calculate the attention weights across all input information, and the second step is to weight the feature information of all inputs based on the attention weights to select the key information from the input. First, let's define the feature information of the attention input as... The query vector related to the current task is represented as Then choose the first one. The probability of each input feature information is represented as: , The attention scoring function is shown in the following formula:

[0065] ;

[0066] By making selections based on factors such as the current task and network conditions, the filtering of feature information can be achieved. This process can be represented as follows:

[0067] .

[0068] In a specific embodiment, the attention mechanism module is the SENet network. The SENet network introduces a residual-like structure, including a global average pooling layer, two fully connected layers and a sigmoid layer connected in sequence. The feature fusion data is input into the SENet network to obtain the global features of each channel of the feature map and to activate the global features. By obtaining the weights of different channels, the relationship between the channels is learned. Finally, the fine-grained features are obtained by multiplying by the original feature map.

[0069] Specifically, the SENet network can improve network performance at the channel level. SENet is a typical channel attention-based model that improves the feature extraction capability of a neural network by considering the relationships between feature channels and the importance of different channels. The main working mechanism of SENet is to compress the feature maps of each convolutional layer, compressing the length, width, and number of channels [H, W, C] of the feature map into features of size [1, 1, C]. This obtains the global features of all channels in the feature map. Then, the global features are excitationd, and by obtaining the weights of different channels, the relationships between channels are learned. Finally, the final features are obtained by multiplying by the original feature map.

[0070] Because fine-grained images often exhibit highly similar features, an attention mechanism is desired to extract more detailed image features. Therefore, the SENet network is employed as the attention mechanism module of the neural network. Features extracted by the pre-trained VGG16 network are input into the SENet network to obtain global features across different channels of the feature map, thereby improving the accuracy of fine-grained image recognition. To obtain a suitable neural network structure, the SENet model is modified by replacing the original Inception module with feature vectors extracted by the pre-trained network as the feature input to the neural network. The structure diagram of the newly constructed SENet network is shown below. Figure 8 As shown, the SENet network introduces a residual-like structure and uses global average pooling to extract global information about features and reduce the number of parameters and computational cost of the neural network. Then, two fully connected layers (FC) are added to limit the model complexity and help increase the model's generalization ability.

[0071] This experimental case study uses fine-grained image classification of citrus Huanglongbing (HLB) as an example. The overall dataset includes three categories: healthy citrus, moderate HLB, and severe HLB. The overall dataset has relatively small feature differences, falling within the scope of fine-grained image classification. However, its features differ significantly from the spatial features on the ImageNet dataset. Although transfer learning can avoid the overfitting problem caused by training a large number of parameters in deep learning, it is still a process of transferring trained parameters from the source domain to the target domain. If the distribution of data in the source and target domains differs too much, it can lead to negative transfer during training. Furthermore, the VGG_Res-Inception model based on the attention mechanism has a very complex network structure containing a large number of untrained parameters. Direct application of this model may cause overfitting during training.

[0072] To avoid potential overfitting and negative transfer problems in transfer learning, a two-step transfer learning method is adopted. As the name suggests, the two-step transfer learning method involves two transfer learning operations and introduces the concept of a "transition domain".

[0073] The transition domain is derived from the Plantvillage leaf lesion dataset, which contains images of leaf lesions from various plants. However, these images are coarse-grained, providing only the general category of each crop disease without specifying the degree of disease. Nevertheless, the leaf and disease characteristics of these crops are similar to the fine-grained image features of a particular plant leaf disease studied in this application. Using this as the transition domain for training reduces the risk of negative transfer during model training. Furthermore, the abundance of this dataset helps reduce the risk of overfitting to randomly initialized model parameters during training.

[0074] For ease of training, during the construction of the transition domain, related lesions are combined into a single major category. That is, the leaves of each crop are considered as a major category, and the leaf lesion images of a certain crop are merged to generate a major category. The Plantview dataset contains multiple crop categories. In this embodiment, the categories with more and more types and quantities of lesions are selected. Finally, eight crops are selected to form the dataset, including apple, cherry, corn, grape, peach, pepper, potato, and tomato. 600 images of each crop are randomly selected as the training set, 200 images as the validation set, and 200 images as the test set.

[0075] In a specific embodiment, the transfer learning between the source domain and the transition domain, and the transfer learning between the transition domain and the target domain during the two-step transfer learning training process, specifically include:

[0076] The VGG16 model is pre-trained using the source domain to achieve convolutional layer parameter transfer, resulting in a pre-trained VGG16 model.

[0077] The weights and parameters of the pre-trained VGG16 model are fixed. The pre-trained VGG16 model is used as a feature extractor. The network parameters of the attention-based VGG_Res-Inception model are initialized through the transition domain, resulting in the initialized attention-based VGG_Res-Inception model.

[0078] The initial attention-based VGG_Res-Inception model is fine-tuned based on the transition domain and the target domain to achieve feature transfer between the transition domain and the target domain, resulting in a trained attention-based VGG_Res-Inception model.

[0079] Specifically, the first step is to perform transfer learning between the source and transition domains. Although the source domain (ImageNet dataset) consists of real-world images, the convolutional layer parameters of the pre-trained VGG16 model on the ImageNet dataset are fixed. Considering that the pre-trained VGG16 model has already learned many low-level image features from the ImageNet dataset, only the Res-Inception and attention mechanisms of the VGG_Res-Inception model are opened in the early stages of transfer learning training. When the model loss stabilizes and no longer decreases, in order for the model to learn high-level semantic features, the parameters of all layers after Block 5_1 of the pre-trained VGG16 model, as well as the Res-Inception and attention mechanisms of the VGG_Res-Inception model, are opened. The model is trained for 600 epochs with these parameters fixed, allowing it to form a multi-layered feature extractor. To improve the interpretability of the model, the Class activation mapping (CAM) method is used to locate the salient regions of orange leaf lesions identified by the model.

[0080] The second step involves transfer learning between the transition domain and the target domain. The parameters of the Res-Inception and attention mechanisms in the VGG_Res-Inception model are decoupled and trained for 200 epochs, allowing the model's feature extraction part to learn the data features of the target dataset. A detailed two-step transfer learning diagram is shown below. Figure 9 As shown.

[0081] S2: Obtain fine-grained images of the degree of plant leaf disease and input them into the trained VGG_Res-Inception model based on the attention mechanism to output the classification results.

[0082] In a specific embodiment, the fine-grained image of the degree of plant leaf disease undergoes multiple feature extractions and feature fusions through a pre-trained VGG16 model and the Res-Inception part to obtain feature fusion data; the feature fusion data is then input into the attention mechanism part to extract fine-grained features and perform classification.

[0083] Further reference Figure 10 As an implementation of the methods shown in the above figures, this application provides an embodiment of a fine-grained image recognition device based on weakly supervised learning. This device embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0084] This application provides a fine-grained image recognition method based on weakly supervised learning, including:

[0085] Model building and training module 1 is configured to build an attention-based VGG_Res-Inception model and train it through two steps of transfer learning to obtain a trained attention-based VGG_Res-Inception model. The attention-based VGG_Res-Inception model includes a pre-trained VGG16 model, a Res-Inception part, and an attention mechanism part. The Res-Inception part includes a fourth batch normalization layer, multiple Res-Inception modules, a third convolutional layer, and a third batch normalization layer. The Res-Inception module includes a first convolutional layer based on residual connections, a first batch normalization layer, an Inception-A unit, a second convolutional layer, and a second batch normalization layer. The attention mechanism part includes an attention mechanism module, a global average pooling layer, a fully connected layer, and a softmax layer. The two-step transfer learning training process includes transfer learning between the source domain and the transition domain, as well as transfer learning between the transition domain and the target domain, wherein the transition domain is a coarse-grained image dataset.

[0086] The output module 2 is configured to acquire fine-grained images of the degree of disease on plant leaves and input them into the trained VGG_Res-Inception model based on the attention mechanism to output classification results.

[0087] The following is for reference. Figure 11 It illustrates an electronic device suitable for implementing embodiments of this application (e.g., Figure 1 A schematic diagram of the structure of a computer device 1100 (shown as a server or terminal device). Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0088] like Figure 11 As shown, the computer device 1100 includes a central processing unit (CPU) 1101 and a graphics processing unit (GPU) 1102, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1103 or programs loaded from storage portion 1109 into random access memory (RAM) 1104. The RAM 1104 also stores various programs and data required for the operation of the device 1100. The CPU 1101, GPU 1102, ROM 1103, and RAM 1104 are interconnected via a bus 1105. An input / output (I / O) interface 1106 is also connected to the bus 1105.

[0089] The following components are connected to I / O interface 1106: an input section 1107 including a keyboard, mouse, etc.; an output section 1108 including an LCD, speakers, etc.; a storage section 1109 including a hard disk, etc.; and a communication section 1110 including a network interface card, such as a LAN card, modem, etc. The communication section 1110 performs communication processing via a network such as the Internet. A drive 1111 may also be connected to I / O interface 1106 as needed. Removable media 1112, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1111 as needed so that computer programs read from them can be installed into storage section 1109 as needed.

[0090] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1110, and / or installed from removable medium 1112. When the computer program is executed by central processing unit (CPU) 1101 and graphics processing unit (GPU) 1102, the functions defined in the methods of this application are performed.

[0091] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0092] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using dedicated hardware-based apparatus to perform the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The modules described in the embodiments of this application can be implemented in software or hardware. These modules can also be located within a processor.

[0095] On another front, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: construct a VGG_Res-Inception model based on an attention mechanism and train it through two steps of transfer learning to obtain a trained VGG_Res-Inception model based on an attention mechanism. The VGG_Res-Inception model based on an attention mechanism includes a pre-trained VGG16 model, a Res-Inception part, and an attention mechanism part. The Res-Inception part includes a fourth batch normalization layer, multiple Res-Inception modules, and a third batch normalization layer. The Res-Inception module consists of three convolutional layers and a third batch normalization layer. It includes a first convolutional layer based on residual connections, a first batch normalization layer, an Inception-A unit, a second convolutional layer, and a second batch normalization layer. The attention mechanism module comprises an attention mechanism module, a global average pooling layer, a fully connected layer, and a softmax layer. The two-step transfer learning training process includes transfer learning between the source and transition domains, as well as between the transition and target domains. The transition domain is a coarse-grained image dataset. Fine-grained images of plant leaf disease severity are acquired and input into the trained attention-based VGG_Res-Inception model, which outputs classification results.

[0096] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A fine-grained image recognition method based on weakly supervised learning, characterized in that, Includes the following steps: A VGG_Res-Inception model based on an attention mechanism is constructed and trained through a two-step transfer learning process to obtain a trained VGG_Res-Inception model based on an attention mechanism. This model includes a pre-trained VGG16 model, a Res-Inception part, and an attention mechanism part. The Res-Inception part includes a fourth batch normalization layer, multiple Res-Inception modules, a third convolutional layer, and a third batch normalization layer. Each Res-Inception module includes a first convolutional layer based on residual connections, a first batch normalization layer, an Inception-A unit, a second convolutional layer, and a second batch normalization layer. The attention mechanism part includes an attention mechanism module, a global average pooling layer, a fully connected layer, and a softmax layer. The two-step transfer learning training process includes transfer learning between the source and transition domains, and transfer learning between the transition and target domains. The transition domain is a coarse-grained image dataset. The transfer learning between the source and transition domains specifically includes: The VGG16 model is pre-trained using the source domain to achieve convolutional layer parameter transfer, thereby obtaining the pre-trained VGG16 model. The weights and parameters of the pre-trained VGG16 model are fixed. Using the pre-trained VGG16 model as a feature extractor, the network parameters of the attention-based VGG_Res-Inception model are initialized through the transition domain, resulting in the initialized attention-based VGG_Res-Inception model. The transfer learning between the transition domain and the target domain specifically includes: Based on the transition domain and the target domain, the initialized attention-based VGG_Res-Inception model is fine-tuned to achieve feature transfer between the transition domain and the target domain, thereby obtaining the trained attention-based VGG_Res-Inception model. A fine-grained image of the degree of disease on plant leaves is obtained and input into the trained VGG_Res-Inception model based on the attention mechanism, and the classification result is output.

2. The fine-grained image recognition method based on weakly supervised learning according to claim 1, characterized in that, The fine-grained images of plant leaf disease severity are subjected to multiple feature extractions and feature fusions using a pre-trained VGG16 model and the Res-Inception part to obtain feature fusion data; the feature fusion data is then input into the attention mechanism part to extract fine-grained features and perform classification.

3. The fine-grained image recognition method based on weakly supervised learning according to claim 2, characterized in that, The attention mechanism module is an SENet network. The SENet network introduces a residual-like structure, including a global average pooling layer, two fully connected layers, and a sigmoid layer connected in sequence. The feature fusion data is input into the SENet network to obtain the global features of each channel of the feature map and to activate the global features. By obtaining the weights of different channels, the relationship between the channels is learned. Finally, the fine-grained features are obtained by multiplying by the original feature map.

4. The fine-grained image recognition method based on weakly supervised learning according to claim 2, characterized in that, The Res-Inception section includes a first Res-Inception module, a second Res-Inception module, and a third Res-Inception module. The output of the first batch of normalization layers is input into the first Res-Inception module. The output of the first batch of normalization layers is combined with the output of the first Res-Inception module for feature fusion to obtain first feature fusion data. The first feature fusion data is input into the second Res-Inception module. The first feature fusion data is combined with the output of the second Res-Inception module for feature fusion to obtain second feature fusion data. The second feature fusion data is input into the third Res-Inception module. The second feature fusion data is combined with the output of the third Res-Inception module for feature fusion to obtain third feature fusion data. The third feature fusion data passes through a second convolutional layer and a second batch of normalization layers to obtain fourth feature fusion data. The output of the first batch of normalization layers is combined with the fourth feature fusion data for feature fusion to obtain fifth feature fusion data.

5. The fine-grained image recognition method based on weakly supervised learning according to claim 1, characterized in that, The Inception-A unit comprises a bottleneck structure network consisting of multiple convolutional layers with a kernel size of 1*1, convolutional layers with a kernel size of 3*3, and an average pooling layer with a kernel size of 1*1.

6. A fine-grained image recognition device based on weakly supervised learning, characterized in that, include: The model construction and training module is configured to build an attention-based VGG_Res-Inception model and train it through two steps of transfer learning to obtain a trained attention-based VGG_Res-Inception model. The attention-based VGG_Res-Inception model includes a pre-trained VGG16 model, a Res-Inception part, and an attention mechanism part. The Res-Inception part includes a fourth batch normalization layer, multiple Res-Inception modules, a third convolutional layer, and a third batch normalization layer. Each Res-Inception module includes a first convolutional layer based on residual connections, a first batch normalization layer, an Inception-A unit, a second convolutional layer, and a second batch normalization layer. The attention mechanism part includes an attention mechanism module, a global average pooling layer, a fully connected layer, and a softmax layer. The two-step transfer learning training process includes transfer learning between the source domain and the transition domain, and transfer learning between the transition domain and the target domain. The transition domain is a coarse-grained image dataset. The transfer learning between the source domain and the transition domain specifically includes: The VGG16 model is pre-trained using the source domain to achieve convolutional layer parameter transfer, thereby obtaining the pre-trained VGG16 model. The weights and parameters of the pre-trained VGG16 model are fixed. Using the pre-trained VGG16 model as a feature extractor, the network parameters of the attention-based VGG_Res-Inception model are initialized through the transition domain, resulting in the initialized attention-based VGG_Res-Inception model. The transfer learning between the transition domain and the target domain specifically includes: Based on the transition domain and the target domain, the initialized attention-based VGG_Res-Inception model is fine-tuned to achieve feature transfer between the transition domain and the target domain, thereby obtaining the trained attention-based VGG_Res-Inception model. The output module is configured to acquire fine-grained images of the degree of plant leaf disease and input them into the trained VGG_Res-Inception model based on the attention mechanism, and output the classification results.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images