Aphid counting method and device based on heterogeneous frame search
Through the combination of heterogeneous framework search module and feature stabilizer, the problem of insufficient accuracy of aphid counting in the prior art is solved, and efficient aphid counting under complex conditions is achieved, which improves the accuracy and stability of the counting.
Patent Information
- Application Number
- CN202311863533.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2043-12-29
AI Technical Summary
The existing aphid counting technology is not accurate in the case of target occlusion, complex background and huge scale changes, making it difficult to achieve efficient automation.
Using aphid counting method based on heterogeneous framework search, aphid counting network model is constructed by constructing aphid training data set, data preprocessing, and constructing aphid counting network model, including basic network, heterogeneous framework search module and feature stabilizer module, using multi-scale and lightweight search Feature extraction and stability constraints are performed in the space, and density maps are generated for counting.
The accuracy of aphid counting in different scales and complex backgrounds is achieved, and the calculation overhead is maintained, which avoids feature loss and improves the accuracy of counting.
Smart Images

Figure CN117809186B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision, and in particular to an aphid counting method and device based on heterogeneous frame search. Background Art
[0002] Aphids are widely regarded as serious agricultural pests, causing damage to a variety of crops such as wheat, soybeans, fruit trees and vegetables. Aphids may indirectly damage plants by spreading pathogenic bacteria that infect plants during aphid feeding. Their presence leads to large crop yield losses and economic difficulties. Accurately handling these pests is an urgent need to achieve planting automation and develop scientific and reasonable prevention and control strategies, and counting these aphids is a crucial prerequisite.
[0003] Aphid counting has been continuously improved in recent years. With the development of deep learning technology, from the original detection method to the use of neural network training to infer aphid density maps. However, there are some problems in the process of aphid counting, such as: severe target occlusion, complex background, and huge changes in scale. Summary of the invention
[0004] Purpose of the invention: To address the problems existing in the prior art, the present invention provides a more accurate aphid counting method and device based on heterogeneous frame search.
[0005] Technical solution: The aphid counting method based on heterogeneous frame search described in the present invention comprises:
[0006] S1: Constructing an aphid training dataset, including aphid image collection and labeling. The labeling is used to mark the aphid targets in each image, and convolution operation is performed on the point markings using Gaussian kernels to generate Gaussian density maps.
[0007] S2: Data preprocessing of aphid images;
[0008] S3: construct an aphid counting network model, including a basic network, a heterogeneous framework search module and a feature stabilizer module, wherein the basic network is used to extract basic features of the network input, the heterogeneous framework search module is used to search for the types of internal units and the operations of nodes to generate a heterogeneous framework search network, and the feature stabilizer module is used to stabilize the learning of features within the heterogeneous framework search module;
[0009] S4: using the preprocessed aphid image and the corresponding density map as samples to train the aphid counting model;
[0010] S5: Finally, the aphid image to be counted is input into the trained aphid counting model, and a density map is obtained as output. The values in the density map are summed to obtain the aphid count value.
[0011] Furthermore, the S2 specifically includes:
[0012] S2-1: resize the aphid image to a given size;
[0013] S2-2: randomly scale the aphid image with a preset probability;
[0014] S2-3: Randomly crop the aphid image;
[0015] S2-4: randomly flip the aphid image horizontally with a preset probability;
[0016] S2-5: Perform random gamma transformation on the aphid image with a preset probability;
[0017] S2-6: randomly convert the aphid image into a grayscale image with a preset probability.
[0018] Furthermore, the heterogeneous frame search module in S3 specifically includes:
[0019] The heterogeneous framework search module is composed of φ basic units, each of which is composed of two internal units, namely a lightweight search unit and a multi-scale search unit. For each basic unit, two internal unit weights are constructed in turn to measure the weights of the two internal units; the lightweight search unit and the multi-scale search unit search for candidate operations in their respective search spaces; the internal unit is composed of The input of each node comes not only from the output of the previous node, but also from the input of the current basic unit. Each input received by the node is formed by adding the candidate operations measured by the node weight. The candidate operations are generated by two search spaces, the multi-scale search space and the lightweight search space. The specific formula is:
[0020]
[0021] In the formula, P represents the number of candidate operations, i represents the index of the node in the internal unit, and x i represents the output of the node in the inner unit, represents the mth candidate operation under index i, express The weight of represents the output of the inner unit, represents the output of a node in a specific type of inner unit, l represents the depth of the basic unit, t represents the type of inner unit, including MS and LW, F l LW and F l MSrepresent lightweight search unit and multi-scale search unit respectively, concat(·) is the feature aggregation operation along the channel dimension, N is a positive integer, α l and β l They correspond to the weights of the lightweight search unit and the multi-scale search unit respectively.
[0022] Furthermore, the multi-scale search space is composed of convolution kernels of different sizes and convolution layers of different expansion rates. The specific candidate operations include:
[0023] 1×1 convolution;
[0024] 3×3 convolution;
[0025] 5×5 convolution;
[0026] 7×7 convolution;
[0027] 3×3 convolution with 2x dilation rate;
[0028] 5×5 convolution with 2x dilation rate;
[0029] Skip connections;
[0030] Zero operation.
[0031] Furthermore, the lightweight search space is composed of lightweight convolutions, and specific candidate operations include:
[0032] 1×1 convolution;
[0033] 3×3 convolution;
[0034] 3×3 grouped convolution;
[0035] 5×5 grouped convolution;
[0036] 3×3 depthwise separable convolution;
[0037] 5×5 depthwise separable convolution;
[0038] Skip connections;
[0039] Zero operation.
[0040] Furthermore, the feature stabilizer module in S3 specifically includes:
[0041] The feature extractor consists of a series of convolutional and max pooling layers;
[0042] A feature channel converter, using 1×1 convolution to align the number of channels of the output features in each basic unit in S3 with the number of channels in the corresponding stage in the feature extractor;
[0043] Search network feature constraints, and perform feature constraints on features obtained by the feature channel converter and features corresponding to the feature extractor in turn.
[0044] Furthermore, the characteristic channel converter is implemented by the following formula:
[0045] T={t i |i=1,2,...,n},S={θ(s i )|i=1,2,...,n}
[0046] Where, t i is the feature of the i-th stage in the feature extractor, s i is the feature of the i-th stage of the feature channel converter, T and S are the feature groups of the feature extractor and the feature channel converter, respectively. i ) represents a 1×1 convolution operation.
[0047] Furthermore, the search network feature constraint is implemented by the following formula:
[0048]
[0049] In the formula, is the feature stability loss, and SSIM(·) represents the structural similarity metric.
[0050] Furthermore, the S4 specifically includes:
[0051] S4-1: Divide the preprocessed aphid images and corresponding density maps into training sets and validation sets;
[0052] S4-2: Initialize learning rate;
[0053] S4-3: Input the training set into the aphid counting network, calculate the network loss based on the output of the heterogeneous framework search network, the output of the feature extractor, and the corresponding density map, and optimize the structural parameters using the SGD optimizer based on the network loss, and use the Adam optimizer to optimize the network parameters;
[0054] S4-4: Validate the output of the aphid counting network using the validation set, and save the network parameters with the minimum loss in the validation set;
[0055] S4-5: When the number of iterations is less than or equal to the given threshold, S4-3 is executed; when the number of iterations is greater than the given threshold, the network search phase is completed;
[0056] S4-6: After the network search phase is over, the internal units are screened by retaining the maximum value of the internal unit weights, and the candidate operation with the largest weight of each node input is screened. The input of the largest and second largest nodes and the corresponding candidate operation with the largest weight are further retained to obtain a lightweight aphid counting network;
[0057] S4-7: Input the training set into the lightweight aphid counting network and use Calculate network loss and use Adam optimizer to optimize network parameters;
[0058] S4-8: The output of the lightweight aphid counting network is verified using the validation set, and the network parameters with the minimum loss in the validation set are saved;
[0059] S4-9: When the number of iterations is less than or equal to the given threshold, S4-7 is executed. When the number of iterations is greater than the given threshold, the network training phase is completed.
[0060] Furthermore, the method for calculating the network loss in S4-3 specifically includes:
[0061] The loss is calculated as follows:
[0062]
[0063] Where M gt represents the density map, M s represents the output of the heterogeneous framework search network, M t represents the output of the feature extractor, s is a hyperparameter, is the feature stability loss.
[0064] The aphid counting device based on heterogeneous frame search described in the present invention comprises a processor and a computer program stored in a memory and executable on the processor, and the method is implemented when the processor executes the program.
[0065] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:
[0066] (1) The heterogeneous frame search module of the present invention utilizes multi-scale search space and lightweight search space to perform multi-scale convolution and lightweight convolution screening, thereby achieving the perception of aphids of different scales while maintaining low computational overhead.
[0067] (2) The feature stabilizer module of the present invention uses the features in the feature extractor to constrain the features in the heterogeneous frame search module, thereby avoiding feature loss due to the simplification of candidate operations. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1It is a schematic diagram of the process of the aphid counting method provided by the present invention;
[0069] Figure 2 It is a schematic diagram of two search spaces and search processes of a heterogeneous frame search module in the aphid counting method provided by the present invention;
[0070] Figure 3 It is a schematic diagram of the characteristic stabilizer structure of the aphid counting method provided by the present invention. DETAILED DESCRIPTION
[0071] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0072] This embodiment specifically provides a method for counting aphids based on heterogeneous frame search, such as Figure 1 , Figure 2 and Figure 3 As shown, the following steps are included:
[0073] S1: Constructing an aphid training dataset, including aphid image collection and labeling. The labeling is used to mark the aphid targets in each image, and convolution operation is performed on the point markings using Gaussian kernels to generate Gaussian density maps.
[0074] S2: Data preprocessing of aphid images.
[0075] The preprocessing method specifically includes:
[0076] S2-1: Adjust the resolution of the aphid image to a given size of 256×256; S2-2: Randomly scale the aphid image with a preset probability; S2-3: Randomly crop the aphid image; S2-4: Randomly flip the aphid image horizontally with a preset probability; S2-5: Randomly gamma transform the aphid image with a preset probability; S2-6: Randomly convert the aphid image to a grayscale image with a preset probability.
[0077] S3: Construct an aphid counting network, including a basic network, a heterogeneous framework search module, and a feature stabilization module.
[0078] Among them, the basic network is used to extract the basic features of the network.
[0079] The heterogeneous framework search module is used to search for the types of internal units and node operations to generate a heterogeneous framework search network. Specifically, it includes:
[0080] The heterogeneous framework search module is composed of φ basic units, each of which is composed of two internal units, namely a lightweight search unit and a multi-scale search unit. For each basic unit, two internal unit weights are constructed in turn to measure the weights of the two internal units; the lightweight search unit and the multi-scale search unit search for candidate operations in their respective search spaces; the internal unit is composed of The input of each node comes not only from the output of the previous node, but also from the input of the current basic unit. Each input received by the node is formed by adding the candidate operations measured by the node weight. The candidate operations are generated by two search spaces, the multi-scale search space and the lightweight search space. The specific formula is:
[0081]
[0082] In the formula, P represents the number of candidate operations, i represents the index of the node in the internal unit, and x i represents the output of the node in the inner unit, represents the mth candidate operation under index i, express The weight of represents the output of the inner unit, represents the output of a node in a specific type of inner unit, l represents the depth of the basic unit, t represents the type of inner unit, including MS and LW, F l LW and F l MS represent lightweight search unit and multi-scale search unit respectively, concat(·) is the feature aggregation operation along the channel dimension, N is a positive integer, α l and β l They correspond to the weights of the lightweight search unit and the multi-scale search unit respectively.
[0083] The multi-scale search space is composed of convolution kernels of different sizes and convolution layers with different dilation rates. The specific candidate operations include:
[0084] 1×1 convolution;
[0085] 3×3 convolution;
[0086] 5×5 convolution;
[0087] 7×7 convolution;
[0088] 3×3 convolution with 2x dilation rate;
[0089] 5×5 convolution with 2x dilation rate;
[0090] Skip connections;
[0091] Zero operation.
[0092] The lightweight search space consists of lightweight convolutions, and specific candidate operations include:
[0093] 1×1 convolution;
[0094] 3×3 convolution;
[0095] 3×3 grouped convolution;
[0096] 5×5 grouped convolution;
[0097] 3×3 depthwise separable convolution;
[0098] 5×5 depthwise separable convolution;
[0099] Skip connections;
[0100] Zero operation.
[0101] The feature stabilizer module is used to stabilize the learning of features within the search framework, and specifically includes:
[0102] The feature extractor consists of a series of convolutional and max pooling layers;
[0103] A feature channel converter, using 1×1 convolution to align the number of channels of the output features in each basic unit in S3 with the number of channels in the corresponding stage in the feature extractor;
[0104] Search network feature constraints, and perform feature constraints on features obtained by the feature channel converter and features corresponding to the feature extractor in turn.
[0105] The characteristic channel converter is implemented by the following formula:
[0106] T={t i |i=1,2,...,n},S={θ(s i )|i=1,2,...,n}
[0107] Where, t i is the feature of the i-th stage in the feature extractor, s i is the feature of the i-th stage of the feature channel converter, T and S are the feature groups of the feature extractor and the feature channel converter, respectively. i ) represents a 1×1 convolution operation.
[0108] The search network feature constraint is implemented by the following formula:
[0109]
[0110] In the formula, is the feature stability loss, and SSIM(·) represents the structural similarity metric.
[0111] S4: using the preprocessed aphid image and the corresponding density map as samples to train the aphid counting model;
[0112] The S4 specifically includes:
[0113] S4-1: Divide the preprocessed aphid images and the corresponding density maps into training and validation sets.
[0114] In addition, in order to test the effect, in addition to the training set and the validation set, this embodiment also divides the test set into 180 pairs of training data and 53 pairs of test data.
[0115] S4-2: In the search phase, initialize the SGD optimizer weight_decay to 3e -5 , momentum is 0.9, and the initial learning rate is 1e -4 ; Initialize Adam optimizer weight_decay to 1e -3 , initialize the learning rate to 1e -3 During the training phase, initialize the Adam optimizer weight_decay to 5e -4 , initialize the learning rate to 1e -3 .
[0116] S4-3: Input the training set into the aphid counting network, calculate the network loss based on the output of the heterogeneous framework search network, the output of the feature extractor, and the corresponding density map, and optimize the structural parameters using the SGD optimizer based on the network loss, and use the Adam optimizer to optimize the network parameters;
[0117] S4-4: Validate the output of the aphid counting network using the validation set, and save the network parameters with the minimum loss in the validation set;
[0118] S4-5: When the number of iterations is less than or equal to the given threshold, S4-3 is executed; when the number of iterations is greater than the given threshold, the network search phase is completed;
[0119] S4-5: Train the model obtained in the network search phase, using Calculate network loss and use Adam optimizer to optimize network parameters;
[0120] S4-6: After the network search phase is over, the internal units are screened by retaining the maximum value of the internal unit weights, and the candidate operation with the largest weight of each node input is screened. The input of the largest and second largest nodes and the corresponding candidate operation with the largest weight are further retained to obtain a lightweight aphid counting network;
[0121] S4-7: Input the training set into the lightweight aphid counting network and use Calculate network loss and use Adam optimizer to optimize network parameters;
[0122] S4-8: The output of the lightweight aphid counting network is verified using the validation set, and the network parameters with the minimum loss in the validation set are saved;
[0123] S4-9: When the number of iterations is less than or equal to the given threshold, S4-7 is executed. When the number of iterations is greater than the given threshold, the network training phase is completed.
[0124] The calculation method of network loss specifically includes:
[0125] S4-3-1: Calculate the loss in the search phase according to the following formula:
[0126]
[0127] Where M gt represents the density map, M s represents the output of the heterogeneous framework search network, M t represents the output of the feature extractor, σ is a hyperparameter, is the feature stability loss.
[0128] Among them, Adam and SGD optimization algorithms are used to find the back-propagation gradient of the error of the network loss function to each network weight parameter, and new network weight parameters are obtained through the parameter update process.
[0129] S4-5-1: Calculate the feature stability loss during the training phase according to the following formula:
[0130]
[0131] Where M gt represents the density map, M s represents the output of the heterogeneous framework search network, M t represents the output of the feature extractor, σ is a hyperparameter, is the feature stability loss.
[0132] Among them, the Adam optimization algorithm is used to find the back-propagation gradient of the error of the network loss function to each network weight parameter, and the new network weight parameters are obtained through the parameter update process.
[0133] S5: Finally, the aphid image to be counted is input into the trained aphid counting model, and a density map is obtained as output. The values in the density map are summed to obtain the aphid count value.
[0134] This embodiment also provides an aphid counting device based on heterogeneous frame search, including a processor and a computer program stored in a memory and executable on the processor, and the processor implements the above method when executing the program.
[0135] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A method for counting aphids based on heterogeneous frame search, Features The method includes: S1: Constructing an aphid training dataset, including aphid image collection and labeling. The labeling is used to mark the aphid targets in each image, and convolution operation is performed on the point markings using Gaussian kernels to generate Gaussian density maps. S2: Data preprocessing of aphid images; S3: construct an aphid counting network model, including a basic network, a heterogeneous framework search module and a feature stabilizer module, wherein the basic network is used to extract basic features of the network input, the heterogeneous framework search module is used to search for the types of internal units and the operations of nodes to generate a heterogeneous framework search network, and the feature stabilizer module is used to stabilize the learning of features within the heterogeneous framework search module; S4: using the preprocessed aphid image and the corresponding density map as training samples, and performing network structure search and training on the aphid counting network model; S5: Finally, the aphid image to be counted is input into the trained aphid counting model, and a density map is outputted. The values in the density map are summed to obtain the aphid count value; The heterogeneous frame search module in S3 specifically includes: The heterogeneous framework search module is composed of φ basic units, each of which is composed of two internal units, namely a lightweight search unit and a multi-scale search unit. For each basic unit, two internal unit weights are constructed in turn to measure the weights of the two internal units; the lightweight search unit and the multi-scale search unit search for candidate operations in their respective search spaces; the internal unit is composed of The input of each node comes not only from the output of the previous node, but also from the input of the current basic unit. Each input received by the node is formed by adding the candidate operations measured by the node weight. The candidate operations are generated by two search spaces, the multi-scale search space and the lightweight search space. The specific formula is: In the formula, P represents the number of candidate operations, i represents the index of the node in the internal unit, and x i represents the output of the node in the inner unit, represents the mth candidate operation under index i, express The weight of represents the output of the inner unit, Represents the output of the node in the specific type of internal unit, l represents the depth of the basic unit, t represents the type of internal unit, including MS and LW types, and represent lightweight search unit and multi-scale search unit respectively, concat(·) is the feature aggregation operation along the channel dimension, N is a positive integer, α l and β l The weights corresponding to the lightweight search unit and the multi-scale search unit respectively; The multi-scale search space in S3 is composed of convolution kernels of different sizes and convolution layers of different expansion rates, and the lightweight search space in S3 is composed of lightweight convolutions; The feature stabilizer module in S3 specifically includes: The feature extractor consists of a series of convolutional and max pooling layers; The feature channel converter uses 1×1 convolution to align the number of channels of the output features in each basic unit in S3 with the number of channels in the corresponding stage in the feature extractor, which is implemented by the following formula: T={t i |i=1,2,...,n},S={θ(s i )|i=1,2,...,n} Where, t i is the feature of the i-th stage in the feature extractor, s i is the feature of the i-th stage of the feature channel converter, T and S are the feature groups of the feature extractor and the feature channel converter respectively, θ(s i ) represents a 1×1 convolution operation; Search network feature constraints, where the features obtained by the feature channel converter are sequentially constrained with the features corresponding to the feature extractor; The search network feature constraint is implemented by the following formula: In the formula, is the feature stability loss, and SSIM(·) represents the structural similarity metric.
2. The aphid counting method based on heterogeneous frame search according to claim 1, Features: The data preprocessing in S2 specifically includes: S2-1: resize the aphid image to a given size; S2-2: randomly scale the aphid image with a preset probability; S2-3: Randomly crop the aphid image; S2-4: randomly flip the aphid image horizontally with a preset probability; S2-5: Perform random gamma transformation on the aphid image with a preset probability; S2-6: randomly convert the aphid image into a grayscale image with a preset probability.
3. The aphid counting method based on heterogeneous frame search according to claim 1, Features: The candidate operations corresponding to the multi-scale search space include: 1×1 convolution; 3×3 convolution; 5×5 convolution; 7×7 convolution; 3×3 convolution with 2x dilation rate; 5×5 convolution with 2x dilation rate; Skip connections; Zero operation.
4. The aphid counting method based on heterogeneous frame search according to claim 1, Features: The candidate operations corresponding to the lightweight search space include: 1×1 convolution; 3×3 convolution; 3×3 grouped convolution; 5×5 grouped convolution; 3×3 depthwise separable convolution; 5×5 depthwise separable convolution; Skip connections; Zero operation.
5. The aphid counting method based on heterogeneous frame search according to claim 1, Features: The S4 specifically includes: S4-1: Divide the preprocessed aphid images and corresponding density maps into training sets and validation sets; S4-2: Initialize learning rate; S4-3: Input the training set into the aphid counting network, calculate the network loss based on the output of the heterogeneous framework search network, the output of the feature extractor, and the corresponding density map, and optimize the structural parameters using the SGD optimizer based on the network loss, and use the Adam optimizer to optimize the network parameters; S4-4: Validate the output of the aphid counting network using the validation set, and save the network parameters with the minimum loss in the validation set; S4-5: When the number of iterations is less than or equal to the given threshold, execute step S4-3; when the number of iterations is greater than the given threshold, complete the network search phase; S4-6: After the network search phase is over, the internal units are screened by retaining the maximum value of the internal unit weights, and the candidate operation with the largest weight of each node input is screened. The input of the largest and second largest nodes and the corresponding candidate operation with the largest weight are further retained to obtain a lightweight aphid counting network; S4-7: Input the training set into the lightweight aphid counting network and use Calculate network loss and use Adam optimizer to optimize network parameters; S4-8: The output of the lightweight aphid counting network is verified using the validation set, and the network parameters with the minimum loss in the validation set are saved; S4-9: When the number of iterations is less than or equal to the given threshold, S4-7 is executed. When the number of iterations is greater than the given threshold, the network training phase is completed.
6. The aphid counting method based on heterogeneous frame search according to claim 5, Features: The method for calculating the network loss in S4-3 specifically includes: Where M gt represents the density map, M s represents the output of the heterogeneous framework search network, M t represents the output of the feature extractor, s is a hyperparameter, is the feature stability loss.
7. An aphid counting device based on heterogeneous frame search, comprising a processor and a computer program stored in a memory and executable on the processor, Features: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Crowd counting method based on multi-stage mixed attention network
CN112668532A
Automatic grouping multi-scale lightweight deep convolutional neural network optimization method
CN113011091A
