Tobacco worm identifying and counting method and device based on coarse and fine granularity fusion
By using coarse and fine particle fusion tobacco insect identification and counting method and three-party strategy to enhance the discrimination module, the problem of distinguishing between tobacco insects and impurities is solved, the accuracy and robustness of tobacco insect counting is improved, and the factory's intelligence is met Monitoring requirements.
Patent Information
- Application Number
- CN202411043013.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-29
AI Technical Summary
The prior art has problems of inefficiency and misjudgment in tobacco insect identification counting in cigarette factories, and it is difficult to effectively distinguish between tobacco insects and impurities such as dust and debris, and cannot meet the factory's intelligent needs.
The tobacco insect identification and counting method based on coarse and fine particle fusion is adopted. By extracting image features of different particle sizes and performing feature fusion, the model's ability to identify and distinguish tobacco insects and impurities is enhanced, and the robustness to dust and debris interference is improved. The three-party strategy enhances the discriminant module assists the training model.
It improves the accuracy of tobaccoworm counting and the robustness of the model, reduces impurity interference, and realizes more discriminant tobaccoworm image feature extraction, meeting the insect situation monitoring needs of cigarette factories.
Smart Images

Figure CN119049081A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of cigarette production, and more specifically, to a method and device for identifying and counting tobacco insects based on coarse and fine granularity fusion. Background Art
[0002] Cigarettes are a kind of smokable commodity, and the quality of cigarette raw materials is one of the important factors that determine the quality of cigarettes. Tobacco insects feed on cigarette raw materials. The constant temperature and humidity conditions in the workshop, as well as the accumulated tobacco and tobacco scraps, provide tobacco insects with a comfortable growth and reproduction environment and rich nutrients. When tobacco insects infect cigarette raw materials, it will have a negative impact on the internal and external quality of the product, resulting in damage to the brand image and economic losses. Therefore, by setting up tobacco insect traps in the cigarette factory, accurately identifying and counting the insects at each monitoring point plays a vital role in the monitoring of the insect situation in the entire factory and the subsequent analysis of the distribution trend of tobacco insects. However, the current identification and counting of tobacco insects mainly relies on manual regular visual identification and manual counting of tobacco insects on tobacco insect traps, which is inefficient and prone to misjudgment, and cannot meet the requirements of factory intelligence.
[0003] The target detection task has always been a research focus in the field of computer vision. In recent years, with the rise of technologies such as artificial intelligence and deep learning, target detection methods based on deep neural networks have been widely used in face recognition, traffic violation detection, industrial defect detection, medical image analysis and other fields. Two-stage methods such as Region-CNN and SPP-Net and one-stage methods such as YOLO have achieved remarkable results. These methods often use convolutional neural networks as the basic model, design complex mapping structures and perception mechanisms to achieve multi-scale image context feature capture, so that the model can learn the size, texture and other information of entities in the image, and achieve accurate target detection.
[0004] However, these deep neural network-based target detection methods have certain difficulties in the task of identifying and counting tobacco insects in actual tobacco factories. The reason is that the traps have a certain degree of stickiness. In the process of trapping tobacco insects, impurities such as dust and debris floating in the air or carried into the air by the operation of factory equipment are often collected. The size and color are often very similar to tobacco insects. They exist in the trap images at the same time as real tobacco insect individuals. The interference of these impurities makes it difficult to identify and count tobacco insects. Therefore, how to improve the robustness of the model to impurity interference and obtain more discriminative tobacco insect image features is an urgent problem to be solved for insect monitoring in cigarette factories. Summary of the invention
[0005] The present application provides a method and device for identifying and counting tobacco insects based on coarse and fine granularity fusion, which extracts image features of different coarse and fine granularity from tobacco insect images to capture the morphological features of tobacco insects of different granularity in tobacco insect images, and through feature fusion, enables the model to have stronger recognition and distinction capabilities for small-scale objects such as tobacco insects and impurities, improves the robustness of the model to interference from impurities such as dust and debris, obtains more discriminative tobacco insect image features, and improves the accuracy of tobacco insect counting.
[0006] The present application provides a tobacco insect identification and counting method based on coarse and fine granularity fusion, including:
[0007] receiving a first tobacco insect image;
[0008] Inputting the first tobacco insect image into a tobacco insect recognition and positioning model, and outputting the number of tobacco insects in the first tobacco insect image;
[0009] The tobacco insect recognition and positioning model processes the first tobacco insect image, specifically including:
[0010] Obtaining a first fused feature map containing image features of different coarse and fine granularities based on the first tobacco insect image;
[0011] The number of tobacco worms is identified based on the first fused feature map.
[0012] Preferably, obtaining a first fused feature map containing image features of different coarse and fine granularities based on the first tobacco insect image comprises the following steps:
[0013] Converting the first tobacco insect image into a grayscale image, and binarizing the grayscale image based on a grayscale threshold to obtain a mask;
[0014] Downsampling the first tobacco insect image multiple times to obtain multiple first tobacco insect image feature maps with different coarse and fine granularities;
[0015] Using the mask to enhance each first tobacco insect picture feature map to obtain a second tobacco insect picture feature map;
[0016] All second tobacco insect picture feature maps generated by the same first tobacco insect image are feature fused to obtain a first fused feature map.
[0017] Preferably, identifying the number of tobacco insects based on the first fused feature map specifically includes:
[0018] Identify the anchor frame containing tobacco worms in the first fused feature map;
[0019] The number of anchor boxes is counted as the number of tobacco insects in the first tobacco insect image.
[0020] Preferably, when training the tobacco insect recognition and positioning model, the second tobacco insect image is input into the tobacco insect recognition and positioning model, and the output data includes anchor frame information and classification information of whether the second tobacco insect image contains tobacco insects, and the first loss value is the weighted sum of the regression loss calculated based on the anchor frame information and the classification loss calculated based on the classification information.
[0021] Preferably, obtaining the grayscale threshold specifically includes:
[0022] For each threshold from 1 to 254, calculate the classification of each pixel in the grayscale image and calculate the inter-class variance corresponding to the threshold;
[0023] The threshold with the largest inter-class variance is taken as the grayscale threshold.
[0024] The present application also provides a tobacco insect identification and counting device based on coarse and fine granularity fusion, comprising an image receiving module and a prediction module;
[0025] The image receiving module is used for receiving the first tobacco insect image;
[0026] The prediction module is used for inputting the first tobacco insect image into the tobacco insect recognition and positioning model, and outputting the number of tobacco insects in the first tobacco insect image;
[0027] Among them, the prediction module includes a fusion feature acquisition module and a tobacco insect recognition module;
[0028] The fusion feature acquisition module is used to obtain a first fusion feature map containing image features of different coarse and fine granularities based on the first tobacco insect image;
[0029] The tobacco insect identification module is used to identify the number of tobacco insects based on the first fusion feature map.
[0030] Preferably, the fusion feature acquisition module includes a mask acquisition module, a downsampling module, an enhancement module and a fusion module;
[0031] The mask acquisition module is used to convert the first tobacco insect image into a grayscale image, and binarize the grayscale image based on a grayscale threshold to obtain a mask;
[0032] The downsampling module is used to perform multiple downsampling on the first tobacco insect image to obtain multiple first tobacco insect image feature maps with different coarse and fine granularity;
[0033] The enhancement module is used to enhance each first tobacco insect picture feature map using a mask to obtain a second tobacco insect picture feature map;
[0034] The fusion module is used to perform feature fusion on all second tobacco insect picture feature maps generated by the same first tobacco insect image to obtain a first fused feature map.
[0035] Preferably, the tobacco insect recognition module includes an anchor frame recognition module and a statistical module;
[0036] The anchor frame recognition module is used to recognize the anchor frame containing tobacco insects in the first fusion feature map;
[0037] The statistical module is used to count the number of anchor boxes as the number of tobacco insects in the first tobacco insect image.
[0038] Preferably, the tobacco insect identification and counting device based on coarse and fine granularity fusion also includes a training module. When the training module trains the tobacco insect identification and positioning model, the second tobacco insect image is input into the tobacco insect identification and positioning model, and the output data includes anchor frame information and classification information of whether the second tobacco insect image contains tobacco insects. The first loss value is the weighted sum of the regression loss calculated based on the anchor frame information and the classification loss calculated based on the classification information.
[0039] Preferably, when the mask acquisition module obtains the grayscale threshold, firstly, for each threshold from 1 to 254, the classification of each pixel in the grayscale image is calculated, and the inter-class variance corresponding to the threshold is calculated; then, the threshold with the largest inter-class variance is used as the grayscale threshold.
[0040] Other features and advantages of the present application will become apparent from the following detailed description of exemplary embodiments of the present application with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.
[0042] Figure 1 A flowchart of the tobacco insect identification and counting method based on coarse and fine granularity fusion provided in this application;
[0043] Figure 2 A flowchart for obtaining a first fusion feature map provided by this application;
[0044] Figure 3 A flowchart of the tobacco insect identification and positioning model provided in this application using a tripartite strategy to enhance the discrimination module to assist in training;
[0045] Figure 4 This is a schematic diagram of a tobacco insect image;
[0046] Figure 5 A schematic diagram of tobacco insect identification and counting results provided for this application;
[0047] Figure 6 This is a structural diagram of the tobacco insect identification and counting device based on coarse and fine granularity fusion provided in this application. DETAILED DESCRIPTION
[0048] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present application.
[0049] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.
[0050] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0051] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0052] The present application provides a method and device for identifying and counting tobacco insects based on coarse and fine granularity fusion, which extracts image features of different coarse and fine granularity from tobacco insect images to capture the morphological features of tobacco insects of different granularity in tobacco insect images, and through feature fusion, enables the model to have stronger recognition and discrimination capabilities for small-scale objects such as tobacco insects and impurities, improves the robustness of the model to interference from impurities such as dust and debris, obtains more discriminative tobacco insect image features, and improves the accuracy of tobacco insect counting. In addition, the present application uses a three-party strategy to enhance the discrimination module to assist in training the tobacco insect identification and positioning model, specifically by training the object discriminator to distinguish whether there are tobacco insects after the tobacco insect fusion features, noise background features, and impurity features are combined in pairs, to assist in improving the discrimination and discrimination of the features extracted by the tobacco insect identification and positioning model, and further improve the accuracy of tobacco insect counting.
[0053] like Figure 1 As shown, the tobacco insect identification and counting method based on coarse and fine granularity fusion provided in the present application includes:
[0054] S110: Receive a first tobacco insect image.
[0055] The tobacco insect image is provided by the cigarette factory and contains the image data of sticky insect boards and tobacco insects, such as Figure 4 shown.
[0056] S120: Input the first tobacco insect image into a tobacco insect recognition and positioning model, and output the number of tobacco insects in the first tobacco insect image.
[0057] As an embodiment, the backbone network of the tobacco insect recognition and positioning model includes a feature encoding network for obtaining a coarse and fine granularity combination of fused feature maps, a feature pyramid composite network for coarse and fine granularity feature fusion, and a feature decoding network for distinguishing tobacco insects and selecting anchor frame positions. The feature encoding network performs steps P2 and P3 in S1201 below, and the feature pyramid composite network performs step P4 in S1201 below. The feature decoding network performs S1202 below.
[0058] Specifically, the tobacco insect recognition and positioning model processes the first tobacco insect image, specifically including:
[0059] S1201: Obtain a first fused feature map containing image features of different coarse and fine granularities based on the first tobacco insect image.
[0060] As an example, Figure 2 As shown, obtaining a first fusion feature map containing image features of different coarse and fine granularities based on the first tobacco insect image includes the following steps:
[0061] P1: The first tobacco insect image (RGB image) Convert to grayscale image The grayscale image is binarized based on the grayscale threshold th to obtain the mask A. Wherein, H and W are the length and width of the image respectively.
[0062] Among them, the grayscale threshold is calculated as follows:
[0063]
[0064] Among them, OTSU is the maximum inter-class variance method, and the grayscale threshold th is obtained by calculating the inter-class variance.
[0065] Specifically, when obtaining the grayscale threshold, for each threshold from 1 to 254, calculate the classification of each pixel in the grayscale image. If the grayscale value of the pixel is less than or equal to the threshold, the pixel belongs to the C1 class, indicating the background of the image; if the grayscale value of the pixel is greater than the threshold, the pixel belongs to the C2 class, indicating the target; then calculate the inter-class variance corresponding to the threshold. Finally, the threshold with the largest inter-class variance is taken as the grayscale threshold th.
[0066] After obtaining the grayscale threshold th, the grayscale threshold th is used to obtain the mask A. Taking the RGB image with an input scale of 1920,1080 as an example, the mask A is as follows:
[0067]
[0068] If the gray value of the pixel (i, j) is less than or equal to the gray threshold th, the pixel belongs to class C1, and mask A is i,j=0; if the gray value of the pixel is greater than the gray threshold th, the pixel belongs to class C2, and a in mask A i,j =1.
[0069] P2: The feature encoding network downsamples the first tobacco insect image multiple times to obtain multiple first tobacco insect image feature maps with different coarse and fine granularity.
[0070] Specifically, as an embodiment, the feature encoding network combining coarse and fine granularity is a multi-layer neural network composed of convolutional layers, batch normalization layers, activation functions, etc. Here, the CSPDarknet network model is selected as the feature encoding network of the backbone network. Downsampling operation is performed in the convolutional layer, and the length and width of the image are halved through convolution with a step size of 2. After each downsampling, the number of channels of the image doubles, and the length and width become half of the original. If the batch size of the tobacco insect image input each time is N, for The feature map after downsampling is:
[0071]
[0072] Among them, F i and F i+1 They represent the first tobacco insect image features of different coarse and fine granularities obtained after i and i+1 times of downsampling respectively, and C is the number of channels.
[0073] P3: The feature encoding network uses a mask to enhance each first tobacco insect image feature map to obtain a second tobacco insect image feature map.
[0074] Specifically, after rescaling the mask A, the mask A is superimposed and averaged with the feature maps of each channel one by one in the spatial scale to obtain the feature map of the second tobacco insect image after mask enhancement. i For example, the feature map of the second tobacco insect image obtained after mask enhancement The calculation method is:
[0075]
[0076] Among them, Resize(·) represents the scale reconstruction operation, and Add(·) represents the superposition averaging operation.
[0077] P4: Perform feature fusion on all second tobacco insect image feature maps generated by the same first tobacco insect image to obtain a first fused feature map.
[0078] The second tobacco insect image feature maps of different coarse and fine granularity are input into the feature pyramid composite network to perform coarse and fine granularity feature fusion, and the first tobacco insect fusion feature map F of a specific size is output. The calculation method is as follows.
[0079]
[0080] Among them, Merg(·) represents the feature fusion operation; Proj(·) represents the mask enhancement of the image, which is a general term for feature encoding and coarse-grained and fine-grained feature fusion operations; x is the tobacco insect image.
[0081] Therefore, the tobacco insect fusion feature map integrates the coarse and fine granularity tobacco insect image information, enabling the model to have a stronger ability to recognize and distinguish small-scale objects such as tobacco insects and impurities.
[0082] S1202: The feature decoding network is used to identify the number of tobacco insects based on the first fused feature map.
[0083] In the prediction stage, the feature decoding network is used to locate the anchor box of the first fused feature map.
[0084] As an embodiment, in the prediction stage, identifying the number of tobacco insects based on the first fused feature map specifically includes:
[0085] Q1: Identify the anchor boxes containing tobacco worms in the first fused feature map.
[0086] When locating the anchor frame, we first use a convolutional layer of size 1 to uniformly adjust the number of channels of the first fusion feature map to 256. Then, after passing through two scale-invariant feature integration modules, we use a convolutional layer of size 1 to adjust the number of output channels to 4, that is, f ,W f ) feature map, adjusted to (N,4,H f ,W f ), and swap the positions of the second channel with the third and fourth channels to obtain a size of (N,H f ,W f ,4) output results, each set of 4 variable values corresponds to an anchor box in the image, where N is the batch size of the incoming tobacco insect image, H f ,W f is the length and width scale of the feature map. The feature map is regarded as a f ·W f The four variables of the anchor box obtained by regression are (a x ,a y ,a w ,a h ), a x ,a y Indicates the offset distance between the center of the anchor box and the corresponding upper left corner of the grid, a w ,a h Represents the width and length of the anchor box. The anchor box on the scale of the first fusion feature map is mapped to the scale of the first tobacco insect image according to the compression step size, and H can be obtained on the first tobacco insect image.f ·W f pre-selected anchor boxes, that is, one grid corresponds to one anchor box prediction result.
[0087] Among them, the scale-invariant feature integration module consists of a convolutional layer with a size of 3, a step size of 1, and an output channel number of 256, a batch normalization layer, and an activation function. This scale-invariant feature integration module is used to reduce the aliasing effect caused by the upsampling interpolation method in the feature pyramid and improve the final detection effect.
[0088] Q2: Count the number of anchor boxes as the number of tobacco insects in the first tobacco insect image. Figure 5 The tobacco insect counting results are shown in the upper right corner.
[0089] In the training phase, the data used is the image data of the sticky insect board and tobacco insects collected by the cigarette factory. Due to the small number of images, data generation is performed on this type of image to expand the data set. The specific method is as follows: Use labelme and other annotation software to annotate the tobacco insect data in the image to obtain the segmented tobacco insect image E = [e1, e2, e3, …, e m ], fill the image after the tobacco worms are removed, and obtain the background image B = [b1, b2, b3, ..., b k ]. Select n tobacco insect image samples, randomly flip them, and paste them onto any background image b. i (where 1≤i≤k), generate new image data x and corresponding target detection label y, repeat this step to obtain the expanded tobacco insect recognition count dataset {x i ,y i} 1≤i≤l , where l is the total number of samples in the dataset.
[0090] When training the tobacco insect identification and positioning model, during the training phase, the feature decoding network is used to simultaneously predict the two branches of anchor box positioning and classification for the first fusion feature map.
[0091] Specifically, in the training stage, the second tobacco insect image is input into the tobacco insect recognition and positioning model, and the output data includes anchor frame information and classification information of whether the second tobacco insect image contains tobacco insects.
[0092] For the anchor frame positioning principle, please refer to the description of Q1 above. For the classification branch, we first use a convolutional layer with a size of 1 to uniformly adjust the number of channels of the first fusion feature map to 256. Then, after passing through two scale-invariant feature integration modules, we use a convolutional layer with a size, step size, and output channel number of 1 to determine whether the grid contains tobacco worms. The final output result (N, H f ,W f ,1) is the confidence that the anchor frame corresponding to each grid contains tobacco worms.
[0093] Based on the above, the first loss value of the tobacco insect recognition and positioning model The regression loss calculated based on the anchor box information And the classification loss calculated based on the classification information The weighted sum of .
[0094] As an example, the classification loss Using cross entropy loss:
[0095]
[0096] Among them, y is the label, p is the prediction result, and CE(·) is the cross entropy loss function.
[0097] The regression loss is used to measure the positioning effect of the anchor box. The IOU (intersection over union) loss is selected as the regression loss. And the true value y = (y1, y2, y3, y4), respectively represents the two anchor boxes on the image, IoU loss IoUloss and the total regression loss for:
[0098]
[0099] Among them, H and W are the spatial scales of the fusion feature map corresponding to the tobacco insect image input during the training process.
[0100] First loss value for:
[0101] Among them, λ is set to 5.
[0102] Preferably, a three-party strategy enhancement discrimination module is used to assist in training the tobacco insect identification and positioning model. The three-party strategy enhancement discrimination module includes a convolutional neural network model (such as a deep CNN network) and an object discriminator (multi-layer perceptron). The convolutional neural network model is used to train the noisy background image X noi and the impurity image X dis Encode and obtain the noise background feature map F noi and impurity profile F dis Among them, the background image is continuously noised to obtain the noise background image X noi , by randomly sprinkling dust particles and tobacco leaf fragments on the sticky board to form impurities Image Library X dis .
[0103] The object discriminator is used to classify the noise background feature map F noi , Impurity characteristic diagram F dis and the second fusion feature map F obtained by inputting the second tobacco insect image into the tobacco insect recognition and positioning model in(Please refer to S1201) to form a third fusion feature map for classification prediction. noi , Impurity characteristic diagram F dis and the second fusion feature map F in Keep the scale consistent.
[0104] Specifically,
[0105]
[0106] The three are simply added to each other to obtain three third fusion feature maps: F mix1 Combining the features of smoke bugs and noise background, F mix2 Combining the characteristics of tobacco worms and impurities, F mix3 Combining the features of noise and impurities:
[0107]
[0108] As an embodiment, after the third fused feature map is input into the object discriminator, it passes through two feature integration modules and then into the global average pooling layer to flatten the spatial dimension, and then into a three-layer MLP network. After the softmax operation, the output layer outputs the classification prediction result of whether the third fused feature map contains tobacco insects.
[0109] The loss function of the object discriminator is the binary cross entropy loss:
[0110]
[0111] where y i is a label, whose value is 0 or 1, p(y i ) is the probability that the classification prediction result belongs to the corresponding label. For the fusion result containing tobacco insect features (F mix1 and F mix2 ), the expected object discriminator outputs the category of tobacco worms; for the fusion result of noise background features and impurity features (F mix3 ), the class output by the object discriminator is expected to be smokeless insects.
[0112] As an example, Figure 3 As shown in FIG. 1 , when the tobacco insect identification and positioning model is assisted in training using the tripartite strategy enhanced discrimination module, after initializing the tobacco insect identification and positioning model and the tripartite strategy enhanced discrimination module using the pre-trained weights, the network parameters are alternately iterated and trained. Specifically, the following steps are repeated until the total loss and the prediction accuracy are fully converged:
[0113] S310: Freeze all parameters of the three-party strategy enhanced discrimination module, input the second tobacco insect image, the noise background image and the impurity image into the entire system, and calculate the first loss value and the second loss value of the object discriminator The sum of The total loss is used to train the tobacco insect recognition and positioning model to improve the model's recognition performance for tobacco insects.
[0114] in,
[0115] S320: Freeze all parameters of the tobacco insect recognition and positioning model, input the second tobacco insect image, the noise background image and the impurity image into the entire system, and calculate the third loss value of the object discriminator The third loss value is used to train the convolutional neural network model and object discriminator of the tripartite strategy enhanced discrimination module to enhance the object discriminator's ability to distinguish different features.
[0116] S310 and S320 are repeatedly executed until the total loss and the prediction accuracy are fully converged.
[0117] Among them, the validation set is used to calculate the prediction accuracy of the tobacco insect identification and positioning model.
[0118] Specifically, the tobacco insect image dataset {x i ,y i} 1≤i≤l 10% of the data is extracted as a validation set. For each image x input into the validation set i , set the threshold to 0.7. If the intersection of the data output by the tobacco insect recognition and positioning model and the actual label anchor frame of the tobacco insect image is above the threshold, the tobacco insect recognition and positioning model is considered to be correctly positioned. Count all the correctly positioned anchor frames, which is the predicted total number of tobacco insects in the tobacco insect image. If the predicted total number of tobacco insects in the tobacco insect image is the same as the number of labels, the prediction is considered correct, otherwise it is considered wrong. The prediction accuracy of the tobacco insect recognition and positioning model is:
[0119]
[0120] where n correct To predict the correct number of images, n total The total number of pictures.
[0121] In the prediction stage, the trained tobacco insect recognition and positioning model is used to realize the recognition and positioning of tobacco insects, and the number of anchor frames of tobacco insects is counted to realize the counting function.
[0122] In order to more comprehensively and reliably demonstrate the superior performance of the tobacco insect identification and counting method with coarse and fine granularity fusion under the three-party enhancement strategy designed in this application in the actual tobacco insect counting task, the traditional OTSU threshold segmentation, the residual network Resnet50 for classification tasks, and the target detection network YOLOX were selected as comparison methods for performance comparison, and the results are shown in Table 1. For the counting task of each method, 3 parallel experiments were conducted and the 3 results were averaged. It can be found that the counting accuracy of the method proposed in this application is significantly better than other methods, which proves the feasibility and superiority of the method proposed in this application.
[0123] Table 1. Comparison of accuracy in tobacco worm identification and counting tasks
[0124]
[0125] Based on the above, the present application also provides a tobacco insect identification and counting device based on coarse and fine granularity fusion. Figure 6 As shown, the tobacco insect identification and counting device includes an image receiving module 610 and a prediction module 620 .
[0126] The image receiving module 610 is used to receive a first tobacco insect image.
[0127] The prediction module 620 is used to input the first tobacco insect image into the tobacco insect recognition and positioning model, and output the number of tobacco insects in the first tobacco insect image.
[0128] The prediction module 620 includes a fusion feature acquisition module 6201 and a tobacco insect recognition module 6202 .
[0129] The fusion feature acquisition module 6201 is used to obtain a first fusion feature map containing image features of different coarse and fine granularities based on the first tobacco insect image.
[0130] The tobacco insect identification module 6202 is used to identify the number of tobacco insects based on the first fusion feature map.
[0131] Preferably, the fusion feature acquisition module 6201 includes a mask acquisition module 62011, a downsampling module 62012, an enhancement module 62013 and a fusion module 62014.
[0132] The mask acquisition module 62011 is used to convert the first tobacco insect image into a grayscale image, and binarize the grayscale image based on a grayscale threshold to obtain a mask.
[0133] The downsampling module 62012 is used to perform multiple downsampling on the first tobacco insect image to obtain multiple first tobacco insect image feature maps with different coarse and fine granularities.
[0134] The enhancement module 62013 is used to enhance each first tobacco insect picture feature map using a mask to obtain a second tobacco insect picture feature map.
[0135] The fusion module 62014 is used to perform feature fusion on all second tobacco insect picture feature maps generated by the same first tobacco insect image to obtain a first fused feature map.
[0136] Preferably, the tobacco insect recognition module 6202 includes an anchor frame recognition module 62021 and a statistical module 62022 .
[0137] The anchor frame recognition module 62021 is used to identify the anchor frame containing tobacco insects in the first fused feature map.
[0138] The statistical module 62022 is used to count the number of anchor frames as the number of tobacco insects in the first tobacco insect image.
[0139] Preferably, the tobacco insect identification and counting device based on coarse and fine granularity fusion also includes a training module 630. When the training module 630 trains the tobacco insect identification and positioning model, the second tobacco insect image is input into the tobacco insect identification and positioning model, and the output data includes anchor frame information and classification information of whether the second tobacco insect image contains tobacco insects. The first loss value is the weighted sum of the regression loss calculated based on the anchor frame information and the classification loss calculated based on the classification information.
[0140] Preferably, when the mask acquisition module 62011 obtains the grayscale threshold, it first calculates the classification of each pixel in the grayscale image for each threshold from 1 to 254, and calculates the inter-class variance corresponding to the threshold. Then, the threshold with the largest inter-class variance is used as the grayscale threshold.
[0141] Preferably, the training module 630 is also used to utilize the three-party strategy enhanced discrimination module to assist in training the tobacco insect identification and positioning model, and the three-party strategy enhanced discrimination module includes a convolutional neural network model and an object discriminator.
[0142] The convolutional neural network model is used to encode the noise background image and the impurity image to obtain the noise background feature map and the impurity feature map.
[0143] The object discriminator is used to classify and predict the noise background feature map, the impurity feature map and the third fused feature map formed by combining the second fused feature map obtained by inputting the second tobacco insect image into the tobacco insect recognition and positioning model.
[0144] Preferably, the training module 630 includes an initialization module 6301 , a first training module 6302 , a second training module 6303 and a calculation module 6304 .
[0145] The initialization module 6301 is used to initialize the tobacco insect identification and positioning model and the three-party strategy enhanced discrimination module using pre-trained weights.
[0146] The first training module 6302 is used to freeze all parameters of the three-party strategy enhanced discrimination module, input the second tobacco insect image, the noise background image and the impurity image, calculate the sum of the first loss value and the second loss value of the object discriminator as the total loss, and use the total loss to train the tobacco insect recognition and positioning model.
[0147] The second training module 6303 is used to freeze all parameters of the tobacco insect recognition and positioning model, input the second tobacco insect image, the noise background image and the impurity image, calculate the third loss value of the object discriminator, and use the third loss value to train the convolutional neural network model and the object discriminator.
[0148] The calculation module 6304 is used to calculate the total loss and use the verification set to calculate the prediction accuracy of the tobacco insect identification and positioning model.
[0149] This application uses a grayscale threshold-based image segmentation method and a target detection network to perform coarse-grained and fine-grained feature fusion, reduce information loss in the process of extracting high-level semantic information by the convolutional neural network, and extract more effective features; and uses a three-party enhancement strategy to distinguish the fusion feature map of tobacco insects, the noise background feature map, and the impurity feature map, thereby improving the robustness of the model to impurity interference. Therefore, this application can effectively improve the quality and discrimination of representation and achieve better performance in tobacco insect identification and counting tasks.
[0150] Although some specific embodiments of the present application have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are only for illustration, not for limiting the scope of the present application. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.
Claims
1. A tobacco insect identification and counting method based on coarse and fine granularity fusion, characterized in that: include: receiving a first tobacco insect image; Inputting the first tobacco insect image into a tobacco insect recognition and positioning model, and outputting the number of tobacco insects in the first tobacco insect image; The tobacco insect identification and positioning model processes the first tobacco insect image, specifically including: Obtaining a first fused feature map containing image features of different coarse and fine granularities based on the first tobacco insect image; The number of tobacco insects is identified based on the first fused feature map.
2. The tobacco insect identification and counting method based on coarse and fine granularity fusion according to claim 1 is characterized in that: Obtaining a first fused feature map containing image features of different coarse and fine granularities based on the first tobacco insect image includes the following steps: Converting the first tobacco insect image into a grayscale image, and binarizing the grayscale image based on a grayscale threshold to obtain a mask; Downsampling the first tobacco insect image multiple times to obtain multiple first tobacco insect image feature maps with different coarse and fine granularities; Using the mask to enhance each first tobacco insect picture feature map to obtain a second tobacco insect picture feature map; All second tobacco insect picture feature maps generated by the same first tobacco insect image are feature fused to obtain the first fused feature map.
3. The tobacco insect identification and counting method based on coarse and fine granularity fusion according to claim 1 is characterized in that: Identifying the number of tobacco insects based on the first fused feature map specifically includes: identifying an anchor frame containing tobacco insects in the first fused feature map; The number of the anchor frames is counted as the number of tobacco insects in the first tobacco insect image.
4. The tobacco insect identification and counting method based on coarse and fine granularity fusion according to claim 2 is characterized in that: When training the tobacco insect recognition and positioning model, the second tobacco insect image is input into the tobacco insect recognition and positioning model, and the output data includes anchor frame information and classification information of whether the second tobacco insect image contains tobacco insects. The first loss value is the weighted sum of the regression loss calculated based on the anchor frame information and the classification loss calculated based on the classification information.
5. The tobacco insect identification and counting method based on coarse and fine granularity fusion according to claim 2 is characterized in that: Obtaining the grayscale threshold specifically includes: For each threshold value from 1 to 254, calculate the classification of each pixel in the grayscale image, and calculate the inter-class variance corresponding to the threshold value; The threshold with the largest inter-class variance is used as the grayscale threshold.
6. A tobacco insect identification and counting device based on coarse and fine granularity fusion, characterized in that: It includes an image receiving module and a prediction module; The image receiving module is used to receive a first tobacco insect image; The prediction module is used to input the first tobacco insect image into a tobacco insect identification and positioning model, and output the number of tobacco insects in the first tobacco insect image; Wherein, the prediction module includes a fusion feature acquisition module and a tobacco insect recognition module; The fusion feature acquisition module is used to obtain a first fusion feature map containing image features of different coarse and fine granularities based on the first tobacco insect image; The tobacco insect identification module is used to identify the number of tobacco insects based on the first fusion feature map.
7. The tobacco insect identification and counting device based on coarse and fine granularity fusion according to claim 6 is characterized in that: The fusion feature acquisition module includes a mask acquisition module, a downsampling module, an enhancement module and a fusion module; The mask acquisition module is used to convert the first tobacco insect image into a grayscale image, and binarize the grayscale image based on a grayscale threshold to obtain a mask; The downsampling module is used to perform multiple downsampling on the first tobacco insect image to obtain multiple first tobacco insect image feature maps with different coarse and fine granularities; The enhancement module is used to enhance each first tobacco insect picture feature map using the mask to obtain a second tobacco insect picture feature map; The fusion module is used to perform feature fusion on all second tobacco insect picture feature maps generated by the same first tobacco insect image to obtain the first fused feature map.
8. The tobacco insect identification and counting device based on coarse and fine granularity fusion according to claim 6 is characterized in that: The tobacco insect recognition module includes an anchor frame recognition module and a statistical module; The anchor frame recognition module is used to recognize the anchor frame containing tobacco insects in the first fusion feature map; The statistical module is used to count the number of the anchor frames as the number of tobacco insects in the first tobacco insect image.
9. The tobacco insect identification and counting device based on coarse and fine granularity fusion according to claim 7 is characterized in that: It also includes a training module. When the training module trains the tobacco insect recognition and positioning model, the second tobacco insect image is input into the tobacco insect recognition and positioning model, and the output data includes anchor frame information and classification information of whether the second tobacco insect image contains tobacco insects. The first loss value is a weighted sum of the regression loss calculated based on the anchor frame information and the classification loss calculated based on the classification information.
10. The tobacco insect identification and counting device based on coarse and fine granularity fusion according to claim 7, characterized in that: When the mask acquisition module obtains the grayscale threshold, it first calculates the classification of each pixel in the grayscale image for each threshold from 1 to 254, and calculates the inter-class variance corresponding to the threshold; then the threshold with the largest inter-class variance is used as the grayscale threshold.
