Early intelligent diagnosis method for silkworm disease by fusing behavior and visual features

Abstract

The present application belongs to the field of agricultural informatization, and particularly relates to a kind of early intelligent diagnosis method of silkworm pyemia fusion behavior and visual feature, according to the early stage of silkworm infection pyemia, will show the behavior characteristics of frenzied crawling, and will appear the visual characteristics such as segment swelling and pus outflow, using the deep learning in artificial intelligence and computer vision technology, the visual features and behavior characteristics of silkworm are extracted respectively, and then the two are fused, further identify whether the model diagnoses silkworm infection pyemia, the result is also more objective and accurate. The present application can realize early detection of silkworm pyemia, and further develop application software to be applied to intelligent breeding equipment, which has very good application prospect.

Description

Technical Field

[0001] This invention belongs to the field of agricultural informatization, and in particular relates to an intelligent early diagnosis method for silkworm pustular disease that integrates behavioral and visual characteristics. Background Technology

[0002] The silkworm is an important economic insect, raised throughout my country. However, silkworms are highly susceptible to diseases, among which silkworm septicemia is a major disease with a high incidence and strong infectivity, accounting for more than 60% of all diseases annually, seriously threatening the safety of my country's sericulture production.

[0003] Early screening of diseased silkworms in real-world environments is crucial for precise prevention and control, reducing losses. However, existing silkworm disease identification methods based on artificial intelligence and computer vision are limited to visual features of diseased silkworms. These methods use images of both diseased and healthy silkworms as datasets and then employ convolutional neural networks to build recognition or localization models. However, visual features are easily affected by silkworm breed, collection environment, and collection equipment, resulting in very low reliability and effectiveness only when obvious disease characteristics are present.

[0004] This invention is based on the abnormal behavioral characteristics of silkworms when they are infected, namely, frantic crawling, and in the middle and late stages of infection, swollen segments and pus discharge. Deep learning technology is used to extract these two characteristics and then perform identification. The method is not affected by factors such as variety, environment, and equipment, and can provide an accurate and reliable intelligent diagnostic method for early septicemia. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an intelligent early diagnosis method for silkworm septicemia that integrates behavioral and visual characteristics, aiming to achieve intelligent early diagnosis of septicemia in the breeding environment.

[0006] To achieve the above objectives, the present invention provides an intelligent early diagnosis method for silkworm septicemia that integrates behavioral and visual features, which keyly includes the following steps:

[0007] S1. Detect silkworms in video images using an individual detection model and extract behavioral features;

[0008] S2. Detect silkworms in video images and extract visual features using an individual identity re-identification model;

[0009] S3. The extracted visual and behavioral features are fused and input into a recognition model that combines CNN and Vision Transformer to predict whether each silkworm is infected with septicemia. Since silkworms infected with septicemia exhibit manic crawling behavior and visual features such as swollen segments and pus discharge in the early stages, this invention utilizes deep learning and computer vision techniques from artificial intelligence to extract the visual and behavioral features of the silkworms, then fuses them, and further uses a recognition model that combines CNN and Vision Transformer to diagnose whether the silkworms are infected with septicemia.

[0010] Furthermore, when implementing this method, first use a camera to capture images of the silkworms from directly above. There is no limit to the number of silkworms captured. Alternatively, a video of the silkworms can be pre-recorded for use in diagnosis.

[0011] For any frame (image) captured, the first frame image is used. The pre-trained individual detection model is used to locate the silkworm. The position of each silkworm in the image is represented by the four vertices of the bounding rectangle of the image. The position information of each silkworm is stored in a linked list.

[0012] For each silkworm in the first frame image, it is cropped out of the image based on its individual information, and the deep convolutional features of each silkworm are extracted using a pre-trained individual identity re-identification model. The results are then saved to a linked list.

[0013] Use a Kalman filter to predict the position information of each silkworm in the second frame of the first frame image, and store the prediction results in a linked list.

[0014] A pre-trained individual detection model is used to locate individual silkworms in the second frame image, and the location results are stored in a linked list. Kalman filtering is then used to predict their possible locations in the third frame image.

[0015] Calculate the intersection-union ratio (IUU) of the bounding rectangle of each silkworm in the second frame with the predicted position information of each silkworm in the first frame. Match silkworms with an IUU ≥ 0.5 as the same silkworm and assign them an identification number.

[0016] For detection results that fail to match in the second frame and prediction results in the first frame, the cosine similarity of their deep convolutional features is calculated using a pre-trained individual identity re-identification model. Those with a result ≥0.5 are considered to be the same silkworm and assigned an identity label.

[0017] Individual detection, location prediction, identity matching, etc. are repeated on each subsequent frame until the last frame or video recording stops.

[0018] Finally, based on the matched identity tags, the behavioral trajectory of each silkworm in the video frame is output as its behavioral feature.

[0019] The behavior trajectory of the silkworm is output once every 100 frames, and the behavior trajectory of each silkworm is drawn separately on a single frame.

[0020] The graph neural network was used to extract depth features from the behavioral trajectory of each silkworm. Then, 3 frames were randomly selected from these 100 frames, and the features obtained by individual identity re-identification were used as the depth features of the silkworm.

[0021] The extracted deep features and behavioral features are concatenated in parallel, and a pre-trained CNN and Vision Transformer model is used to predict whether each silkworm is infected with septicemia. The prediction results are then labeled onto the original frame images.

[0022] Furthermore, this method requires pre-training a silkworm individual detection model, with the detection target being the silkworm's head rather than the entire silkworm.

[0023] Furthermore, when constructing the individual detection model, it is necessary to collect silkworm images in a real environment, with a number of images ≥10,000 and an image size ≥640×640. Then, an annotation tool is used to mark the position of each silkworm's head in the image, using the bounding rectangle of the head to represent its position. The detector is based on YOLO v8, with a channel correction module added after the feature extraction network, and a spatial enhancement module added between the backbone network and the feature fusion network. Its residuals are connected to the detection head to enhance its ability to detect small targets. During training, the detection accuracy is improved to ≥97% through optimization methods and data augmentation, thus forming a strong detection capability.

[0024] Furthermore, this method requires pre-training a silkworm individual identification re-recognition model. When constructing the model, a dataset needs to be pre-created containing more than 5000 individual silkworms, with each silkworm having more than 100 images from different angles, postures, and instars. The recognizer model uses pyramid-structured convolutional kernels to enhance the network's receptive field and feature representation ability, and also adds a channel attention mechanism to enhance the network's discrimination ability. During training, images are enhanced using methods such as rotation and adding noise to achieve a recognition accuracy of ≥95%, thus forming a strong discrimination capability.

[0025] Furthermore, this method operates on video data, which can be pre-captured video files stored on a computer or real-time video captured by a camera, simultaneously performing detection, matching, and diagnosis. Since the silkworm's position in the video does not undergo sudden changes, diagnosis can be performed every 5 to 10 frames to improve the method's efficiency, and the diagnostic results are annotated on the video in real time.

[0026] Furthermore, for the behavioral characteristics of each silkworm, the behavioral trajectory of the silkworm was obtained from the video data, and the behavioral trajectory of each silkworm was plotted separately on a single image frame, which should contain trajectory information from 100 frames of images in the time series. Then, a graph neural network was used to extract deep features from the trajectory images. For the visual characteristics of each silkworm, three images were randomly selected from the frames extracted for behavioral characteristics, and deep features were extracted using an individual identity re-identification model. The behavioral features and deep features were then concatenated in parallel for identification.

[0027] Furthermore, the fused behavioral and visual features are input into a classifier that integrates CNN and Vision Transformer architectures to determine whether silkworms are infected with septicemia.

[0028] Furthermore, this method requires training before practical application. Training can be performed using only healthy silkworms to generate classifier weights; alternatively, healthy and diseased silkworms can be manually labeled for model training. In subsequent applications, diagnosis can be made based on the classifier's output; silkworms exceeding a threshold are identified as diseased.

[0029] Beneficial Effects: This invention provides an intelligent early diagnosis method for silkworm septicemia that integrates behavioral and visual features. Based on the early behavioral characteristics of silkworms infected with septicemia, such as frantic crawling and visual features like swollen segments and pus discharge, this invention utilizes deep learning and computer vision techniques from artificial intelligence to extract the visual and behavioral features of the silkworms, then fuses them to further identify and diagnose whether the silkworms are infected with septicemia. The results are more objective and accurate. This invention enables early detection of silkworm septicemia, and further development of application software allows for its application in intelligent breeding equipment, showing great promise for future applications. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the structure of the present invention;

[0031] Figure 2 This is a flowchart of the method of the present invention;

[0032] Figure 3 This is a schematic diagram of the individual detection model of the present invention;

[0033] Figure 4 This is a schematic diagram of the channel correction module of the present invention;

[0034] Figure 5 This is a schematic diagram of the space enhancement module of the present invention;

[0035] ] Figure 6 This is a schematic diagram of the identity re-identification network of the present invention;

[0036] Figure 7 This is a schematic diagram of the graph neural network of the present invention;

[0037] Figure 8 This is a schematic diagram of the feature blocks of the classifier of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention are described clearly and completely below. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0039] Please see Figure 1 A novel intelligent diagnostic method for early silkworm septicemia that integrates behavioral and visual features is proposed. This method uses video data to acquire behavioral and visual characteristics of silkworms for early diagnosis of septicemia. The videos can be pre-collected and stored in a computer or captured in real time using a camera.

[0040] Please see Figure 2 As shown, the main flow of the method includes:

[0041] 1) Individual detection: Detect all silkworms from each frame (or every 5-10 frames) of the video using a pre-trained individual detection model. The detection target is the head of the silkworm, not the entire body.

[0042] Specifically, when acquiring behavioral features:

[0043] 2) Individual localization: The relative coordinates of the four vertices of the bounding rectangle of each silkworm's head predicted by the individual detection model in the image.

[0044] 3) Position prediction: Based on the position coordinates of each silkworm's head, a Kalman filter is used to predict its possible position coordinates in the next frame image.

[0045] 4) Feature extraction: Use a pre-trained individual identity re-identification model to extract deep convolutional features of each silkworm head.

[0046] 5) Individual matching: Hungarian and cosine similarity are used to match and associate the individual detection results of silkworms in the current frame with the location prediction results. Those that are successfully matched share an identity label. The accurate location of the silkworm in the frame is also corrected.

[0047] 6) Based on the position information of each silkworm's head on each frame of the video, obtain the head movement trajectory of each silkworm.

[0048] 7) Use a pre-trained graph neural network to extract the depth information (behavioral features) of the activity trajectory of each silkworm.

[0049] When acquiring visual features:

[0050] 8) Randomly extract frame images from the video and crop out each individual silkworm.

[0051] 9) Use a pre-trained individual identity re-identification network to extract deep convolutional features (visual features) of each silkworm head.

[0052] Fusion and Diagnosis:

[0053] 10) After fusing the behavioral and visual characteristics of each silkworm, input them into the classifier to diagnose whether the silkworm is infected with septicemia.

[0054] 11) Draw and dynamically display the diagnostic results of each silkworm on the original image frame.

[0055] In this embodiment, a pre-trained individual detection model is required to detect individual silkworms from selected frames of the video.

[0056] In this embodiment, the individual detection targets the head of each silkworm, rather than the entire silkworm body. This is because, in a real breeding environment, silkworms are often covered by mulberry leaves or other silkworm bodies. Since silkworms have the habit of crawling upwards, their heads are usually not covered and are very active, making it easy to extract their movement trajectory.

[0057] Please see Figure 3 The individual detection model is built on YOLO v8. To improve persistence performance and suppress image interference, a Channel Correction Module (CRM) is added after the feature extraction block of the original network to enhance the model's feature representation capability. Figure 3 In this code, "3×3Conv, stride=2" represents a convolution operation with a kernel size of 3×3 and a stride of 2. "Conv" represents a convolutional layer, "BN" represents Batch Normalization, "SiLU" is the activation function, "E-ELAN" represents the original feature extraction block of YOLO v8, "CRM" represents the channel correction network, "UpS" represents upsampling, "NMS" represents non-maximum suppression, "MP" represents max pooling, and "ReC" represents Rep convolution operation.

[0058] In this embodiment, the head of a silkworm is used as the detection target, which is a small target detection. A spatial enhancement module (SEM) is added between the main body of the original principle and feature fusion, and directly connected to the detection head to enhance the detection capability of small targets.

[0059] Please see Figure 5 In the diagram, "X" represents the input, "Y" represents the output, "W," "H," and "C" represent the width, height, and number of channels of the feature map, respectively. "GAP" represents global average pooling, "Linear Transform" represents linear mapping, "Soft feature aggregation" represents flexible feature aggregation, "element-wise product" represents pixel multiplication, and "1D Conv" represents 1D convolution. The channel correction network uses flexible feature aggregation to preserve the inherent relationships between the original features, such as relative spatial location and shallow semantic information. Flexible feature aggregation is achieved through depthwise separable convolutions with a kernel size of 3×3. Then, global average pooling is used to aggregate the features between each channel. Further, 1D convolution is used to obtain the attention weights between each channel, and a normalization function is used to activate the features. Finally, the feature weights are multiplied by the features of the flexible mapping to achieve channel correction.

[0060] Please see Figure 5 In the figure, "DConv" represents dilated convolution, "Concatenate" represents feature merging, and "C / 2" represents half the number of input channels. The spatial enhancement module uses dilated convolution to enhance the receptive field of the model. In the three cascaded convolutional layers, the dilation coefficient gradually increases to 1, 3, and 5, respectively.

[0061] In this embodiment, when the individual detection model is used, silkworm images need to be collected in a real environment and the model needs to be trained. The main method is to use a handheld device or camera to collect ≥10,000 images in a real breeding environment, with an image size ≥640×640. Then, the position of each silkworm's head in the image is marked using a labeling tool, and its position is represented by the bounding rectangle of the head.

[0062] In this embodiment, when training an individual detection model, the detection accuracy can be improved to ≥97% by means of optimization and data augmentation, so as to form a strong detection capability.

[0063] In this embodiment, a pre-trained individual identity re-identification model is needed to extract the deep convolutional features of each silkworm.

[0064] Please see Figure 6 The basic extraction module of the individual identity re-identification model uses a pyramid-structured convolutional kernel with sizes of 7×7, 5×5 and 3×3. This structure can significantly enhance the receptive field of the recognition model and obtain fine-grained features of the silkworm head from coarse to fine in spatial dimension. Then, the image attention mechanism (SENet) is used to improve the ability to extract key information and channel correction.

[0065] Please see Figure 7The “CSP-SE block” in the diagram is... Figure 6 The individual identity re-identification model is based on an image classification model, but is implemented by removing the classification layer.

[0066] In a more preferred embodiment, the individual identity re-identification model needs to be pre-trained to develop a strong feature extraction capability.

[0067] In this embodiment, the individual identity re-identification model needs to collect images of individual silkworms in a real environment to create a dataset. The dataset contains more than 5,000 individual silkworms, and each silkworm contains more than 100 images of different angles, postures, and instars.

[0068] In this embodiment, the input for training the individual identification model is two images. If the two images belong to the same silkworm, the training objective is to make the cosine similarity of the two vectors output by the network equal to 1. If they do not belong to the same silkworm, the output vector is made to approach 0. The ternary loss function is used as the optimization method for the individual identification model.

[0069] During training, images are enhanced using methods such as rotation and adding noise to achieve a recognition accuracy of ≥95% in order to develop strong recognition capabilities.

[0070] In this embodiment, based on the individual detection results of the current frame, a Kalman filter is used to predict the possible location of each silkworm in the next frame.

[0071] In a more preferred embodiment, for the detection result of the current frame, the intersection-union ratio (IUR) of the bounding rectangle of each silkworm in the current frame and the predicted bounding rectangle of the previous frame is calculated. If the IUR is ≥0.5, it is considered to be the same silkworm, and its position in the two frames constitutes a trajectory.

[0072] In a better embodiment, there may be a mismatch between the prediction of the previous frame and the detection result of the current frame. In this case, it is necessary to use the individual identity re-identification model to calculate the detection depth feature vector, and then use the Hungarian algorithm to perform centralized matching on the cosine similarity between all unmatched or undetected heads, and regard those with a similarity ≥ 0.5 as the same silkworm.

[0073] In this embodiment, if a silkworm does not match in 5 consecutive frames, it is considered to be occluded, and its information is retained. The matching is queried in every frame until the end of the video. For newly appearing detection results, an identification number can be added.

[0074] In this embodiment, the input is video data, and after individual detection and object matching, the output is the head movement trajectory of each silkworm.

[0075] In a more preferred embodiment, the acquired head trajectory of each silkworm is used as graph data, and a graph neural network is used to extract the depth features of the trajectory.

[0076] In this embodiment, the graph neural network is a graph convolutional neural network, and the position coordinates of the silkworm on each frame of the graph data are node data.

[0077] Except for the input layer, which is a graph, the other parts of the network adopt the same network structure as the individual identity re-identification, and the network output is a 1-dimensional vector.

[0078] In this embodiment, when extracting the visual features of silkworms, three frames are randomly selected from the video frame, and then the individual identity re-identification model is used to extract the deep convolutional features of each silkworm.

[0079] After extracting the behavioral and visual features of the silkworm separately, a 1D convolution operation is used to fuse the two.

[0080] Please see Figure 8 The recognizer is a hybrid structure that combines CNN and Vision Transformer. The Transformer is used to obtain global features, while the CNN is used to obtain local features. To improve efficiency, the Transformer's feature extraction adopts a sliding window self-attention mechanism.

[0081] The model structure of the recognizer is the same as that of individual identity re-identification, and its output consists of two categories, representing either health or illness.

[0082] In this embodiment, a supervised learning method is used to pre-train the classifier. First, healthy silkworms are used as the output to establish the threshold for the discrimination criteria. Later, the classifier learns the accurate parameters and can be directly deployed to the breeding environment.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the foregoing technical solutions, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the various technical solutions of the present invention.

Claims

1. A method for early intelligent diagnosis of silkworm septicemia integrating behavioral and visual features, characterized in that... Includes the following steps: S1. Detect silkworms in video images using an individual detection model and extract behavioral features; S2. Detect silkworms in video images and extract visual features using an individual identity re-identification model; S3. The extracted visual and behavioral features are fused and input into a recognition model that combines CNN and Vision Transformer to predict whether each silkworm is infected with septicemia. The individual detection model in S1 is based on YOLO v8 architecture. A channel correction module is added after the feature extraction network, and a spatial enhancement module is added between the backbone network and the feature fusion network. Channel correction is achieved through the channel correction network. The trained individual detection model has a detection accuracy of ≥97%. The individual identity re-identification model is based on a pyramid-structured convolutional kernel and incorporates a channel attention mechanism to enhance the extraction of key information and channel correction. The trained silkworm individual identity re-identification model achieves an accuracy of ≥95%. Specifically, S1 is: S1.1 Take any frame image in the video as the first frame image, use a pre-trained individual detection model to locate the head of the silkworm, use the four vertices of the bounding rectangle of each silkworm in the image to represent its position in the image, and store the position information of each silkworm in a linked list. S1.2 Based on the position coordinates of each silkworm's head, use a Kalman filter to predict its possible position coordinates in the next frame image; S1.3 Use a pre-trained individual identity re-identification model to extract deep convolutional features from each silkworm head; S1.4 Use Hungarian and cosine similarity to match and associate the individual detection results of silkworms in the current frame with the location prediction results. If the association is successful, they share an identity label and correct the accurate position of the silkworm in the frame. S1.5 Based on the position information of each silkworm's head on each frame of the video, obtain the head movement trajectory of each silkworm. S1.6 Use a pre-trained graph neural network to extract behavioral features of the activity trajectory of each silkworm; In the recognition model of S3, Transformer is used to obtain global features, and the feature extraction of Transformer adopts the sliding window self-attention mechanism. CNN is used to obtain local features.

2. The method for early intelligent diagnosis of silkworm septicemia by integrating behavioral and visual features according to claim 1, characterized in that: The video images can be pre-shot video files stored on a computer, or videos captured in real time by a camera, which are then simultaneously detected, matched, and diagnosed.

3. The method for early intelligent diagnosis of silkworm septicemia by integrating behavioral and visual features according to claim 1, characterized in that: When constructing the individual detection model, silkworm images need to be collected in a real environment, with a number of images ≥10,000 and an image size ≥640×640. Then, the position of each silkworm's head in the image is marked using a labeling tool, and its position is represented by the bounding rectangle of the head.

4. The method for early intelligent diagnosis of silkworm septicemia by integrating behavioral and visual features according to claim 1, characterized in that: When constructing the silkworm individual identity re-identification model, a dataset needs to be prepared in advance. The dataset contains more than 5,000 individual silkworms, and each silkworm contains more than 100 images from different angles, postures and instars.

5. The method for early intelligent diagnosis of silkworm septicemia by integrating behavioral and visual features according to claim 1, characterized in that: The identification model in S3 needs to be trained using a supervised learning method. That is, firstly, all outputs are healthy silkworms to establish the threshold of the discrimination criteria. Later, the classifier learns the accurate parameters and can be directly deployed to the breeding environment.

6. The method for early intelligent diagnosis of silkworm septicemia by integrating behavioral and visual features according to claim 1, characterized in that... Specifically, S2 is: S2.1 Randomly extract frame images from the video and crop out each individual silkworm from them; S2.2 Use a pre-trained individual identity re-recognition model to extract visual features of each silkworm head.

Images