Method for identifying individual silkworms across time series under dense conditions
By combining deep learning and visual technology with Kalman filtering to predict the location of silkworms, the problem of individual silkworm identification in dense environments has been solved, achieving high-precision and high-speed individual silkworm identification.
Patent Information
- Application Number
- CN202310774280.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-28
- Publication Date
- 2025-11-28
- Estimated Expiration
- 2043-06-28
AI Technical Summary
Existing technologies cannot effectively identify the individual identities of silkworms in dense environments, and existing methods are mainly used for pedestrians or livestock, failing to adapt to the high-density rearing scenarios of silkworms.
By employing deep learning and vision technologies, an initial individual detection is performed using an object detection model, the location of silkworms is predicted using Kalman filtering, and object matching is performed by combining a retrieval and recognition model, thereby achieving cross-time series identification of individual silkworms.
High-precision and high-speed identification of individual silkworms was achieved under dense conditions, and the results are more objective and accurate, making it suitable for individual silkworm identification in dense environments.
Smart Images
Figure CN116740148B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of agricultural informatization, and particularly relates to a silkworm individual identity retrieval and identification method under dense conditions and across time sequences. BACKGROUND
[0002] Silkworms belong to a kind of insects that are bred in high density, and the identity of each silkworm can be identified in a real environment to carry out early disease diagnosis, accurate feeding and life form analysis.
[0003] Existing individual identity retrieval and identification methods are mainly used for pedestrian re-identification, and since pedestrians are always in a standing state, the pedestrian identity retrieval and identification method cannot be directly applied to silkworms. Some methods achieve individual identity identification of livestock and poultry, but the identification scene is relatively simple and cannot be involved in dense environments.
[0004] The application mainly uses individual detection and matching to retrieve and identify silkworm individuals according to the fact that the position of silkworms will not change, the number of silkworms in the cocoon will not change in the same time period, and the depth features of each silkworm are different. Only a silkworm individual detection model needs to be trained, and the method can be widely used to carry out silkworm individual identity retrieval and identification, and the results are more objective and accurate. SUMMARY
[0005] The application aims to solve the defects of the existing technology, use deep learning and visual technology, and provide a silkworm individual identity retrieval and identification method under dense conditions and across time sequences.
[0006] In order to achieve the above purpose, the application provides a silkworm individual identity retrieval and identification method under dense conditions and across time sequences, which is characterized by comprising the following steps:
[0007] S1. An individual detection model is used to detect silkworms in a fixed area at an initial time, and each detected silkworm is assigned an identity number;
[0008] S2. The position of each silkworm at the next time is predicted using Kalman filtering, and an individual detection model is used to detect the silkworms at the time, and each detected silkworm is assigned a temporary identity number;
[0009] S3. A retrieval and identification model is used to match the detection results at the current time with the detection results at the previous time, and silkworms that are successfully matched share the same identity number, and silkworms that are not successfully matched retain the identity number;
[0010] S4. Repeat S2-S3 until the unmatched silkworms are successfully matched at a certain time to achieve silkworm identity identification.
[0011] Through the above technical solution, firstly, individual detection is performed at the starting moment of the sequence using an individual model, and an identity number is assigned to each detected domestic silkworm, then a Kalman filter is used to predict the position of each domestic silkworm at the next moment, and then individual detection is performed at the second moment using an individual detection model, and the Hungarian algorithm is used to match the prediction result with the detection result, and the domestic silkworm that is successfully matched at consecutive moments is the same domestic silkworm, that is, shares an identity number, and a new identity number is added for a newly detected individual from the second moment. The method realizes the cross-time sequence individual identity retrieval and recognition of domestic silkworms under dense conditions by sequentially performing individual detection, position prediction, identity matching, and position updating operations on the sequence frame image.
[0012] Further, the target detection model in S1 is a deep learning-based target detection model, and the detection accuracy reaches more than 96%, and the detection speed is greater than or equal to 25 frames / second.
[0013] Through the above technical solution, the deep learning-based target detection model requires that the individual detection accuracy of domestic silkworms under dense conditions reaches more than 96%, and the detection speed is greater than or equal to 25 frames / second, compared with mainstream target detection models. The model adds a channel correction network based on feature mapping in the fast feature extraction of the classic method YOLOv7, which has improved the feature extraction capability and detection performance of the network. In the training process, the loss function of the boundary box regression of the betaCIoU function is used to ensure high positioning accuracy.
[0014] Further, the detection environment of the target detection model is that the input size of the image is a 3-channel color image of 1280x640 pixels, each image covers a silkworm foil of 60cmx40cm, and the number of domestic silkworms in the silkworm foil is about 90.
[0015] Further, the target detection model is based on the YOLOv7 framework, and a channel calibration network based on feature mapping is added in the feature extraction and fusion network. In the training process, the βCIoU function is used as the boundary box regression function of the individual detection model training. Specifically, the individual detection model needs to be pre-constructed dataset for model training and verification. The construction method is to collect multiple segments of domestic silkworm growth videos in the actual environment, and each video is ≥1 hour in length. The video frame image contains a whole silkworm leaf of domestic silkworm. Then, every 5 seconds, a frame image is intercepted, and the images intercepted from each video are stored in chronological order. The silkworms are labeled from each image using a labeling tool, and the smallest circumscribed rectangle of each silkworm is taken as the position of the silkworm, and the number of each silkworm is labeled. The total number of images in the dataset should be ≥10,000, and the density of silkworms in each image should be ≥90. Then, the images are randomly selected in the proportion of 6:2:2 to construct the training set, the verification set and the test set. When training the individual detection model, the image sequence and the silkworm number are not required, and the silkworm number is required for identification test to verify the matching effect
[0016] Further, the working process of the channel calibration network is as follows:
[0017] S1.1 Linear mapping is performed on the feature map Q=W Q X, wherein W Q is a learnable parameter;
[0018] S1.2 Feature aggregation is performed using the global average pooling method of formula 1 to obtain feature weights,
[0019] u=AvgPooling(Q) Formula 1
[0020] wherein AvgPool represents global average pooling
[0021] S1.3 The 1D convolution of ECANet is adopted to obtain the dependency relationship between each channel, and formula 2 is used for activation operation,
[0022] w=δ(1DConv(u)) Formula 2
[0023] wherein δ is a Sigmoid activation function, and
[0024] S1.4 Linear mapping and feature value weights are multiplied to obtain the corrected feature map, which is represented by formula 3:
[0025] Y=w*Q Formula 3
[0026] wherein * represents pixel multiplication.
[0027] Further, the boundary box regression function is shown in formula 4,
[0028]
[0029] wherein (x, y) is the center point coordinate of the prediction box, (x gt , y gt ) is the center point coordinate of the real box, W g , H g is the length and width of the union of the real box and the prediction box,
[0030]
[0031] W i , H i is the length and width of the intersection of the real box and the prediction box, S u is the area of the intersection of the prediction box and the real box.
[0032] Further, the dynamic weight (1-IoU) β is in the range of 0≤(1-IoU) β ≤1.
[0033] Further, the object matching method in S3 includes the following two methods:
[0034] When the current frame and the previous frame are continuous sequences, according to the position of the domestic silkworm in the silkworm foil, the position of the domestic silkworm will not change suddenly, the area of the circumscribed rectangle of each domestic silkworm in the current frame and the area of the circumscribed rectangle of each domestic silkworm in the previous frame are calculated, and the maximum intersection and union ratio is the same domestic silkworm, which shares the same number;
[0035] When the current frame and the previous frame are random sequence images, the depth features of each domestic silkworm image in the current frame are calculated, and the Hungarian algorithm is used for object matching, and the depth feature cosine similarity of two different sequence domestic silkworms is the same domestic silkworm, which shares the same number.
[0036] Further, the S3 retrieval identification model is a multi-retrieval model, specifically, the first frame image of the video is given from the video, and each domestic silkworm is matched with the domestic silkworm in the first frame from the subsequent video.
[0037] Further, the S3 retrieval identification model is a single retrieval model, specifically, an image of a domestic silkworm is given, and the domestic silkworm is found from one frame or a group of images.
[0038] Beneficial effects: The silkworm individual identity retrieval and identification method across time series under dense conditions provided by the present application can accurately identify the individual identity of each silkworm at any moment in a dense scene, according to the fact that the position of the silkworm will not change, the number of silkworms in the cocoon will not change in the same time period, and the depth features of each silkworm are different, the individual detection and matching method is used to retrieve and identify the silkworm individuals by using deep learning and visual technology, and through the steps of individual detection, position prediction, identity matching and position updating on the sequence frame image, the silkworm individual identity retrieval and identification across time series under dense conditions is realized, and the result is more objective and accurate. BRIEF DESCRIPTION OF DRAWINGS
[0039] Figure 1 is a method function diagram of the present application;
[0040] Figure 2 is a feature extraction block diagram of the individual detection model of the present application;
[0041] Figure 3 is a channel correction network diagram based on feature mapping of the present application;
[0042] Figure 4 is a bounding box regression diagram of the present application. DETAILED DESCRIPTION
[0043] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.
[0044] Referring to Figure 1 , a silkworm individual identity retrieval and identification method across time series under dense conditions, which realizes the function of accurately identifying the individual identity of each silkworm at any moment in a dense scene.
[0045] A silkworm individual identity retrieval and identification method across time series under dense conditions, which is mainly realized by using target detection and object matching technology in deep learning.
[0046] In this embodiment, the target detection model of the method is a representative algorithm YOLOv7 in the single-step detection algorithm, which is used to ensure the detection speed and detection accuracy. The basic structure of the feature extraction network is as shown in Figure 2 , which mainly controls the longest and shortest gradient paths of the network to ensure a certain accuracy while having a faster inference speed.
[0047] The workflow of this extraction block is: for the input feature map X R W×H×CWhere W, H, and C represent the length, width, and height of the feature map, respectively. First, two 1×1 convolutional layers are used to compress the channels, resulting in two sub-feature maps Y1 and Y2 with dimensions W×H×C / 2, expressed by the formula:
[0048] Y1 = Conv 1×1 (X)
[0049] Y2 = Conv 1×1 (X)
[0050] Among them, Conv 1×1 This represents a convolution operation with a kernel size of 1×1.
[0051] Then, one sub-feature map (Y1) is a cross-stage connection, acting similarly to a residual connection, while the other sub-feature map (Y2) uses stacked 3×3 convolutional layers to increase network depth, acquiring more diverse features and enhancing feature representation capabilities. This can be expressed as:
[0052] Y n =Conv 3×3 (Y n-1 )
[0053] Where n = 3, 4.
[0054] Next, feature concatenation is used to fuse all sub-feature maps Y1, Y2, Y3, and Y4 to obtain feature blocks with different depth and semantic information. Finally, a 1×1 convolution is used to compress the number of channels. This can be expressed by the formula:
[0055] Y = Conv 1×1 (Concat(Y1, Y2, Y3, Y4))
[0056] Concat represents feature concatenation.
[0057] In this embodiment, in order to improve the detection effect, a channel correction network based on feature transfer is designed to address the problems of small individual silkworms, small differences between individuals, and a large number of detections in each image. This network is then added to the feature extraction block of the original YOLOv7 network and used in the individual detection network.
[0058] See Figure 3 As shown, the channel correction network based on feature transfer is effective for input feature maps X∈R. W×H×C First, a linear mapping is performed on the feature map, the formula of which is:
[0059] Q = W Q X
[0060] Among them, W Q These are learnable parameters.
[0061] In actual operation, we found that the feature map linear mapping can be realized by depth separable convolution operation, since depth separable convolution is a linear multiplication on each feature channel, so that the inherent relationship between each channel of the feature map is well preserved. Then use the global average pooling method for feature aggregation, get the feature weight u, expressed as:
[0062] u = AvgPooling (Q)
[0063] Where, AvgPooling represents global average pooling.
[0064] In order to distinguish from the single AvgPooling feature aggregation method, we call the feature aggregation method after linear feature mapping and AvgPooling as soft aggregator, subsequent experiments show that using the proposed feature aggregation method instead of traditional, can preserve the inherent relationship between the original feature map, enrich the channel feature expression ability (neuron diversity), so as to realize better channel correction effect. Subsequently, the 1D convolution proposed by ECANet is used to obtain the dependence between each channel and perform activation operation, and the calculation formula is:
[0065] w = δ (1DConv (u))
[0066] Where, δ is the Sigmoid activation function, and its formula is:
[0067]
[0068] Finally, multiply the linear mapping and the feature value weight to get the corrected feature map, expressed as:
[0069] Y = w * Q
[0070] Where, * represents pixel multiplication.
[0071] Compared with the existing channel calibration network, the channel calibration network of the present method uses flexible feature mapping, which is realized by using depth separable convolution, which not only improves the operation efficiency, but also avoids directly using global average on feature map, which simply ignores the dependence between channels.
[0072] In this embodiment, the βCIoU function is used as the boundary box regression function of the individual detection model training, which is realized on the basis of CIoU, and its function expression is: As shown in Figure 4 , wherein: (x, y) is the center point coordinate of the prediction box, (x gt , y gt) is the center point coordinate of the real frame, W g , H g is the length and width of the real frame and the predicted frame union,
[0073]
[0074]
[0075] ) is the center point coordinate of the real frame, W i , H i is the length and width of the real frame and the predicted frame intersection, S u is the area of the predicted frame and the real frame intersection.
[0076] Due to the addition of dynamic weight (1-IoU) β , wherein: 0≤1-IoU≤1,
[0077] 0≤(1-IoU) β ≤1
[0078] . In the early stage of training, the value of IoU is close to 0, (1-IoU) β is close to 1, L βCIoU ≈L CIoU , the regression operation contributes more to the overall loss function, and the value of the predicted frame obtains a larger gradient and updates faster. As the training continues, to the later stage of training, the degree of coincidence of the two frames gradually increases, the value of IoU is close to 1, (1-IoU) β is close to 0, the regression operation contributes less to the overall loss function, and the predicted frame becomes slow, and the training focuses on target detection and classification, thereby achieving dynamic adjustment of the loss function, which can also greatly improve the efficiency of back propagation, thereby shortening the training time.
[0079] In this embodiment, the detection environment of the target detection model is: the input size of the image is 1280×640 pixels of 3-channel color image, each image covers 60cm×40cm of silkworm foil, and the number of silkworms in the silkworm foil is about 90. The detection accuracy of the detection model for individuals reaches more than 96%, and the detection speed on the GPU is ≥25 images / second, that is, the dynamic detection capability is achieved, so as to perform silkworm individual identity recognition and retrieval.
[0080] In this embodiment, the target detection model needs to be trained using a data set to achieve detection ability, and the data set needs to be collected in an actual environment. The method is to raise silkworms in silkworm foil, the size of the silkworm foil is 60cm*40cm, and there are about 90 silkworms in each silkworm foil. Then use a camera to collect video above the silkworm foil, the length of each video segment is ≥1 hour, the more video segments collected, the stronger the generalization ability of the model will be. Then take a frame of image every 5 seconds for each segment, and store the intercepted images in the folder according to the time sequence. Use the image labeling tool labeling to label the outer rectangular frame of each silkworm in each image in turn, and assign a number to each silkworm. The total number of labeled images should be ≥10000, and then randomly select images in the ratio of 6:2:2 to construct the training set, validation set and test set for model training.
[0081] In this embodiment, during the training of the target detection model, parameter adjustment can be carried out from the aspects of data enhancement (such as Mosica data enhancement, generated images), dynamic learning rate, optimizer, positive and negative sample matching, etc. to improve the detection accuracy of the model. When the accuracy of the model on the test set reaches 96% or more, and the detection speed on the GPU is ≥25 images / second, it means that the training is complete, and it can be used for individual identity recognition and retrieval.
[0082] It should be noted that the detection accuracy of silkworm individuals is closely related to the number of silkworms in the image, the occlusion situation, and the size of the silkworm individuals. When constructing the data set of the target detection model, this method tries to choose a time period after the silkworms have been fed for a period of time and have basically eaten all the mulberry leaves in the silkworm foil, so as to avoid the influence of mulberry leaf occlusion on detection. For the occlusion between silkworms and silkworms, continuous video frame images are used for model training. According to the normal situation, the individual crawling of silkworms will not be occluded all the time, and through data enhancement methods and image processing methods, silkworms with high coincidence degree are separated out during model training, so as to ensure that the model has strong detection ability.
[0083] In this embodiment, two kinds of retrieval and recognition models are provided. One is to give the first frame image of a video from the video, and to match each silkworm in the first frame with the silkworm from the subsequent video, which can also be called multi-retrieval. The other is to give an image of a silkworm, and to find out the silkworm from a frame or a group of images, which is also called single retrieval.
[0084] Further, multi-retrieval can be performed on a group of images in continuous time sequence, or on a group of images in random sequence.
[0085] The workflow on the continuous sequence is: first, all silkworm individuals are detected simultaneously on the first frame image using a target detection model, the position information of each silkworm is stored in a linked list, and the linked list number is the silkworm number, and each silkworm is extracted from the original image according to the position information of each detected silkworm, the deep features of each silkworm are extracted using the image recognition network SwinTransformer, MobileNet or ResNet-50 model without the Softmax classification layer, and are stored in the form of a one-dimensional vector in the linked list; further, the possible position of each silkworm in the second frame is predicted using Kalman filtering; all silkworm individuals are detected on the second frame image using the target detection model, the detection results and the prediction results of Kalman filtering are used for position correction, and the deep feature vectors of the silkworms in the second frame image are extracted using the image classification network; the position information of each silkworm after correction is compared with the position information of each silkworm in the first frame by intersection over union, according to the principle that the silkworm moves slowly and the position in the continuous sequence frame image does not change suddenly, the intersection over union value of two silkworms ≥0.85 is identified as the same silkworm.
[0086] In this embodiment, after intersection over union, there will be silkworms that cannot be matched in the detection results of the first frame and the second frame, at this time, the cosine similarity between all unmatched silkworms in the first frame and the second frame is calculated, and the similarity ≥0.9 is identified as the same silkworm. For all matched silkworms in the second frame, only the position information is saved to the silkworm chain table information with the same identity in the previous frame, after the cosine similarity calculation, for the silkworms in the first frame that cannot be matched, it is identified that the target disappears in the second frame, and it may be blocked, and the silkworms in the second frame that cannot be matched are identified as new identities, and a new chain table is built to store the information. According to the principle that the number of silkworms in the cocoon does not change suddenly, the information of all unmatched silkworms in the previous frame is retained, and when there are unmatched silkworms in the detection results of each subsequent frame, identity matching is continuously performed. After matching the second frame image, the possible position of the silkworm in the third frame is predicted using Kalman filtering, and the detection, position correction, deep feature vector extraction, intersection over union matching, cosine similarity matching and other operations are sequentially performed on the subsequent sequence images to realize the identity retrieval and recognition of multiple silkworms.
[0087] In this embodiment, when searching on a random sequence, the method is basically the same as on a continuous sequence, the difference is that since it is a non-continuous sequence, the position of the silkworm in the previous and subsequent two frames may fluctuate greatly, at this time, the intersection over union matching is no longer reliable, and the cosine similarity is mainly used for identity matching.
[0088] The single retrieval workflow is: for a given silkworm that needs to be retrieved, first use the image recognition network SwinTransformer, MobileNet or ResNet-50 model without the Softmax classification layer to extract the deep feature vector of the silkworm, and store it in a linked list in the form of a one-dimensional vector. Then use the target detection model in the sequence image to detect all silkworms in each frame of image, then extract the deep feature vector of each silkworm, and finally calculate the similarity in turn, and the one with the highest similarity value is the retrieval result.
[0089] In this embodiment, when extracting the deep feature vector of the individual detected by the target detection model, a convolutional neural network model without a classification layer is used. The deep feature vector extraction model also needs to be trained after the data set is prepared to form the extraction ability.
[0090] The method for preparing the data set of the deep feature vector is: for the same silkworm, collect its images in different time periods and different backgrounds when the silkworm is in different postures, each image contains only one silkworm, and the images of each silkworm become a class of image groups. At least 1000 silkworm images are needed for model training. Take 500 silkworm images as the training set and 500 silkworm images as the test set. The more silkworms in the data set, the stronger the generalization ability of the model.
[0091] The deep feature extraction model inputs 2 images each time. If the two images are of the same silkworm, the deep feature vector similarity value is minimized, and if they are not the same silkworm, the network is trained to maximize the feature vector similarity value. Use the triplet loss function to train the model. When the accuracy of the model on the test set is ≥98%, it means that the deep feature vector of each silkworm can be accurately extracted. During model training, you can use learning rate adjustment, optimizer and transfer learning to enhance the extraction ability of the model.
[0092] In this embodiment, the deep feature vector extraction model can also use the backbone extraction network in the target detection model.
[0093] For all silkworm deep feature vectors stored in the linked list, continuous saving can be used to increase the data set of the method and provide digital support for silkworm life form research.
[0094] It should be noted that the above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the foregoing described technical solutions can still be modified, or some of the technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the present application.
Claims
1. A method for identifying individual silkworms across time series under dense conditions, characterized by Comprising the following steps: S1. Perform individual detection on the silkworms in the fixed area at the initial time by a target detection model, and assign an identity number to each detected silkworm; The target detection model is based on YOLOv7 framework, and a channel calibration network based on feature mapping is added in the feature extraction and fusion network. In the training process, the βCIoU function is used as the boundary box regression function for individual detection model training. The workflow of the channel calibration network is as follows: S1.1, to the feature map is a linear mapping, where, are learnable parameters; S1.2, using the global average pooling method of formula 1 for feature convergence, obtaining feature weights, Formula 1 wherein, represents global average pooling; S1.3, using the 1D convolution of ECANet to obtain the dependency between each channel, and using formula 2 for activation operation, Formula 2 wherein is a Sigmoid activation function, and ; S1.4, multiply the linear mapping with the feature value weight to obtain the corrected feature map, which is represented by formula 3: Formula 3 wherein represents a multiplication of the pixels; The boundary box regression function is shown in formula 4, Formula 4 in,( x, y ) are the coordinates of the center point of the prediction box, ( x gt ，y gt () represents the coordinates of the center point of the true bounding box. W g ，H g It is the length and width of the union of the ground truth bounding box and the predicted bounding box. , , , W i ，H i The length and width of the intersection of the ground truth bounding box and the predicted bounding box. S u It is the area of the intersection of the predicted bounding box and the ground truth bounding box; S2. Predict the position of each silkworm at the next time using Kalman filter, and perform individual detection at that time by a target detection model, and assign a temporary identity number to each detected silkworm; S3. Use the search and recognition model to match the detection results of the current time with the detection results of the last time. The silkworms that match successfully share the same identity number, and the silkworms that do not match successfully keep the identity number. The object matching method in S3 includes the following two methods: When the current frame and the last frame are continuous sequences, the position of the silkworm in the cocoon foil will not change suddenly. The intersection ratio of the area of the silkworm in the current frame and the area of the silkworm in the last frame is calculated. The maximum value is the same silkworm, which shares the same number. When the current frame and the last frame are random sequence images, the depth features of each silkworm image in the current frame are calculated, and the Hungarian algorithm is used for object matching. The cosine similarity of the depth features of two different sequence silkworms is the same, which is the same silkworm, which shares the same number. S4. Repeat S2-S3 until the unmatched silkworms are matched successfully at a certain time, and realize the identity recognition of the silkworms.
2. The method according to claim 1, wherein The target detection model in S1 is a deep learning-based target detection model, with a detection accuracy of more than 96% and a detection speed of ≥25 frames / second.
3. The method according to claim 1, wherein The detection environment of the target detection model is: the input size of the image is 1280×640 pixels of 3-channel color image, each image covers 60cm×40cm of cocoon, and the number of silkworms in the cocoon is about 90.
4. The method according to claim 1, wherein Dynamic weights The value range of .
5. The method according to claim 1, wherein The search and recognition model is a multi-search model, which specifically matches each silkworm with the silkworm in the first frame of the video from the first frame of the video to the subsequent video.
6. The method according to claim 1, wherein The search and recognition model is a single search model, which specifically finds the image of a silkworm from one frame or a group of images.