Single tomato flower dynamic counting method based on target tracking counting

Abstract

The invention relates to a single tomato flower dynamic counting method based on target tracking counting, and relates to the field of agricultural yield prediction.The method comprises the steps that single tomato flowers on tomato vines are recognized through a depth camera and a tomato flower target detection algorithm; obtaining a single tomato flower target frame through the target detection model, extracting appearance characteristics and motion characteristics of a single tomato flower in combination with a tracking counting algorithm, and matching the appearance characteristics and the motion characteristics with the target frame; the tomato flower ID jump in the tracing process is slowed down through a multi-section region area method; according to the method, the tomato flowers can be accurately and effectively detected by using the target detection algorithm, tracking counting of the single tomato flowers can be realized by combining the target detection model and the tracking counting model, dynamic counting of the single tomato flowers is effectively completed, and a reference is provided for yield estimation of a planter.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and tomato flower counting, and more specifically to a single tomato flower dynamic counting method based on target tracking and counting. Background Art

[0002] Counting tomato flowers can predict tomato yields, and accurate yield estimates can help growers plan agricultural production effectively. Progress in automated flower counting is relatively slow. Currently, most flower counting activities are still performed manually, which limits the improvement of production efficiency. Traditional manual counting methods are time-consuming and impractical in the field of large-scale tomato cultivation. Due to the large number of tomato flowers, manual counting not only requires a lot of manpower, but is also prone to counting errors and uncertainties. In this context, automatic counting of tomato flowers has become a key technology that is expected to improve production efficiency, reduce labor costs, and ultimately increase agricultural output value.

[0003] Tomato flowers are small in size and are small targets for detection. In addition, the branches and leaves of tomatoes seriously block the tomato flowers, which increases the difficulty of flower detection and tracking. With the help of target detection models, real-time tracking algorithms and multi-segment area methods, these problems can be effectively solved, and dynamic tracking and counting of single tomato flowers can be achieved, thus providing reliable technical support for agricultural yield forecasting. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is: in view of the problem that tomato flowers are small in size and easily blocked by stems and leaves, a single tomato flower dynamic counting method based on target tracking and counting is proposed to achieve dynamic counting of tomato flowers and help growers estimate tomato yields.

[0006] (II) Technical solution

[0007] The technical solution adopted by the present invention: To solve the above problems, the present invention provides a single tomato flower dynamic counting method based on target tracking counting:

[0008] A single tomato flower dynamic counting method based on target tracking counting includes the following steps:

[0009] S1. Collect tomato flower image video data through a depth camera, obtain an image dataset from it, perform annotation and dataset division operations on the target detection model, and divide the image dataset into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0010] The target detection model is trained using the corresponding data set to obtain a single tomato flower target detection model;

[0011] S2. Detect each frame of the tomato flower image video data according to the single tomato flower target detection model described in S1 to obtain the target frame position and size information of each tomato flower;

[0012] Extracting appearance feature information of each tomato flower based on the appearance feature extraction network in the deep learning target tracking algorithm, wherein the acquired image data set and annotation information are sent to the appearance feature extraction model for training, so as to train the appearance feature extraction network in the appearance feature extraction model;

[0013] The motion feature information of tomato flowers is predicted based on the deep learning target tracking algorithm model, where the ReID model is retrained to adapt to the target tomato flower tracking and counting tasks;

[0014] Matching the trajectory of each tomato flower with the target frame by combining the appearance feature information and the motion feature information;

[0015] Combine multiple area segments to solve the ID jump problem of tomato flowers;

[0016] S4. Input the test image into the trained single tomato flower dynamic counting network based on target tracking and counting, and output the corresponding counting result.

[0017] In a possible implementation, step S1 specifically includes the following steps:

[0018] The tomato flower image video data is collected by the depth camera, the image data set is obtained from the tomato flower image video data, and a labeling and data set division operation is performed on the target detection model to divide the image data set into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0019] The deep learning-based target detection model is trained using the corresponding data set to obtain the single tomato flower target detection model.

[0020] In a possible implementation, step S2 specifically includes the following steps:

[0021] Detect each frame of the tomato flower image video data according to the single tomato flower target detection model in S1 to obtain the target frame position and size information of each tomato flower;

[0022] Extracting appearance feature information of each tomato flower based on the appearance feature extraction network in the deep learning target tracking algorithm, wherein the extracted appearance feature of the tomato flower is represented by a 512-dimensional vector, and each tomato flower tracking trajectory has a separate corresponding appearance feature library for storing the appearance information of the successfully matched tomato flower;

[0023] Calculating the minimum cosine distance between the appearance feature library and each frame of the tomato flower appearance feature to measure the similarity between the trajectory and the tomato flower appearance, the more similar, the smaller the minimum cosine distance;

[0024] The deep learning target tracking algorithm model first uses linear matching to match the confirmed trajectory with the tomato flower in the current frame using the appearance feature information and the motion feature information of the tomato flower. Three results are generated: successful trajectory matching, the tomato flower motion trajectory does not match the tomato flower target frame, and the tomato flower target frame does not match the tomato flower motion trajectory. Then, IoU matching is used to match the tomato flower motion trajectory that does not match the tomato flower target frame, the tomato flower target frame that does not match the tomato flower motion trajectory, and the unconfirmed tomato flower motion trajectory again. After matching, the above three outputs are still obtained. When the tomato flower is correctly tracked, all trajectories are matched to the corresponding tomato flower after global linear matching and IoU matching.

[0025] Among the tomato flowers with smaller size, there are tomato flowers that do not match the movement trajectory of the tomato flowers, and they are given new IDs;

[0026] The multi-segment area method is used to divide the detection field of view into three areas. The width of each area is larger than the size of the detection frame and is distributed in the front, middle and back parts of the image respectively.

[0027] Calculate the center position of the detected tomato flower target frame. If the coordinates of the center position are within the defined area, store the ID number of the tomato flower to avoid repeated counting in the next frame, and add 1 to the tomato flower number count.

[0028] The three areas were counted independently, and then the number of tomato flowers in the three areas was counted separately, and the largest number was taken as the final number of tomato flowers.

[0029] (III) Beneficial effects

[0030] Beneficial effects of the invention: The invention provides a single tomato flower dynamic counting method based on target tracking counting, which has the following beneficial effects:

[0031] Using the object detection algorithm, tomato flowers can be detected accurately and effectively;

[0032] To solve the problem of tomato flowers being blocked by many stems and leaves, a method combining target tracking algorithm and multi-segment area method was proposed to track and count tomato flowers. The dynamic counting of single tomato flowers was effectively completed, providing a reference for growers to estimate yield. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of the method provided by the present invention;

[0035] Figure 2 A schematic diagram of the structure of the tracking and counting algorithm based on the YOLOX-s target provided by the present invention;

[0036] Figure 3 This is a rendering of the multi-segment area method provided by the present invention. DETAILED DESCRIPTION

[0037] The embodiments of the present invention are described below in conjunction with the drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention. It is known to those of ordinary skill in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present invention are also applicable to similar technical problems. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present invention.

[0038] like Figure 1 As shown, the following steps are included:

[0039] Obtain an image dataset for pre-training. Specifically, collect tomato flower image video data through a depth camera, obtain an image dataset from it, perform annotation and dataset division operations for the target detection model, and divide the image dataset into a training set, a validation set, and a test set in a ratio of 8:1:1;

[0040] Tomato flower target detection model, the target detection model is trained using the corresponding data set. Mosaic data enhancement is used during the training process with a 50% usage probability. The pre-trained weights are frozen for the first 50 epochs, and the batch size is set to 16. After that, the weight parameters are unfrozen for fine-tuning, and the batch size is set to 8. After the training is completed, a single tomato flower target detection model based on the YOLOX-s network is obtained;

[0041] Construct a dynamic counting method network architecture. Specifically, detect each frame of the tomato flower video according to the above single tomato flower target detection model to obtain the target frame position and size information of each tomato flower, and track and count the YOLOX-s target based on the deep learning target tracking algorithm model. Figure 2 As shown;

[0042] Based on the appearance feature extraction network in the deep learning target tracking algorithm, the appearance feature information of each tomato flower is extracted. The extracted appearance features of the tomato flower are represented by a 512-dimensional vector. For each tomato flower tracking trajectory, there will be a separate corresponding appearance feature library to save the appearance information of the tomato flower that has been successfully matched recently. The minimum cosine distance between the appearance feature library and the appearance features of each frame of tomato flower is calculated to measure the similarity between the trajectory and the tomato flower. The more similar (the smaller the appearance similarity), the smaller the minimum cosine distance.

[0043] Based on the deep learning target tracking algorithm model, the motion feature information of tomato flowers is predicted;

[0044] The deep learning target tracking algorithm model first uses linear matching to match the confirmed trajectory with the tomato flower in the current frame based on the appearance feature information and motion feature information of the tomato flower. Three results are generated: successful trajectory matching, the tomato flower trajectory does not match the tomato flower target frame, and the tomato flower target frame does not match the tomato flower trajectory. Then, IoU matching is used to match the tomato flower trajectory that does not match the tomato flower target frame, the tomato flower target frame that does not match the tomato flower trajectory, and the unconfirmed tomato flower trajectory again. After matching, the above three outputs are still obtained. When the tomato flower is correctly tracked, all trajectories can match the corresponding tomato flower after global linear matching and IoU matching.

[0045] Tomato flowers with small volumes that do not match the motion trajectory will be assigned new IDs, using the multi-segment area method. Figure 3 As shown, the detection field of view is divided into three areas. The width of each area is slightly larger than the size of the detection frame and is distributed in the front, middle and back parts of the image. The center position of the detected tomato flower target frame is calculated. If the coordinates of the center position are within the defined area, the ID number of the tomato flower is stored to avoid repeated counting in the next frame. At the same time, the number of tomato flowers is counted by 1, and the three areas are counted independently. Then, the number of tomato flowers in the three areas is counted separately, and the largest number is taken as the final number of tomato flowers. The multi-segment area method is used to alleviate the ID jump problem and effectively avoid the overestimation of the number of tomato flowers.

[0046] Training network. Specifically, for the tomato flower tracking and counting model training, the acquired data set and annotation information are sent to the BOT+ResNeSt50 model for training, which is used to train the appearance feature extraction network;

[0047] Retrain the ReID model to adapt to the target tomato flower tracking and counting task;

[0048] Input a test image to track and count tomato flowers. Specifically, according to the tomato flower tracking and counting model obtained above, input a test video and output the corresponding tomato flower counting result;

[0049] The main features and beneficial effects of the present invention are shown and described above. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is limited by the appended claims rather than the above description. Therefore, it is intended to include all changes within the meaning and scope of the equivalent elements of the claims in the present invention, and any figure marks in the claims should not be regarded as limiting the claims involved.

[0050] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A single tomato flower dynamic counting method based on target tracking counting, characterized in that: The steps include: S1. Collect tomato flower image video data through a depth camera, obtain an image dataset from it, perform annotation and dataset division operations on the target detection model, and divide the image dataset into a training set, a validation set, and a test set in a ratio of 8:1:1; The target detection model is trained using the corresponding data set to obtain a single tomato flower target detection model; S2. Detect each frame of the tomato flower image video data according to the single tomato flower target detection model described in S1 to obtain the target frame position and size information of each tomato flower; Extracting appearance feature information of each tomato flower based on the appearance feature extraction network in the deep learning target tracking algorithm, wherein the acquired image data set and annotation information are sent to the appearance feature extraction model for training, so as to train the appearance feature extraction network in the appearance feature extraction model; The motion feature information of tomato flowers is predicted based on the deep learning target tracking algorithm model, where the ReID model is retrained to adapt to the target tomato flower tracking and counting tasks; Matching the trajectory of each tomato flower with the target frame by combining the appearance feature information and the motion feature information; Combine multiple area segments to solve the ID jump problem of tomato flowers; S4. Input the test image into the trained single tomato flower dynamic counting network based on target tracking and counting, and output the corresponding counting result.

2. The method for dynamic counting of single tomato flowers based on target tracking and counting according to claim 1, characterized in that: Step S1 specifically includes the following steps: The tomato flower image video data is collected by the depth camera, the image data set is obtained from the tomato flower image video data, and a labeling and data set division operation is performed on the target detection model to divide the image data set into a training set, a validation set, and a test set in a ratio of 8:1:1; The deep learning-based target detection model is trained using the corresponding data set to obtain the single tomato flower target detection model.

3. The method for dynamic counting of single tomato flowers based on target tracking and counting according to claim 1, characterized in that: Step S2 specifically includes the following steps: Detect each frame of the tomato flower image video data according to the single tomato flower target detection model in S1 to obtain the target frame position and size information of each tomato flower; Extracting appearance feature information of each tomato flower based on the appearance feature extraction network in the deep learning target tracking algorithm, wherein the extracted appearance feature of the tomato flower is represented by a 512-dimensional vector, and each tomato flower tracking trajectory has a separate corresponding appearance feature library for storing the appearance information of the successfully matched tomato flower; Calculating the minimum cosine distance between the appearance feature library and the appearance feature of each frame of tomato flower to measure the similarity between the trajectory and the tomato flower appearance, the smaller the appearance similarity is, the smaller the minimum cosine distance is; The deep learning target tracking algorithm model first uses linear matching to match the confirmed trajectory with the tomato flower in the current frame using the appearance feature information and the motion feature information of the tomato flower. Three results are generated: successful trajectory matching, the tomato flower motion trajectory does not match the tomato flower target frame, and the tomato flower target frame does not match the tomato flower motion trajectory. Then, IoU matching is used to match the tomato flower motion trajectory that does not match the tomato flower target frame, the tomato flower target frame that does not match the tomato flower motion trajectory, and the unconfirmed tomato flower motion trajectory again. After matching, the above three outputs are still obtained. When the tomato flower is correctly tracked, all trajectories are matched to the corresponding tomato flower after global linear matching and IoU matching. Among the tomato flowers with smaller size, there are tomato flowers that do not match the movement trajectory of the tomato flowers, and they are given new IDs; The multi-segment area method is used to divide the detection field of view into three areas. The width of each area is larger than the size of the detection frame and is distributed in the front, middle and back parts of the image respectively. Calculate the center position of the detected tomato flower target frame. If the coordinates of the center position are within the defined area, store the ID number of the tomato flower to avoid repeated counting in the next frame, and add 1 to the tomato flower number count. The three areas were counted independently, and then the number of tomato flowers in the three areas was counted separately, and the largest number was taken as the final number of tomato flowers.

Images