Citrus green fruit tracking and counting method and system based on improved YOLOv7 model
By improving the YOLOv7 model and combining the SimAM attention mechanism and the Ivy intelligent optimization algorithm, combined with the improved StrongSORT target tracking algorithm, the existing citrus green fruit counting methods are solved, and high-precision and high-efficiency citrus green fruit tracking counting is achieved, reducing the need and security risks of manual counting.
Patent Information
- Application Number
- CN202411731246.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-07
AI Technical Summary
The existing citrus green fruit counting methods mainly rely on manual inventory, resulting in low detection accuracy and low counting efficiency, especially in hilly areas with complex environments.
The citrus green fruit tracking and counting method based on the improved YOLOv7 model is adopted. By introducing the SimAM attention mechanism module and the Ivy intelligent optimization algorithm, the detection model is trained and hyperparameter optimization. Combined with the improved StrongSORT target tracking algorithm, it is transplanted to the edge computing platform for real-time tracking and counting.
It improves the accuracy and efficiency of citrus green fruit detection, reduces the need for manual counting, reduces the security risks in complex environments, and realizes a portable mobile detection device.
Smart Images
Figure CN119580244A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target counting, and in particular to a citrus fruit tracking and counting method and system based on an improved YOLOv7 model. Background Art
[0002] In agricultural production, counting immature fruits can provide data support for subsequent orchard yield forecasts. Real-time detection of citrus orchards can provide reliable support for production management links such as fruit thinning, sunburn prevention and control, and yield forecasts. Since citrus orchards are mostly located in hilly areas with complex environments, counting immature citrus fruits is dangerous. In addition, since citrus fruits are similar in color to leaves, a large amount of human resources are required for counting, which is inefficient.
[0003] Therefore, an intelligent counting method is urgently needed to improve the recognition accuracy and detection counting efficiency of production measurement tasks. Summary of the invention
[0004] In view of this, in order to solve the technical problem that most of the existing fruit counting methods are manual counting, which leads to low detection accuracy and low counting efficiency, the present invention proposes a citrus green fruit tracking and counting method based on an improved YOLOv7 model, and the method comprises the following steps:
[0005] Collect training images in different environments and construct training sets;
[0006] Based on the YOLOv7 model, the SimAM attention mechanism module is introduced into the first branch and the second branch of the ELAN-H module in front of the P3 detection head to build a detection model;
[0007] Using the Ivy intelligent optimization algorithm, the detection model is trained and hyperparameters are optimized based on the training set to obtain a trained detection model;
[0008] The trained detection model is combined with the target tracking algorithm to detect and track the video to be tested.
[0009] In some embodiments, it also includes:
[0010] Data augmentation is performed on the training image.
[0011] In some embodiments, the step of using the Ivy intelligent optimization algorithm to train the detection model and optimize hyperparameters based on the training set to obtain a trained detection model specifically includes:
[0012] Initialize parameters and population and update the initial historical optimum;
[0013] Update the growth rate of each individual;
[0014] Hyperparameter optimization and position update;
[0015] Update the historical optimum and determine the optimal model parameters.
[0016] In some embodiments, it also includes:
[0017] The StrongSORT target tracking algorithm is improved to obtain the OStrongSORT target tracking algorithm.
[0018] The feature extractor BoT+ResNeSt50 in StrongSORT is changed to the lightweight OSNet for feature extraction, and the residual connection blocks of multiple convolutions in OSNet are used to realize full-scale feature learning, thereby improving StrongSORT's ability to capture the key features of dynamic citrus green fruits.
[0019] In the motion branch, the Hungarian algorithm is used to match and associate the tracking box with the motion trajectory, excluding the detection box with low correlation with the average tracking result;
[0020] In the appearance branch, the minimum cosine distance is used as the appearance metric, and the minimum cosine distance is combined with the Mahalanobis distance to avoid interference from the citrus green fruit or leaf background with high similarity.
[0021] In some embodiments, it also includes:
[0022] The YOLOv7S model and OStrongSORT target tracking algorithm were transplanted to the edge computing platform, and the cameras of multiple tablets and mobile phones were called through wireless IP to capture citrus fruit images in real time, and detect, track and count the citrus fruits.
[0023] The present invention also proposes a citrus fruit tracking and counting system based on an improved YOLOv7 model, the system comprising:
[0024] Image acquisition module, used to acquire images and build training sets;
[0025] The model building module is based on the YOLOv7 basic framework, introduces the SimAM attention mechanism module, and builds a detection model;
[0026] A model training module is used to train the detection model and optimize hyperparameters based on the training set to obtain a trained detection model;
[0027] The model application module processes the video to be tested based on the trained detection model and target tracking algorithm to generate a counting result.
[0028] Based on the above scheme, the present invention provides a citrus green fruit tracking and counting method and system based on the improved YOLOv7 model, which takes YOLOv7S as the benchmark target detection model, and uses the ivy intelligent optimization algorithm to optimize the hyperparameters of the YOLOv7S model to improve its detection accuracy; then this optimized model is combined with the improved StrongSORT target tracking algorithm OStrongSORT and transplanted to the edge platform to form a portable mobile citrus green fruit tracking and counting device; the citrus green fruit is tracked and counted in real time by calling the camera of the mobile phone or tablet through the wireless IP, which can improve the detection and counting efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is a flowchart of the steps of a citrus green fruit tracking and counting method based on an improved YOLOv7 model of the present invention;
[0030] Figure 2 It is an improved ELAN-HS module;
[0031] Figure 3 It is an improved StrongSORT target tracking algorithm;
[0032] Figure 4 The present invention is a structural block diagram of a citrus green fruit tracking and counting system based on an improved YOLOv7 model. DETAILED DESCRIPTION
[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0034] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.
[0035] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.
[0036] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.
[0037] In the description of the embodiments of the present application, "plurality" means two or more than two. The following terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.
[0038] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.
[0039] Reference Figure 1 , is a flow chart of an optional example of a citrus green fruit tracking and counting method based on an improved YOLOv7 model proposed in the present invention. The method can be applied to a computer device. The tracking and counting method proposed in this embodiment may include but is not limited to the following steps:
[0040] Step S1, acquiring images and constructing a training set;
[0041] Step S2: Based on the YOLOv7 basic framework, the SimAM attention mechanism module is introduced to build a detection model;
[0042] Step S3: training the detection model and optimizing hyperparameters based on the training set to obtain a trained detection model;
[0043] Step S4: Process the video to be tested based on the trained detection model and target tracking algorithm to generate a counting result.
[0044] In some feasible embodiments, the step S1 specifically includes:
[0045] S1.1, acquiring images and enhancing the images;
[0046] The citrus fruit images obtained mainly include scenes under sunny and cloudy conditions, with direct light and backlight;
[0047] The data is expanded by adding salt and pepper noise, image inversion, mirroring and other data enhancement methods to the citrus fruit images. Finally, the original images and the data-enhanced images are used to construct a citrus fruit dataset.
[0048] S1.2. Build a training set.
[0049] The Labelimg tool was used to annotate the citrus fruit dataset, and the annotated data was divided into a training set and a validation set in a ratio of 8:2.
[0050] In some feasible embodiments, the step S2 specifically includes:
[0051] Based on the YOLOv7 basic framework;
[0052] In the first branch of the ELAN-H module before the P3 detection head, a SimAM attention mechanism module is added after channel and spatial convolution;
[0053] In the second branch of the ELAN-H module before the P3 detection head, a SimAM attention mechanism module is added after channel convolution;
[0054] Build a detection model.
[0055] The addition in the first branch is to further extract features in the deep layer, and the addition in the second branch is because the shallow feature information retained by a channel convolution is less. The SimAM attention mechanism is used to strengthen the feature weight of citrus green fruit, so that features can be better extracted in complex backgrounds. The improved ELAN-H module is named ELAN-HS module. After the model is trained, it has a good detection effect in the citrus green fruit target detection task. Due to the addition of the SimAM module, the computational complexity and parameter quantity of the ELAN-HS module are almost the same as those of the ELAN-H module.
[0056] The present invention takes YOLOv7 as the benchmark model and proposes a citrus fruit target detection model YOLOv7S (YOLOv7-SimAM). A SimAM attention mechanism module is added to the ELAN-H module in front of the P3 detection head, which is designed as the ELAN-HS module. This module improves the feature extraction capability of citrus fruit in small targets and complex backgrounds and reduces the computational complexity and parameter amount of the model.
[0057] In the ELAN-H module in YOLOv7, it controls the shortest and longest gradient paths, allowing the network to learn more features and have stronger robustness. Figure 2As shown in the figure, the ELAN-H module consists of two branches ① and ②, and the activation function is SiLU. In ①, four residual connections are used to fuse the shallow and deep feature information. Finally, it is connected with the results of ① and ②, and a complete ELAN-H module is formed through CBS. Thanks to the ELAN-H module, it has the advantages of fast detection speed and high detection accuracy. Therefore, the present invention selects the YOLOv7 model for optimization to better complete the subsequent citrus green fruit tracking and counting tasks.
[0058] In order to mimic the collaborative attention of humans, the SimAM attention mechanism module assigns each neuron a unique weight.
[0059] In order to achieve better attention, it is necessary to evaluate the importance of each neuron. In neurology, information-rich neurons usually show different discharge states from surrounding neurons, and these activated neurons will inhibit the surrounding neurons to a certain extent, that is, spatial inhibition. This shows that activated neurons with spatial inhibition effects are more important. Measuring the linear separability between neurons can make it easier to find important neurons. Therefore, the energy function is obtained:
[0060]
[0061] in Minimizing the above formula is equivalent to training the linear separability between neuron t and other neurons in the same channel. For simplicity, binary labels are used and regularization is added. The final energy function is defined as follows:
[0062]
[0063] Theoretically, each channel has an energy function, and for the above formula, there is the following analytical solution:
[0064]
[0065] Therefore, the minimum energy formula can be obtained from the above formula:
[0066]
[0067] The lower the energy, the greater the difference between neuron t and surrounding neurons, and the higher its importance. Therefore, important neurons can be get.
[0068] It can be concluded that according to the definition of the attention mechanism, the features need to be enhanced:
[0069]
[0070] According to the definition of this energy function, we can get the SimAM attention mechanism.
[0071] In some feasible embodiments, the step S3 specifically includes:
[0072] In this step, the YOLOv7S model is optimized using the Ivy intelligent optimization algorithm.
[0073] S3.1. Using the training set as input, the detection model is trained and hyperparameters are optimized in combination with a hyperparameter optimization algorithm;
[0074] S3.2. Set relevant parameters and initialize the population, including the population number N, the number of iterations Iter, and the upper and lower limits C. max With C min Initialize the initial position of each individual and save the initial historical optimal solution C Best .
[0075] The first step is to initialize the population.
[0076] C i =C min +rand(1,D)⊙(C max -C min ),i=1,...,Npop
[0077] Among them, rand(1,D) is a vector of uniformly distributed random numbers [0,1] with dimension D, where C max and C min are the upper and lower limits of the search space, ⊙ is the product of two vector elements, and Npop represents the number of all individuals in the population.
[0078] S3.3, update the growth rate of individuals;
[0079] Orderly growth of population:
[0080] ΔGv i (t+1)=rand 2 ⊙(N(1,D)⊙ΔGv i (t))
[0081] Among them, ΔGv i (t+1) and ΔGv i (t) represents the growth rate at time t+1 and time t, rand 2 is a random number of a random variable, and its probability density function is N(1,D) represents a random vector with dimension D, which is a random number from a standard normal distribution.
[0082] S3.4, use individuals to perform global search and local search respectively, and update the growth rate again;
[0083] In order to find sunlight to grow, ivy needs to climb something like walls, rocks or trees. In the natural forest, some ivy can find nearby trees to climb, and they look for old ivy that has already found a support point to climb. In the end, only the strongest ivy can survive. The following formula shows that the new ivy uses the old ivy to climb and move towards the light source. The global search is performed through the following formula:
[0084]
[0085] Among them, |N(1,D)| is the absolute value of N(1,D), is the element-wise division of two vectors. C i represents the hyperparameter group of the ith individual, C ii This represents the hyperparameter set of Old Ivy that has found a good support point.
[0086] All ivy plants in the population will follow and evolve towards the best growing ivy, that is, search for a better optimal solution near the best growing ivy, and perform local search near the optimal solution using the following formula:
[0087]
[0088] Among them, rand(1,D) is a vector of uniformly distributed random numbers [0,1] with dimension D, C Best Represents the historical optimal solution.
[0089] The value of the growth rate is then updated.
[0090]
[0091] Finally, use the following formula to make comparisons and take relevant actions.
[0092]
[0093] The ivy will then begin to expand the width of its branches and leaves, otherwise, the ivy continues to climb upward.
[0094] S3.5. Compare the optimal solution in the current iteration with the historical optimal solution. If C Best If it is greater than the historical optimal solution, the historical optimal solution is updated, otherwise it remains unchanged;
[0095] S3.6. If the termination condition is reached, the hyperparameters corresponding to the historical optimal solution are fixed to obtain the trained detection model.
[0096] In some feasible embodiments, the step S4 specifically includes:
[0097] Since most citrus orchards are located in hilly areas with complex natural environments, there are many green citrus fruits in the orchards, and the production measurement work requires manual counting, which consumes a lot of manpower costs. Therefore, the present invention uses the OStrongSORT+ edge computing platform to track and count green citrus fruits. The equipment of the present invention includes Jetson AGX Xavier, a touch screen display, a mobile power supply, and multiple tablets or mobile phones. The edge computing platform equipment has the advantages of strong portability and fast detection speed.
[0098] The SORT target tracking algorithm combines a target detector and a tracker. The StrongSORT algorithm improves on the SORT algorithm and introduces re-identification technology. While paying attention to the target movement, it uses the model to extract the target's appearance features and compares them with the features stored in the previous frame to determine whether it is the same object, thereby improving tracking accuracy.
[0099] During the real-time detection of green citrus fruits in the citrus orchard, the green citrus fruits may be lost due to overlapping fruits or obstructed by leaves, which will cause the tracking ID to change continuously; and before the green citrus fruit target trajectory is associated, the targets with low confidence will be filtered, and only the detection frames with confidence greater than 0.5 will be retained. Therefore, in this embodiment, the feature extractor BoT+ResNeSt50 in StrongSORT is changed to a lightweight OSNet for feature extraction, and the residual connection blocks of multiple convolutions in OSNet are used to realize full-scale feature learning, thereby improving StrongSORT's ability to capture key features of dynamic green citrus fruits. Multi-level trajectory association matching is used to retain as many detection frames as possible and detect confidence for matching, which includes motion branches and target appearance branches. The structure of the improved OStrongSORT refers to Figure 3 .
[0100] In the motion branch, the Hungarian algorithm is used to match and associate the tracking frame with the motion trajectory. The Mahalanobis distance calculates the degree of correlation between the predicted target and the actual target motion state, which can exclude the detection frame with low correlation with the average tracking result. However, the Mahalanobis distance association method fails when encountering fruit overlap or leaf occlusion, so in addition to the motion branch, the target appearance information is introduced for matching and association. In the appearance branch, the minimum cosine distance is used as the appearance metric. Combining the minimum cosine distance with the Mahalanobis distance can avoid interference from the citrus green fruit or leaf background with high similarity.
[0101] This embodiment uses the OStrongSORT algorithm to track, detect and count citrus fruits. The target detection performed by the YOLOv7S model is used to assign IDs to the detected citrus fruits through the OStrongSORT algorithm, and each citrus fruit with a different ID is tracked during movement.
[0102] In some feasible embodiments, it also includes:
[0103] Since most citrus orchards are located in hilly areas with complex terrain and require real-time detection and counting, and considering the limitations of wired cameras, multi-channel wireless IP cameras are used to collect images in real time. Multi-channel wireless IP cameras can improve the efficiency of tracking and counting, and can perform portable detection in citrus orchards with complex terrain, solving the problem of excessive labor costs in citrus yield measurement work, and preparing for the subsequent refined management and protection of citrus orchards.
[0104] The YOLOv7S model is transplanted to the edge computing platform, and the cameras of multiple tablets and mobile phones are called through wireless IP to capture the images of citrus fruits in real time. Then, the OStrongSORT target tracking algorithm is used to track and count the citrus fruits. Finally, the real-time processed data results can be observed from the touch screen display, tablet or mobile phone.
[0105] Based on the above scheme and in combination with actual application scenarios, the present invention provides the following adaptive settings:
[0106] Since most citrus orchards are located in hilly areas with complex natural environments, server equipment is not suitable for field use. Therefore, the optimized YOLOv7S model is combined with OStrongSORT and transplanted to the edge computing platform to form a portable citrus green fruit tracking and counting device for real-time tracking and counting of citrus green fruit.
[0107] Due to the large number of citrus fruits in the orchard, the production measurement work needs to be manually counted, which consumes a lot of manpower costs. The use of multi-channel wireless IP cameras can improve the efficiency of real-time tracking and counting, and can perform portable detection in citrus orchards with complex terrain, solving the problem of excessive manpower costs in citrus production measurement work, and preparing for the subsequent refined management and protection of citrus orchards.
[0108] As shown in Figure 4, a citrus fruit tracking and counting system based on an improved YOLOv7 model includes:
[0109] Image acquisition module, used to acquire images and build training sets;
[0110] The model building module is based on the YOLOv7 basic framework, introduces the SimAM attention mechanism module, and builds a detection model;
[0111] A model training module is used to train the detection model and optimize hyperparameters based on the training set to obtain a trained detection model;
[0112] The model application module processes the video to be tested based on the trained detection model and target tracking algorithm to generate a counting result.
[0113] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0114] A citrus fruit tracking and counting device based on an improved YOLOv7 model:
[0115] at least one processor;
[0116] at least one memory for storing at least one program;
[0117] When the at least one program is executed by the at least one processor, the at least one processor implements the citrus green fruit tracking and counting method based on the improved YOLOv7 model as described above.
[0118] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0119] A storage medium stores processor-executable instructions, which are used to implement a citrus fruit tracking and counting method based on an improved YOLOv7 model when executed by the processor.
[0120] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0121] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A citrus fruit tracking and counting method based on an improved YOLOv7 model, characterized in that: The following steps are involved: Get images and build a training set; Based on the YOLOv7 basic framework, the SimAM attention mechanism module is introduced to build a detection model; The detection model is trained and hyperparameters are optimized based on the training set to obtain a trained detection model; the video to be tested is processed based on the trained detection model and the target tracking algorithm to generate a counting result.
2. According to claim 1, a citrus fruit tracking and counting method based on an improved YOLOv7 model is characterized in that: The step of introducing the SimAM attention mechanism module based on the YOLOv7 basic framework and building a detection model specifically includes: Based on the YOLOv7 basic framework, the SimAM attention mechanism modules are introduced into the first branch and the second branch of the ELAN-H module in front of the P3 detection head to build a detection model.
3. According to claim 1, a citrus fruit tracking and counting method based on an improved YOLOv7 model is characterized in that: The step of training the detection model and optimizing hyperparameters based on the training set to obtain a trained detection model specifically includes: Taking the training set as input, the detection model is trained and hyperparameters are optimized in combination with a hyperparameter optimization algorithm; Set relevant parameters and initialize the population; Update the growth rate of individuals; Use individuals to perform global search and local search respectively, and update the growth rate again; Compare the optimal solution in the current iteration with the historical optimal solution, and update the historical optimal solution based on the comparison result; If the termination condition is reached, the hyperparameters corresponding to the historical optimal solution are fixed to obtain the trained detection model.
4. According to claim 3, a citrus fruit tracking and counting method based on an improved YOLOv7 model is characterized in that: The formula for global search of the model's hyperparameter set is as follows: Among them, C i represents the hyperparameter group of the i-th individual, N(1,D) represents a random vector with dimension D, C ii represents the hyperparameter group of old ivy that has found a good support point, Npop represents the number of individuals in the population, C max represents the upper limit of the search space, C min represents the lower limit of the search space, ⊙ represents the product of two vector elements, and rand 2 Represents a random number that is a random variable.
5. According to claim 4, a citrus fruit tracking and counting method based on an improved YOLOv7 model is characterized in that: The formula for local search of the model's hyperparameter set is as follows: Among them, rand(1,D) is a vector of uniformly distributed random numbers [0,1] with dimension D, C Best Represents the historical optimal solution.
6. According to claim 1, a citrus fruit tracking and counting method based on an improved YOLOv7 model is characterized in that: The target tracking algorithm specifically includes: Based on the StrongSORT target tracking algorithm; Introducing OSNet in the feature extractor; In the motion branch, the Hungarian algorithm is used to match and associate the tracking box with the motion trajectory; The minimum cosine distance is used as the appearance metric in the appearance branch.
7. According to claim 1, a citrus fruit tracking and counting method based on an improved YOLOv7 model is characterized in that: Also includes: The trained detection model and target tracking algorithm are deployed to the edge platform.
8. A citrus fruit tracking and counting system based on an improved YOLOv7 model, characterized in that: include: Image acquisition module, used to acquire images and build training sets; The model building module is based on the YOLOv7 basic framework, introduces the SimAM attention mechanism module, and builds a detection model; A model training module is used to train the detection model and optimize hyperparameters based on the training set to obtain a trained detection model; The model application module processes the video to be tested based on the trained detection model and target tracking algorithm to generate a counting result.
9. A citrus fruit tracking and counting device based on an improved YOLOv7 model, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the citrus green fruit tracking and counting method based on the improved YOLOv7 model as described in any one of claims 1-7.