A method and system for automatic counting of dense small targets

Abstract

The application discloses a kind of dense small target automatic counting method, comprising: obtaining the image of fruit to be counted;Image is input to target detection model, and the position information and confidence information of fruit are obtained;The position information and confidence information of fruit are input to target tracking model to obtain fruit target tracking result;Based on fruit target tracking result, the counting result of image is determined.A kind of dense small target automatic counting system, comprising sequentially connected in order: data acquisition module, information acquisition module, target tracking module and result output module.Realize the automatic counting of fruit on the basis of guaranteeing accurate and efficient, solve the problem that prior art lacks accurate and efficient automatic fruit counting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart agriculture, and more particularly to a method and system for automatically counting small and dense targets. BACKGROUND

[0002] At present, with the development of smart agriculture, orchard planting is becoming more and more sophisticated, and fruit counting plays an important role in agricultural production management. In addition to estimating the overall yield of the orchard, fruit counting can also be used to calculate the fruit maturity rate, so as to determine the picking period and improve the utilization rate of resources and planting benefits.

[0003] However, today's orchard growers still rely on manual field counting operations, which has the disadvantages of low counting efficiency, low counting accuracy and high labor input. The current fruit counting method is to detect the number of fruits in the image through a target detection model, but in real agricultural scenarios, fruits are easily blocked, the objects selected by the model are large target fruits, and the model cannot be applied to tracking of small and dense targets. There is a lack of accurate and efficient automatic fruit counting method.

[0004] Therefore, how to realize automatic fruit counting while ensuring precision and efficiency is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a method and system for automatically counting small and dense targets, which realizes automatic fruit counting while ensuring precision and efficiency, and solves the problem of lack of accurate and efficient automatic fruit counting in the prior art.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0007] A method for automatically counting small and dense targets, comprising:

[0008] obtaining an image of the fruit to be counted;

[0009] inputting the image into a target detection model to obtain fruit position information and confidence information;

[0010] inputting the fruit position information and the confidence information into a target tracking model to obtain a fruit target tracking result;

[0011] determining the counting result of the image based on the fruit target tracking result.

[0012] Preferably, the fruit position information and confidence information are obtained, specifically comprising:

[0013] preprocessing the input image to obtain a plurality of preprocessed sub-images;

[0014] A pre-trained model EfficientNetv2-B0 is used as a backbone network to extract features of the sub-images and obtain feature maps;

[0015] The feature maps are sequentially up-sampled and down-sampled at the neck part, and the up-sampled and down-sampled feature maps are fused by connection, each layer after up-sampling and down-sampling passes through a CBS-1 module to obtain the final feature map extracted by the neck part;

[0016] The detection head adopts a Decoupled Head structure, and after each layer of feature map output by the neck part passes through a CBS-3 module, a texture attention DeepTEN is added, then two 3x3 convolutions are performed, an anchor-free anchor box generation strategy is adopted, and after passing through the detection head, the image is predicted and output to obtain the detected fruit position information and confidence information, and the detected fruit is determined based on the fruit position information and confidence information.

[0017] Preferably, the input image is preprocessed, specifically including:

[0018] The input image is divided into a plurality of sub-images with the same resolution;

[0019] The sub-images and the input image are connected into a batch and input into the target detection model for parallel processing to obtain a detection box;

[0020] The detection box is mapped to the input image, and for the overlapping boxes, the overlap degree is determined by the formula IOS:

[0021]

[0022] Wherein a and b represent the corresponding detection boxes in the sub-image and the input image respectively, Inter represents the intersection between the two boxes, min represents the smallest area of the two box areas, bbox represents the coordinate information of the box, and area represents the area of the box;

[0023] The boxes with an overlap degree greater than 0.7 are merged to obtain the preprocessed sub-image.

[0024] Preferably, the fruit target tracking result is obtained, specifically including:

[0025] Each detected fruit is assigned an ID and a tracking state; the tracking state includes: an uncertain state, a determined state and a deleted state;

[0026] When managing the tracking states, the difference between adjacent frames is described by camera motion compensation, all determined state positions in the current frame are predicted using Kalman filtering, and when a tracking state matches a fruit in a new frame, the Kalman filtering parameters in the tracking state are updated according to the fruit position in the current frame;

[0027] The target frame with a confidence score greater than 0.1 is reserved, and the target frame is divided into two categories of high-score target and low-score target, the fruit to be added to the counting task is determined according to the two-stage data association strategy of the target tracking trajectory and the target frame, and the tracking result of the counted fruit is obtained;

[0028] The target tracking model counts the fruits according to the tracking result, and realizes automatic counting of the fruits.

[0029] Preferably, the tracking state assignment specifically includes:

[0030] When the fruit detected by the target tracking model does not belong to the tracking state established in the previous frame, the fruit is set as an uncertain state;

[0031] When the same fruit is detected in three consecutive frames, the tracking state of the fruit is set as a determined state, otherwise the tracking state is set as a deleted state;

[0032] When the fruit is in a determined state and fails to match the corresponding fruit for a long time, it is considered that the fruit has disappeared from the current image, and the fruit is set as a deleted state.

[0033] Preferably, the data association strategy specifically includes:

[0034] The high-score target fruit with a confidence score greater than 0.5 is matched with the fruit in a determined state, the position of the frame after prediction of each tracking state to be matched using Kalman filtering is bbox t , the position of each target frame to be matched is bbox d , and the target distance dist t,d between the high-score target fruit and the fruit in the determined state is calculated.

[0035] dist t,d =1-IoU(bbox t ,bbox d )*score d

[0036] Wherein, IoU represents the intersection over union between two frames, and score d represents the score of the detection frame to be matched.

[0037] When the target distance is less than a set threshold, it is determined that the two are matched successfully, otherwise the matching fails.

[0038] After the first matching, three types of data are obtained: matched tracking count targets and detection targets, unmatched tracking count targets and unmatched detection targets;

[0039] The unmatched detection targets after the first matching are matched with the tracking count targets in the uncertain state for the second time, and the calculation formula of the target distance is the same as dist t,d ;

[0040] The tracking count targets without matching after the second matching are marked as a deletion state, and the detection targets without matching after the second matching are marked as tracking count targets in the uncertain state;

[0041] The low-score target fruits with a confidence between 0.1 and 0.5 are matched with the tracking count targets without matching after the first matching for the third time, and the calculation formula of the target distance for the third time is:

[0042] dist t,d '=1-IoU(bbox t ,bbox d )

[0043] When the target distance is greater than 0.5, the third matching fails, the last appearance frame of the tracking count target without matching after the matching failure is searched, if the disappearance time is greater than the preset reserved frame time, it is considered that the fruit has disappeared from the current video or image, and the state is set to a deletion state, and the fruit is no longer matched, otherwise, it is reserved.

[0044] A dense small target automatic counting system comprises, which are sequentially connected: a data acquisition module, an information acquisition module, a target tracking module and a result output module;

[0045] The data acquisition module is used for acquiring an image of fruits to be counted.

[0046] The information acquisition module is used for inputting the image into a target detection model to obtain fruit position information and confidence information.

[0047] The target tracking module is used for inputting the fruit position information and the confidence information into a target tracking model to obtain fruit target tracking results.

[0048] The result output module is used for determining a counting result of the image based on the fruit target tracking results.

[0049] According to the technical solution, compared with the prior art, the present application provides a dense small target automatic counting method and system, which has the following advantages:

[0050] 1. By setting a lightweight one-stage target detection model LitchiDet, the problem of insufficient detection of occluded objects in dense small target detection is solved.

[0051] 2. By setting a two-stage multi-target tracking model LitchiTrack, more accurate data association matching and trajectory tracking are achieved without using a ReID model.

[0052] 3. The high-resolution image is preprocessed by image segmentation, which effectively avoids the problem of not being able to fully utilize the information in the image, and can more accurately identify small targets. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0054] Figure 1 The automatic counting method flowchart provided by the present application.

[0055] Figure 2 The image segmentation flowchart provided by the present application.

[0056] Figure 3 The dense small target detection algorithm structure diagram provided by the present application.

[0057] Figure 4 The target tracking model LitchiTrack data association strategy flowchart provided by the present application.

[0058] Figure 5 The automatic counting system structure diagram provided by the present application embodiment 4.

[0059] Figure 6 The automatic counting system structure diagram provided by the present application embodiment 5.

[0060] Figure 7 The automatic counting system structure diagram provided by the present application embodiment 6. DETAILED DESCRIPTION

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Example 1

[0063] like Figure 1 As shown, this embodiment of the invention discloses an automatic counting method for dense small targets, including:

[0064] Obtain images of the fruits to be counted;

[0065] The image is input into the target detection model to obtain the fruit location information and confidence information;

[0066] The fruit location information and confidence level information are input into the target tracking model to obtain the fruit target tracking result;

[0067] The counting results of the image are determined based on the fruit target tracking results.

[0068] Example 2

[0069] Acquire image or video data of the fruits to be counted;

[0070] Input image or video data into the automatic fruit counting model in the embedded platform to perform counting operations;

[0071] Preferably, the training process for the automatic fruit counting model is as follows:

[0072] A large number of fruit images or videos were collected using drones and then cropped and filtered to obtain pre-processed fruit images.

[0073] The preprocessed fruit images were sequentially subjected to data augmentation and data annotation operations to obtain an annotated dataset.

[0074] The labeled dataset includes a training set and a validation set. The automatic fruit counting model is trained and tested using the labeled dataset.

[0075] Preferably, the performance of the counting model is evaluated by having three different researchers manually count the fruits in each image or video and taking the average value as the true total number of fruits in each image or video, which is then compared with the predicted value of the automatic fruit counting model.

[0076] Preferably, the automatic fruit counting model includes: the target detection model LitchiDet and the target tracking model LitchiTrack.

[0077] Preferably, the target detection model LitchiDet is used to detect the positions of all fruits in the fruit image, and outputs the detected fruit positions and confidence information.

[0078] Preferably, the target detection model LitchiDet outputs the detected fruit position information and confidence information, specifically including:

[0079] The input image or video is preprocessed to obtain a plurality of preprocessed sub-images;

[0080] A pre-trained model EfficientNetv2-B0 is used as a backbone network to extract features of the sub-images to obtain feature maps;

[0081] The feature maps are sequentially upsampled and downsampled at the neck, the upsampled and downsampled feature maps are fused by connection, a CBS-1 module is added after each layer of the upsampled and downsampled feature maps after feature fusion, a CBS-3 module is added after each layer of the downsampled feature maps after feature fusion, and a CBS-1 module is used after each layer after upsample and downsample to normalize the channel number of all feature maps, further filter effective feature information, and obtain the final extracted feature maps at the neck;

[0082] Preferably, CBS-n represents n x n convolution + batch normalization + SiLU activation function, which is used to filter data information in the feature maps after feature fusion.

[0083] The detection head adopts a Decoupled Head structure, and a 1x1 convolution is used to predict classification and anchor box regression results respectively, which can effectively improve the performance and convergence speed of the model; after each layer of feature maps output by the neck passes through a CBS-3 module, a texture attention DeepTEN is added, so that the model focuses on the texture characteristics of the fruit, improves the detection accuracy, and then performs two 3x3 convolutions; an anchor-free anchor box generation strategy is adopted, which does not need to determine the size of the generated anchor box in advance, realizes end-to-end detection, and reduces the calculation cost of model migration to different data sets; after passing through the detection head, the image is predicted and output to obtain the detected fruit position information and confidence information, and the detected fruit is determined based on the fruit position information and confidence information.

[0084] Preferably, the input image is preprocessed, specifically including:

[0085] As shown in Figure 2 , taking an image with a resolution of 1920x1080 as an example, the input image is divided into 8 sub-images with a size of 640x640;

[0086] The sub-image and the input image are connected Concat into a batch and input into the target detection model for parallel processing. After the model completes the detection, for the detected frame, the offset during the cropping is adjusted, and the frame is mapped back to the original 1920x1080 resolution image. For the frames that overlap with each other, the overlap degree is determined by the formula IOS:

[0087]

[0088] Where a and b represent the corresponding detection frames in the sub-image and the input image respectively, Inter represents the intersection between the two frames, min represents the minimum area of the two frames, bbox represents the coordinate information of the frame, and area represents the area of the frame.

[0089] Frames with an overlap degree greater than 0.7 are merged to obtain the final preprocessed sub-image.

[0090] Preferably, the fusion method used in the present application is the Concat method.

[0091] Preferably, for each image, the overlap rate is set to 20% of 640 pixels, i.e. there is an overlap of 128 pixels between adjacent cropped images. This overlap setting does not take effect on the right and bottom end regions of the final image, to ensure that each sub-image is 640x640 in size, avoiding the problem of incomplete targets caused by edge cropping.

[0092] Preferably, in the present embodiment, EfficientNetv2 uses F-MBConv composed of two 3x3 convolutions as the shallow part of the network, and MBConv composed of 1x1 convolution and 3x3 depth separable convolution as the deep part of the network. As shown in Fig. 3, EfficientNetv2-B0 is used as the backbone network in the present application, and in the dashed box of BackBone on the left side of the figure, each flow block is divided into three parts. The number on the left represents the number of times the module is reused, and Pn on the right represents the output feature map with a size of 1 / 2^n. After image segmentation, the input image is transmitted into the backbone network to obtain feature maps of different convolution layers. Figure 1

[0093] Then the feature maps are input into the Neck dashed box from bottom to top. Each layer of feature maps is divided into two parts, one part is subjected to convolution operation, and the other part is subjected to feature fusion. Pn represents the feature maps of the same layer, which are subjected to upsampling and downsampling and then a CBS-1 module to obtain the final extracted feature maps of each layer. Finally, the input is input into the Head dashed box, which is subjected to a CBS-3 module and a ConvTEN module in turn to obtain the output result.

[0094] ​Preferably, the application increases the texture attention in the part of the detection head, that is, a learnable residual coding layer-Texture Encoding Network (DeepTEN) is introduced; DeepTEN can effectively capture the texture features of the fruit, making the fruit more easily distinguished from the surrounding branches and leaves or other backgrounds; the addition of DeepTEN can also make the model pay more attention to the irregular edge contour of the fruit, which is very effective for positioning objects in dense small target detection.

[0095] Preferably, DeepTEN combines traditional texture recognition methods, integrates dictionary learning and residual coding into a convolutional module, and learns coding parameters and intrinsic dictionaries in a fully supervised manner during training. Through the encoding representation, the extracted convolutional features help improve the model training efficiency and prediction accuracy.

[0096] Preferably, the intrinsic dictionary is learned from a set of N descriptors X = {x1,...x N} distribution by assigning weights to the gradient, and the gradient information is transmitted to each component during backpropagation, so as to adjust each component for the task at hand. The weight assignment adopts the Soft-assignment method, and the formula is as follows:

[0097]

[0098] Where a ik represents the descriptor x i , r ik represents the residual vector, S k represents the smoothing factor, K represents the number of coding parameters, r ij represents the jth coding corresponding residual vector, S j represents the jth coding corresponding smoothing factor.

[0099] Preferably, the target tracking model LitchiTrack performs fruit target tracking based on fruit position information and confidence information, and realizes automatic fruit counting.

[0100] Preferably, the automatic fruit counting is implemented, specifically including:

[0101] An ID and a tracking state track are given to each detected fruit; the tracking state track includes: an uncertain state tentative, a confirmed state confirmed, and a deleted state deleted;

[0102] When managing the tracking states, the difference between adjacent frames is described by camera motion compensation, all determined state positions in the current frame are predicted using Kalman filter, when a tracking state matches the fruit in a new frame, the Kalman filter parameters in the tracking state are updated according to the fruit position in the current frame, so as to better match the fruit in the next frame;

[0103] The target frame with a confidence score greater than 0.1 is reserved, and the target frame is divided into two categories of high-score target and low-score target, the fruit added to the counting task is determined by a two-stage data association strategy of track and target frame, and the tracking result of the counted fruit is obtained.

[0104] The target tracking model LitchiTrack counts the fruits according to the tracking result, and realizes automatic counting of the fruits.

[0105] Preferably, the tracking state assignment specifically includes:

[0106] When the fruit detected by the target tracking model LitchiTrack does not belong to the tracking state established in the previous frame, it is set as an uncertain state tentative;

[0107] When the same fruit is detected in three consecutive frames, the tracking state of the fruit is set as a determined state confirmed, otherwise the tracking state of the fruit is set as a deleted state deleted;

[0108] When the fruit is in the determined state and fails to match the corresponding fruit for a long time, it is considered that the fruit has disappeared from the current video, and the fruit is set as the deleted state deleted.

[0109] Preferably, as shown in Figure 4 , the data association strategy specifically includes:

[0110] The high-score target fruit with a confidence score greater than 0.5 is matched with the fruit in the determined state, the position of the frame after prediction of each tracking state to be matched using Kalman filter is bbox t , the position of each target frame to be matched is bbox d , and the target distance dist t,d between the high-score target fruit and the fruit in the determined state is calculated.

[0111] dist t,d =1-IoU(bbox t ,bbox d )*score d

[0112] Wherein, IoU represents the intersection over union between two frames, and score d represents the score of the detection frame to be matched;

[0113] When the target distance is less than the set threshold, it is determined that the two are matched successfully, otherwise the matching fails;

[0114] After the first matching, three types of data are obtained: matched tracking count targets and detection targets MatchedTracks&Objects, unmatched tracking count targets Unmatched Tracks and unmatched detection targets Unmatched Objects;

[0115] The unmatched detection targets after the first matching are matched with the tracking count targets in an uncertain state for the second time, and the calculation formula of the target distance is the same as dist t,d The calculation formula is the same;

[0116] The tracking count targets that are not matched after the second matching are marked as deleted, and the detection targets that are not matched after the second matching are marked as tracking count targets in an uncertain state;

[0117] The low-score target fruits with a confidence between 0.1 and 0.5 are matched with the tracking count targets that are not matched after the first matching for the third time, and the calculation formula of the target distance for the third time is:

[0118] dist t,d '=1-IoU(bbox t ,bbox d )

[0119] When the target distance is greater than 0.5, the third matching fails, the last appearance frame of the tracking count target that is not matched after the matching is found, if the disappearance time is greater than the preset frame retention time, it is considered that the fruit has disappeared from the current video or image, and the state is set to deleted, and the fruit is no longer matched, otherwise it is retained.

[0120] Preferably, after each round of three times of matching, the fruit information matched successfully is obtained, the fruit automatic counting task is updated according to the determined state and ID of the fruit, and a real-time counting result is obtained.

[0121] Preferably, the target distance matching thresholds in the three times of matching are set to 0.1, 0.3 and 0.5 respectively, if the target distance is lower than or equal to the set threshold, it indicates that the matching is successful, and if the target distance is higher than the set threshold, it indicates that the matching fails; different matching thresholds are set by different target distance matching, which ensures the effectiveness of data association.

[0122] Preferably, the preset frame retention time in the embodiment is set to 90 frames, and if the fruit disappearance time is greater than 90 frames, the fruit will not be matched.

[0123] Preferably, Kalman filtering can only predict the next motion state according to the last stored motion state, and as the time of the fruit disappearing in the picture increases, the uncertainty of the Kalman filtering prediction also increases accordingly. In the present application, additional weight is added to the time of the track count target track disappearing to offset this part of uncertainty when data association, improving the prediction accuracy.

[0124] Preferably, the target tracking model LitchiTrack of the present application considers low-score targets for three times of matching, and even if the target is lost in a certain frame, it can be found back in the subsequent frame with the same ID, improving the accuracy of fruit counting, reducing missed detection and false detection, and maintaining the persistence of tracking; since the image segmentation method is used for fruit prediction, the target tracking model LitchiTrack performs more accurate target detection on the image, and it is easier to find small targets in the same frame.

[0125] Preferably, the fruit automatic counting model is transplanted to an embedded platform, specifically including:

[0126] The training task of the fruit automatic counting model is completed through the server to obtain an initial target detection model;

[0127] The initial target detection model file is converted into an offline network model file supported by the Ascend AI processor and transplanted into the running environment of the Atlas 200DK embedded platform;

[0128] The inference execution of the algorithm is completed, the input data calls the fruit automatic counting model for inference, and the inference result is saved to the output buffer.

[0129] Preferably, the training task of the fruit automatic counting model is completed through the server, specifically including:

[0130] The input fruit image size is set to 640x640 resolution, and the target detection model LitchiDet is trained;

[0131] The trained target detection model LitchiDet is called, and the high-score target box and low-score target box thresholds in the target tracking model LitchiTrack are set to 0.5 and 0.1 respectively, and the target distance matching threshold in three times of matching is set to 0.1, 0.3 and 0.5 respectively;

[0132] The input fruit image or video realizes automatic counting to obtain the final fruit automatic counting model.

[0133] Preferably, the inference execution of the algorithm is completed, specifically including:

[0134] The acl.init interface is used for ACL initialization, the acl.media interface is used for data preprocessing resource application, and the acl.mdl interface is used for fruit automatic counting model loading resource application;

[0135] The DVPP module based on the Ascend AI processor realizes picture format conversion and cropping through the convet function and the resize function in the acl.media;

[0136] The acl.mdl interface is called, the image is sent into the fruit automatic counting model for model loading and execution, and the inference task of the model is completed;

[0137] The inference result is obtained and result output is performed.

[0138] Embodiment 3

[0139] A mobile terminal APP is developed based on the DJI MSDK software development kit, real-time transmission of the collected video stream to the development board is performed, and development and application of a real-time automatic counting system and remote monitoring of an orchard are realized.

[0140] Preferably, the mobile terminal APP is developed based on the MSDK, and specifically includes:

[0141] The unmanned aerial vehicle, the computer and the mobile terminal are ensured to be in the same local area network, and the mobile terminal APP is developed based on the Android system by using the developer tool MSDK;

[0142] S1, the video collected by the unmanned aerial vehicle is obtained in real time through the APP, is encoded into a video code stream in H.265 format, and then is sent to the Ascend 310 processor in the Atlas 200DK embedded platform through the RTSP protocol;

[0143] S2, the Ascend 310 processor receives the original video data from the APP through a network port, then calls FFmpeg to capture the raw code stream, then decodes, and then places the decoded image into an output buffer for subsequent processing;

[0144] S3, the decoded image is preprocessed in combination with the hardware driving capability of the Ascend 310, and functions such as scaling and cropping and color gamut conversion are completed;

[0145] S4, the algorithm analysis result is obtained, is solved to obtain detection confidence, position and other information, and the original image data is encoded, and finally the detection result is output;

[0146] S5, the output module receives the image data in jpge format, encapsulates the image data into data in protobuf format, and sends the data to the server through a socket network interface, the target frame position, the number and the counting result are received by the Android mobile terminal APP in real time, the remote client browser can display the detection result through the established connection, and the orchard remote monitoring is realized.

[0147] Preferably, the DJI MSDK software development kit is a software development kit of a DJI unmanned aerial vehicle, through the MSDK, a developer can use various functions of the DJI unmanned aerial vehicle, and can also develop functions according to own requirements.

[0148] Embodiment 4

[0149] As shown in Figure 5 The embodiment of the application discloses a dense small target automatic counting system, which comprises, which are connected in sequence: a data acquisition module, an information acquisition module, a target tracking module and a result output module.

[0150] The data acquisition module is used for acquiring images of fruits to be counted.

[0151] The information acquisition module is used for inputting the images into a target detection model to obtain fruit position information and confidence information.

[0152] The target tracking module is used for inputting the fruit position information and the confidence information into a target tracking model to obtain fruit target tracking results.

[0153] The result output module is used for determining counting results of the images based on the fruit target tracking results.

[0154] Embodiment 5

[0155] As shown in Figure 6 Another preferred embodiment of the application discloses a dense small target automatic counting system, which comprises, which are connected in sequence: an unmanned aerial vehicle, an Android mobile application terminal and an Atlas 200DK developer kit; further comprising a ground host monitoring terminal connected with the Atlas 200DK developer kit through a USB-typeC interface.

[0156] The embodiment is applied to litchi counting, the unmanned aerial vehicle is used for shooting litchi video data, and the litchi video data is transmitted to the Android mobile application terminal.

[0157] The Android mobile application terminal is used for transmitting the litchi video data to the Atlas 200DK developer kit.

[0158] The Atlas 200DK developer kit pre-deploys a fruit automatic counting model for inference on the acquired video stream, obtains the target frame information and counting result, and transmits the result to the Android mobile application in real time.

[0159] The ground host monitoring terminal views the target frame information and counting result in real time through the Firefox browser.

[0160] Preferably, the unmanned aerial vehicle shooting task is selected to be performed in a sunny, cloudless or less cloudy environment with little or small wind speed.

[0161] Preferably, the unmanned aerial vehicle used in the embodiment is a DJI Mini 3 unmanned aerial vehicle produced by DJI Company, the camera of which adopts a 1 / 1.3-inch sensor, supports double-native ISO and chip-level HDR technology, a f / 1.7 large aperture cooperates with a 2.4-micron four-in-one large pixel, supports 4x zoom, and can record 4K HDR video.

[0162] Preferably, the Huawei Atlas 200DK developer kit is used in the embodiment, the model of which is 3000, and the Ascend 310 AI processor is integrated, the single-card computing power of which can reach 88 TOPS INT8, and 80 high-definition video real-time analysis is supported; the Ascend 310 adopts the AI Core of the Davinci architecture, supports high-speed interfaces such as PCIE3.0, RGMII, USB3.0, and low-speed interfaces such as GPIO, UART, I2C, SPI, and greatly improves the overall performance of the neural network algorithm.

[0163] Preferably, the development environment system used by the ground host monitoring terminal in the embodiment is the Linux system of ubuntu18.04.1, the developer kit version used is 20.0.RC1, a lightweight Web version video server software Presenter Server developed by Huawei is used to realize video streaming, and the inference result is displayed in real time through the Firefox browser.

[0164] Embodiment 6

[0165] As shown in Figure 7 Another preferred embodiment of the application discloses a dense small target automatic counting system, which comprises front-end visual acquisition modules, mobile terminal control modules, streaming media processing modules, application modules and remote monitoring modules connected in sequence; the application module is connected with the mobile terminal control module.

[0166] The front-end visual acquisition module is used for collecting video and transmitting the video to the mobile terminal control module through the APP.

[0167] The mobile terminal control module is configured to control the front-end visual acquisition module to acquire a video, acquire a real-time video stream through a Mobile SDK, and transmit the real-time video stream to the streaming media processing module.

[0168] The streaming media processing module is configured to call a hardware driver layer interface to uniformly manage all tasks related to image data, complete processing of streaming media data, and include video input, video preprocessing, video postprocessing, and video output.

[0169] The application module is configured to deploy a trained fruit automatic counting model to an Atlas 200DK device, realize real-time automatic counting of a fruit video stream, and transmit target frame information and counting results to the mobile terminal control module and the remote monitoring module, respectively.

[0170] The remote monitoring module is configured to remotely and real-timely view the target frame information and the counting results, and perform orchard monitoring and management.

[0171] According to the technical solution, compared with the prior art, the application provides a dense small target automatic counting method and system, which has the following advantages.

[0172] 1. A lightweight One-stage target detection model LitchiDet is set, and the problem of insufficient detection of occluded objects in the model in dense small target detection is solved.

[0173] 2. A two-stage multi-target tracking model LitchiTrack is set, and more accurate data association matching and track tracking are realized without using a ReID model.

[0174] 3. High-resolution images are preprocessed in an image segmentation manner, which effectively avoids the problem of insufficient use of information in the images and enables more accurate identification of small targets.

[0175] 4. In order to improve the detection speed while maintaining the accuracy, an embedded platform is used to replace the traditional acceleration device to quickly and efficiently complete the dense small target automatic counting task in the actual scene.

[0176] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0177] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for automatic counting of dense small objects, characterized in that, The method comprises the following steps: obtaining an image of fruits to be counted; inputting the image into a target detection model to obtain fruit position information and confidence information; inputting the fruit position information and the confidence information into a target tracking model to obtain a fruit target tracking result; the fruit target tracking result comprises the following steps: assigning an ID and a tracking state to each detected fruit; the tracking state comprises an uncertain state, a determined state and a deleted state; when managing the tracking state, describe the difference between adjacent frames through camera motion compensation, predict the position of all determined state fruits in the current frame using Kalman filtering, and update the Kalman filtering parameters in the tracking state when a tracking state matches a fruit in a new frame, and update the Kalman filtering parameters in the tracking state according to the fruit position in the current frame; retain all target boxes with a confidence score greater than 0.1, and divide the target boxes into high-score target boxes and low-score target boxes, determine the fruits to be counted according to the two-stage data association strategy of target tracking trajectory and target box, and obtain the tracking result of the counted fruits; the target tracking model counts the fruits according to the tracking result to realize automatic counting of the fruits; the data association strategy comprises the following steps: Match the high-score target fruit with a confidence greater than 0.5 with the determined state fruit, the position of the bounding box of each to-be-matched tracking state after Kalman filtering prediction is bbox t , the position of each to-be-matched target box is bbox d , and the target distance dist t,d between the high-score target fruit and the determined state fruit is calculated dist t,d = 1 - IoU(bbox t , bbox d ) * score d wherein, IoU represents the intersection over union between two boxes, score d represents the score of the detection box to be matched; when the target distance is less than a set threshold, it is determined that the two are matched successfully, otherwise the matching fails; after the first matching, three types of data are obtained: matched tracking counting targets and detected targets, unmatched tracking counting targets and unmatched detected targets; The first matching of the detection target and the tracking count target in the uncertain state is performed, and the distance between the targets is calculated according to the following formula: t,d The same formula is used to calculate the distance between the targets. after the second matching, the unmatched tracking counting targets are marked as deleted state, and the unmatched detected targets are marked as uncertain state tracking counting targets; the low-score target fruits with a confidence score between 0.1 and 0.5 are matched with the unmatched tracking counting targets after the first matching for the third time, and the target distance calculation formula for the third matching is: dist t,d ' = 1 - IoU(bbox t , bbox d ) when the target distance is greater than 0.5, the third matching fails, the last frame where the unmatched tracking counting target appears is searched, if the disappearance time is greater than the preset frame retention time, it is considered that the fruit has disappeared from the current video or image, and the state of the fruit is set to deleted state, and the fruit is no longer matched, otherwise the fruit is retained; determine the counting result of the image based on the fruit target tracking result.

2. The method of claim 1, wherein, the fruit position information and the confidence information are obtained by the following steps: preprocessing the input image to obtain a plurality of preprocessed sub-images; using a pre-trained model EfficientNetv2-B0 as a backbone network to extract features of the sub-images to obtain feature maps; performing upsampling and downsampling on the feature maps in the neck in turn, using a connection method to fuse the upsampled and downsampled feature maps, and obtaining a final extracted feature map in the neck through a CBS-1 module after each layer is upsampled and downsampled; The detection head adopts a Decoupled Head structure, after each layer of feature maps output through the neck passes through a CBS-3 module, texture attention DeepTEN is added, then 3x3 convolution is performed twice, an anchor-free anchor box generation strategy is adopted, prediction and output image are performed after the detection head, fruit position information and confidence information are obtained, and the detected fruits are determined based on the fruit position information and the confidence information.

3. The method of claim 2, wherein, The input image is preprocessed, specifically including: The input image is divided into a plurality of sub-images with the same resolution; The sub-images and the input image are connected as a batch and input into the target detection model for parallel processing to obtain a detection box; The detection box is mapped to the input image, and for the overlapping boxes, the overlapping degree is determined by the formula IOS: Where a and b represent the corresponding detection boxes in the sub-image and the input image respectively, Inter represents the intersection between the two boxes, min represents the smallest area of the two box areas, bbox represents the coordinate information of the box, and area represents the area of the box; The boxes with an overlapping degree greater than 0.7 are merged to obtain a preprocessed sub-image.

4. The method of claim 1, wherein, The tracking state is assigned, specifically including: When the fruit detected by the target tracking model does not belong to the tracking state established in the previous frame, it is set to an uncertain state; When the same fruit is detected in three consecutive frames, the tracking state is set to a determined state, otherwise it is set to a deleted state; When in the determined state and unable to match the corresponding fruit for a long time, it is considered that the fruit has disappeared from the current image, and it is set to a deleted state.

5. A system for performing an automatic counting method of small objects in a dense population according to any one of claims 1 to 4, characterized in that, It comprises, in sequence, a data acquisition module, an information acquisition module, a target tracking module and a result output module; The data acquisition module is configured to acquire an image of a fruit to be counted; The information acquisition module is configured to input the image into a target detection model to obtain fruit position information and confidence information; The target tracking module is configured to input the fruit position information and the confidence information into a target tracking model to obtain a fruit target tracking result; The result output module is configured to determine a counting result of the image based on the fruit target tracking result.

Images