Target Dynamic Tracking Method, System, Device and Medium Based on DeepSORT

Abstract

The present invention relates to the field of computer vision technology, and provides a target dynamic tracking method, system, device and medium based on DeepSORT, which solves the problem in the prior art that targets in the same area appear in different images and are repeatedly counted, directly affecting the counting accuracy. Among them, the method includes: inputting video stream data into a target detector so that the target detector sequentially outputs detection box information of the target frame by frame; assigning a target ID to the target corresponding to each target detection box in the t-th frame; predicting the predicted box information of the target in the (t + 1)-th frame; performing IOU matching between the predicted box and the detection box in the (t + 1)-th frame; assigning the target ID corresponding to the t-th frame to the target corresponding to the detection box with successful IOU matching in the (t + 1)-th frame; when IOU matching is performed continuously for N frames, performing cascade matching between the targets in each frame after the N-th frame and the targets after continuous IOU matching for N frames; counting the target IDs with successful cascade matching as the detection result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target dynamic tracking method, system, device and medium based on DeepSORT. Background Art

[0002] In recent years, various intelligent devices integrating modern agriculture with computer vision technology have been increasingly popularized in agricultural production and have become a research hotspot in the agricultural field. Specifically in apple production, based on deep learning object detection technology, apples can be identified, yield predicted, and pests and diseases identified through pictures or videos taken by cameras or drones. Also, the quality of apple fruits can be graded according to image information. These technologies have greatly promoted the intelligent development of apple production management. Among them, accurately identifying apple fruits using object detection technology and then predicting the yield is of great significance for the operation and management of orchards and can provide information on orchard yields for fruit farmers.

[0003] Traditional technologies for crop yield estimation still have many problems to a certain extent, such as unclear image boundaries caused by sliding windows, complex target feature design, poor portability, and cumbersome manual design. Therefore, more and more deep learning-based object detection algorithms are applied to agricultural production. These algorithms can be roughly divided into single-stage and two-stage algorithms according to the algorithm implementation process. Classic two-stage object detection algorithms include R-CNN, Fast R-CNN, Faster R-CNN, etc. Compared with two-stage algorithms, single-stage algorithms have less redundant calculation, faster detection speed, and lower requirements for hardware performance. Representative single-stage algorithms include the YOLOV series, SSD, etc. Single-stage algorithms omit the step of generating candidate target regions and directly predict the category or position of the target on the input image, resulting in a significant improvement in detection speed, so they are widely used in the agricultural field.

[0004] Most of the existing apple quantity detection methods are based on static images for experiments. The static images taken have a single perspective and limited field of view. In addition, during the shooting process, apples in the same area may appear in different images, resulting in duplicate counting, making it difficult to automatically and accurately count apples, which directly affects the accuracy of yield estimation. Summary of the Invention

[0005] The purpose of the present invention is to provide a target dynamic tracking method, system, device and medium based on DeepSORT to solve the technical problem in the prior art that targets in the same area appear in different images and are repeatedly counted, directly affecting the counting accuracy.

[0006] To solve the above technical problems, as an aspect of the present invention, a method for dynamically tracking targets based on DeepSORT is provided, including the following steps:

[0007] Input the video stream data into the target detector so that the target detector outputs the detection box information of the targets frame by frame in sequence;

[0008] Assign a target ID to the target corresponding to each target detection box in the t-th frame; where t is an integer greater than 0;

[0009] Predict the predicted box information of the target in the (t + 1)-th frame according to the detection box information of the target in the t-th frame;

[0010] Match the predicted box and the detection box in the (t + 1)-th frame by IOU according to the detection box information and the predicted box information in the (t + 1)-th frame;

[0011] Assign the target ID corresponding to the t-th frame to the target corresponding to the detection box with successful IOU matching in the (t + 1)-th frame;

[0012] When the predicted box and the detection box of a target are continuously IOU-matched for N frames, cascade-match the target in each frame after the N-th frame with the target after continuous N-frame IOU matching; where N is an integer greater than 1;

[0013] Count the target IDs with successful cascade matching as the detection result.

[0014] Further, the detection box information includes the width, height and the coordinates of the center point of the detection box, and the predicted box information includes the width, height and the coordinates of the center point of the predicted box; the predicting the predicted box information of the target in the (t + 1)-th frame according to the detection box information of the target in the t-th frame includes:

[0015] Calculate the state matrix of the target according to the width, height and the coordinates of the center point of the detection box of the target in the t-th frame; where the state matrix includes the coordinates of the center point of the detection box of the target in the t-th frame, the aspect ratio of the width to the height of the detection box, the height, and the change rates of the coordinates of the center point, the aspect ratio and the height;

[0016] Calculate the predicted box information of the target in the (t + 1)-th frame through Kalman filtering according to the state matrix of the target.

[0017] Further, the cascade-matching the target in each frame after the N-th frame with the target after continuous N-frame IOU matching includes:

[0018] Select one frame from the continuous N frames to extract the first appearance feature information of the target after IOU matching in this frame by using the ReID network, and obtain the first position information of the target in this frame through the detection box information of the target;

[0019] In the current frame after N frames, use the ReID network to extract the second appearance feature information of all targets in the current frame, and obtain the second position information of all targets in the current frame through the prediction box information;

[0020] Calculate a cost matrix according to the first appearance feature information and the second appearance feature information, as well as the first position information and the second position information;

[0021] Complete cascade matching through the Hungarian matching algorithm according to the cost matrix.

[0022] Further, the calculating a cost matrix according to the first appearance feature information and the second appearance feature information, as well as the first position information and the second position information includes:

[0023] Calculate the cosine distance of the appearance feature information according to the first appearance feature information and the second appearance feature information;

[0024] Calculate the Mahalanobis distance of the position information according to the first position information and the second position information;

[0025] Construct a cost matrix according to the cosine distance of the appearance feature information and the Mahalanobis distance of the position information.

[0026] Further, the first appearance feature information and the second appearance feature information at least both include the size, shape and color of the target.

[0027] Further, it further includes: when multiple targets complete consecutive N-frame IOU matching, sort the multiple targets according to the number of frames of consecutive IOU matching to prioritize the targets in each frame after N frames for cascade matching with the targets with higher priority; among them, the more frames of consecutive IOU matching, the higher the priority.

[0028] Further, it further includes: deleting the prediction boxes with failed IOU matching.

[0029] As the second aspect of the present invention, a target dynamic tracking system based on DeepSORT is provided, including:

[0030] A detection module, configured to input video stream data into a target detector so that the target detector outputs the detection box information of the target frame by frame;

[0031] A first ID assignment module, configured to assign a target ID to the target corresponding to each target detection box in the t-th frame; where t is an integer greater than 0;

[0032] A prediction module, configured to predict the prediction box information of the target in the (t + 1)-th frame according to the detection box information of the target in the t-th frame;

[0033] The IOU matching module is used to perform IOU matching between the predicted bounding boxes and the detection bounding boxes in the (t + 1)-th frame according to the detection bounding box information and the predicted bounding box information in the (t + 1)-th frame;

[0034] The second ID assignment module is used to assign the target ID corresponding to the t-th frame to the target corresponding to the detection bounding box with successful IOU matching in the (t + 1)-th frame;

[0035] The cascade matching module is used to perform cascade matching between the target in each frame after the N-th frame and the target after continuous N-frame IOU matching when the predicted bounding box and the detection bounding box of a target are continuously IOU-matched for N frames; where N is an integer greater than 1;

[0036] The statistics module is used to count the target IDs with successful cascade matching as the detection result.

[0037] As the third aspect of the present invention, there is provided an electronic device, including:

[0038] One or more processors;

[0039] A storage device for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of the above first aspects.

[0041] As the fourth aspect of the present invention, there is provided a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, the method according to any one of the above first aspects is implemented.

[0042] The beneficial effects of the present invention are:

[0043] The present invention counts the target video (instead of static images), solving the problems of single perspective and limited field of view due to shooting static images; in addition, the present invention assigns the same ID to two targets with successful IOU matching in different images, and performs cascade matching after continuous multi-frame IOU matching. Only the target ID with successful cascade matching is counted into the detection result, reducing the impact of ID jump on the calculation accuracy and solving the problem that the targets in the same area appear in different images and are repeatedly counted, directly affecting the counting accuracy. Description of the Drawings

[0044] Figure 1 Schematically shows a step diagram of a target dynamic tracking method based on DeepSORT.

[0045] Figure 2 Schematically shows the structural diagram of a multi-size target model improved based on YOLOV7.

[0046] Figure 3 is Figure 2 The structural diagram of the CBS module in

[0047] Figure 4 is Figure 2 The structural diagram of the ELAN module in

[0048] Figure 5 is Figure 2 The structural diagram of the MPConv module in

[0049] Figure 6 is Figure 2 The structural diagram of the SPPCSPC module in

[0050] Figure 7 is Figure 2 The structural diagram of the Concat-Conv module in

[0051] Figure 8 Schematically shows a block diagram of a target dynamic tracking system based on DeepSORT.

[0052] [[ID=z36]] Figure 9 Schematically shows a block diagram of an electronic device.

[0053] Figure 10 Schematically shows a block diagram of a computer-readable medium. Detailed implementation manners

[0054] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Like reference numerals in the figures denote like or similar parts, and thus their repetitive description will be omitted.

[0055] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of this application. However, those skilled in the art will realize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of this application.

[0056] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0057] The flowcharts shown in the drawings are only exemplary illustrations, not necessarily including all contents and operations / steps, nor necessarily executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0058] It should be understood that although terms such as first, second, and third may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concept of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more of them.

[0059] Those skilled in the art can understand that the drawings are only schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing the present application. Therefore, they cannot be used to limit the protection scope of the present application.

[0060] The present application improves the DeepSORT algorithm. Please refer to Figure 1 , Figure 1 which schematically shows a step diagram of a target dynamic tracking method based on DeepSORT. According to the first specific embodiment of the present invention, as Figure 1 shown, the present invention provides a target dynamic tracking method based on DeepSORT, including the following steps:

[0061] S11, input video stream data into a target detector so that the target detector outputs detection box information of the target frame by frame.

[0062] Among them, the target detector is a target detection algorithm model based on deep learning. For example, classic two-stage target detection algorithms include: R-CNN, Fast R-CNN, Faster R-CNN, etc.; and for example, representative one-stage algorithms include: YOLOV series, SSD, etc. The detection box is a detection box of different sizes output by the target detection algorithm model containing the target. The detection box information (c x , c y , w, h) includes the width and height of the detection box and the coordinates of the center point of the detection box. Among them, (c x , c y) is the center point coordinate of the detection box, w is the width of the detection box, and h is the height of the detection box.

[0063] Since most of the existing apple quantity detection methods are based on static images for experiments. However, in the natural environment, when shooting apples, due to factors such as too far shooting distance and leaf occlusion, some detection target pixels are too small, resulting in a large difference in different target scales. Although YOLOV7 has three detection heads to detect large, medium, and small targets respectively. But for tiny targets with little pixel information, the detection effect is poor only relying on these three detection heads.

[0064] To solve the technical problem that the YOLOV series detection networks in the prior art have a poor detection effect on tiny targets with little pixel information, the target detector can be a multi-size target detection model improved based on YOLOV7.

[0065] First, construct a multi-size target detection model improved based on YOLOV7. Figure 2 Schematically shows the constellation configuration diagram of the multi-size target detection model improved based on YOLOV7. As Figure 2 shown, the multi-size target detection model includes a Backbone network, a Neck network, and a Head network. The Backbone network is used to perform four times of feature extraction on the feature map input to the multi-size target detection model. The Neck network is used to fuse the features extracted four times by the Backbone network to form four fusion branches, and input the four fusion branches into the Head network to detect four different size targets. The Head network outputs the detection results of the four different size targets.

[0066] Please refer to Figure 2 and Figure 3 , Figure 3 is Figure 2 the CBS module structure diagram in Figure 2 and Figure 3 shown, the CBS combination module includes four different CBS modules. The CBS module is used to change the size and number of the feature map. Stacking multiple CBS modules or combining them with other modules can achieve purposes such as feature extraction and feature fusion. The CBS module is composed of a two-dimensional convolutional layer Conv, a batch normalization layer BN, and an activation function layer SiLU connected in sequence.

[0067] As Figure 2As shown in the figure, the CBS combination module includes a fifth CBS module, a sixth CBS module, a seventh CBS module, and an eighth CBS module connected in sequence. Among them, the fifth CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 3X3, and the number of output feature maps is 32; the sixth CBS module is a two-dimensional convolution with a step size of 2 and a convolution kernel size of 3X3, and the number of output feature maps is 64; the seventh CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 3X3, and the number of output feature maps is 64; the eighth CBS module is a two-dimensional convolution with a step size of 2 and a convolution kernel size of 3X3, and the number of output feature maps is 128.

[0068] Assume that the size of the input image of the multi-size object detection model is H*W*C, where H represents the length, W represents the width, and C represents the number of channels. As Figure 2 shown, if the size of the input image is 640*640*3, the size of the extracted feature map after passing through the CBS combination module becomes 160*160*128.

[0069] Continue to refer to Figure 2 , after the feature map passing through the CBS combination module enters the first SIMAM combination module, and the first SIMAM combination module is used to perform the first feature extraction on the input feature map. The features extracted by the first SIMAM combination module for the first time are respectively input into the second SIMAM combination module and the Neck network. As Figure 2 shown, the first SIMAM combination module includes an ELAN module and a SIMAM module in sequence.

[0070] The ELAN module is an efficient layer aggregation network, and its function is to perform multi-scale feature extraction and fusion on the image. Next, please refer to Figure 4 , Figure 4 is Figure 2 the structure diagram of the ELAN module in Figure 4 . The ELAN module aggregates features at four different levels to better capture multi-scale information. The ELAN module includes four branches. As Figure 2 shown, both the first branch and the fourth branch have gone through 1 CBS module operation, the second branch has continuously gone through 5 CBS module operations, the third branch has continuously gone through 3 CBS module operations, and then the outputs of these 4 branches are passed through the Concat module operation, that is, the features output by these 4 branches are fused in the channel dimension, and finally passed through one more CBS module, so that the operation of the ELAN module is completed. As

[0071] As Figure 2As shown, the first SIMAM combination further includes a SIMAM module. The feature map after passing through the CBS combination module enters the SIMAM module after passing through the ELAN module. The SIMAM module is used to incorporate a three-dimensional attention mechanism during feature extraction to improve the feature extraction ability of the multi-scale object detection model network. The SIMAM modules in the four SIMAM combination modules all input the extracted features to the Neck network.

[0072] Inspired by neuroscience, the SIMAM module proposes that neurons containing more information often behave differently from surrounding neurons. Therefore, an energy function e is proposed to measure the importance of each neuron. The smaller the value of the energy function e, the more important the information contained in the neuron, and the higher the weight it should occupy.

[0073] According to research in visual neuroscience, the most informative neurons are usually those with activation patterns that are significantly different from surrounding neurons. In addition, an active neuron can also inhibit the activity of surrounding neurons, a phenomenon known as spatial inhibition. To calculate the importance of each neuron, the following energy function is defined:

[0074]

[0075] where e t represents the neuron energy function, and are linear transformations of t and x i , w t and b t refer to the weight and bias during the transformation of a certain neuron respectively. t and x i refer to the target neuron and other neurons in a single channel of the input feature X respectively. i is the index in the spatial dimension, and M is the number of neurons in a channel. y represents a constant value, and y t and y o are two different constant values used to simplify the calculation. By minimizing this energy function, the linear separability between the target neuron t and other neurons in the same channel can be found. To simplify the calculation, a regularization term is introduced, and the final energy function is as follows:

[0076]

[0077] Theoretically, for each channel, M energy functions need to be solved. However, through some assumptions, the mean and variance of all neurons in the channel can be calculated and reused, thus avoiding the computational cost of iteratively calculating the mean and variance for each position. Therefore, the minimum energy can be calculated by the following formula:

[0078]

[0079] Among them, and are the mean and variance calculated on all neurons in the channel respectively, representing the minimum value of the energy function. The lower the energy value, the greater the difference between neuron t and its surrounding neurons, and the higher the importance. According to the definition of the attention mechanism, the SIMAM module can be represented by the following formula:

[0080]

[0081] Among them, is the output of the SIMAM module, X is the input of the SIMAM module, and E groups all in the channel and spatial dimensions. The sigmoid function is used to limit too large values in e.

[0082] As Figure 2 shown, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128, and the size of the feature map after passing through the SIMAM module in the first SIMAM combination is 160*160*256. The feature map after passing through the SIMAM module in the first SIMAM combination has two output directions. As Figure 2 shown, one is to input this feature map into the Neck network, and the other is to input this feature map into the second SIMAM combination.

[0083] Continuing to refer to Figure 2 , the second SIMAM combination module sequentially includes an MPConv module, an ELAN module, and a SIMAM module for the second feature extraction. The features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network. As Figure 2 shown, compared with the first SIMAM combination module, the second SIMAM combination module has an additional MPConv module.

[0084] Among them, the MPConv module is used to perform pooling on the feature map. Please refer to Figure 5 , Figure 5 is Figure 2 the structure diagram of the MPConv module in Figure 5As shown in the figure, the MPConv module includes two branches: the first branch first passes through a max pooling layer Maxpoolk2 (k2 indicates that the pooling kernel size is (2*2) and the stride is 2), and then passes through a 1*1 convolution module to adjust the output feature dimension; the second branch first passes through a 1*1 convolution module to adjust the feature dimension, and then passes through a 3*3 convolution module with a stride of 2 to achieve downsampling; finally, the results of the first branch and the second branch are concatenated to obtain the final output of the MPConv module.

[0085] The features first extracted by the first SIMAM combination module will sequentially pass through the MPConv module, ELAN module, and SIMAM module in the second SIMAM combination module. Since the structures of the ELAN module and SIMAM module have been described in the relevant paragraphs of the first SIMAM combination module, for the sake of brevity of the specification, they will not be elaborated here.

[0086] As Figure 2 shown, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128, the size of the feature map after passing through the SIMAM module in the first SIMAM combination is 160*160*256, and the size of the feature map after passing through the SIMAM module in the second SIMAM combination is 80*80*512. The feature map after passing through the SIMAM module in the second SIMAM combination has two output directions, as Figure 2 shown, one is to input this feature map into the Neck network, and the other is to input this feature map into the third SIMAM combination.

[0087] Continue to refer to Figure 2 , the third SIMAM combination module sequentially includes an MPConv module, an ELAN module, and a SIMAM module for the third feature extraction, and the features extracted for the third time by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network.

[0088] The features second extracted by the second SIMAM combination module will sequentially pass through the MPConv module, ELAN module, and SIMAM module in the third SIMAM combination module. Since the structures of the MPConv module, ELAN module, and SIMAM module have been described in the relevant paragraphs of the first SIMAM combination module and the second SIMAM combination module, for the sake of brevity of the specification, they will not be elaborated here.

[0089] As Figure 2As shown, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128. After passing through the SIMAM module in the first SIMAM combination, the size of the feature map is 160*160*256. After passing through the SIMAM module in the second SIMAM combination, the size of the feature map is 80*80*512. After passing through the SIMAM module in the third SIMAM combination, the size of the feature map is 40*40*1024. The feature map after passing through the SIMAM module in the third SIMAM combination has two output directions, as Figure 2 shown, one is to input this feature map into the Neck network, and the other is to input this feature map into the fourth SIMAM combination.

[0090] Continue to refer to Figure 2 , the fourth SIMAM combination module sequentially includes an MPConv module, an ELAN module, an SPPCSPC module, and a SIMAM module to extract features for the fourth time and input the extracted features into the Neck network. The features extracted by the third SIMAM combination module for the third time will sequentially pass through the MPConv module, ELAN module, SPPCSPC module, and SIMAM module in the fourth SIMAM combination module. Since the structures of the MPConv module, ELAN module, and SIMAM module have been described in the relevant paragraphs of the first SIMAM combination module and the second SIMAM combination module, for the sake of simplicity of the specification, they will not be elaborated again.

[0091] Among them, the SPPCSPC module is used to extract features after being manifested in different sizes. Please refer to Figure 6 , Figure 6 for Figure 2 the structure diagram of the SPPCSPC module in Figure 6 As shown, the SPPCSPC module consists of 7 CBS modules (the number of output feature maps of each CBS module is 512, only the two-dimensional convolution kernel size and convolution stride are different), and 3 max pooling layers (Maxpool k5, Maxpool k9, Maxpool k13). The stride of the 3 max pooling layers is 1. The pooling kernel size of the max pooling layer Maxpool k5 is 5*5, the pooling kernel size of the max pooling layer Maxpool k95 is 9*9, and the pooling kernel size of the max pooling layer Maxpool k13 is 13*13.

[0092] As Figure 6As shown, the SPPCSPC module divides the input into two branches. One branch performs feature extraction, then passes through a feature pyramid, performs max pooling on features of different sizes, reveals the extracted feature information, and concatenates the result with the feature map that has not passed through the feature pyramid pooling (i.e., the other branch). Through this operation, the network can extract features of different sizes after manifestation without losing the information of the original feature map, so as to achieve the effect of improving the detection accuracy.

[0093] As Figure 2 shown, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128. After passing through the SIMAM module in the first SIMAM combination, the size of the feature map is 160*160*256. After passing through the SIMAM module in the second SIMAM combination, the size of the feature map is 80*80*512. After passing through the SIMAM module in the third SIMAM combination, the size of the feature map is 40*40*1024. After passing through the SIMAM module in the fourth SIMAM combination, the size of the feature map is 20*20*512. The feature map after passing through the SIMAM module in the fourth SIMAM combination is input into the Neck network.

[0094] Continue to refer to Figure 2 , the four groups of feature maps P1 - P4 output by the BackBone network will respectively enter the Neck network to fuse the top - layer and bottom - layer features from top to bottom for feature fusion, obtain four groups of fused feature results, and input them into the Head network. As Figure 2 shown, the Neck network includes 2 first CBS modules, 2 second CBS modules, 2 third CBS modules, 1 fourth CBS module, 3 up - sampling modules, 6 Concat modules, 6 Concat - Conv modules, and 3 MPConv modules.

[0095] Among them, the first CBS module, the second CBS module, the third CBS module, and the fourth CBS module are four different CBS modules. The first CBS module is a two - dimensional convolution with a stride of 1 and a kernel size of 1X1, and the number of output feature maps is 64; the second CBS module is a two - dimensional convolution with a stride of 1 and a kernel size of 1X1, and the number of output feature maps is 128; the third CBS module is a two - dimensional convolution with a stride of 1 and a kernel size of 1X1, and the number of output feature maps is 256; the fourth CBS module is a two - dimensional convolution with a stride of 1 and a kernel size of 1X1, and the number of output feature maps is 512.

[0096] Among them, the Concat module is used for feature fusion in the channel dimension. The Concat-Conv module is used for multi-scale feature extraction and fusion of images. Please refer to Figure 7 , Figure 7 is Figure 2 the structural diagram of the Concat-Conv module in Figure 7 As shown, the Concat-Conv module aggregates features at 6 different levels to better capture multi-scale information. The Concat-Conv module includes 6 branches. The 1st branch and the 6th branch both go through 1 CBS module operation. The 2nd branch continuously goes through 5 CBS module operations. The 3rd branch continuously goes through 4 CBS module operations. The 4th branch continuously goes through 3 CBS module operations. The 5th branch continuously goes through 2 CBS module operations. Then, the outputs of these 6 branches are passed through the Concat module operation, that is, the features output by these 6 branches are fused in the channel dimension. Finally, it goes through one more CBS module, thus completing the operation of the Concat-Conv module.

[0097] As Figure 2 shown, the Head network includes the first size detection head, the second size detection head, the third size detection head, and the fourth size detection head; among them, the first size > the second size > the third size > the fourth size. The features extracted by the fourth SIMAM combination module are successively passed through the fourth CBS module, the first third CBS module, the first upsampling module, the first Concat module, the first Concat-Conv module, the second CBS module, the second upsampling module, the second Concat module, the second Concat-Conv module, the first CBS module, the third upsampling module, the third Concat module, and the third Concat-Conv module; the features extracted by the third SIMAM combination module are passed through the second third CBS module and the first Concat module; the features extracted by the second SIMAM combination module are passed through the second second CBS module and the second Concat module; the features extracted by the first SIMAM combination module are passed through the second first CBS module and the third Concat module; the output of the third Concat-Conv module is used as the input of the first size detection head.

[0098] The output of the third Concat-Conv module also sequentially passes through the first MPConv module, the fourth Concat module, the fourth Concat-Conv module, the second MPConv module, the fifth Concat module, the fifth Concat-Conv module, the third MPConv module, the sixth Concat module, and the sixth Concat-Conv module; the output of the second Concat-Conv module also serves as the input to the fourth Concat module; the output of the first Concat-Conv module also serves as the input to the fifth Concat module; the output of the fourth CBS module also serves as the input to the sixth Concat module.

[0099] The output of the fourth Concat-Conv module also serves as the input to the second size detection head; the output of the fifth Concat-Conv module also serves as the input to the third size detection head; the output of the sixth Concat-Conv module serves as the input to the fourth size detection head.

[0100] The Head network also includes four RepConv modules. The RepConv module is a reparameterized convolution module, which is used to reparameterize the network to reduce the model's computational complexity and improve the model's performance. Among them, the output of the third Concat-Conv module serves as the input to the first RepConv module, and the output of the first RepConv module serves as the input to the first size detection head; the output of the fourth Concat-Conv module serves as the input to the second RepConv module, and the output of the second RepConv module serves as the input to the second size detection head; the output of the fifth Concat-Conv module serves as the input to the third RepConv module, and the output of the third RepConv module serves as the input to the third size detection head; the output of the sixth Concat-Conv module serves as the input to the fourth RepConv module, and the output of the fourth RepConv module serves as the input to the fourth size detection head.

[0101] Optionally, the loss function of the multi-size object detection model improved based on YOLOV7 is a hybrid bounding box regression loss function based on the CIOU loss function and the normalized Gaussian Wasserstein distance.

[0102] In the prior art, YOLOV7 uses CIOU as the bounding box regression loss function. However, a single CIOU is too sensitive to the pixel position changes of tiny objects, which will lead to a sharp decline in the detection performance of the model for tiny objects. In contrast, the Normalized Wasserstein Distance (NWD) is smoother with respect to the changes in position offsets, and NWD can measure the similarity between non-overlapping or overlapping tiny bounding boxes.

[0103] Model the bounding box as a Gaussian distribution as follows: To determine a bounding box, it is necessary to know its center point coordinates (c x , c y ), width (w), and height (h). Any horizontal bounding box R = (c x , c y , w, h), and the equation of its inscribed ellipse can be expressed as:

[0104]

[0105] The probability density function of the two-dimensional Gaussian distribution is as follows:

[0106]

[0107] where x is the coordinate of the two-dimensional Gaussian distribution, is the covariance matrix, is the mean vector. When , the inscribed ellipse of any bounding box R is the density contour of the two-dimensional Gaussian distribution. Therefore, the bounding box R = (c x , c y , w, h) can be modeled as a Gaussian distribution . Thus far, the similarity between two bounding boxes can be converted into the distribution distance between the Gaussian distributions corresponding to the two bounding boxes. Use the normalized wasserstein distance as the distance between the two Gaussian distributions. In an actual detection task, the Gaussian distributions corresponding to the predicted box P and the ground truth box T are . Define the wasserstein distance between these two Gaussian distributions as follows:

[0108]

[0109] When the covariance matrix satisfies the transformation: , substitute , into it, and the following formula is obtained:

[0110]

[0111] Since the Wasserstein distance is a distance metric and cannot be directly used to measure the similarity between bounding boxes, it will cause the loss function to be difficult to converge. Therefore, we perform a normalization process on the Wasserstein distance:

[0112]

[0113] where C is a constant closely related to the training dataset. For example, in the apple detection task, the empirical value 12.8 can be taken. After obtaining the NWD, multiply it by the weight factor λ and then add it to CIOU to obtain the new bounding box regression loss function:

[0114]

[0115] If the proportion of small targets in the task is relatively large, the weight λ of NWD can be increased accordingly. For example, in the apple detection task, λ can be taken as 0.5.

[0116] As Figure 2 shown, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128. After passing through the SIMAM module in the first SIMAM combination, the size of the feature map is 160*160*256. After passing through the SIMAM module in the second SIMAM combination, the size of the feature map is 80*80*512. After passing through the SIMAM module in the third SIMAM combination, the size of the feature map is 40*40*1024. After passing through the SIMAM module in the fourth SIMAM combination, the size of the feature map is 20*20*512. The feature map after passing through the SIMAM module in the fourth SIMAM combination is input to the Neck network. The Neck network fuses the top-layer and bottom-layer features from top to bottom, obtains four sets of fused feature results and inputs them to the Head network. Among them, the size of the feature map input by the Neck network to the first size detection head is 160*160*64, and the size of the feature map output by the first size detection head is 160*160; the size of the feature map input by the Neck network to the second size detection head is 80*80*128, and the size of the feature map output by the second size detection head is 80*80; the size of the feature map input by the Neck network to the third size detection head is 40*40*256, and the size of the feature map output by the third size detection head is 40*40; the size of the feature map input by the Neck network to the fourth size detection head is 20*20*512, and the size of the feature map output by the fourth size detection head is 40*40.

[0117] After completing the construction of the multi-scale object detection model improved based on YOLOV7, an image data training set and a validation set are established. The image data training set is input into the multi-scale object detection model improved based on YOLOV7 for training, and the validation set is detected to output the multi-scale object detection results. Then, the video stream data is input into the trained multi-scale object detection model, and the multi-scale object detection model will detect the objects that meet the four sizes in each frame of the image, and finally output the detection box information of the objects frame by frame.

[0118] To verify the influence of each improved part on the performance of the multi-scale object detection model improved based on YOLOV7, the original YOLOV7 is used as the benchmark model, and the improved parts are added in turn, and ablation experiments are carried out on the Apple (self-made apple data set) and Drone (public data set) data sets respectively. The results are shown in Table 1. Among them, Method A means that the SimAM attention mechanism is incorporated in the feature extraction stage, Method B means adding an additional small object detection head, and Method C means adding the NWD loss on the basis of the original bounding box loss of the network. It can be seen from Table 1 that the methods proposed in this paper all improve the detection ability of the model. Especially, Method C improves the detection ability of the model without adding any network parameters. Therefore, it fully proves the effectiveness of the NWD loss for small object detection. For the improved network, on the Apple data set, mAP@0.5 is improved by 1.61%, and on the Drone data set, map@0.5 is improved by 3.95%, and the improvement effect is obvious.

[0119] Table 1

[0120]

[0121] In this application, corresponding improvements are made to the Backbone network, Neck network and Head network of the YOLOV7 model according to the characteristics of small and dense detection targets; the Backbone network adds the extraction of small-size object features, that is, four-scale feature maps are output, and the three-dimensional attention mechanism SimAM is added during the extraction process of the four scales to improve the feature extraction ability of the network, and output branches are added to the Neck network and Head network, solving the technical problem that the YOLOV7 detection network in the prior art has poor detection effect on small objects with less pixel information.

[0122] After the object detector outputs the detection box information of the object frame by frame, step S12 is executed.

[0123] S12, assign an object ID to the object corresponding to each object detection box in the t-th frame.

[0124] Among them, t is an integer greater than 0. That is to say, whenever the target detector outputs the detection box information of the target in a frame of image according to the video stream order, a target ID is assigned to the target corresponding to each target detection box.

[0125] S13. Predict the predicted box information of the target in the (t + 1)-th frame according to the detection box information of the target in the t-th frame.

[0126] Among them, the predicted box information includes the width, height of the predicted box and the coordinates of the center point of the predicted box. In one embodiment, step S13 may include the following steps:

[0127] S131. Calculate the state matrix of the target according to the width, height of the detection box and the coordinates of the center point of the detection box of the target in the t-th frame.

[0128] Among them, the state matrix includes the coordinates of the center point of the detection box of the target in the t-th frame, the width-to-height ratio of the detection box, the height, and the change rates of the center point coordinates, width-to-height ratio, and height.

[0129] According to the detection box information (c x , c y , w, h) of the target in the t-th frame, calculate the state matrix [c x , c y , r, h, v x , v y , v r , v h . Among them, (c x , c y ) is the center point of the detection box, r is the width-to-height ratio w / h of the detection box, h is the height, and v x , v y , v r , v h are the change rates of these four values. For example, in the first frame of the image, the detector detects three apples, and first assigns them IDs 1, 2, and 3, and then calculates the state matrices of these three apples.

[0130] S132. Calculate the predicted box information of the target in the (t + 1)-th frame through Kalman filtering according to the state matrix of the target.

[0131] According to the target state matrix obtained in step S131, through Kalman filtering, the estimated position of the target in the next frame of the image, that is, the predicted box, can be calculated. For example, in the first frame of the image, three apples are detected. According to the state matrices of the three apples obtained in step S131, through Kalman filtering, the estimated positions of the three apples in the second frame of the image, that is, the predicted box information of the three apples in the second frame of the image, can be calculated.

[0132] After obtaining the predicted box information of the target in the (t + 1)-th frame, execute step S14.

[0133] S14. Match the predicted bounding boxes and detected bounding boxes of the (t + 1)-th frame according to the detected bounding box information and predicted bounding box information of the (t + 1)-th frame using IOU.

[0134] In computer vision, IOU (Intersection Over Union) is a metric used to measure the matching degree between predicted bounding boxes and actual bounding boxes. IOU is also known as the intersection-over-union ratio or overlap ratio. When evaluating a model in object detection or image segmentation tasks, IOU is one of the very important evaluation metrics. In object detection tasks, IOU is usually used to measure the matching degree between predicted bounding boxes and actual objects (detected bounding boxes). If the IOU value of a predicted bounding box is higher than a certain threshold (such as 0.5), then the predicted box is considered correct.

[0135] In step S14, the detected bounding box information of the (t + 1)-th frame is output after the object detector (a multi-scale object detection model improved based on YOLOV7) detects the (t + 1)-th frame image; the predicted bounding box information of the (t + 1)-th frame is obtained from step S13.

[0136] It should be noted that the predicted bounding boxes that fail the IOU match are directly deleted.

[0137] S15. Assign the target corresponding to the detected bounding box that successfully matches using IOU in the (t + 1)-th frame to the target ID corresponding to the t-th frame.

[0138] For example, in the first frame image, the object detector detects three apples. First, assign IDs 1, 2, and 3 to them; then, calculate the state matrices of these three apples; next, through Kalman filtering, the estimated positions of the three apples in the second frame image can be calculated, that is, three predicted bounding boxes are output; suppose the detector detects four apples in the second frame image and outputs four detected bounding boxes at the same time; then, perform IOU matching between the four detected bounding boxes and the three predicted bounding boxes. If the three predicted bounding boxes and three detected bounding boxes successfully match using IOU, that is, three apples in the second frame image match successfully with the three apples in the first frame image, then the predicted bounding boxes corresponding to the three apples in the first frame image are used as their positions in the second frame, and assign the same IDs as in the first frame (1, 2, 3) to them again. For the fourth apple detected in the second frame image, since it appears for the first time and there is no predicted bounding box, there is no need to perform IOU matching and a new ID 4 is directly assigned.

[0139] S16. When the predicted bounding box and detected bounding box of a target are continuously matched using IOU for N frames, cascade-match the target in each frame after the N-th frame with the target after continuous IOU matching for N frames.

[0140] Among them, N is an integer greater than 1. Cascade matching is actually a feature matching. The main idea is to use the ReID network to extract the appearance feature information and position information of the tracked target and the detected target in the current frame, then calculate the cosine distance of the appearance feature information and the Mahalanobis distance of the position information to construct a cost matrix, and finally complete the matching of the tracked target and the detected target through the Hungarian matching algorithm.

[0141] Optionally, step S16 may include the following steps:

[0142] S161, select one frame from N consecutive frames to use the ReID network to extract the first appearance feature information of the target after IOU matching in this frame, and obtain the first position information of the target in this frame through the detection box information of the target.

[0143] S162, in the current frame after N frames, use the ReID network to extract the second appearance feature information of all targets in the current frame, and obtain the second position information of all targets in the current frame through the prediction box information.

[0144] S163, calculate a cost matrix according to the first appearance feature information and the second appearance feature information, and the first position information and the second position information.

[0145] Further, step S163 may include the following steps:

[0146] S1631, calculate the cosine distance of the appearance feature information according to the first appearance feature information and the second appearance feature information.

[0147] S1632, calculate the Mahalanobis distance of the position information according to the first position information and the second position information.

[0148] S1633, construct a cost matrix according to the cosine distance of the appearance feature information and the Mahalanobis distance of the position information.

[0149] After completing the construction of the cost matrix, execute step S164.

[0150] S164, complete the cascade matching through the Hungarian matching algorithm according to the cost matrix.

[0151] The appearance feature information is extracted by a pre-trained ReID network. The appearance feature information is actually a 128-dimensional feature vector. The appearance feature information refers to the appearance information of the target, such as appearance features such as size, shape, and color.

[0152] It should be noted that when multiple targets complete consecutive N-frame IOU matching, the multiple targets are sorted according to the number of frames of consecutive IOU matching to prioritize them, so that the targets in each frame after the Nth frame are preferentially cascade-matched with the targets with higher priority; among them, the more frames of consecutive IOU matching, the higher the priority.

[0153] For example, there are 3 apples in the 58th frame. Among them, apple 1 has been tracked in each of the previous 57 frames, apple 2 has not been tracked for two frames, and apple 3 has not been tracked for five frames. Then the 3 apples detected by the target detector will first perform cascade matching with apple 1, and then the remaining two apples will perform cascade matching with apple 2. Cascade matching is actually a priority matching mechanism that will preferentially match the targets with fewer frames of tracking loss.

[0154] After completing the cascade matching, step S17 is executed.

[0155] S17. Count the target IDs successfully cascade-matched as the detection result.

[0156] When using the DeepSort algorithm model for multi-target tracking, the problem of ID jumping is likely to occur. The main reason is that targets that are not successfully matched are also assigned IDs. If only the ID numbers are used as the number of detected apples, the error will be relatively large. Therefore, an additional statistical step is added to only include the target IDs successfully cascade-matched in the statistical quantity as the final detection result. During this process, the ReID network can re-identify the target. Even if the target is lost in some frames of images, it can be re-identified by the ReID network in subsequent images, solving the problem of the same target being double-counted.

[0157] This application counts the target video (rather than static images), solving the problems of single perspective and limited field of view due to shooting static images; in addition, this application assigns the same ID to two targets with successful IOU matching in different images, and performs cascade matching after consecutive multi-frame IOU matching. Only the target IDs successfully cascade-matched are counted in the detection result, reducing the impact of ID jumping on the calculation accuracy and solving the problem that the targets in the same area appear in different images and are double-counted, directly affecting the counting accuracy.

[0158] According to the second specific embodiment of the present invention, the present invention provides a target dynamic tracking system based on DeepSORT, adopting the method of the first specific embodiment, as Figure 8 shown, the target dynamic tracking system 400 based on DeepSORT includes:

[0159] A detection module 410, configured to input video stream data into a target detector so that the target detector outputs detection box information of a target frame by frame in sequence;

[0160] A first ID assignment module 420, configured to assign a target ID to a target corresponding to each target detection box in the t-th frame; where t is an integer greater than 0;

[0161] A prediction module 430, configured to predict predicted box information of a target in the (t + 1)-th frame according to the detection box information of the target in the t-th frame;

[0162] An IOU matching module 440, configured to perform IOU matching between the predicted boxes and the detection boxes in the (t + 1)-th frame according to the detection box information and the predicted box information in the (t + 1)-th frame;

[0163] A second ID assignment module 450, configured to assign the target ID corresponding to the t-th frame to the target corresponding to the detection box with successful IOU matching in the (t + 1)-th frame;

[0164] A cascade matching module 460, configured to, when the predicted box and the detection box of a target are continuously IOU-matched for N frames, perform cascade matching between the target in each frame after the N-th frame and the target after continuous IOU matching for N frames; where N is an integer greater than 1;

[0165] A statistics module 470, configured to count the target IDs with successful cascade matching as detection results.

[0166] According to the third specific embodiment of the present invention, the present invention provides an electronic device, as Figure 9 shown, Figure 9 is a block diagram of an electronic device shown according to an exemplary embodiment.

[0167] Next, refer to Figure 9 to describe the electronic device 200 according to this embodiment of the present application. Figure 9 The shown electronic device 200 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0168] As Figure 9 shown, the electronic device 200 is presented in the form of a general computing device. The components of the electronic device 200 may include but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0169] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 210, so that the processing unit 210 executes the steps according to various exemplary embodiments of the present application described in this specification. For example, the processing unit 210 can execute as Figure 1 the steps shown therein.

[0170] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 2201 and / or a cache storage unit 2202, and may further include a read-only storage unit (ROM) 2203.

[0171] The storage unit 220 may also include a program / utilities 2204 having a set (at least one) of program modules 2205. Such program modules 2205 include but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include the implementation of a network environment.

[0172] The bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0173] The electronic device 200 may also communicate with one or more external devices 200' (such as a keyboard, a pointing device, a Bluetooth device, etc.), so that the device that enables a user to interact with the electronic device 200 communicates, and / or the electronic device 200 can communicate with any device that can communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 250. And, the electronic device 200 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 260. The network adapter 260 can communicate with other modules of the electronic device 200 through the bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0174] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by the way of software combined with necessary hardware.

[0175] Therefore, according to the fourth specific embodiment of the present invention, the present invention provides a computer-readable medium. AsFigure 10 As shown, the technical solution according to an embodiment of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a portable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to an embodiment of the present invention.

[0176] The software product can adopt any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0177] The computer-readable storage medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium can also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted by any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the above.

[0178] The program code for performing the operations of the present invention can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages - such as Java, C++, etc., and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0179] The above computer-readable medium carries one or more programs which, when executed by the device, cause the computer-readable medium to implement the functions of the first embodiment.

[0180] Those skilled in the art can understand that the above-mentioned modules can be distributed in the device according to the description of the embodiments, or can be correspondingly changed and distributed in one or more devices that are uniquely different from this embodiment. The modules of the above embodiments can be combined into one module, or further split into multiple sub-modules.

[0181] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present invention.

[0182] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A target dynamic tracking method based on DeepSORT, characterized in that, Including the following steps: Input the video stream data into the target detector so that the target detector outputs the detection box information of the target frame by frame; wherein, the target detector is a multi-size target detection model improved based on YOLOV7 after training, and the multi-size target detection model includes a Backbone network, a Neck network, and a Head network; the Backbone network is used to perform four times of feature extraction on the feature map input into the multi-size target detection model, the Neck network is used to fuse the features extracted four times by the Backbone network to form four fusion branches, and input the four fusion branches into the Head network to detect four different-size targets, and the Head network outputs the detection results of the four different-size targets; the feature map sizes of the four different-size targets are 160*160, 80*80, 40*40, and 20*20 respectively; Assign a target ID to the target corresponding to each target detection box in the t-th frame; wherein, t is an integer greater than 0; Predict the predicted box information of the target in the (t + 1)-th frame according to the detection box information of the target in the t-th frame; According to the detection box information in the (t + 1)-th frame and the predicted box information in the (t + 1)-th frame, perform IOU matching on the predicted box and the detection box in the (t + 1)-th frame and delete the predicted boxes with failed IOU matching; Assign the target ID corresponding to the t-th frame to the target corresponding to the detection box with successful IOU matching in the (t + 1)-th frame. For the target that appears for the first time, no IOU matching is performed, and a new ID is directly assigned; When the predicted box and the detection box of a target are continuously IOU-matched for N frames, perform cascade matching on the target in each frame after the N-th frame and the target after continuous N-frame IOU matching; wherein, N is an integer greater than 1; Count the target IDs with successful cascade matching as the detection result; Among them, the performing cascade matching on the target in each frame after the N-th frame and the target after continuous N-frame IOU matching includes: selecting one frame from the continuous N frames to extract the first appearance feature information of the target after IOU matching in this frame by using the ReID network, and obtaining the first position information of the target in this frame through the detection box information of the target; in the current frame after the N-th frame, extract the second appearance feature information of all targets in the current frame by using the ReID network, and obtain the second position information of all targets in the current frame through the predicted box information; calculate the cost matrix according to the first appearance feature information and the second appearance feature information, and the first position information and the second position information; complete the cascade matching through the Hungarian matching algorithm according to the cost matrix; Among them, the Backbone network sequentially includes a CBS combination module, a first SIMAM combination module, a second SIMAM combination module, a third SIMAM combination module, and a fourth SIMAM combination module; the first SIMAM combination module sequentially includes an ELAN module and a SIMAM module for extracting features for the first time, and the features extracted by the first SIMAM combination module for the first time are respectively input into the second SIMAM combination module and the Neck network; the second SIMAM combination module sequentially includes an MPConv module, an ELAN module, and a SIMAM module for extracting features for the second time, and the features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network; the third SIMAM combination module sequentially includes an MPConv module, an ELAN module, and a SIMAM module for extracting features for the third time, and the features extracted by the third SIMAM combination module for the third time are respectively input into the fourth SIMAM combination module and the Neck network; the fourth SIMAM combination module sequentially includes an MPConv module, an ELAN module, an SPPCSPC module, and a SIMAM module for extracting features for the fourth time and inputting the extracted features into the Neck network; Among them, the CBS combination module includes four different CBS modules, and the CBS module is used to change the size and number of feature maps; the ELAN module is an efficient layer aggregation network for multi-scale feature extraction and fusion of images; the SIMAM module is used to incorporate a three-dimensional attention mechanism during feature extraction to improve the feature extraction ability of the multi-size object detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features into the Neck network; the MPConv module is used to perform pooling on the feature map; the SPPCSPC module is used to extract features after being made explicit with different sizes.

2. The target dynamic tracking method based on DeepSORT according to claim 1, wherein The detection box information includes the width, height, and the coordinates of the center point of the detection box, and the prediction box information includes the width, height, and the coordinates of the center point of the prediction box; predicting the prediction box information of the target in the (t + 1)th frame based on the detection box information of the target in the tth frame includes: Based on the width, height, and the coordinates of the center point of the detection box of the target in the tth frame, calculate the state matrix of the target; among them, the state matrix includes the coordinates of the center point of the detection box of the target in the tth frame, the aspect ratio of the width to the height of the detection box, the height, and the change rates of the center point coordinates, the aspect ratio, and the height. Based on the state matrix of the target, calculate the prediction box information of the target in the (t + 1)th frame through Kalman filtering.

3. The object dynamic tracking method based on DeepSORT according to claim 1, characterized in that Calculating the cost matrix based on the first appearance feature information, the second appearance feature information, the first position information, and the second position information includes: Based on the first appearance feature information and the second appearance feature information, calculate the cosine distance of the appearance feature information. Based on the first position information and the second position information, calculate the Mahalanobis distance of the position information. Construct a cost matrix based on the cosine distance of the appearance feature information and the Mahalanobis distance of the position information.

4. The target dynamic tracking method based on DeepSORT according to claim 3, wherein, The first appearance feature information and the second appearance feature information at least both include the size, shape, and color of the target.

5. The target dynamic tracking method based on DeepSORT according to claim 1, characterized in that It also includes: When multiple targets complete consecutive N-frame IOU matching, sort the multiple targets according to the number of frames of consecutive IOU matching to prioritize the targets, so that the targets in each frame after N frames are preferentially cascade-matched with the targets with higher priorities; among them, the more frames of consecutive IOU matching, the higher the priority.

6. A target dynamic tracking system based on DeepSORT, characterized in that, It includes: A detection module, configured to input video stream data into a target detector so that the target detector sequentially outputs the detection box information of the target frame by frame; wherein, the target detector is a multi-size target detection model improved based on YOLOV7 after training, and the multi-size target detection model includes a Backbone network, a Neck network, and a Head network; the Backbone network is used to perform four times of feature extraction on the feature map input to the multi-size target detection model, the Neck network is used to fuse the features extracted four times by the Backbone network to form four fusion branches, and input the four fusion branches into the Head network to detect four different-size targets, and the Head network outputs the detection results of the four different-size targets; A first ID assignment module, configured to assign a target ID to the target corresponding to each target detection box in the t-th frame; wherein, t is an integer greater than 0; A prediction module, configured to predict the predicted box information of the target in the (t + 1)-th frame according to the detection box information of the target in the t-th frame; An IOU matching module, configured to perform IOU matching on the predicted boxes and detection boxes in the (t + 1)-th frame according to the detection box information and the predicted box information in the (t + 1)-th frame; A second ID assignment module, configured to assign the target ID corresponding to the t-th frame to the target corresponding to the detection box with successful IOU matching in the (t + 1)-th frame and delete the predicted boxes with failed IOU matching; A cascade matching module, configured to, when the predicted box and the detection box of a target are continuously IOU-matched for N frames, perform cascade matching on the targets in each frame after N frames and the target after continuous N-frame IOU matching; wherein, N is an integer greater than 1; A statistics module, configured to count the target IDs with successful cascade matching as the detection result; Among them, the cascade matching module is further configured to: select one frame from the consecutive N frames to extract the first appearance feature information of the target after IOU matching in this frame by using the ReID network, and obtain the first position information of the target in this frame through the detection box information of the target; in the current frame after N frames, extract the second appearance feature information of all targets in the current frame by using the ReID network, and obtain the second position information of all targets in the current frame through the predicted box information; calculate a cost matrix according to the first appearance feature information and the second appearance feature information, and the first position information and the second position information; and complete cascade matching through the Hungarian matching algorithm according to the cost matrix; Among them, the Backbone network sequentially includes a CBS combination module, a first SIMAM combination module, a second SIMAM combination module, a third SIMAM combination module, and a fourth SIMAM combination module; the first SIMAM combination module sequentially includes an ELAN module and a SIMAM module for first feature extraction, and the features first extracted by the first SIMAM combination module are respectively input into the second SIMAM combination module and the Neck network; the second SIMAM combination module sequentially includes an MPConv module, an ELAN module, and a SIMAM module for second feature extraction, and the features second extracted by the second SIMAM combination module are respectively input into the third SIMAM combination module and the Neck network; the third SIMAM combination module sequentially includes an MPConv module, an ELAN module, and a SIMAM module for third feature extraction, and the features third extracted by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network; the fourth SIMAM combination module sequentially includes an MPConv module, an ELAN module, an SPPCSPC module, and a SIMAM module for fourth feature extraction and inputting the extracted features into the Neck network; Among them, the CBS combination module includes four different CBS modules, and the CBS module is used to change the size and number of feature maps; the ELAN module is an efficient layer aggregation network for multi-scale feature extraction and fusion of images; the SIMAM module is used to incorporate a three-dimensional attention mechanism during feature extraction to improve the feature extraction ability of the multi-size object detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features into the Neck network; the MPConv module is used to perform pooling on the feature map; the SPPCSPC module is used to extract features after different sizes are made explicit.

7. An electronic device, characterized in that, Comprising: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-5.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the method according to any one of claims 1-5 is implemented.

Images