Multi-target tracking method, system and medium

Abstract

The invention relates to the technical field of computer vision, provides a multi-target tracking method and system and a medium, and solves the problem that in the prior art, targets in the same area appear in different images and are counted repeatedly, and the counting accuracy is directly influenced. The method comprises the following steps: inputting video stream data into a trained multi-size target detection model based on YOLOV7 improvement, so that the multi-size target detection model outputs detection frame information of a target in sequence according to frames; allocating a target ID to a target corresponding to each target detection frame in the t frame; predicting prediction frame information of the target in a (t + 1) frame; performing IOU matching on the prediction frame and the detection frame of the t + 1 frame; a target ID corresponding to the t frame is allocated to a target corresponding to the detection frame with successful IOU matching in the t + 1 frame; after continuous N frames of IOU matching of the prediction frame and the detection frame of a target are carried out, cascade matching is carried out on the target in each frame after N frames and the target after continuous N frames of IOU matching; and performing statistics on the target ID which is successfully cascaded and matched to serve as a detection result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-target tracking method, system and medium. Background Art

[0002] In recent years, the application of various intelligent devices combined with computer vision technology in modern agriculture has become increasingly popular in agricultural production and has become a research hotspot in the agricultural field. Specifically in apple production, deep learning target detection technology can be used to identify apples, predict yields, and identify pests and diseases through pictures or videos taken by cameras or drones. Apples can also be graded based on image information. These technologies have greatly promoted the intelligent development of apple production management. Among them, using target detection technology to accurately identify apples and then predict yields is of great significance to the operation and management of orchards, and can provide fruit farmers with information on orchard yields.

[0003] Traditional technologies used for crop yield estimation still have many problems to a certain extent, such as unclear image boundaries caused by sliding windows, complex target feature design, poor portability, and cumbersome manual design. Therefore, more and more deep learning-based target detection algorithms are applied to agricultural production. These algorithms can be roughly divided into single-stage and two-stage algorithms according to the algorithm implementation process. Classic two-stage target detection algorithms include R-CNN, Fast R-CNN, FasterR-CNN, etc. Compared with the two-stage algorithm, the single-stage algorithm has less redundant calculations, faster detection speed, and lower hardware performance requirements. Representative single-stage algorithms include YOLOV series, SSD, etc. The single-stage algorithm omits the step of generating candidate target areas and directly predicts the category or position of the target on the input image. The detection speed has been greatly improved, so it has been widely used in the agricultural field.

[0004] Most of the existing apple quantity detection methods are based on static images for experiments. However, in the natural environment, the density of apples on apple trees is high and there is a problem of apples blocking or overlapping each other. The static images taken have a single perspective and a limited field of view. In addition, during the shooting process, apples in the same area may appear in different images, resulting in repeated counting. It is difficult to automatically and accurately count apples, which directly affects the accuracy of yield estimation. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-target tracking method, system and medium for solving the technical problems in the prior art that the single viewing angle and limited field of view of shooting static images, as well as the repeated counting of targets in the same area in different images directly affect the counting accuracy.

[0006] To solve the above technical problem, as one aspect of the present invention, a multi-target tracking method is provided, comprising the following steps: Input the video stream data into the trained multi-size target detection model based on YOLOV7, so that the multi-size target detection model outputs the detection frame information of the target frame by frame; wherein the multi-size target detection model includes a Backbone network, a Neck network and a Head network; the Backbone network is used to extract features four times from the feature map input to the multi-size target detection model, the Neck network is used to fuse the features extracted four times by the Backbone network to form four fusion branches, and the four fusion branches are input into the Head network to detect four targets of different sizes, and the Head network outputs the detection results of four targets of different sizes; Assign a target ID to the target corresponding to each target detection box in t frames; where t is an integer greater than 0; According to the detection box information of the target in frame t, predict the prediction box information of the target in frame t+1; According to the detection box information of frame t+1 and the prediction box information of frame t+1, perform IOU matching between the prediction box and the detection box of frame t+1; The target corresponding to the detection box with successful IOU matching in frame t+1 is assigned the target ID corresponding to frame t; When the prediction box and detection box of an object have been matched by IOU for N consecutive frames, the object in each frame after N frames is cascade matched with the object after IOU matching for N consecutive frames; where N is an integer greater than 1; Count the target IDs that have successfully matched the cascade as the detection result; Wherein, the Backbone network includes a CBS combination module, a first SIMAM combination module, a second SIMAM combination module, a third SIMAM combination module and a fourth SIMAM combination module in sequence; The first SIMAM combination module includes an ELAN module and a SIMAM module in sequence for first feature extraction, and the features extracted by the first SIMAM combination module are respectively input into the second SIMAM combination module and the Neck network; The second SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for extracting features for the second time, and the features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network; The third SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for third feature extraction, and the features extracted for the third time by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network; The fourth SIMAM combination module includes an MPConv module, an ELAN module, a SPPCSPC module and a SIMAM module in sequence for extracting features for the fourth time and inputting the extracted features into the Neck network; Among them, the CBS combination module includes four different CBS modules, and the CBS module is used to change the size and number of feature maps; the ELAN module is an efficient layer aggregation network for multi-scale feature extraction and fusion of images; the SIMAM module is used to incorporate a three-dimensional attention mechanism when extracting features to improve the feature extraction capability of the multi-scale target detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features to the Neck network; the MPConv module is used to pool the feature maps; and the SPPCSPC module is used to extract features after being manifested in different sizes.

[0007] Furthermore, the loss function of the multi-scale object detection model is a hybrid border regression loss function based on the CIOU loss function and the normalized Gaussian Wasserstein distance.

[0008] Furthermore, the expression of the mixed border regression loss function lbox is: ; Among them, λ is the weight factor and NWD is the normalized Gaussian Wasserstein distance.

[0009] Further, the Neck network includes 2 first CBS modules, 2 second CBS modules, 2 third CBS modules, 1 fourth CBS module, 3 upsampling modules, 6 Concat modules, 6 Concat-Conv modules and 3 MPConv modules; Among them, the first CBS module, the second CBS module, the third CBS module and the fourth CBS module are four different CBS modules; the Concat module is used to perform feature fusion in the channel dimension; and the Concat-Conv module is used to extract and fuse multi-scale features of the image.

[0010] Further, the Head network includes a first size detection head, a second size detection head, a third size detection head and a fourth size detection head; wherein the first size>the second size>the third size>the fourth size; The features extracted by the fourth SIMAM combination module are sequentially passed through the fourth CBS module, the first third CBS module, the first upsampling module, the first Concat module, the first Concat-Conv module, the second CBS module, the second upsampling module, the second Concat module, the second Concat-Conv module, the first CBS module, the third upsampling module, the third Concat module and the third Concat-Conv module; the features extracted by the third SIMAM combination module are passed through the second third CBS module and the first Concat module; the features extracted by the second SIMAM combination module are passed through the second second CBS module and the second Concat module; the features extracted by the first SIMAM combination module are passed through the second first CBS module and the third Concat module; the output of the third Concat-Conv module is used as the input of the first size detection head; The output of the third Concat-Conv module is also sequentially passed through the first MPConv module, the fourth Concat module, the fourth Concat-Conv module, the second MPConv module, the fifth Concat module, the fifth Concat-Conv module, the third MPConv module, the sixth Concat module and the sixth Concat-Conv module; the output of the second Concat-Conv module is also used as the input of the fourth Concat module; the output of the first Concat-Conv module is also used as the input of the fifth Concat module; the output of the fourth CBS module is also used as the input of the sixth Concat module; The output of the fourth Concat-Conv module is also used as the input of the second-size detection head; the output of the fifth Concat-Conv module is also used as the input of the third-size detection head; the output of the sixth Concat-Conv module is used as the input of the fourth-size detection head.

[0011] Furthermore, the Head network also includes four RepConv modules, which are re-parameterized convolution modules for re-parameterizing the Head network; wherein, the output of the third Concat-Conv module is used as the input of the first RepConv module, and the output of the first RepConv module is used as the input of the first size detection head; the output of the fourth Concat-Conv module is used as the input of the second RepConv module, and the output of the second RepConv module is used as the input of the second size detection head; the output of the fifth Concat-Conv module is used as the input of the third RepConv module, and the output of the third RepConv module is used as the input of the third size detection head; the output of the sixth Concat-Conv module is used as the input of the fourth RepConv module, and the output of the fourth RepConv module is used as the input of the fourth size detection head.

[0012] Further, the CBS combination module includes a fifth CBS module, a sixth CBS module, a seventh CBS module and an eighth CBS module; the feature map input to the multi-size target detection model passes through the fifth CBS module, the sixth CBS module, the seventh CBS module and the eighth CBS module in sequence; Among them, the first CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of output feature maps is 64; the second CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of output feature maps is 128; the third CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of output feature maps is 256; the fourth CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of output feature maps is 5 12; the fifth CBS module is a two-dimensional convolution with a stride of 1 and a convolution kernel size of 3X3, and the number of output feature maps is 32; the sixth CBS module is a two-dimensional convolution with a stride of 2 and a convolution kernel size of 3X3, and the number of output feature maps is 64; the seventh CBS module is a two-dimensional convolution with a stride of 1 and a convolution kernel size of 3X3, and the number of output feature maps is 64; the eighth CBS module is a two-dimensional convolution with a stride of 2 and a convolution kernel size of 3X3, and the number of output feature maps is 128.

[0013] Further, the detection box information includes the width, height and coordinates of the center point of the detection box, and the prediction box information includes the width, height and coordinates of the center point of the prediction box; and predicting the prediction box information of the target in the t+1 frame based on the detection box information of the target in the t frame includes: Calculate the state matrix of the target according to the detection frame width, height and detection frame center coordinates of the target in frame t; wherein the state matrix includes the detection frame center coordinates of the target in frame t, the aspect ratio and height of the detection frame, and the change rate of the center coordinates, aspect ratio and height; According to the state matrix of the target, the predicted box information of the target in the t+1 frame is calculated through Kalman filtering.

[0014] Furthermore, the target in each frame after N frames is cascade matched with the target after IOU matching of consecutive N frames, including: Select a frame from the N consecutive frames to use the ReID network to extract the first appearance feature information of the target after IOU matching in the frame, and obtain the first position information of the target in the frame through the detection box information of the target; In the current frame after N frames, the ReID network is used to extract the second appearance feature information of all objects in the current frame, and the second position information of all objects in the current frame is obtained through the prediction box information; Calculating a cost matrix according to the first appearance feature information and the second appearance feature information, and the first position information and the second position information; According to the cost matrix, cascade matching is completed through the Hungarian matching algorithm.

[0015] Further, calculating a cost matrix according to the first appearance feature information and the second appearance feature information, and the first position information and the second position information includes: Calculating a cosine distance of appearance feature information according to the first appearance feature information and the second appearance feature information; Calculating the Mahalanobis distance of the position information according to the first position feature information and the second position feature information; A cost matrix is ​​constructed according to the cosine distance of the appearance feature information and the Mahalanobis distance of the position information.

[0016] Furthermore, the first appearance feature information and the second appearance feature information both include at least the size, shape and color of the target.

[0017] Furthermore, it also includes: when multiple targets complete N consecutive IOU matching, the multiple targets are prioritized according to the number of frames of consecutive IOU matching, so that the targets in each frame after N frames are preferentially cascade matched with targets with high priority; wherein, the more frames of consecutive IOU matching, the higher the priority.

[0018] Furthermore, it also includes: deleting the predicted box that fails IOU matching.

[0019] As a second aspect of the present invention, a multi-target tracking system is provided, comprising: A detection module, used for inputting video stream data into a multi-size target detection model improved based on YOLOV7 so that the multi-size target detection model outputs detection frame information of the target frame by frame; wherein the multi-size target detection model includes a Backbone network, a Neck network and a Head network; the Backbone network is used for performing four feature extractions on the feature map input into the multi-size target detection model, the Neck network is used for fusing the features extracted four times by the Backbone network to form four fusion branches, and the four fusion branches are input into the Head network to detect four targets of different sizes, and the Head network outputs the detection results of four targets of different sizes; A first ID assignment module is used to assign a target ID to a target corresponding to each target detection frame in t frames; wherein t is an integer greater than 0; A prediction module is used to predict the prediction box information of the target in frame t+1 based on the detection box information of the target in frame t; The IOU matching module is used to perform IOU matching between the prediction box and the detection box of the t+1 frame according to the detection box information of the t+1 frame and the prediction box information of the t+1 frame; The second ID assignment module is used to assign the target ID corresponding to the t frame to the target corresponding to the detection frame with successful IOU matching in the t+1 frame; The cascade matching module is used to perform cascade matching on the target in each frame after N frames after the prediction box and the detection box of a target have been matched with each other for N consecutive frames; wherein N is an integer greater than 1; The statistics module is used to count the target IDs that have been successfully matched in the cascade as the detection results; Wherein, the Backbone network includes a CBS combination module, a first SIMAM combination module, a second SIMAM combination module, a third SIMAM combination module and a fourth SIMAM combination module in sequence; The first SIMAM combination module includes an ELAN module and a SIMAM module in sequence for first feature extraction, and the features extracted by the first SIMAM combination module are respectively input into the second SIMAM combination module and the Neck network; The second SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for extracting features for the second time, and the features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network; The third SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for third feature extraction, and the features extracted for the third time by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network; The fourth SIMAM combination module includes an MPConv module, an ELAN module, a SPPCSPC module and a SIMAM module in sequence for extracting features for the fourth time and inputting the extracted features into the Neck network; Among them, the CBS combination module includes four different CBS modules, and the CBS module is used to change the size and number of feature maps; the ELAN module is an efficient layer aggregation network for multi-scale feature extraction and fusion of images; the SIMAM module is used to incorporate a three-dimensional attention mechanism when extracting features to improve the feature extraction capability of the multi-scale target detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features to the Neck network; the MPConv module is used to pool the feature maps; and the SPPCSPC module is used to extract features after being manifested in different sizes.

[0020] As a third aspect of the present invention, there is provided an electronic device, comprising: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect above.

[0021] As a fourth aspect of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.

[0022] The beneficial effects of the present invention are: 1) In view of the characteristics that the detection targets are small and dense, the present invention makes corresponding improvements to the Backbone network, Neck network and Head network of the YOLOV7 model; the Backbone network adds feature extraction of small-size targets, that is, outputs four scale feature maps, and adds a three-dimensional attention mechanism SimAM to the four scale feature extraction processes to improve the feature extraction ability of the network, and adds output branches to the Neck network and the Head network, solving the technical problem in the prior art that the YOLOV7 detection network has poor detection effect on small targets with little pixel information.

[0023] 2) The present invention counts target videos (rather than static images), which solves the problem of single viewing angle and limited field of view of static images. In addition, the present invention assigns the same ID to two targets with successful IOU matching in different images, and performs cascade matching after multiple consecutive frames of IOU matching. Only the target IDs with successful cascade matching are counted in the detection results, reducing the impact of ID jumps on calculation accuracy, and solving the problem that targets in the same area appear in different images and are repeatedly counted, which directly affects the counting accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A step diagram of a multi-target tracking method is schematically shown.

[0025] Figure 2 The structural diagram of the multi-scale target model improved based on YOLOV7 is schematically shown.

[0026] Figure 3 for Figure 2 CBS module structure diagram.

[0027] Figure 4 for Figure 2 ELAN module structure diagram.

[0028] Figure 5 for Figure 2 Figure 2. The MPConv module structure diagram.

[0029] Figure 6 for Figure 2 SPC module structure diagram.

[0030] Figure 7 for Figure 2 Concat-Conv module structure diagram.

[0031] Figure 8 A block diagram of a multi-target tracking system is schematically shown.

[0032] Fig. 9 A block diagram of an electronic device is schematically shown.

[0033] Fig.10 A block diagram of a computer-readable medium is schematically shown. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0035] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0038] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another component. Therefore, the first component discussed below can be referred to as the second component without departing from the teachings of the concepts of the present application. As used herein, the term "and / or" includes any one of the associated listed items and all combinations of one or more.

[0039] Those skilled in the art will appreciate that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0040] This application improves the YOLOV7 algorithm and DeepSORT algorithm, please refer to Figure 1 , Figure 1 The following schematically shows a step diagram of a multi-target tracking method. Figure 1As shown, according to a first specific embodiment of the present invention, the present invention provides a multi-target tracking method, comprising the following steps: S11, inputting the video stream data into a multi-scale target detection model improved based on YOLOV7 so that the multi-scale target detection model outputs the detection frame information of the target frame by frame.

[0041] The detection frame is a detection frame of different sizes output by the multi-size object detection model that contains the object. x ,c y ,w,h) includes the width, height and center coordinates of the detection box. x ,c y ) is the coordinate of the center point of the detection box, w is the width of the detection box, and h is the height of the detection box.

[0042] Since most existing apple quantity detection methods are based on static images for experiments, in natural environments, when photographing apples, some detection target pixels are too small due to factors such as long shooting distance and occlusion by leaves, resulting in large differences in the scales of different targets. Although YOLOV7 has three detection heads to detect large, medium, and small targets respectively, for small targets with little pixel information, the detection effect of these three detection heads alone is poor.

[0043] In order to solve the technical problem that the YOLOV series detection network in the prior art has poor detection effect on small targets with little pixel information, this application constructs a multi-size target detection model based on the improvement of YOLOV7.

[0044] Figure 2 The constellation configuration diagram of the multi-scale target detection model improved based on YOLOV7 is schematically shown. Figure 2 As shown, the multi-size target detection model includes a Backbone network, a Neck network and a Head network. The Backbone network is used to extract features four times from the feature map input into the multi-size target detection model, the Neck network is used to fuse the features extracted four times by the Backbone network to form four fusion branches, and the four fusion branches are input into the Head network to detect four targets of different sizes, and the Head network outputs the detection results of four targets of different sizes.

[0045] Please refer to Figure 2 and Figure 3 , Figure 3 for Figure 2 The CBS module structure diagram is shown in Figure 1. Figure 2 and Figure 3As shown in the figure, the CBS combination module includes four different CBS modules. The CBS module is used to change the size and number of feature maps. Multiple CBS modules can be superimposed or matched with other modules to achieve the purpose of feature extraction and feature fusion. The CBS module is composed of a two-dimensional convolution layer Conv, a batch normalization layer BN and an activation function layer SiLU connected in sequence.

[0046] like Figure 2 As shown, the CBS combination module includes a fifth CBS module, a sixth CBS module, a seventh CBS module and an eighth CBS module connected in sequence. Among them, the fifth CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 3X3, and the number of feature maps output is 32; the sixth CBS module is a two-dimensional convolution with a step size of 2 and a convolution kernel size of 3X3, and the number of feature maps output is 64; the seventh CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 3X3, and the number of feature maps output is 64; the eighth CBS module is a two-dimensional convolution with a step size of 2 and a convolution kernel size of 3X3, and the number of feature maps output is 128.

[0047] Assume that the image size of the input multi-scale object detection model is H*W*C, where H represents length, W represents width, and C represents the number of channels. Figure 2 As shown in the figure, if the size of the input image is 640*640*3, the size of the feature map after the CBS combination module becomes 160*160*128.

[0048] Continue to refer Figure 2 The feature graph after the CBS combination module enters the first SIMAM combination module, which is used to perform the first feature extraction on the input feature graph. The features extracted by the first SIMAM combination module are respectively input into the second SIMAM combination module and the Neck network. Figure 2 As shown, the first SIMAM combination module includes an ELAN module and a SIMAM module in sequence.

[0049] The ELAN module is an efficient layer aggregation network that extracts and fuses multi-scale features from images. Figure 4 , Figure 4 for Figure 2 The ELAN module aggregates features at four different levels to better capture multi-scale information. The ELAN module consists of four branches, such as Figure 4As shown in the figure, the first branch and the fourth branch have been operated by one CBS module, the second branch has been operated by five CBS modules in succession, and the third branch has been operated by three CBS modules in succession. The outputs of these four branches are then operated by the Concat module, that is, the features of the outputs of these four branches are fused in the channel dimension, and finally passed through a CBS module, thus completing the operation of the ELAN module. Figure 2 As shown in the figure, the improved multi-size target detection model based on YOLOV7 uses 4 ELAN modules, and the parameters of the CBS module in each ELAN module are different.

[0050] like Figure 2 As shown, the first SIMAM combination also includes a SIMAM module, and the feature map after passing through the CBS combination module enters the SIMAM module after passing through the ELAN module. The SIMAM module is used to incorporate a three-dimensional attention mechanism when extracting features to improve the feature extraction capability of the multi-size target detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features to the Neck network.

[0051] The SIMAM module is inspired by neuroscience: neurons that contain more information often behave differently from surrounding neurons, so an energy function e is proposed to measure the importance of each neuron. The smaller the energy function value e, the more important the information contained in the neuron, and the higher its weight should be.

[0052] According to research in visual neuroscience, the most informative neurons are usually those that show significantly different activation patterns from surrounding neurons. In addition, an active neuron can also inhibit the activity of surrounding neurons, a phenomenon known as spatial inhibition. In order to calculate the importance of each neuron, the following energy function is defined:

[0053] Among them, e t represents the neuron energy function, and is t and x i The linear transformation of w t and b t Respectively refer to the weight and bias of a neuron when it is transformed. t and x i They refer to the target neuron and other neurons in a single channel of the input feature X, respectively. i is the index in the spatial dimension, and M is the number of neurons in a channel. y represents a constant value, y t and o are two different constant values ​​used to simplify calculations. By minimizing this energy function, the linear separability between the target neuron t and other neurons in the same channel can be found. In order to simplify calculations, the regularization term is introduced , the final energy function is as follows:

[0054] In theory, for each channel, there are M energy functions to be solved. However, with some assumptions, the mean and variance can be calculated for all neurons in the channel and reused, thus avoiding the computational cost of iteratively calculating the mean and variance for each position. Therefore, the minimum energy can be calculated by the following formula:

[0055] in, and are the mean and variance calculated on all neurons in the channel, Represents the minimum value of the energy function. The lower the energy value, the greater the difference between neuron t and surrounding neurons, and the higher its importance. According to the definition of the attention mechanism, the SIMAM module can be expressed by the following formula:

[0056] in, is the output of the SIMAM module, X is the input of the SIMAM module, and E is all Grouped on the channel and spatial dimensions, sigmoid is used to limit values ​​in e that are too large.

[0057] like Figure 2 As shown in Figure 1, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128, and the size of the feature map after passing through the SIMAM module in the first SIMAM combination is 160*160*256. The feature map after passing through the SIMAM module in the first SIMAM combination has two directional outputs, such as Figure 2 As shown, one is to input the feature map to the Neck network, and the other is to input the feature map to the second SIMAM combination.

[0058] Continue to refer Figure 2 The second SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for extracting features for the second time. The features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network. Figure 2 As shown, compared with the first SIMAM combination module, the second SIMAM combination module has an additional MPConv module.

[0059] The MPConv module is used to pool the feature map. Figure 5 , Figure 5 for Figure 2 The MPConv module structure diagram is shown in Figure 2. Figure 5 As shown in the figure, the MPConv module consists of two branches: the first branch first passes through a maximum pooling layer Maxpoolk2 (k2 means the pooling kernel size is (2*2), and the step size is 2), and then passes through a 1*1 convolution module to adjust the output feature dimension; the second branch first passes through a 1*1 convolution module to adjust the feature dimension, and then passes through a 3*3 convolution module with a step size of 2 to achieve down sampling; finally, the results of the first branch and the second branch are spliced ​​together to obtain the final output of the MPConv module.

[0060] The features extracted by the first SIMAM combination module will pass through the MPConv module, ELAN module and SIMAM module in the second SIMAM combination module in sequence. Since the structures of the ELAN module and the SIMAM module have been described in the relevant paragraphs of the first SIMAM combination module, they will not be repeated for the sake of brevity.

[0061] like Figure 2 As shown in Figure 1, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128, and the size of the feature map after passing through the SIMAM module in the first SIMAM combination is 160*160*256, and the size of the feature map after passing through the SIMAM module in the second SIMAM combination is 80*80*512. The feature map after passing through the SIMAM module in the second SIMAM combination has two directional outputs, such as Figure 2 As shown, one is to input the feature map to the Neck network, and the other is to input the feature map to the third SIMAM combination.

[0062] Continue to refer Figure 2 The third SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for the third feature extraction, and the features extracted for the third time by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network.

[0063] The features extracted by the second SIMAM combination module for the second time will pass through the MPConv module, ELAN module and SIMAM module in the third SIMAM combination module in sequence. Since the structures of the MPConv module, ELAN module and SIMAM module have been described in the relevant paragraphs of the first SIMAM combination module and the second SIMAM combination module, they will not be repeated for the sake of brevity.

[0064] like Figure 2As shown, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128, and the size of the feature map after passing through the SIMAM module in the first SIMAM combination is 160*160*256, and the size of the feature map after passing through the SIMAM module in the second SIMAM combination is 80*80*512, and the size of the feature map after passing through the SIMAM module in the third SIMAM combination is 40*40*1024. The feature map after passing through the SIMAM module in the third SIMAM combination has two directional outputs, such as Figure 2 As shown, one is to input the feature map to the Neck network, and the other is to input the feature map to the fourth SIMAM combination.

[0065] Continue to refer Figure 2 , the fourth SIMAM combination module includes MPConv module, ELAN module, SPPCSPC module and SIMAM module in sequence for extracting features for the fourth time and inputting the extracted features into the Neck network. The features extracted for the third time by the third SIMAM combination module will pass through the MPConv module, ELAN module, SPPCSPC module and SIMAM module in the fourth SIMAM combination module in sequence. Since the structures of the MPConv module, ELAN module and SIMAM module have been described in the relevant paragraphs of the first SIMAM combination module and the second SIMAM combination module, they will not be repeated for the sake of brevity of the specification.

[0066] Among them, the SPPCSPC module is used to extract features after manifestation at different sizes. Please refer to Figure 6 , Figure 6 for Figure 2 SPC SPC module structure diagram. Figure 6 As shown in the figure, the SPPCSPC module consists of 7 CBS modules (the number of feature maps output by each CBS module is 512, but the size of the 2D convolution kernel and the convolution step size are different) and 3 maximum pooling layers (Maxpool k5, Maxpoolk9, Maxpool k13). The step size of the 3 maximum pooling layers is 1, the size of the pooling kernel of the maximum pooling layer Maxpool k5 is 5*5, the size of the pooling kernel of the maximum pooling layer Maxpool k9 is 9*9, and the size of the pooling kernel of the maximum pooling layer Maxpool k13 is 13*13.

[0067] like Figure 6As shown in the figure, the SPPCSPC module divides the input into two branches, one of which performs feature extraction, and then passes through the feature pyramid and performs maximum pooling on features of different sizes, visualizes the extracted feature information and concatenates the result with the feature map that has not been pooled through the feature pyramid (that is, the other branch). Through this operation, the network can extract features of different sizes after visualization without losing the information of the original feature map, so as to achieve the effect of improving the detection accuracy.

[0068] like Figure 2 As shown, if the size of the input image is 640*640*3, the size of the feature map after passing through the CBS combination module becomes 160*160*128, and the size of the feature map after passing through the SIMAM module in the first SIMAM combination is 160*160*256, and the size of the feature map after passing through the SIMAM module in the second SIMAM combination is 80*80*512, and the size of the feature map after passing through the SIMAM module in the third SIMAM combination is 40*40*1024, and the size of the feature map after passing through the SIMAM module in the fourth SIMAM combination is 20*20*512. The feature map after passing through the SIMAM module in the fourth SIMAM combination is input into the Neck network.

[0069] Continue to refer Figure 2 , the four feature maps P1-P4 output by the BackBone network will enter the Neck network to fuse the top and bottom features from top to bottom, obtain four sets of fusion feature results and input them to the Head network. Figure 2 As shown, the Neck network includes 2 first CBS modules, 2 second CBS modules, 2 third CBS modules, 1 fourth CBS module, 3 upsampling modules, 6 Concat modules, 6 Concat-Conv modules and 3 MPConv modules.

[0070] Among them, the first CBS module, the second CBS module, the third CBS module and the fourth CBS module are four different CBS modules. The first CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of feature maps output is 64; the second CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of feature maps output is 128; the third CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of feature maps output is 256; the fourth CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of feature maps output is 512.

[0071] The Concat module is used to perform feature fusion in the channel dimension. The Concat-Conv module is used to extract and fuse multi-scale features of the image. Figure 7 , Figure 7 for Figure 2 The Concat-Conv module structure diagram in Figure 2. Figure 7 As shown in the figure, the Concat-Conv module aggregates features at 6 different levels to better capture multi-scale information. The Concat-Conv module includes 6 branches. The first and sixth branches have been operated by 1 CBS module, the second branch has been operated by 5 CBS modules in succession, the third branch has been operated by 4 CBS modules in succession, the fourth branch has been operated by 3 CBS modules in succession, and the fifth branch has been operated by 2 CBS modules in succession. The outputs of these 6 branches are then operated by the Concat module, that is, the features of the outputs of these 6 branches are fused in the channel dimension, and finally passed through a CBS module, thus completing the operation of the Concat-Conv module.

[0072] like Figure 2 As shown, the Head network includes a first size detection head, a second size detection head, a third size detection head and a fourth size detection head; wherein, the first size> the second size> the third size> the fourth size. The features extracted by the fourth SIMAM combination module are sequentially passed through the fourth CBS module, the first third CBS module, the first upsampling module, the first Concat module, the first Concat-Conv module, the second CBS module, the second upsampling module, the second Concat module, the second Concat-Conv module, the first CBS module, the third upsampling module, the third Concat module and the third Concat-Conv module; the features extracted by the third SIMAM combination module are passed through the second third CBS module and the first Concat module; the features extracted by the second SIMAM combination module are passed through the second second CBS module and the second Concat module; the features extracted by the first SIMAM combination module are passed through the second first CBS module and the third Concat module; the output of the third Concat-Conv module is used as the input of the first size detection head.

[0073] The output of the third Concat-Conv module also passes through the first MPConv module, the fourth Concat module, the fourth Concat-Conv module, the second MPConv module, the fifth Concat module, the fifth Concat-Conv module, the third MPConv module, the sixth Concat module and the sixth Concat-Conv module in sequence; the output of the second Concat-Conv module is also used as the input of the fourth Concat module; the output of the first Concat-Conv module is also used as the input of the fifth Concat module; the output of the fourth CBS module is also used as the input of the sixth Concat module.

[0074] The output of the fourth Concat-Conv module is also used as the input of the second-size detection head; the output of the fifth Concat-Conv module is also used as the input of the third-size detection head; the output of the sixth Concat-Conv module is used as the input of the fourth-size detection head.

[0075] The Head network also includes four RepConv modules, which are re-parameterized convolution modules. Their function is to re-parameterize the network to reduce the computational complexity of the model and improve the model performance. Among them, the output of the third Concat-Conv module is used as the input of the first RepConv module, and the output of the first RepConv module is used as the input of the first-size detection head; the output of the fourth Concat-Conv module is used as the input of the second RepConv module, and the output of the second RepConv module is used as the input of the second-size detection head; the output of the fifth Concat-Conv module is used as the input of the third RepConv module, and the output of the third RepConv module is used as the input of the third-size detection head; the output of the sixth Concat-Conv module is used as the input of the fourth RepConv module, and the output of the fourth RepConv module is used as the input of the fourth-size detection head.

[0076] Optionally, the loss function of the improved multi-size object detection model based on YOLOV7 is a hybrid bounding box regression loss function based on the CIOU loss function and the normalized Gaussian Wasserstein distance.

[0077] YOLOV7 in the prior art uses CIOU as the bounding box regression loss function. However, the single CIOU is too sensitive to the change of the pixel position of small targets, which will cause the model's small target detection performance to drop rapidly. In contrast, the normalized Wasserstein distance (NWD) is smoother to the change of position offset, and NWD can measure the similarity between non-overlapping or overlapping small bounding boxes.

[0078] The bounding box is modeled as a Gaussian distribution as follows: To determine a bounding box, we need to know the coordinates of its center point (c x , c y ), width (w) and height (h). Any horizontal bounding box R = (c x , c y , w, h), the equation of the inscribed ellipse can be expressed as:

[0079] The probability density function of the two-dimensional Gaussian distribution is as follows:

[0080] Where x is the coordinate of the two-dimensional Gaussian distribution, is the covariance matrix, is the mean vector. When , the inscribed ellipse of any bounding box R is the density contour of the two-dimensional Gaussian distribution, so the bounding box R = (c x , c y , w, h), can be modeled as a Gaussian distribution . At this point, the similarity between the two bounding boxes can be converted into the distribution distance between the Gaussian distributions corresponding to the two bounding boxes. The normalized Wasserstein distance is used as the distance between the two Gaussian distributions. In the actual detection task, the Gaussian distribution corresponding to the predicted box P and the true box T is The Wasserstein distance between these two Gaussian distributions is defined as follows:

[0081] When the covariance matrix satisfies the transformation: When , Substituting in, we get the following formula:

[0082] Because the Wasserstein distance is a distance metric and cannot be directly used to measure the similarity between bounding boxes, it will make it difficult for the loss function to converge. Therefore, the Wasserstein distance is normalized:

[0083] Where C is a constant that is closely related to the training data set. For example, in the apple detection task, the empirical value can be 12.8. After obtaining NWD, multiply it by the weight factor λ and add it to CIOU to obtain the new bounding box regression loss function:

[0084] If the proportion of small targets in the task is large, the weight λ of NWD can be increased accordingly. For example, the λ of the apple detection task can be 0.5.

[0085] like Figure 2 As shown, if the size of the input image is 640*640*3, the size of the feature map after the CBS combination module becomes 160*160*128, and the size of the feature map after the SIMAM module in the first SIMAM combination is 160*160*256, and the size of the feature map after the SIMAM module in the second SIMAM combination is 80*80*512, and the size of the feature map after the SIMAM module in the third SIMAM combination is 40*40*1024, and the size of the feature map after the SIMAM module in the fourth SIMAM combination is 20*20*512, and the feature map after the SIMAM module in the fourth SIMAM combination is input to the Neck network. The Neck network fuses the top and bottom features from top to bottom, obtains four sets of fused feature results and inputs them to the Head network. Among them, the feature map size input by the Neck network to the first size detection head is 160*160*64, and the feature map size output by the first size detection head is 160*160; the feature map size input by the Neck network to the second size detection head is 80*80*128, and the feature map size output by the second size detection head is 80*80; the feature map size input by the Neck network to the third size detection head is 40*40*256, and the feature map size output by the third size detection head is 40*40; the feature map size input by the Neck network to the fourth size detection head is 20*20*512, and the feature map size output by the fourth size detection head is 20*20.

[0086] After completing the construction of the multi-size target detection model based on the improvement of YOLOV7, establish the image data training set and verification set, input the image data training set into the multi-size target detection model based on the improvement of YOLOV7 for training, and detect the verification set and output the multi-size target detection results. Then, input the video stream data into the multi-size target detection model after training, and the multi-size target detection model will detect the targets of four sizes in each frame of the image, and finally output the detection frame information of the target frame by frame.

[0087] In order to verify the impact of each improved part on the performance of the multi-size target detection model based on YOLOV7, the original YOLOV7 is used as the baseline model, and the improved parts are added in turn. Ablation experiments are performed on the Apple (self-made Apple dataset) and Drone (public dataset) datasets, and the results are shown in Table 1. Among them, method A means that the SimAM attention mechanism is integrated into the feature extraction stage, method B means adding an additional small target detection head, and method C means adding NWD loss to the original border loss of the network. It can be seen from Table 1 that the methods proposed in this paper all improve the detection ability of the model, especially method C improves the detection ability of the model without increasing any network parameters. Therefore, it fully proves the effectiveness of NWD loss for small object detection. For the improved network, mAP@0.5 is improved by 1.61% on the Apple dataset, and map@0.5 is improved by 3.95% on the Drone dataset, with obvious improvement effect.

[0088] Table 1

[0089] This application makes corresponding improvements to the Backbone network, Neck network and Head network of the YOLOV7 model in view of the characteristics that the detection targets are small and dense; the Backbone network adds feature extraction of tiny-sized targets, that is, outputs four scale feature maps, and adds a three-dimensional attention mechanism SimAM in the four-scale feature extraction process to improve the feature extraction ability of the network, and adds output branches to the Neck network and the Head network, solving the technical problem in the prior art that the YOLOV7 detection network has poor detection effect on tiny targets with little pixel information.

[0090] After the multi-size target detection model improved based on YOLOV7 outputs the detection frame information of the target frame by frame, step S12 is executed.

[0091] S12, assigning a target ID to the target corresponding to each target detection box in frame t.

[0092] Among them, t is an integer greater than 0. That is to say, every time the multi-size target detection model improved based on YOLOV7 outputs the detection frame information containing the target in a frame of image in the order of the video stream, the target ID is assigned to the target corresponding to each target detection frame.

[0093] S13: predict the prediction box information of the target in frame t+1 according to the detection box information of the target in frame t.

[0094] The prediction box information includes the width, height and center coordinates of the prediction box. In one embodiment, step S13 may include the following steps: S131, calculate the state matrix of the target according to the detection frame width, height and detection frame center coordinates of the target in frame t.

[0095] The state matrix includes the center point coordinates of the detection frame of the target in frame t, the aspect ratio and height of the detection frame, and the rate of change of the center point coordinates, aspect ratio and height.

[0096] According to the detection box information of the target in frame t (c x ,c y ,w,h), calculate the state matrix [c x ,c y ,r,h,v x ,v y ,v r ,v h ]. Among them, (c x ,c y ) is the center point of the detection frame, r is the aspect ratio w / h of the detection frame, h is the height, v is the x ,v y ,v r ,v h is the rate of change of these four values. For example, in the first frame of the image, the detector detects three apples. First, they are assigned IDs 1, 2, and 3, and then the state matrix of these three apples is calculated.

[0097] S132, according to the state matrix of the target, calculate the predicted frame information of the target in the t+1 frame through Kalman filtering.

[0098] According to the target state matrix obtained in step S131, the estimated position of the target in the next frame image, that is, the prediction frame, can be calculated through Kalman filtering. For example, three apples are detected in the first frame image, and the state matrix of the three apples is obtained according to step S131. Through Kalman filtering, the estimated position of the three apples in the second frame image can be calculated, that is, the prediction frame information of the three apples in the second frame image can be obtained.

[0099] After obtaining the predicted frame information of the target in frame t+1, execute step S14.

[0100] S14, according to the detection box information of the t+1 frame and the prediction box information of the t+1 frame, perform IOU matching on the prediction box and the detection box of the t+1 frame.

[0101] In computer vision, IOU (Intersection Over Union) is a metric used to measure the degree of match between a predicted bounding box and an actual bounding box. IOU is also known as intersection over union or overlap. When evaluating models in object detection or image segmentation tasks, IOU is one of the most important evaluation metrics. In object detection tasks, IOU is usually used to measure the degree of match between a predicted bounding box and the actual object (detection box). If the IOU value of a predicted bounding box is higher than a certain threshold (such as 0.5), the predicted box is considered correct.

[0102] In step S14, the detection box information of the t+1 frame is output after the t+1 frame image is detected by the multi-size target detection model improved based on YOLOV7 (multi-size target detection model improved based on YOLOV7); the prediction box information of the t+1 frame is obtained by step S13.

[0103] It should be noted that the prediction boxes that fail the IOU matching are directly deleted.

[0104] S15, the target corresponding to the detection box with successful IOU matching in frame t+1 is assigned the target ID corresponding to frame t.

[0105] For example, in the first frame, the multi-scale object detection model based on the improved YOLOV7 detects three apples. First, they are assigned IDs 1, 2, and 3. Then, the state matrix of the three apples is calculated. Then, through Kalman filtering, the estimated positions of the three apples in the second frame can be calculated, that is, three prediction boxes are output. If the detector detects four apples in the second frame, four detection boxes are output at the same time. Then, the four detection boxes and the three prediction boxes are matched by IOU. If the three prediction boxes and the three detection boxes are successfully matched by IOU, that is, the three apples in the second frame match the three apples in the first frame, then the prediction boxes corresponding to the three apples in the first frame are used as their positions in the second frame, and the same IDs (1, 2, 3) as in the first frame are assigned to them again. The fourth apple detected in the second frame has no prediction box because it appears for the first time, so there is no need to do IOU matching, and a new ID 4 is directly assigned.

[0106] S16, when the prediction box and the detection box of a target have been matched by IOU for N consecutive frames, the target in each frame after N frames is cascade matched with the target after IOU matching for N consecutive frames.

[0107] Where N is an integer greater than 1. Cascade matching is actually a feature matching. The main idea is to use the ReID network to extract the appearance feature information and location information of the tracked target and the detection target in the current frame, and then calculate the cosine distance of the appearance feature information and the Mahalanobis distance of the location information to construct the cost matrix, and finally complete the matching of the tracked target and the detection target through the Hungarian matching algorithm.

[0108] Optionally, step S16 may include the following steps: S161, selecting a frame from N consecutive frames to use the ReID network to extract the first appearance feature information of the target after IOU matching in the frame, and obtaining the first position information of the target in the frame through the detection box information of the target.

[0109] S162, in the current frame after N frames, the second appearance feature information of all objects in the current frame is extracted using the ReID network, and the second position information of all objects in the current frame is obtained through the prediction box information.

[0110] S163: Calculate a cost matrix according to the first appearance feature information and the second appearance feature information, and the first position information and the second position information.

[0111] Further, step S163 may include the following steps: S1631: Calculate a cosine distance of appearance feature information according to the first appearance feature information and the second appearance feature information.

[0112] S1632: Calculate the Mahalanobis distance of the position information according to the first position information and the second position information.

[0113] S1633: construct a cost matrix according to the cosine distance of the appearance feature information and the Mahalanobis distance of the position information.

[0114] After the cost matrix is ​​constructed, step S164 is executed.

[0115] S164, completing cascade matching through the Hungarian matching algorithm according to the cost matrix.

[0116] The appearance feature information is extracted by the pre-trained ReID network. The appearance feature information is actually a 128-dimensional feature vector. The appearance feature information refers to the appearance information of the target, such as size, shape, color and other appearance features.

[0117] It should be noted that when multiple targets complete N consecutive IOU matching frames, the multiple targets are prioritized according to the number of frames of consecutive IOU matching, so that the targets in each frame after N frames are cascade matched with the targets with high priority first; among them, the more frames of consecutive IOU matching, the higher the priority.

[0118] For example, there are three apples in the 58th frame, of which Apple 1 is tracked in every frame of the first 57 frames, Apple 2 is not tracked in two frames, and Apple 3 is not tracked in five frames. Then the three apples detected by the improved multi-scale object detection model based on YOLOV7 will first be cascade matched with Apple 1, and then the remaining two apples will be cascade matched with Apple 2. Cascade matching is actually a priority matching mechanism, which will give priority to matching and tracking targets with fewer lost frames.

[0119] After the cascade matching is completed, step S17 is executed.

[0120] S17, counting the target IDs of successful cascade matching as the detection result.

[0121] When using the DeepSort algorithm model for multi-target tracking, the problem of ID jumping is prone to occur. The main reason is that targets that have not been successfully matched will also be assigned IDs. If the ID number is simply used as the number of apples detected, the error will be large. For this reason, an additional statistical step is added, and only the target IDs that have successfully cascade matched are included in the statistical number as the final detection result. In this process, the ReID network can be used to re-identify the target. Even if the target is lost in certain frames of the image, the ReID network can be used to re-identify the target in subsequent images, solving the problem of the same target being counted repeatedly.

[0122] This application counts target videos (rather than static images), which solves the problem of single viewing angle and limited field of view when shooting static images. In addition, this application assigns the same ID to two targets with successful IOU matching in different images, and performs cascade matching after multiple consecutive frames of IOU matching. Only the target IDs with successful cascade matching are counted in the detection results, reducing the impact of ID jumps on calculation accuracy, and solving the problem of targets in the same area being repeatedly counted in different images, which directly affects the counting accuracy.

[0123] According to a second specific embodiment of the present invention, the present invention provides a multi-target tracking system, using the method of the first specific embodiment, such as Figure 8 As shown, the multi-target tracking system 400 includes: The detection module 410 is used to input the video stream data into the multi-size target detection model improved based on YOLOV7 so that the multi-size target detection model outputs the detection frame information of the target frame in sequence; wherein the multi-size target detection model includes a Backbone network, a Neck network and a Head network; the Backbone network is used to extract features four times from the feature map input into the multi-size target detection model, the Neck network is used to fuse the features extracted four times by the Backbone network to form four fusion branches, and the four fusion branches are input into the Head network to detect four targets of different sizes, and the Head network outputs the detection results of four targets of different sizes; A first ID assignment module 420 is used to assign a target ID to a target corresponding to each target detection frame in t frames; wherein t is an integer greater than 0; A prediction module 430 is used to predict the prediction box information of the target in frame t+1 according to the detection box information of the target in frame t; An IOU matching module 440 is used to perform IOU matching between the prediction box and the detection box of the t+1 frame according to the detection box information of the t+1 frame and the prediction box information of the t+1 frame; The second ID assignment module 450 is used to assign the target ID corresponding to the t frame to the target corresponding to the detection box with successful IOU matching in the t+1 frame; The cascade matching module 460 is used to perform cascade matching on the target in each frame after N frames with the target after N frames of IOU matching after N frames of IOU matching, where N is an integer greater than 1. A statistics module 470 is used to count the target IDs of successful cascade matching as detection results; Wherein, the Backbone network includes a CBS combination module, a first SIMAM combination module, a second SIMAM combination module, a third SIMAM combination module and a fourth SIMAM combination module in sequence; The first SIMAM combination module includes an ELAN module and a SIMAM module in sequence for first feature extraction, and the features extracted by the first SIMAM combination module are respectively input into the second SIMAM combination module and the Neck network; The second SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for extracting features for the second time, and the features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network; The third SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for third feature extraction, and the features extracted for the third time by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network; The fourth SIMAM combination module includes an MPConv module, an ELAN module, a SPPCSPC module and a SIMAM module in sequence for extracting features for the fourth time and inputting the extracted features into the Neck network; Among them, the CBS combination module includes four different CBS modules, and the CBS module is used to change the size and number of feature maps; the ELAN module is an efficient layer aggregation network for multi-scale feature extraction and fusion of images; the SIMAM module is used to incorporate a three-dimensional attention mechanism when extracting features to improve the feature extraction capability of the multi-scale target detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features to the Neck network; the MPConv module is used to pool the feature maps; and the SPPCSPC module is used to extract features after being manifested in different sizes.

[0124] According to a third specific embodiment of the present invention, the present invention provides an electronic device, such as Fig. 9 As shown, Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment.

[0125] Refer to the following Fig. 9 The electronic device 200 according to this embodiment of the present application is described. Fig. 9 The electronic device 200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0126] like Fig. 9 As shown, the electronic device 200 is in the form of a general computing device. The components of the electronic device 200 may include, but are not limited to: at least one processing unit 210, at least one storage unit 220, a bus 230 connecting different system components (including the storage unit 220 and the processing unit 210), a display unit 240, etc.

[0127] The storage unit stores a program code, which can be executed by the processing unit 210, so that the processing unit 210 performs the steps described in this specification according to various exemplary embodiments of the present application. For example, the processing unit 210 can perform the following steps: Figure 1 Follow the steps shown in .

[0128] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 2201 and / or a cache memory unit 2202 , and may further include a read-only memory unit (ROM) 2203 .

[0129] The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0130] Bus 230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0131] The electronic device 200 may also communicate with one or more external devices 200' (e.g., keyboards, pointing devices, Bluetooth devices, etc.) so that a user can communicate with the device that interacts with the electronic device 200, and / or the electronic device 200 can communicate with any device (e.g., routers, modems, etc.) that communicates with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface 250. In addition, the electronic device 200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 through a bus 230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0132] Through the description of the above implementations, those skilled in the art can easily understand that the example implementations described here can be implemented by software, or by combining software with necessary hardware.

[0133] Therefore, according to a fourth specific embodiment of the present invention, the present invention provides a computer readable medium. Fig.10As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0134] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0135] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0136] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0137] The computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the computer-readable medium implements the functions of the first embodiment.

[0138] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0139] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0140] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-target tracking method, characterized in that: The following steps are involved: Input the video stream data into the trained multi-size target detection model based on YOLOV7, so that the multi-size target detection model outputs the detection frame information of the target frame by frame; wherein the multi-size target detection model includes a Backbone network, a Neck network and a Head network; the Backbone network is used to extract features four times from the feature map input to the multi-size target detection model, the Neck network is used to fuse the features extracted four times by the Backbone network to form four fusion branches, and the four fusion branches are input into the Head network to detect four targets of different sizes, and the Head network outputs the detection results of four targets of different sizes; Assign a target ID to the target corresponding to each target detection box in t frames; where t is an integer greater than 0; According to the detection box information of the target in frame t, predict the prediction box information of the target in frame t+1; According to the detection box information of frame t+1 and the prediction box information of frame t+1, perform IOU matching between the prediction box and the detection box of frame t+1; The target corresponding to the detection box with successful IOU matching in frame t+1 is assigned the target ID corresponding to frame t; When the prediction box and detection box of an object have been matched by IOU for N consecutive frames, the object in each frame after N frames is cascade matched with the object after IOU matching for N consecutive frames; where N is an integer greater than 1; Count the target IDs that have been successfully matched in the cascade as the detection result; Wherein, the Backbone network includes a CBS combination module, a first SIMAM combination module, a second SIMAM combination module, a third SIMAM combination module and a fourth SIMAM combination module in sequence; The first SIMAM combination module includes an ELAN module and a SIMAM module in sequence for first feature extraction, and the features extracted by the first SIMAM combination module are respectively input into the second SIMAM combination module and the Neck network; The second SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for extracting features for the second time, and the features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network; The third SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for third feature extraction, and the features extracted for the third time by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network; The fourth SIMAM combination module includes an MPConv module, an ELAN module, a SPPCSPC module and a SIMAM module in sequence for extracting features for the fourth time and inputting the extracted features into the Neck network; Among them, the CBS combination module includes four different CBS modules, and the CBS module is used to change the size and number of feature maps; the ELAN module is an efficient layer aggregation network for multi-scale feature extraction and fusion of images; the SIMAM module is used to incorporate a three-dimensional attention mechanism when extracting features to improve the feature extraction capability of the multi-scale target detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features to the Neck network; the MPConv module is used to pool the feature maps; and the SPPCSPC module is used to extract features after being manifested in different sizes.

2. The multi-target tracking method according to claim 1, characterized in that: The loss function of the multi-scale object detection model is a hybrid bounding box regression loss function based on the CIOU loss function and the normalized Gaussian Wasserstein distance.

3. The multi-target tracking method according to claim 2, characterized in that: The expression of the mixed border regression loss function lbox is: ； Among them, λ is the weight factor and NWD is the normalized Gaussian Wasserstein distance.

4. The multi-target tracking method according to claim 1, characterized in that: The Neck network includes 2 first CBS modules, 2 second CBS modules, 2 third CBS modules, 1 fourth CBS module, 3 upsampling modules, 6 Concat modules, 6 Concat-Conv modules and 3 MPConv modules; Among them, the first CBS module, the second CBS module, the third CBS module and the fourth CBS module are four different CBS modules; the Concat module is used to perform feature fusion in the channel dimension; and the Concat-Conv module is used to extract and fuse multi-scale features of the image.

5. The target dynamic tracking method based on DeepSORT according to claim 4, characterized in that: The Head network includes a first size detection head, a second size detection head, a third size detection head and a fourth size detection head; wherein the first size>the second size>the third size>the fourth size; The features extracted by the fourth SIMAM combination module are sequentially passed through the fourth CBS module, the first third CBS module, the first upsampling module, the first Concat module, the first Concat-Conv module, the second CBS module, the second upsampling module, the second Concat module, the second Concat-Conv module, the first CBS module, the third upsampling module, the third Concat module and the third Concat-Conv module; the features extracted by the third SIMAM combination module are passed through the second third CBS module and the first Concat module; the features extracted by the second SIMAM combination module are passed through the second second CBS module and the second Concat module; the features extracted by the first SIMAM combination module are passed through the second first CBS module and the third Concat module; the output of the third Concat-Conv module is used as the input of the first size detection head; The output of the third Concat-Conv module is also sequentially passed through the first MPConv module, the fourth Concat module, the fourth Concat-Conv module, the second MPConv module, the fifth Concat module, the fifth Concat-Conv module, the third MPConv module, the sixth Concat module and the sixth Concat-Conv module; the output of the second Concat-Conv module is also used as the input of the fourth Concat module; the output of the first Concat-Conv module is also used as the input of the fifth Concat module; the output of the fourth CBS module is also used as the input of the sixth Concat module; The output of the fourth Concat-Conv module is also used as the input of the second-size detection head; the output of the fifth Concat-Conv module is also used as the input of the third-size detection head; the output of the sixth Concat-Conv module is used as the input of the fourth-size detection head.

6. The multi-target tracking method according to claim 5, characterized in that: The Head network also includes four RepConv modules, which are re-parameterized convolution modules for re-parameterizing the Head network; wherein, the output of the third Concat-Conv module is used as the input of the first RepConv module, and the output of the first RepConv module is used as the input of the first size detection head; the output of the fourth Concat-Conv module is used as the input of the second RepConv module, and the output of the second RepConv module is used as the input of the second size detection head; the output of the fifth Concat-Conv module is used as the input of the third RepConv module, and the output of the third RepConv module is used as the input of the third size detection head; the output of the sixth Concat-Conv module is used as the input of the fourth RepConv module, and the output of the fourth RepConv module is used as the input of the fourth size detection head.

7. The multi-target tracking method according to claim 4, characterized in that: The CBS combination module includes a fifth CBS module, a sixth CBS module, a seventh CBS module and an eighth CBS module; the feature map input to the multi-size target detection model passes through the fifth CBS module, the sixth CBS module, the seventh CBS module and the eighth CBS module in sequence; wherein the first CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of feature maps output is 64; the second CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of feature maps output is 128; the third CBS module is a two-dimensional convolution with a step size of 1 and a convolution kernel size of 1X1, and the number of feature maps output is 128. The number of feature maps is 256; the fourth CBS module is a two-dimensional convolution with a stride of 1 and a convolution kernel size of 1X1, and the number of output feature maps is 512; the fifth CBS module is a two-dimensional convolution with a stride of 1 and a convolution kernel size of 3X3, and the number of output feature maps is 32; the sixth CBS module is a two-dimensional convolution with a stride of 2 and a convolution kernel size of 3X3, and the number of output feature maps is 64; the seventh CBS module is a two-dimensional convolution with a stride of 1 and a convolution kernel size of 3X3, and the number of output feature maps is 64; the eighth CBS module is a two-dimensional convolution with a stride of 2 and a convolution kernel size of 3X3, and the number of output feature maps is 128.

8. The multi-target tracking method according to claim 1, characterized in that: Also includes: When multiple targets complete N consecutive IOU matching, the multiple targets are prioritized according to the number of frames of consecutive IOU matching, so that the targets in each frame after N frames are cascade matched with the targets with high priority first; the more frames of consecutive IOU matching, the higher the priority.

9. A multi-target tracking system, characterized in that: include: A detection module, used for inputting video stream data into a multi-size target detection model improved based on YOLOV7 so that the multi-size target detection model outputs detection frame information of the target frame by frame; wherein the multi-size target detection model includes a Backbone network, a Neck network and a Head network; the Backbone network is used for performing four feature extractions on the feature map input into the multi-size target detection model, the Neck network is used for fusing the features extracted four times by the Backbone network to form four fusion branches, and the four fusion branches are input into the Head network to detect four targets of different sizes, and the Head network outputs the detection results of four targets of different sizes; A first ID assignment module is used to assign a target ID to a target corresponding to each target detection frame in t frames; wherein t is an integer greater than 0; A prediction module is used to predict the prediction box information of the target in frame t+1 based on the detection box information of the target in frame t; The IOU matching module is used to perform IOU matching between the prediction box and the detection box of the t+1 frame according to the detection box information of the t+1 frame and the prediction box information of the t+1 frame; The second ID assignment module is used to assign the target ID corresponding to the t frame to the target corresponding to the detection frame with successful IOU matching in the t+1 frame; The cascade matching module is used to perform cascade matching on the target in each frame after N frames after the prediction box and the detection box of a target have been matched with each other for N consecutive frames; wherein N is an integer greater than 1; The statistics module is used to count the target IDs that have been successfully matched in the cascade as the detection results; Wherein, the Backbone network includes a CBS combination module, a first SIMAM combination module, a second SIMAM combination module, a third SIMAM combination module and a fourth SIMAM combination module in sequence; The first SIMAM combination module includes an ELAN module and a SIMAM module in sequence for first feature extraction, and the features extracted by the first SIMAM combination module are respectively input into the second SIMAM combination module and the Neck network; The second SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for extracting features for the second time, and the features extracted by the second SIMAM combination module for the second time are respectively input into the third SIMAM combination module and the Neck network; The third SIMAM combination module includes an MPConv module, an ELAN module and a SIMAM module in sequence for third feature extraction, and the features extracted for the third time by the third SIMAM combination module are respectively input into the fourth SIMAM combination module and the Neck network; The fourth SIMAM combination module includes an MPConv module, an ELAN module, a SPPCSPC module and a SIMAM module in sequence for extracting features for the fourth time and inputting the extracted features into the Neck network; Among them, the CBS combination module includes four different CBS modules, and the CBS module is used to change the size and number of feature maps; the ELAN module is an efficient layer aggregation network for multi-scale feature extraction and fusion of images; the SIMAM module is used to incorporate a three-dimensional attention mechanism when extracting features to improve the feature extraction capability of the multi-scale target detection model network, and the SIMAM modules in the four SIMAM combination modules all input the extracted features to the Neck network; the MPConv module is used to pool the feature maps; and the SPPCSPC module is used to extract features after being manifested in different sizes.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images