A multi-target recognition and tracking method and recognition and tracking system

Abstract

The present invention provides a multi-target recognition and tracking method and recognition and tracking system, the tracking method comprising: merging the detection result information, depth feature information and motion feature information of each target in the video image to be recognized and tracked to form a target feature descriptor; calculating the similarity between each target and the existing target according to the target feature descriptor of each target in the video image to be recognized and tracked, and obtaining the initial result of multi-target matching; optimizing the initial result of multi-target matching based on a multi-layer cascade matching strategy, and obtaining the final multi-target tracking result information. The present invention introduces a feature descriptor composed of multiple features during multi-target association matching, which makes up for the lack of information for single feature discrimination and improves the accuracy of target matching.

Description

Technical Field

[0001] The present invention relates to the field of target recognition and tracking, and more specifically, to a multi-target recognition and tracking method and a recognition and tracking system. Background Art

[0002] Multi-target tracking technology is an important research field in computer vision. It can obtain the number of targets in the area of ​​interest and the state and motion trajectory of each target, and realize the tracking and positioning of multiple targets. Multi-target tracking technology can be applied to various fields.

[0003] At present, the multi-target tracking algorithm is mainly based on online association tracking based on detection. This type of algorithm decomposes the multi-target tracking task into two subtasks, namely target detection and target association. Target detection mainly uses the target detector to detect each frame of the video image, and then constructs a candidate target set for tracking targets based on the detection results, turning the multi-target tracking problem into an association problem based on the detection results; target association is to calculate the data association between the detection results and the target feature model, and associate the detection results of multiple targets into the target's trajectory and unique ID by setting the feature similarity distance threshold.

[0004] Target association usually uses a single feature to achieve the matching of targets in adjacent frames. Common target features mainly include motion information and appearance features. Motion information is generally obtained by linear filter prediction, but the motion of the target is nonlinear and disordered. Therefore, when the target changes its motion state, it is difficult to correctly associate the target, and it cannot cope with the situation of target occlusion. Appearance features are divided into traditional features and deep features. Traditional manual features rely on manually formulated strategies, which are fast and simple to implement, but cannot flexibly cope with complex and changing scenes; deep features have abstract, high-level semantic information, which is conducive to describing the appearance characteristics of the target, but usually require the construction of an additional deep convolutional network, which is computationally intensive and time-consuming to extract features. Summary of the invention

[0005] The present invention aims at the technical problems existing in the prior art and provides a multi-target recognition and tracking method and a recognition and tracking system.

[0006] According to a first aspect of the present invention, a multi-target recognition and tracking method is provided, comprising:

[0007] The video image to be identified and tracked is input into the target detection network, and the detection result information and depth feature information of multiple targets output by the target detection network are obtained; the motion feature information of each target is predicted and output by the Kalman filter algorithm; the detection result information, depth feature information and motion feature information of each target are combined to form a target feature descriptor; according to the target feature descriptor of each target in the video image to be identified and tracked, the similarity between each target and the existing targets is calculated to obtain the initial result of multi-target matching; the initial result of multi-target matching is optimized based on the multi-layer cascade matching strategy to obtain the final multi-target tracking result information.

[0008] According to a second aspect of the present invention, a multi-target recognition and tracking system is provided, comprising: an acquisition module, used to input a video image to be identified and tracked into a target detection network, and obtain detection result information and depth feature information of multiple targets output by the target detection network; and obtain motion feature information of each target predicted and output by a Kalman filter algorithm; a merging module, used to merge the detection result information, depth feature information and motion feature information of each target to form a target feature descriptor; an initial matching module, used to calculate the similarity between each target and an existing target based on the target feature descriptor of each target in the video image to be identified and tracked, and obtain a preliminary result of multi-target matching; and an optimization matching module, used to optimize the preliminary result of multi-target matching based on a multi-layer cascade matching strategy, and obtain the final multi-target tracking result information.

[0009] According to a third aspect of the present invention, there is provided an electronic device comprising a memory and a processor, wherein the processor is configured to implement the steps of a multi-target recognition and tracking method when executing a computer management program stored in the memory.

[0010] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium on which a computer management program is stored, and when the computer management program is executed by a processor, the steps of the multi-target recognition and tracking method are implemented.

[0011] The present invention provides a multi-target recognition and tracking method and recognition and tracking system, which combines the detection result information, depth feature information and motion feature information of each target in the video image to be recognized and tracked to form a target feature descriptor; according to the target feature descriptor of each target in the video image to be recognized and tracked, the similarity between each target and the existing target is calculated to obtain the initial result of multi-target matching; based on the multi-layer cascade matching strategy, the initial result of multi-target matching is optimized to obtain the final multi-target tracking result information. The present invention introduces a feature descriptor composed of multiple features during multi-target association matching, which makes up for the lack of information for single feature discrimination and improves the accuracy of target matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flow chart of a multi-target recognition and tracking method provided by the present invention;

[0013] Figure 2 It is a schematic diagram of the structure of the target detection network;

[0014] Figure 3 A flowchart of a multi-target recognition and tracking method;

[0015] Figure 4 A structural block diagram of a multi-target recognition and tracking system provided by the present invention;

[0016] Figure 5 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0017] Figure 6 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0018] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0019] In view of the problem of insufficient target-related feature information in the background technology, the present invention intends to introduce motion information, deep feature information and target detection results to form feature descriptors to describe the target as a whole from multiple angles. In view of the time-consuming problem of deep feature extraction, the present invention intends to extract deep features from the target detection and recognition network to save feature extraction time, but the directly derived target features only contain target category characteristics and cannot highlight the typicality of the characteristics of the target individual.

[0020] In summary, in order to address the problems of independent target feature extraction process, insufficient target feature information and poor anti-occlusion performance in traditional detection-based multi-target tracking methods, the present invention designs a multi-target tracking algorithm (DF-MOT) based on feature descriptors, which effectively improves the accuracy of multi-target association, enhances anti-occlusion performance, and reduces the problems of batch number interleaving when multiple targets intersect and overlap, and the inability to match the original batch number after the target is occluded.

[0021] Embodiment 1

[0022] A multi-target recognition and tracking method, see Figure 1 , the multi-target recognition and tracking method mainly includes the following steps:

[0023] S1, inputting the video image to be identified and tracked into a target detection network, and obtaining detection result information and depth feature information of multiple targets output by the target detection network.

[0024] As an embodiment, the target detection network includes a feature extraction subnetwork, a region pooling subnetwork and an output subnetwork; the feature extraction subnetwork is used to extract feature information of different scales of each target in the video image; the region pooling subnetwork is used to fuse the feature information of different scales of each target in the video image to generate multi-scale fused deep feature information of each target; the output subnetwork is used to output the detection result information of each target based on the multi-scale fused deep feature information of each target, and the detection result information includes category attribute information and location information of each target.

[0025] Specifically, the detection result information and deep feature information of multiple targets in the video image are extracted and identified through the target detection network, wherein the target detection result information mainly includes the category attribute information and position information of each target, wherein the position information is represented by a target detection box, for example.

[0026] It is difficult to obtain deep feature information with typicality of the target by directly extracting the features of the intermediate layer of the target detection network, which is not conducive to the matching of the target appearance features. Therefore, a fast target deep feature extraction method based on multi-task learning is designed. The core idea of ​​the fast target feature extraction method based on multi-task learning is to constrain the target features so that the features have stronger typicality, reuse the features extracted from the target detection link, and map the reused features to a new space with the help of a small amount of calculations, so that it is suitable for multi-target tracking tasks. Considering that the network based on detection and recognition only considers the errors of target detection and recognition during training, and lacks the constraints of multi-target tracking task feedback, directly extracting the deep features in the detection and recognition network for subsequent matching association cannot obtain good association results.

[0027] Therefore, the present invention adds constraints to the intermediate features of the target detection network, expecting that the deep feature information of different targets has certain differences, which is convenient for subsequent identification of different targets. At the same time, in order to enrich the semantic information of the deep features, multiple feature layers are selected for fusion to represent the deep feature information of the target. The way to achieve this goal is to extract the multi-level features in the target detection network with the help of regional pooling operation, and after organically fusing them, add feature constraints to the network objective function, that is, the greater the difference in the deep features of different targets, the better, thereby driving network training. Among them, in the present invention, the target detection network uses the YOLOv5s model.

[0028] The method of selecting multiple feature layers for fusion to represent the deep feature information of the target is to add a regional pooling self-network to the target detection network, see Figure 2The regional pooling subnetwork is used to fuse the feature information of different scales of each target in the video image extracted by the feature extraction subnetwork, generate multi-scale fused deep feature information of each target, and fuse the feature information of different scales to enrich the semantic information of the deep feature information of each target.

[0029] In order to reflect the differences between different targets, the present invention adds feature constraints to the loss function of the target detection network. The designed network target loss function is as follows:

[0030] Loss = L detect +λL fea-measure ;

[0031] Among them, L detect represents the target detection and recognition error, λ is the loss function balance factor, L fea-measure is the newly added feature similarity error, and the calculation formula is as follows:

[0032] L fea-measure =∑g[(F i -F j ) T S ij -1 (F i -F j )];

[0033]

[0034] Among them, F i and F j ∈R N Represent the multi-scale fused deep feature information of the i-th and j-th targets respectively.

[0035] By minimizing the loss function, the spatial distance between deep features of different targets can be maximized, thereby highlighting the typicality of the target features and facilitating subsequent feature association matching.

[0036] The main amount of computation in the target detection network structure is concentrated in the multi-scale feature extraction part. Based on the multi-scale features, only a small amount of computation is required to obtain the target detection and feature extraction results, thus realizing the reuse of multi-scale features. Compared with the original target detection network, the additional amount of computation introduced by the target detection / feature extraction network based on multi-task learning is almost negligible. For high-level features, their discrimination of different targets in the same category is weak. Fusion of low-level features can effectively improve the discrimination of different targets. For scenarios with a large number of targets in the tracking area, this target detection network structure can effectively maintain the real-time performance of multi-target feature extraction.

[0037] S2, uses the Kalman filter algorithm to predict and output the motion feature information of each target.

[0038] It can be understood that the Kalman filter algorithm is used to predict the motion feature information of each target, that is, the motion feature information of the target in the next frame image is predicted based on the motion feature information of the target in the previous frame image, wherein the motion feature information of the target is related to the position information of the target detected in each frame image.

[0039] S3, combining the detection result information, depth feature information and motion feature information of each target to form a target feature descriptor.

[0040] It can be understood that multiple features such as the target's depth feature information, motion feature information, and detection result information are combined to form a target feature descriptor, and then the similarity of each target's multi-feature information is calculated to achieve target association matching and improve the accuracy of target association.

[0041] The target motion information features are predicted by the Kalman filter algorithm, and the predicted motion feature vector is output. The target category attribute information is obtained by inference through the target detection network. In order to achieve accurate association matching of multiple targets, it is necessary to design an effective association matching method for target feature descriptors.

[0042] S4, based on the target feature descriptor of each target in the video image to be identified and tracked, the similarity between each target and the existing targets is calculated to obtain the initial result of multi-target matching.

[0043] As an embodiment, the method calculates the similarity between each target and the existing target based on the target feature descriptor of each target in the to-be-identified tracking video image to obtain the initial result of multi-target matching, including: establishing a first similarity between the motion feature information of each target in the to-be-identified tracking video image and the motion feature information of the existing target, a second similarity between the depth feature information of each target and the depth feature information of the existing target, and a third similarity between the detection result information of each target and the detection result information of the existing target; based on the first similarity, the second similarity and the third similarity, establishing the similarity between the target feature descriptor of each target in the to-be-identified tracking video image and the target feature descriptor of the existing target; based on the established similarity between the target feature descriptor of each target in the to-be-identified tracking video image and the target feature descriptor of the existing target, calculating the similarity between each target and the existing target to obtain the initial result of multi-target matching.

[0044] Specifically, for the target motion feature information, the Mahalanobis distance between the Kalman filter prediction result of the motion state of the existing target and the target detection and recognition result in the tracking video image to be identified is used to associate the target motion feature information, and the expression of the correlation degree is:

[0045]

[0046] Among them, d j represents the position of the jth target detection box in the video image to be identified and tracked, y i represents the predicted position of the i-th existing target, S i represents the covariance matrix between the detected position and the existing target position, d 1 (i,j) represents the Mahalanobis distance between the i-th target and the j-th existing target.

[0047] The target's deep feature information is obtained by a fast target deep feature extraction method based on multi-task learning. In order to make the algorithm have a certain anti-occlusion ability, the target's deep feature information of nearly 50 frames is saved to form a deep feature vector set when obtaining the target's deep feature information. When calculating the similarity of deep feature information, the target deep feature information of the current frame (the video image to be identified and tracked) is associated with the deep feature information of the existing 50 frames of different targets, and the minimum cosine distance between the deep feature vector of the i-th existing target's historical 50 frames and the deep feature vector of the j-th target in the current frame is calculated. The formula is as follows:

[0048]

[0049] Among them, r k (i) represents the deep feature information of the kth frame of the i-th existing target, r j Indicates the depth feature information of the jth detected target in the video image to be identified and tracked. If the minimum cosine distance is less than the set threshold, the depth feature information of the i-th existing target is successfully associated with the depth feature information of the j-th detected target in the video image to be identified and tracked, otherwise, the association fails.

[0050] For the association of the target detection result information, the matching association is performed by comparing the category attribute information and confidence of each target in the video image to be identified and tracked with the existing target. When the categories are the same, the confidence similarity is calculated. The formula is as follows:

[0051] d 3 (i,j)=(1+αs i s j P(cls(i),cls(j)));

[0052]

[0053] Among them, α is the confidence scaling factor, s i s jis the confidence product of the i-th detected target in the to-be-identified tracking video image and the j-th target that already exists, cls(i) is the category attribute information of the i-th detected target in the to-be-identified tracking video image, and cls(j) is the category attribute information of the j-th target that already exists.

[0054] The similarity measurement formula of the multi-target feature descriptor is formed by integrating the three feature similarity measurement methods, as shown below:

[0055] c i,j =(λd 1 (i,j)+(1-λ)d 2 (i,j))·d 3 (i,j);

[0056] Among them, d 1 (i,j),d 2 (i,j),d 3 (i, j) are the first similarity, the second similarity and the third similarity respectively, λ is the weight coefficient, c i,j It is the similarity between the target feature descriptor of each target in the to-be-identified and tracked video image and the target feature descriptor of the existing target.

[0057] Based on the target feature descriptor measurement formula of multiple targets, the Hungarian algorithm and IOU are used to associate the targets and obtain the initial results of multi-target matching.

[0058] S5, optimizing the initial multi-target matching result based on a multi-layer cascade matching strategy to obtain final multi-target tracking result information.

[0059] Specifically, in the process of using the Hungarian algorithm and IOU (Intersection over Union) to associate the targets and obtain the initial results of multi-target matching. Among them, IoU is a standard for measuring the accuracy of detecting corresponding objects in a specific data set. IoU is a simple measurement standard. As long as a task with a predicted range (bounding box ex) in the output can be measured by IoU.

[0060] However, when the target is blocked for a long time, the probability mass is scattered in the target state space, the peak value of the observed likelihood is reduced, and the Kalman filter cannot accurately predict. At this time, when the two tracking trajectories are associated with the detection result of the current frame, the Mahalanobis distance calculated by the tracking trajectory with a longer blocking time is smaller, resulting in false association. In order to solve this problem, the present invention designs a multi-layer cascade matching strategy, which associates the current frame detection target according to the target category and the length of the tracking trajectory blocking time, and gives priority to the more frequently occurring targets of the same category. Specifically, by tracking and identifying multiple targets in a multi-frame video image, for the target whose blocking time exceeds the set threshold time, first identify and track other targets, and then identify and track the target whose blocking time exceeds the set threshold time. By optimizing the initial association result of multiple targets, the final multi-target tracking result information (the total number of targets in the video image to be identified and tracked, the ID sequence number of each target, and the trajectory of each target) is obtained.

[0061] Embodiment 2

[0062] A multi-target recognition and tracking method, see Figure 3 , the multi-target recognition and tracking method includes:

[0063] (1) For the input video sequence, read the image frame by frame and input it into the object detection network;

[0064] (2) The target detection network uses the YOLOv5s model, which can quickly and accurately detect all targets in the area of ​​interest and obtain the target category and location. The network structure is as follows: Figure 2 As shown;

[0065] (3) A fast target deep feature extraction method based on multi-task learning is used to extract target deep feature information;

[0066] (4) The Kalman filter algorithm is used to predict the motion feature vector of the output target, and the target's depth feature information, motion feature information, and detection result information are combined to form a target feature descriptor. The similarity of each target's multi-feature information (target feature descriptor) is calculated, and the Hungarian algorithm and IOU are used to associate the targets to obtain the initial results of multi-target matching.

[0067] (5) The multi-layer cascade matching strategy optimizes the initial association results of multiple targets to obtain the final multi-target tracking result information (number of targets, ID numbers, and trajectories).

[0068] Embodiment 3

[0069] A multi-target recognition and tracking system, see Figure 4The multi-target recognition and tracking system includes an acquisition module 401, a merging module 402, an initial matching module 403 and an optimized matching module 404, wherein:

[0070] The acquisition module 401 is used to input the video image to be identified and tracked into the target detection network, and obtain the detection result information and depth feature information of multiple targets output by the target detection network; and obtain the motion feature information of each target predicted and output by the Kalman filter algorithm; the merging module 402 is used to merge the detection result information, depth feature information and motion feature information of each target to form a target feature descriptor; the initial matching module 403 is used to calculate the similarity between each target and the existing target according to the target feature descriptor of each target in the video image to be identified and tracked, and obtain the initial result of multi-target matching; the optimization matching module 403 is used to optimize the initial result of multi-target matching based on the multi-layer cascade matching strategy to obtain the final multi-target tracking result information.

[0071] It can be understood that the multi-target recognition and tracking system provided by the present invention corresponds to the multi-target recognition and tracking method provided by the aforementioned embodiments. The relevant technical features of the multi-target recognition and tracking system can refer to the relevant technical features of the multi-target recognition and tracking method, which will not be repeated here.

[0072] Embodiment 4

[0073] See also Figure 5 , Figure 5 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 5 As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520 and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented: inputting the video image to be identified and tracked into a target detection network, obtaining detection result information and depth feature information of multiple targets output by the target detection network; using a Kalman filter algorithm to predict and output motion feature information of each target; merging the detection result information, depth feature information and motion feature information of each target to form a target feature descriptor; according to the target feature descriptor of each target in the video image to be identified and tracked, calculating the similarity between each target and the existing target, and obtaining a preliminary result of multi-target matching; optimizing the preliminary result of multi-target matching based on a multi-layer cascade matching strategy, and obtaining final multi-target tracking result information.

[0074] Embodiment 5

[0075] See also Figure 6 , Figure 6 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 6As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, the following steps are implemented: inputting the video image to be identified and tracked into the target detection network, and obtaining the detection result information and depth feature information of multiple targets output by the target detection network; using the Kalman filter algorithm to predict and output the motion feature information of each target; merging the detection result information, depth feature information and motion feature information of each target to form a target feature descriptor; according to the target feature descriptor of each target in the video image to be identified and tracked, calculating the similarity between each target and the existing target, and obtaining the initial result of multi-target matching; optimizing the initial result of multi-target matching based on the multi-layer cascade matching strategy, and obtaining the final multi-target tracking result information.

[0076] The multi-target recognition and tracking method and recognition and tracking system provided by the embodiment of the present invention have the following advantages compared with the prior art:

[0077] (1) In multi-target association matching, a feature descriptor composed of multiple features is introduced to make up for the lack of information in single feature discrimination and improve the accuracy of target matching;

[0078] (2) The intermediate features of the target detection network are improved by using a small amount of operation mapping to obtain deep feature information and the loss function of the multi-target detection network, without increasing the amount of additional calculations, thus ensuring the rate of multi-target tracking.

[0079] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0084] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0085] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-target recognition and tracking method, characterized in that: include: Inputting the video image to be identified and tracked into the target detection network, and obtaining detection result information and depth feature information of multiple targets output by the target detection network; The Kalman filter algorithm is used to predict and output the motion feature information of each target; The detection result information, depth feature information and motion feature information of each target are combined to form a target feature descriptor; According to the target feature descriptor of each target in the video image to be identified and tracked, the similarity between each target and the existing targets is calculated to obtain the initial result of multi-target matching; Optimizing the initial multi-target matching results based on a multi-layer cascade matching strategy to obtain final multi-target tracking result information; The target detection network includes a feature extraction subnetwork, a region pooling subnetwork and an output subnetwork; The feature extraction subnetwork is used to extract feature information of different scales of each target in the video image; The regional pooling subnetwork is used to fuse feature information of different scales of each target in the video image to generate multi-scale fused deep feature information of each target; The output subnetwork is used to output the detection result information of each target according to the multi-scale fusion depth feature information of each target, and the detection result information includes the category attribute information and location information of each target; Also includes: The loss function of the target detection network is set as: ； in, L detect represents the target detection and recognition error, λ is the loss function balance factor, L fea-measure is the newly added feature similarity error, and the calculation formula is as follows: in, F i and F j ∈R N Respectively represent i and j Multi-scale fusion deep feature information of each target.

2. The multi-target recognition and tracking method according to claim 1, characterized in that: The method of calculating the similarity between each target and an existing target based on the target feature descriptor of each target in the video image to be identified and tracked, and obtaining the initial result of multi-target matching includes: Establishing respectively a first similarity between motion feature information of each target in the to-be-identified tracking video image and motion feature information of an existing target, a second similarity between depth feature information of each target and depth feature information of an existing target, and a third similarity between detection result information of each target and detection result information of an existing target; Based on the first similarity, the second similarity and the third similarity, establishing the similarity between the target feature descriptor of each target in the to-be-identified tracking video image and the target feature descriptor of the existing target; Based on the similarity between the target feature descriptor of each target in the to-be-identified and tracked video image and the target feature descriptor of the existing target, the similarity between each target and the existing target is calculated to obtain the initial result of multi-target matching.

3. The multi-target recognition and tracking method according to claim 2, characterized in that: The step of establishing a first similarity between the motion feature information of each target in the to-be-identified and tracked video image and the motion feature information of the existing targets includes: According to the Mahalanobis distance between the motion feature information of the existing target and the detection result information of each target in the video image to be identified and tracked, the motion feature information of multiple targets is matched and associated: ； in, d j Indicates the first j The location of the target detection box, y i Indicates i The predicted positions of existing targets, S i represents the covariance matrix between the detected position and the existing target position, Represents the Mahalanobis distance between the i-th target and the j-th existing target.

4. The multi-target recognition and tracking method according to claim 2, characterized in that: The step of establishing a second similarity between the depth feature information of each target in the to-be-identified and tracked video image and the depth feature information of the existing target includes: The deep feature information of each target in the video image to be identified and tracked is associated with the historical 50 frames of deep feature information of different targets, and the first i The depth feature information of the existing target history 50 frames and the first j The minimum cosine distance between the depth feature information of the target is as follows: ； Among them, r k (i) Indicates i Target already exists k The deep feature information of the frame, r j Indicates the first j The deep feature information of the detected target.

5. The multi-target recognition and tracking method according to claim 2, characterized in that: The step of establishing a third similarity between the detection result information of each target in the to-be-identified tracking video image and the detection result information of the existing target includes: The matching association is performed by comparing the category attribute information and confidence of each target in the video image to be identified and tracked with the existing targets. When the categories are the same, the confidence similarity is calculated. The formula is as follows: ； ； in, α is the confidence scaling factor, s i s j is the first i detection target and the existing j The product of the confidence scores of the targets, cls ( i ) is the first i The category attribute information of the detected target, cls ( j ) is the existing j The category attribute information of each target.

6. The multi-target recognition and tracking method according to claim 2, characterized in that: The establishing, based on the first similarity, the second similarity and the third similarity, of the similarity between the target feature descriptor of each target in the to-be-identified tracking video image and the target feature descriptor of the existing target includes: ； in, , , are the first similarity, the second similarity and the third similarity respectively, λ is the weight coefficient, It is the similarity between the target feature descriptor of each target in the to-be-identified and tracked video image and the target feature descriptor of the existing target.

7. The multi-target recognition and tracking method according to claim 1, characterized in that: The multi-target matching initial result is optimized based on the multi-layer cascade matching strategy to obtain the final multi-target tracking result information, including: By tracking and identifying multiple targets in multiple frames of video images, for a target whose occlusion time exceeds a set threshold time, first identify and track other targets, and then identify and track the target whose occlusion time exceeds the set threshold time.

8. A multi-target recognition and tracking system, characterized in that: include: An acquisition module is used to input the video image to be identified and tracked into the target detection network, and obtain the detection result information and depth feature information of multiple targets output by the target detection network; and obtaining the motion feature information of each target predicted and output by the Kalman filter algorithm; A merging module, used to merge the detection result information, depth feature information and motion feature information of each target to form a target feature descriptor; The initial matching module is used to calculate the similarity between each target and the existing targets according to the target feature descriptor of each target in the video image to be identified and tracked, and obtain the initial result of multi-target matching; An optimization matching module is used to optimize the initial multi-target matching result based on a multi-layer cascade matching strategy to obtain final multi-target tracking result information; The target detection network includes a feature extraction subnetwork, a region pooling subnetwork and an output subnetwork; The feature extraction subnetwork is used to extract feature information of different scales of each target in the video image; The regional pooling subnetwork is used to fuse feature information of different scales of each target in the video image to generate multi-scale fused deep feature information of each target; The output subnetwork is used to output the detection result information of each target according to the multi-scale fusion depth feature information of each target, and the detection result information includes the category attribute information and location information of each target; Also includes: The loss function of the target detection network is set as: ； in, L detect represents the target detection and recognition error, λ is the loss function balance factor, L fea-measure is the newly added feature similarity error, and the calculation formula is as follows: in, F i and F j ∈R N Respectively represent i and j Multi-scale fusion deep feature information of each target.

Images