Multi-target tracking method based on two-stage appearance feature learning and related device

Through the two-stage appearance feature learning method, combined with single-frame and multi-frame feature learning models, the occlusion and clutter impact problems of multi-objective tracking in complex scenarios is solved, and the tracking accuracy and reliability are improved.

CN118942010APending Publication Date: 2024-11-12XI AN JIAOTONG UNIV

Patent Information

Application Number
CN202411033843.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing multi-objective tracking methods are susceptible to mutual occlusion and background clutter when dealing with dense crowds and complex scenes, resulting in unsatisfactory tracking results.

Method used

The two-stage appearance feature learning method is adopted to extract short-term appearance features through a single-frame feature learning model and perform short-term correlation. The long-term trajectory-level features are extracted in combination with the multi-frame feature learning model, and the short-term and long-term correlation results are integrated to achieve multi-objective tracking.

Benefits of technology

It improves the accuracy and reliability of multi-objective tracking in complex scenarios, can effectively handle occlusion and loss trajectories, and maintain the consistency of target identity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942010A_ABST
    Figure CN118942010A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-target tracking method based on two-stage appearance feature learning and a related device, and belongs to the technical field of computer vision. The method comprises the steps of obtaining a to-be-processed video and transmitting the to-be-processed video into a Transmitting a video image into a single-frame feature learning model, extracting appearance features of a target single-frame detection level, and matching the appearance features with target features of a previous frame to complete short-term association; inputting candidate trajectories and lost trajectories in the trajectory library into the trained multi-frame feature learning model, and extracting features of a target trajectory level so as to complete association between the trajectories; combining the results of the short-term association and the long-term association to obtain a final tracking result; according to the multi-target tracking method or system provided by the invention, a relatively accurate tracking result can be obtained under the special condition of crowding or serious shielding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a multi-target tracking method based on two-stage appearance feature learning and a related device. Background Art

[0002] Multi-target tracking is a very important task in the field of computer vision. It is the core technology in many other fields, such as autonomous driving, behavior recognition, intelligent monitoring, and medical impact analysis. Its task is to locate each target in a video sequence and identify the continuous identity of each target.

[0003] The purpose of multi-target tracking is to locate the target and keep each target consistent throughout the video sequence. Previous methods usually learn a single discriminative feature representation to associate detections across frames, such as motion and appearance features. However, this is easily affected by mutual occlusion between targets and complex scenes in practical applications, resulting in unsatisfactory tracking results in dense crowd scenes. In summary, a new and more robust multi-target tracking method and system is urgently needed. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-target tracking method and related devices based on two-stage appearance feature learning, so as to solve the technical problem that the appearance feature learning method in the prior art is easily affected by mutual occlusion and background clutter during multi-target tracking.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention discloses a multi-target tracking method based on two-stage appearance feature learning, comprising the following steps:

[0007] Obtaining video image data to be processed;

[0008] The video image data to be processed is input into a pre-trained single-frame feature learning model to extract the short-term appearance features of the target, perform short-term association on the short-term appearance features of the target to obtain short-term association results, and update the candidate trajectory library and the lost trajectory library;

[0009] The candidate trajectories and lost trajectories in the candidate trajectory library and the lost trajectory library are input into the pre-trained multi-frame feature learning model to extract the long-term trajectory level features of the target, and the long-term trajectory level features of the target are subjected to long-term association to obtain the long-term association results;

[0010] Integrate short-term association results and long-term association results to achieve multi-target tracking.

[0011] Furthermore, the steps of inputting the to-be-processed video image data into a pre-trained single-frame feature learning model, extracting the short-term appearance features of the target, performing short-term association on the short-term appearance features of the target to obtain short-term association results, and updating the candidate trajectory library and the lost trajectory library specifically include:

[0012] Detect and locate each target in the video image to obtain the target detection result;

[0013] Obtain the short-term appearance features of each target based on the original image and target detection results in the video;

[0014] Construct the correlation between the short-term appearance features of each target to achieve short-term association. The specific formula is:

[0015]

[0016] Among them, S short is the short-term correlation matrix of trajectories and detections; is the short-term feature of the jth trajectory; is the short-term feature of the i-th detection;

[0017] Update the candidate trajectory library and lost trajectory library based on historical trajectories and short-term association results.

[0018] Furthermore, the step of obtaining the short-term appearance features of each target according to the original image and the target detection result in the video specifically includes:

[0019] Extract the original feature map of the image from the detection network, and get the feature pyramid as

[0020] By interactively enhancing the feature pyramid obtained from The target discriminative feature map is obtained from

[0021]

[0022] Among them, ⊕ represents the splicing operation; represents the flattening operation; ψ k (.) represents a set of 1×1 convolution operations;

[0023] According to the target position, the discriminative feature map of the target is obtained. In the above formula, short-term appearance features are extracted for each track and detection. The specific formula is:

[0024]

[0025] Among them, RoIAlign is the sampling function; is the short-term feature of the jth trajectory; is the short-term feature of the i-th detection.

[0026] Furthermore, the step of updating the candidate trajectory library and the lost trajectory library according to the historical trajectory and the short-term association result specifically includes: the detection without association is initialized as a new trajectory and enters the candidate trajectory library as a candidate trajectory; the trajectory without matching is entered as a lost trajectory and enters the lost trajectory library.

[0027] Furthermore, the training method of the single-frame feature learning model comprises the following steps:

[0028] Calculate the loss function of the single-frame feature learning model during forward inference. The specific formula is:

[0029] L total =L inter +λ1L memo +λ2L inner

[0030] Among them, L total is the total loss function; L inter is the inter-frame loss function; L memo is the memory loss function; L inner is the intra-frame loss function; λ1 represents the memory loss weight; λ2 represents the intra-frame loss weight;

[0031] Optimize network parameters; use Adam optimizer and train the single-frame feature learning model for 80 epochs.

[0032] Furthermore, the step of inputting the candidate trajectories and lost trajectories in the candidate trajectory library and the lost trajectory library into a pre-trained multi-frame feature learning model, extracting the long-term trajectory level features of the target, and performing long-term association on the long-term trajectory level features of the target to obtain a long-term association result specifically includes:

[0033] The long-term trajectory level features of the candidate trajectory are extracted from the candidate trajectory library. The specific formula is:

[0034]

[0035] in, is the long-term trajectory-level feature of the candidate trajectory; W cadi is the weight for fusing temporal features of candidate trajectories; is the feature map corresponding to the candidate trajectory; φ(.) represents 3 attention blocks;

[0036] The long-term trajectory-level features of the lost trajectory are extracted from the lost trajectory library. The specific formula is:

[0037]

[0038] in is the long-term trajectory-level feature of the lost trajectory; W lost is the weight for temporal feature fusion of lost trajectories; is the feature map corresponding to the lost trajectory;

[0039] Construct the correlation between the long-term trajectory-level features of the candidate trajectory and the long-term trajectory-level features of the lost trajectory to complete the long-term association of the long-term trajectory-level features of the target. The specific formula is:

[0040]

[0041] Among them, S long is the similarity between the long-term trajectory-level features of the candidate trajectory and the long-term trajectory-level features of the lost trajectory.

[0042] Furthermore, the training method of the multi-frame feature learning model comprises the following steps:

[0043] Calculate the loss function of the multi-frame feature learning model during forward inference. The specific formula is:

[0044]

[0045] Among them, L asso is the cross entropy loss; n is the total number of samples; y i is the true label of the sample; s i is the predicted output of the sample;

[0046] Optimize network parameters; use Adam optimizer and train the multi-frame feature learning model for 50 epochs.

[0047] In a second aspect, the present invention discloses a multi-target tracking system based on two-stage appearance feature learning, comprising:

[0048] A data acquisition module, used for acquiring video image data to be processed;

[0049] The single-frame feature learning module is used to input the video image data to be processed into a pre-trained single-frame feature learning model, extract the short-term appearance features of the target, perform short-term association on the short-term appearance features of the target to obtain short-term association results, and update the candidate trajectory library and the lost trajectory library;

[0050] The multi-frame feature learning module is used to input the candidate trajectories and lost trajectories in the candidate trajectory library and the lost trajectory library into the pre-trained multi-frame feature learning model, extract the long-term trajectory level features of the target, and perform long-term association on the long-term trajectory level features of the target to obtain a long-term association result;

[0051] The integration module is used to integrate the short-term association results and the long-term association results to achieve multi-target tracking.

[0052] In a third aspect, the present invention discloses a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0053] In a fourth aspect, the present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention discloses a multi-target tracking method and related devices based on two-stage appearance feature learning. A simple and effective two-stage feature learning paradigm is adopted to respectively learn the single-frame detection level features and multi-frame trajectory level features of different targets. For unrelated detections, the present invention designs a new single-frame feature learning model to extract the discriminant features of each detection, which can effectively associate the targets to adjacent frames. For the trajectories that have lost several frames, the present invention designs a multi-frame feature learning model to extract the discriminant features of each trajectory, which can accurately relocate these lost targets after a long time, so that the multi-target tracking technology can have higher accuracy when processing more complex scenes, and improve the reliability when used in the corresponding scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 is a flow chart of the method of the present invention;

[0058] Figure 2 is a schematic diagram of the system of the present invention;

[0059] Figure 3 This is a schematic diagram of the overall tracking framework flow of an embodiment of the present invention;

[0060] Figure 4 It is a structural block diagram of a multi-target tracking method based on two-stage appearance feature learning according to an embodiment of the present invention;

[0061] Figure 5This is a structural diagram of a single-frame feature learning module according to an embodiment of the present invention;

[0062] Figure 6 It is a structural diagram of a multi-frame feature learning module according to an embodiment of the present invention;

[0063] Figure 7 A schematic diagram of the tracking capability of an embodiment of the present invention in various complex scenarios;

[0064] Figure 8 It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0066] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0068] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0069] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0070] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0071] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0072] See also Figure 1 and Figure 3 The embodiment of the present invention discloses a multi-target tracking method based on two-stage appearance feature learning, comprising the following steps:

[0073] Step 1, obtaining the video image data to be processed, specifically includes:

[0074] Video data is generally exported from fixed cameras, vehicle-mounted cameras, or certain application scenarios.

[0075] Step 2: Input the video image data to be processed into a pre-trained single-frame feature learning model, extract the short-term appearance features of the target, perform short-term association on the short-term appearance features of the target to obtain short-term association results, and update the candidate trajectory library and the lost trajectory library;

[0076] Step 2.1, target detection is performed on the current frame image, and the target detection result is used for the following data association. The present invention uses the open source target detection model YOLOX to first detect all targets in the scene, and the result is recorded as The detected target location and category are then passed to the tracker for subsequent data association steps.

[0077] Step 2.2, obtaining the short-term appearance features of each target based on the original image and the target detection results;

[0078] 1) Extract the original feature map of the image from the detection network.

[0079] We record the feature pyramid extracted from the YOLOX backbone network as

[0080] 2) Enhance the feature pyramid obtained from the previous step through interaction The feature map with target discriminativeness is obtained

[0081]

[0082] Where ⊕ represents the concatenation operation, represents the flattening operation, and ψ k (.) represents a set of 1×1 convolution operations.

[0083] 3) According to the target position, the discriminative feature map of the target is obtained. Short-term appearance features are extracted for each tracklet and detection.

[0084]

[0085]

[0086] Among them, RoIAlign is a sampling function proposed by FasterRcnn. and represent the short-term features of the j-th trajectory and the i-th detection, respectively.

[0087] Step 2.3, construct the correlation between short-term features and realize short-term association based on this relationship.

[0088]

[0089] Where S short is the short-term correlation matrix between trajectory and detection, whose value represents the similarity between the short-term features of trajectory and detection. Based on this, we complete the short-term association between detection and trajectory through Hungarian matching.

[0090] Step 2.4: Update the candidate trajectory library and the lost trajectory library according to the historical trajectory and short-term association results.

[0091] Specifically, the detection that is not associated in step 2.3 is initialized as a new track and enters the candidate track library as a candidate track; the track that is not matched is considered a lost track and enters the lost track library.

[0092] Step 3: Input the candidate trajectories and lost trajectories in the candidate trajectory library and the lost trajectory library into a pre-trained multi-frame feature learning model to extract the long-term trajectory level features of the target, and perform long-term association on the long-term trajectory level features of the target to obtain a long-term association result;

[0093] Step 3.1, from the multi-frame feature map of the trajectory and Long-term trajectory-level features are extracted from trajectories in two trajectory libraries.

[0094] Specifically, the trajectories in the two trajectory libraries are input into the multi-frame feature extraction module to extract multi-frame trajectory-level features for each trajectory.

[0095]

[0096] Where φ(.) represents 3 attention blocks, and are the long-term trajectory-level features of the lost trajectory and candidate trajectory, respectively.

[0097] Step 3.2: Build the correlation between the long-term features of the two trajectories and complete the association at the long-term trajectory level.

[0098]

[0099] Where S long is the similarity between long-term trajectory level features, S long Each element of is a correlation score, which represents whether the corresponding candidate track and the lost track belong to the same target. Based on this, we complete the association between the candidate track and the lost track through Hungarian matching.

[0100] Step 4: Integrate the short-term correlation results and the long-term correlation results to achieve a complete tracking step.

[0101] In a feasible implementation manner of the present invention, the training steps of the single-frame feature learning model and the multi-frame feature learning model are:

[0102] 1) Calculate the loss function of the model during forward inference:

[0103] We use the weighted sum of the following losses as the loss function for the single-frame feature learning model during training:

[0104] L total =L inter +λ1L memo +λ2L inner

[0105] For the multi-frame feature learning model, we use cross entropy loss as the final loss function as follows:

[0106]

[0107] 2) Optimize network parameters

[0108] Using the Adam optimizer, the training iterations of the single-frame feature learning model are 80 epochs, and the training iterations of the multi-frame feature learning model are 50 epochs. During training, the learning rate slowly decays with the number of iterations.

[0109] The embodiment of the present invention provides a multi-target tracking method based on two-stage appearance feature learning, which addresses the problem that existing appearance feature learning methods are easily affected by mutual occlusion and background clutter. A simple and effective two-stage feature learning paradigm is proposed to learn the single-frame detection level features and multi-frame trajectory level features of different targets respectively. The framework includes a new single-frame feature learning model to extract the discriminant features of each detection, which can effectively associate targets to adjacent frames. For trajectories that have lost several frames, we designed a multi-frame feature learning model to extract the discriminant features of each trajectory, which can accurately relocate these lost targets after a long time, so that the multi-target tracking technology can have higher accuracy when dealing with more complex scenes, and improve the reliability when used in the corresponding fields.

[0110] See also Figure 2 and Figure 4 The embodiment of the present invention discloses a multi-target tracking system based on two-stage appearance feature learning, comprising:

[0111] The data acquisition module acquires the video image data to be processed; generally, the video data is exported from a fixed camera, a vehicle-mounted camera, or certain application scenarios.

[0112] The single-frame feature learning module performs target detection on the current frame image, and the target detection results are used for the following data association. The present invention uses the open source target detection model YOLOX to first detect all targets in the scene, and the results are recorded as After that, the detected target location and category information is passed to the tracker for subsequent data association steps. Then, the short-term appearance features of each target are obtained based on the original image and the target detection results; the correlation between short-term features is constructed, and short-term association is achieved based on this relationship.

[0113]

[0114] Where S short It is the short-term correlation matrix of trajectory and detection, and its value represents the short-term characteristics of trajectory and detection. Finally, the candidate trajectory library and lost trajectory library are updated according to the historical trajectory and short-term association results.

[0115] Multi-frame feature learning module, from multi-frame feature maps of multiple frames and The long-term trajectory level features of the trajectories in the two trajectory libraries are extracted. The correlation between the long-term features of the two trajectories is constructed to complete the association at the long-term trajectory level.

[0116]

[0117] Where S longis the similarity between long-term trajectory level features, S long Each element of is a correlation score, which represents whether the corresponding candidate track and the lost track belong to the same target. Based on this, we complete the association between the candidate track and the lost track through Hungarian matching.

[0118] The integration module is used to integrate the short-term association results and the long-term association results to achieve multi-target tracking.

[0119] The embodiment of the present invention provides a more general multi-target tracking method with powerful tracking capabilities in complex scenes with frequent occlusions and dense crowds. In view of the problems of identity recognition errors and trajectory interruptions caused by crowded crowds and severe occlusions in existing tracking methods, a simple and effective online tracking framework is proposed, which integrates a short-term inter-frame feature interaction module (see Figure 5 ) and the long-term trajectory association module (see Figure 6 ) is used to extract "interactive motion features" and "historical trajectory motion features" to guide the short-term and long-term associations during multi-target tracking, respectively. This not only makes the motion features of each target more distinguishable in a crowded crowd, thereby enabling the tracker to have a stronger identity retention capability, but also enables the target to be retrieved after its trajectory is interrupted due to severe occlusion, thereby comprehensively improving the frame-by-frame identity recognition capability and cross-frame identity re-recognition capability during tracking.

[0120] See also Figure 7 The embodiments of the present invention provide a comparison of the tracking results of the method of the present invention and the existing representative methods in some scenarios. It can be clearly observed that the method proposed by us has better tracking performance, maintains the consistency of the target identity when facing dense crowds, and effectively retrieves the target lost due to long-term occlusion.

[0121] In one embodiment of the present invention, see Figure 8, a computer device is provided, the computer device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in a computer storage medium to implement a corresponding method flow or a corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a multi-target tracking method based on two-stage appearance feature learning.

[0122] The present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, an extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multi-target tracking method based on two-stage appearance feature learning in the above embodiment.

[0123] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-target tracking method based on two-stage appearance feature learning, characterized in that: The following steps are involved: Obtaining video image data to be processed; Input the video image data to be processed into a pre-trained single-frame feature learning model, extract the short-term appearance features of the target, perform short-term association on the short-term appearance features of the target to obtain short-term association results, and update the candidate trajectory library and the lost trajectory library; The candidate trajectories and lost trajectories in the candidate trajectory library and the lost trajectory library are input into the pre-trained multi-frame feature learning model to extract the long-term trajectory level features of the target, and the long-term trajectory level features of the target are subjected to long-term association to obtain the long-term association results; Integrate short-term association results and long-term association results to achieve multi-target tracking.

2. A multi-target tracking method based on two-stage appearance feature learning according to claim 1, characterized in that: The steps of inputting the video image data to be processed into a pre-trained single-frame feature learning model, extracting the short-term appearance features of the target, performing short-term association on the short-term appearance features of the target to obtain short-term association results, and updating the candidate trajectory library and the lost trajectory library specifically include: Detect and locate each target in the video image to obtain the target detection result; Obtain the short-term appearance features of each target based on the original image and target detection results in the video; Construct the correlation between the short-term appearance features of each target to achieve short-term association. The specific formula is: Among them, S short is the short-term correlation matrix of trajectories and detections; is the short-term feature of the jth trajectory; is the short-term feature of the i-th detection; Update the candidate trajectory library and lost trajectory library based on historical trajectories and short-term association results.

3. The multi-target tracking method based on two-stage appearance feature learning according to claim 2 is characterized in that: The step of obtaining the short-term appearance features of each target according to the original image and the target detection result in the video specifically includes: Extract the original feature map of the image from the detection network, and obtain the feature pyramid as By interactively enhancing the feature pyramid obtained from The target discriminative feature map is obtained from in, Represents a splicing operation; represents the flattening operation; ψ k (.) represents a set of 1×1 convolution operations; According to the target position, the discriminative feature map of the target is obtained. In the above formula, short-term appearance features are extracted for each track and detection. The specific formula is: Among them, RoIAlign is the sampling function; is the short-term feature of the jth trajectory; is the short-term feature of the i-th detection.

4. The multi-target tracking method based on two-stage appearance feature learning according to claim 2 is characterized in that: The step of updating the candidate trajectory library and the lost trajectory library according to the historical trajectory and the short-term association result specifically includes: the detection without association is initialized as a new trajectory and enters the candidate trajectory library as a candidate trajectory; the trajectory without matching is entered as a lost trajectory into the lost trajectory library.

5. The multi-target tracking method based on two-stage appearance feature learning according to claim 1, characterized in that: The training method of the single-frame feature learning model comprises the following steps: Calculate the loss function of the single-frame feature learning model during forward inference. The specific formula is: L total =L inter +λ1L memo +λ2L inner Among them, L total is the total loss function; L inter is the inter-frame loss function; L memo is the memory loss function; L inner is the intra-frame loss function; λ1 represents the memory loss weight; λ2 represents the intra-frame loss weight; Optimize network parameters; use Adam optimizer and train the single-frame feature learning model for 80 epochs.

6. The multi-target tracking method based on two-stage appearance feature learning according to claim 1, characterized in that: The step of inputting the candidate trajectories and lost trajectories in the candidate trajectory library and the lost trajectory library into a pre-trained multi-frame feature learning model, extracting the long-term trajectory level features of the target, and performing long-term association on the long-term trajectory level features of the target to obtain a long-term association result specifically includes: The long-term trajectory level features of the candidate trajectory are extracted from the candidate trajectory library. The specific formula is: in, is the long-term trajectory-level feature of the candidate trajectory; W cadi is the weight for fusing temporal features of candidate trajectories; is the feature map corresponding to the candidate trajectory; φ(.) represents 3 attention blocks; The long-term trajectory-level features of the lost trajectory are extracted from the lost trajectory library. The specific formula is: in is the long-term trajectory-level feature of the lost trajectory; W lost is the weight for temporal feature fusion of lost trajectories; is the feature map corresponding to the lost trajectory; Construct the correlation between the long-term trajectory-level features of the candidate trajectory and the long-term trajectory-level features of the lost trajectory to complete the long-term association of the long-term trajectory-level features of the target. The specific formula is: Among them, S long is the similarity between the long-term trajectory-level features of the candidate trajectory and the long-term trajectory-level features of the lost trajectory.

7. The multi-target tracking method based on two-stage appearance feature learning according to claim 1, characterized in that: The training method of the multi-frame feature learning model comprises the following steps: Calculate the loss function of the multi-frame feature learning model during forward inference. The specific formula is: Among them, L asso is the cross entropy loss; n is the total number of samples; y i is the true label of the sample; s i is the predicted output of the sample; Optimize network parameters; use Adam optimizer and train the multi-frame feature learning model for 50 epochs.

8. A multi-target tracking system based on two-stage appearance feature learning, characterized in that: include: A data acquisition module, used for acquiring video image data to be processed; The single-frame feature learning module is used to input the video image data to be processed into a pre-trained single-frame feature learning model, extract the short-term appearance features of the target, perform short-term association on the short-term appearance features of the target to obtain short-term association results, and update the candidate trajectory library and the lost trajectory library; The multi-frame feature learning module is used to input the candidate trajectories and lost trajectories in the candidate trajectory library and the lost trajectory library into the pre-trained multi-frame feature learning model, extract the long-term trajectory level features of the target, and perform long-term association on the long-term trajectory level features of the target to obtain a long-term association result; The integration module is used to integrate the short-term association results and the long-term association results to achieve multi-target tracking.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Multi-target tracking method and device, computer and storage medium

    CN113409361A

  • Multi-target tracking method, system and device and readable storage medium

    CN116485838A

Cited By

  • Target identifying, tracking and guiding method for photoelectric pod, detection equipment, program product and storage medium

    CN119672325A