A multi-target tracking method and system based on optical flow information and identity prediction

Abstract

This invention discloses a multi-target tracking method and system based on optical flow information and identity prediction. The method includes the following steps: acquiring the target to be tracked and the original image, performing preprocessing to obtain a reference image labeled with the target; extracting optical flow feature maps and cropping the optical flow features of the target to be tracked; stitching and fusing the previous frame image with the current frame image to generate an optical flow-guided feature-enhanced image; cropping the image features of the target to be tracked, inputting them into a feature extraction network, and outputting the target image features; stitching the optical flow features and target image features of the target to be tracked to obtain the target features of the target to be tracked, and stitching them with the identity of the target to be tracked to obtain the tracking target with the identity to be assigned; inputting the data into a multi-target tracking model for prediction to obtain the prediction result of the tracking target. This invention utilizes the optical flow features and image features of the target for identity prediction, realizing multi-target tracking under occlusion and nonlinear motion.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a multi-target tracking method based on optical flow information and identity prediction. Background Technology

[0002] Multi-target tracking is a crucial problem in image processing and computer vision, aiming to identify and track multiple targets from a video sequence. This technology is widely used in fields such as autonomous driving and intelligent surveillance. The task involves assigning unique identities to newly appearing objects in a video sequence and then assigning identities to targets detected in subsequent frames, enabling long-term tracking of the targets.

[0003] Most current popular detection-based multi-target tracking methods utilize Kalman filtering to estimate target motion and generate a cost matrix based on the estimation results for matching the target's identity with the detected object. However, in practical multi-target tracking, target motion is easily occluded, and its complex and often nonlinear patterns make it difficult for Kalman filters, which are based on linear motion assumptions, to accurately predict target motion in most cases. This leads to interference with target matching information and unsatisfactory multi-target tracking results.

[0004] Deep learning-based multi-target tracking methods have been proposed. These algorithms use deep learning techniques to extract historical features of targets or to predict targets, thus assisting in solving the identity assignment problem.

[0005] Therefore, how to effectively extract features and improve the accuracy of identity allocation is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a multi-target tracking method and system based on optical flow information and identity prediction. By extracting optical flow information to assist in image feature enhancement, and using the optical flow features and image features of the target for identity prediction, the method effectively solves the multi-target tracking problem under occlusion and nonlinear motion.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, this invention proposes a multi-target tracking method based on optical flow information and identity prediction, comprising the following steps:

[0009] S1. Acquire the target to be tracked and the original image, perform preprocessing, and obtain a reference image labeled with the target to be tracked;

[0010] S2. Use a lightweight deep optical flow network to extract the optical flow feature map of the current frame in the reference image compared to the previous frame, and crop the optical flow information near the center point of the target to be tracked as the optical flow feature of the target to be tracked.

[0011] S3. Using the optical flow feature map, the previous frame image and the current frame image are stitched and fused to generate an optical flow-guided feature-enhanced image;

[0012] S4. Crop the feature-enhanced image within the bounding box of the target to be tracked as the image feature of the target to be tracked, input the image feature into the feature extraction network, and output the target image feature;

[0013] S5. The optical flow features and target image features of the target to be tracked are spliced ​​together to obtain the target features of the target to be tracked, and then spliced ​​together with the identity of the target to be tracked to obtain the tracking target to be assigned an identity;

[0014] S6. Input the tracking target whose identity is to be assigned into the multi-target tracking model for prediction to obtain the prediction result of the tracking target.

[0015] Secondly, this invention proposes a multi-target tracking system based on optical flow information and identity prediction, comprising the following modules:

[0016] Image acquisition module: used to acquire the target to be tracked and the original image, perform preprocessing, and obtain a reference image labeled with the target to be tracked;

[0017] Optical flow feature extraction module: used to extract the optical flow feature map of the current frame in the reference image compared with the previous frame using a lightweight deep optical flow network, and to crop the optical flow information near the center point of the target to be tracked as the optical flow feature of the target to be tracked;

[0018] Feature enhancement module: used to use the optical flow feature map to stitch and fuse the previous frame image with the current frame image to generate an optical flow-guided feature enhancement image;

[0019] Target image feature extraction module: used to crop the feature-enhanced image within the bounding box of the target to be tracked as the image features of the target to be tracked, input the image features into the feature extraction network, and output the target image features;

[0020] Tracking target stitching module: used to stitch together the optical flow features and target image features of the target to be tracked to obtain the target features of the target to be tracked, and stitch them together with the identity of the target to be tracked to obtain the tracking target to be assigned an identity;

[0021] Tracking target prediction module: used to input the tracking target whose identity is to be assigned into the multi-target tracking model for prediction, and obtain the prediction result of the tracking target.

[0022] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a multi-target tracking method and system based on optical flow information and identity prediction, which has the following beneficial effects:

[0023] 1. This invention combines optical flow features and image features to extract target features, achieving information enhancement and multi-angle feature extraction.

[0024] 2. By combining identity information with the target to be tracked, and utilizing cross-attention and self-attention mechanisms, the target characteristics of historically tracked objects can be better captured, thereby achieving accurate multi-target tracking. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 A flowchart of a multi-target tracking method based on optical flow information and identity prediction is provided for an embodiment of the present invention.

[0027] Figure 2 This is a diagram illustrating the architecture of an optical flow-based image feature enhancement module provided in an embodiment of the present invention.

[0028] Figure 3 This is an architecture diagram of a multi-target tracking method based on optical flow information and identity prediction provided in an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of a multi-target tracking system based on optical flow information and identity prediction, provided as an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example 1

[0032] This invention discloses a multi-target tracking method based on optical flow information and identity prediction, referring to... Figure 1 As shown, it includes the following steps S1 to S6:

[0033] S1. Acquire the target to be tracked and the original image, perform preprocessing, and obtain a reference image labeled with the target to be tracked;

[0034] In this embodiment, successfully detected objects and the original image are acquired. Each frame's successfully detected objects are used as targets to be tracked. Image processing software or libraries are used to uniformly adjust the image and the detected target bounding boxes to the same scaling scale. Specifically, for excessively large images, their side lengths are limited. The detection bounding boxes are scaled according to the image scaling result to ensure accurate selection of the target to be tracked. This step ensures the consistency of subsequent processing and the effectiveness of feature extraction, while also guaranteeing training efficiency.

[0035] S2. Use a lightweight deep optical flow network to extract the optical flow feature map of the current frame compared to the previous frame in the reference image, and crop the optical flow information near the center point of the target to be tracked as the optical flow feature of the target to be tracked.

[0036] In this embodiment, a lightweight pre-trained optical flow extraction network is used. The optical flow extraction network automatically estimates the optical flow field from consecutive video frames through a convolutional neural network. It takes two frames of images as input, learns motion displacement through training, and optimizes the prediction results using a loss function. After training, it can quickly infer optical flow information from two frames of images.

[0037] This lightweight deep optical flow network is used to quickly and efficiently extract optical flow information from the current frame compared to the previous frame, and obtain the optical flow feature map Flow. t Based on the reference image labeled with the target to be tracked, the center position of the target is found. A 3x3 optical flow feature map near the target's center point is cropped, flattened, and then stitched together with the center point coordinates to form the optical flow feature Fcrop. t .

[0038] S3. Using the optical flow feature map, the previous frame image and the current frame image are stitched and fused to generate an optical flow-guided feature-enhanced image;

[0039] In this embodiment, refer to Figure 2 As shown, using optical flow feature maps... t For the previous frame image I t-1 Bilinear sampling is performed to obtain the sampled optical flow characteristics. The formula is expressed as:

[0040] F s (i,j)=BS(I t-1 ,i,j,dx,dy)

[0041] BS(I,i,j,dx,dy)=s x s y I(x,y)+(1-s x )s yI(x+1,y)

[0042] +s x (1-s y )I(x,y+1)+(1-s x (1-s) y )I(x+1,y+1)

[0043] s x = i + dx - [i + dx]

[0044] s y =j+dy-[j+dy]

[0045] BS(I,i,j,dx,dy) is a bilinear sampling operation of image I at position (i,j) with optical flow size (dx,dy), where dx and dy are the x-axis and y-axis offsets of the optical flow at position (i,j), I(x,y) represents the image pixel value at (x,y), x represents [i+dx], y represents [j+dy], and [.] represents the floor function. x ,s y It is an interpolation within the range (0,1).

[0046] stitching optical flow features and the current frame image I along the channel dimension t The concatenated features are fed into a 3x3 convolution to fuse the optical flow features and the current feature image, and the number of output channels is limited to 3, outputting a fused image F with optical flow-guided feature enhancement. fuse Similarly, this feature is fed into a 1x1 convolution with the number of channels limited to 1, and then normalized using a Sigmoid network layer to generate the attention weight F. att Weights and fused image F fuse The result of multiplication and the current image I t The images are added together to obtain the optical flow-guided feature-enhanced image I. aug .

[0047] The formula is expressed as:

[0048] I aug =F fuse *F att +I t

[0049] S4. Crop the feature-enhanced image within the bounding box of the target to be tracked as the image feature of the target to be tracked, input the image feature into the feature extraction network, and output the target image feature;

[0050] In this embodiment, based on the target to be tracked, a detection bounding box of the target is obtained, and a feature enhancement image I within the target bounding box is generated. augThe image is cropped and used as the current frame image feature of the target. Image processing software or libraries are used to uniformly adjust the current frame image feature of the detected target to 32x32 before feeding it into a convolutional neural network module. After being processed by three convolutional modules, the feature is flattened and then fed into a fully connected network to obtain a feature vector of size 256, which is used as the target image feature F. app .

[0051] S5. Combine the optical flow features and target image features of the target to be tracked to obtain the target features of the target to be tracked, and then combine them with the identity of the target to be tracked to obtain the tracking target with the identity to be assigned.

[0052] In this embodiment, refer to Figure 3 As shown, for each target to be tracked, a learnable feature vector is used as the identity feature of the new object. The identity feature is generated by using a one-hot encoded fully connected network to generate a feature vector of size 256. The target image features F generated in steps S2 and S4 for each target to be tracked are then combined. app and optical flow characteristics Fcrop t The data is concatenated along the size dimension to form the target feature of the target to be tracked. Similarly, the identity feature and target feature of the newly created object are concatenated along the size dimension to form the target information for which identity needs to be assigned. All target information is then concatenated along the channel dimension to form the feature vector F of all targets to be tracked. obj The formula is expressed as:

[0053] F id =f id (newborn)

[0054] F obj =concat(F id Fcrop t ,F app )

[0055] Where, newbqrn is the pre-defined one-hot encoded identity identifier of the newly created object, F id The identity characteristics of new subjects.

[0056] S6. Input the tracking target whose identity is to be assigned into the multi-target tracking model for prediction, and obtain the prediction result of the tracking target.

[0057] In this embodiment, optical flow extraction module and image feature enhancement module extract optical flow motion information and image feature information as target features, respectively. The concatenated identity vector serves as a guide for identity prediction. The vector is then fed into the backbone prediction model and uses an attention mechanism to output a target identity vector and feature vector that fuse historical information, thereby achieving identity prediction.

[0058] In this embodiment, the multi-target tracking model is composed of six cross-attention modules and five self-attention modules alternately. The tracking target to be assigned an identity is used as the query part input to the model. For the cross-attention module, the identity features and target features of the tracking results of the previous T frames are concatenated as historical tracking information, and the historical tracking information is used as the key and value input.

[0059] For the cross-attention part, the formula is expressed as:

[0060]

[0061] For the self-attention component, the formula is expressed as:

[0062]

[0063] in, Let F represent the learnable weight matrix. c and F s Q represents the feature vectors generated by cross-attention and self-attention, respectively. obj K represents a query for the object to be tracked. his and V his This represents the key and value corresponding to the historical object, and K. obj and V obj These represent the key and value of the object to be tracked for self-attention, respectively.

[0064] For multi-target tracking models, six cross-attention modules and five self-attention modules Passing F obj and historical information F his The formula for the multi-target tracking model is expressed as:

[0065]

[0066]

[0067] Where F i For the intermediate output, F out This is the output vector of the multi-target tracking model.

[0068] The training process of the model in this embodiment will be described in detail below.

[0069] Reference Figure 3 As shown, Γ t-T Γ represents the historical information of the multi-target tracking model passed in before frame T. t-1This represents the historical information passed to the multi-target tracking model in the previous frame. First, for the historical information part, an identity dictionary is preset based on the tracking targets with assigned identities in the historical T frames. For each historical target with a successfully assigned identity, a learnable 256-dimensional feature vector is mapped to its identity identifier, serving as the identity feature vector of the successfully assigned target. The target information of the tracking targets with assigned identities in the historical T frames is concatenated with the corresponding identity feature vector, resulting in a 512-dimensional vector. For different periods in the historical T frames, T learnable feature vectors of size 512 are generated and added to the historical information of the T frames respectively, serving as the embedding of time information.

[0070] Secondly, the feature vector generated through the final layer of cross-attention is used as the feature vector of the tracking target combined with historical information. The generated feature vector is then split: the first 256 dimensions are used as identity information input to the identity prediction model; the middle 256 dimensions are used as image features input to the image feature extraction model; and the remaining features are used as optical flow features input to the motion prediction model. All three models are composed of fully connected neural networks. The formula is expressed as:

[0071] (F ID ,F app ,F mv ) = split(Γ t )

[0072] R ID =f ID (F ID )

[0073] R app =f app (F app )

[0074] R mv =f mv (F mv )

[0075] Among them, Γ t f represents the feature vector output by the multi-target tracking model. ID f app f mv These represent the identity prediction output head, image feature output head, and motion prediction output head, respectively; R ID R app R mv These are the corresponding identity prediction results, appearance feature results, and displacement prediction results.

[0076] Finally, using an improved loss function, the results of identity prediction, image feature extraction, motion information prediction, and identity-guided feature extraction obtained by the multi-target tracking model are evaluated from multiple perspectives. Specifically, the identity prediction loss uses the cross-entropy loss averaged over T frames, the image feature loss uses the cosine loss, the motion information loss uses the L1 loss of predicted displacement and actual displacement, and the identity feature-guided extraction loss uses the weighted sum of the identity probability multiplied by the feature loss of the corresponding identity. The weights of each loss are adjusted through hyperparameters to form the overall loss.

[0077] The formula is expressed as:

[0078] Loss=λ1L id +λ2L ap +λ3L mv +λ4L id_guide_ap +λ5L id_guide_mv

[0079]

[0080] Where Loss represents the total loss, L id L represents the loss of identity features. ap L represents image feature loss. mv L represents the loss of motion features. id_guide_ap and L id_guide_mv Let represent the identity-guided image feature extraction loss and the identity-guided motion feature extraction loss, respectively; λ1, λ2, λ3, λ4, λ5 are the weight hyperparameters of the corresponding losses; M t p_t represents the number of targets in frame t, 2≤t≤T; m ={p_t m 1 ,p_t m 2 ,...,p_t m k ,p_t m newborn} represents the prediction result of the m-th target in the t-th frame, k is the preset number of targets to be tracked, and gt_t m R represents the actual identity result of the m-th target in the t-th frame; t ={r t 1,r t 2,...,r t Mt} represents the image features of the t-th frame, where r t m His_r represents the image features of the m-th target in the t-th frame. t m ={r t -Tid＝m ,...,r t-2 id＝m r t-1 id＝m} represents all historical image features with the same identity as m in the historical trajectory, D t ={d t 1,d t 2,...,d t Mt} represents the displacement prediction result for frame t, gt_d t m L represents the displacement difference between targets with the same identity and m in frame t and frame (t-1). ap (i,his_r t i L is the minimum image feature loss between the historical target and the current target with identity i. mv (i,gt_d t m ) represents the minimum motion information loss between the historical target and the current target with identity i.

[0081] The model is trained by evaluating the results. Once the model loss converges and the training and validation losses stabilize, it can perform multi-target tracking well. This indicates that the model training is successful, and the trained neural network model is obtained. This multi-target tracking model can accurately assign the identity of the target and complete multi-target tracking.

[0082] This invention combines optical flow information and deep learning. First, optical flow information is extracted to obtain the optical flow features of the current frame. Second, optical flow information is combined with image information to achieve information enhancement. Simultaneously, image features and optical flow features of the target object are extracted using bounding boxes to perform multi-angle feature extraction. Finally, identity information is combined with the target object to output the target object's identity end-to-end, thereby enabling target tracking. By combining optical flow and image features to extract target features and utilizing a cross-attention mechanism, the target features of historically tracked objects are better captured, achieving accurate multi-target tracking.

[0083] Example 2

[0084] This invention discloses a multi-target tracking system based on optical flow information and identity prediction, c refers to Figure 4 As shown, it includes the following modules:

[0085] Image acquisition module: used to acquire the target to be tracked and the original image, perform preprocessing, and obtain a reference image labeled with the target to be tracked;

[0086] Optical flow feature extraction module: used to extract the optical flow feature map of the current frame in the reference image compared with the previous frame using a lightweight deep optical flow network, and to crop the optical flow information near the center point of the target to be tracked as the optical flow feature of the target to be tracked;

[0087] Feature enhancement module: used to use the optical flow feature map to stitch and fuse the previous frame image with the current frame image to generate an optical flow-guided feature enhancement image;

[0088] Target image feature extraction module: used to crop the feature-enhanced image within the bounding box of the target to be tracked as the image features of the target to be tracked, input the image features into the feature extraction network, and output the target image features;

[0089] Tracking target stitching module: used to stitch together the optical flow features and target image features of the target to be tracked to obtain the target features of the target to be tracked, and stitch them together with the identity of the target to be tracked to obtain the tracking target to be assigned an identity;

[0090] Tracking target prediction module: used to input the tracking target whose identity is to be assigned into the multi-target tracking model for prediction, and obtain the prediction result of the tracking target.

[0091] The system of this invention first includes an image acquisition module, used to acquire the target to be tracked and the original image. The image data and the target to be tracked are preprocessed and scaled at the same scale to ensure that the detection result correctly selects the target. Subsequently, in the optical flow feature extraction module, two adjacent frames of images are input into the optical flow feature network to extract the optical flow information of the current frame of the image. The optical flow information shows pixel-level motion information. The optical flow information near the center point of the detected object is used as the coarse motion information of the object for the estimation of subsequent motion information.

[0092] Meanwhile, in the feature enhancement module, bilinear sampling is performed using the previous frame image and optical flow information. The sampling result is then stitched together with the current frame image, and the neural network adaptively enhances the image features to facilitate the subsequent extraction of image features of the tracked target. Secondly, in the target image feature extraction module, the target bounding box position of the target to be tracked is selected based on the target to be tracked, and the obtained feature-enhanced image is cropped as the current frame image feature of the target to be tracked. The resulting image is fused with optical flow information, achieving the effect of image enhancement. After the image size is scaled to a preset size, it is fed into the convolutional neural network module, and the result after the convolution operation is flattened to ensure that each target to be tracked has image features of the same size.

[0093] Then, in the target stitching module, the target prediction module stitches together the image features and optical flow features of each target to be detected, as the target features of that target. At the same time, the target features are stitched together with the pre-generated identity feature vector indicating the identity of the new object, as the target information to be tracked that needs to be assigned an identity. The information obtained in this way contains the motion information and image information of the object to be assigned an identity, which helps in subsequent prediction operations.

[0094] Subsequently, in the target prediction module, self-attention and cross-attention mechanisms are used. The identity information of the target to be tracked is used as guidance, and historical information of successfully assigned identities is utilized to adaptively generate a feature vector of the target combined with historical information. This feature vector is then fed into the prediction module to obtain the identity prediction result and the target features combined with historical information. Finally, based on an improved loss function, the results of identity prediction, image feature extraction, motion information prediction, and identity-guided feature extraction are evaluated from multiple perspectives. The evaluation results are used to train the model, obtaining a trained neural network model that ultimately achieves accurate target identity assignment and completes multi-target tracking. This invention's system can adaptively combine historical target information to achieve more accurate multi-target tracking.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0096] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-target tracking method based on optical flow information and identity prediction, characterized in that, Includes the following steps: S1. Acquire the target to be tracked and the original image, perform preprocessing, and obtain a reference image labeled with the target to be tracked; S2. Use a lightweight deep optical flow network to extract the optical flow feature map of the current frame in the reference image compared to the previous frame, and crop the optical flow information near the center point of the target to be tracked as the optical flow feature of the target to be tracked. S3. Using the optical flow feature map, the previous frame image and the current frame image are stitched and fused to generate an optical flow-guided feature-enhanced image; Specifically, it includes: Using the optical flow feature map, the previous frame image is bilinearly sampled; The obtained sampling results are then stitched together with the current frame image; The obtained stitched features are used to generate a fused image and attention weights through a convolutional network; The attention weights are multiplied by the fused image and then added to the current frame image to obtain the optical flow-guided feature enhancement image. S4. Crop the feature-enhanced image within the bounding box of the target to be tracked as the image feature of the target to be tracked, input the image feature into the feature extraction network, and output the target image feature; S5. The optical flow features and target image features of the target to be tracked are spliced ​​together to obtain the target features of the target to be tracked, and then spliced ​​together with the identity of the target to be tracked to obtain the tracking target to be assigned an identity; S6. Input the tracking target whose identity is to be assigned into the multi-target tracking model for prediction to obtain the prediction result of the tracking target; Specifically, it includes: Input the tracking target whose identity is to be assigned and the tracking targets whose identities have been assigned in historical T frames into the multi-target tracking model; By utilizing cross-attention and self-attention mechanisms, feature vectors of the tracked target that incorporate historical information are generated. Using a fully connected neural network, we obtain the identity prediction results, appearance feature results, and displacement prediction results of the tracked target to be assigned an identity.

2. The multi-target tracking method based on optical flow information and identity prediction as described in claim 1, characterized in that, The preprocessing described in step S1 specifically includes: The original image and the target to be tracked are adjusted to the same scaling scale.

3. The multi-target tracking method based on optical flow information and identity prediction as described in claim 2, characterized in that, In step S2, a lightweight deep optical flow network is used to extract the optical flow feature map of the current frame in the reference image compared to the previous frame, and the optical flow information near the center point of the target to be tracked is cropped as the optical flow feature of the target to be tracked. Specifically, it includes: The current frame and the previous frame of the reference image are input into the convolutional neural network; The motion displacement is learned through training, and the prediction results are optimized using a loss function. Infer the optical flow feature map of the current frame compared to the previous frame from two adjacent frames; The optical flow features around the center point of the target to be tracked are cropped to a size of 3x3 as the optical flow features of the target to be tracked.

4. The multi-target tracking method based on optical flow information and identity prediction as described in claim 1, characterized in that, The obtained splicing features are then used to generate a fused image and attention weights via a convolutional network; specifically, this includes: The obtained stitched features are fed into a 3x3 convolutional network for fusion, and the number of output channels is set to 3 to obtain a fused image with optical flow-guided feature enhancement. Similarly, the obtained spliced ​​features are fed into a 1x1 convolutional network with the number of channels set to 1. After normalization through the network layers, attention weights are generated.

5. The multi-target tracking method based on optical flow information and identity prediction as described in claim 4, characterized in that, In step S4, the image features are input into the feature extraction network, and the target image features are output; specifically, this includes: Use image processing software or libraries to uniformly adjust the image features of the target to be tracked to 32x32 and then input them into the convolutional neural network module; After passing through the three convolutional modules of the convolutional neural network module, the obtained features are flattened and fed into the fully connected network to obtain a feature vector of size 256, which is used as the target image feature.

6. The multi-target tracking method based on optical flow information and identity prediction as described in claim 5, characterized in that, In step S5, the optical flow features and target image features of the target to be tracked are stitched together to obtain the target features of the target to be tracked, and then stitched together with the identity of the target to be tracked to obtain the tracking target to be assigned an identity; specifically including: The optical flow features and target image features of the target to be tracked are stitched together in the size dimension to obtain the target features of the target to be tracked. A learnable feature vector is used as the identity of the target to be tracked, and it is concatenated with the target feature in the size dimension to obtain the tracking target with the identity to be assigned.

7. The multi-target tracking method based on optical flow information and identity prediction as described in claim 1, characterized in that, Step S6 also includes: The identity prediction results, appearance feature results, and displacement prediction results are evaluated from multiple perspectives using a loss function. The multi-target tracking model was optimized and adjusted based on the evaluation results.

8. A multi-target tracking system based on optical flow information and identity prediction, characterized in that, Includes the following modules: Image acquisition module: used to acquire the target to be tracked and the original image, perform preprocessing, and obtain a reference image labeled with the target to be tracked; Optical flow feature extraction module: used to extract the optical flow feature map of the current frame in the reference image compared with the previous frame using a lightweight deep optical flow network, and to crop the optical flow information near the center point of the target to be tracked as the optical flow feature of the target to be tracked; Feature enhancement module: used to use the optical flow feature map to stitch and fuse the previous frame image with the current frame image to generate an optical flow-guided feature enhancement image; Specifically used for: Using the optical flow feature map, the previous frame image is bilinearly sampled; The obtained sampling results are then stitched together with the current frame image; The obtained stitched features are used to generate a fused image and attention weights through a convolutional network; The attention weights are multiplied by the fused image and then added to the current frame image to obtain the optical flow-guided feature enhancement image. Target image feature extraction module: used to crop the feature-enhanced image within the bounding box of the target to be tracked as the image features of the target to be tracked, input the image features into the feature extraction network, and output the target image features; Tracking target stitching module: used to stitch together the optical flow features and target image features of the target to be tracked to obtain the target features of the target to be tracked, and stitch them together with the identity of the target to be tracked to obtain the tracking target to be assigned an identity; Tracking target prediction module: Used to input the tracking targets whose identities are to be assigned into the multi-target tracking model for prediction, and obtain the prediction results of the tracking targets: Specifically used for: Input the tracking target whose identity is to be assigned and the tracking targets whose identities have been assigned in historical T frames into the multi-target tracking model; By utilizing cross-attention and self-attention mechanisms, feature vectors of the tracked target that incorporate historical information are generated. Using a fully connected neural network, we obtain the identity prediction results, appearance feature results, and displacement prediction results of the tracked target to be assigned an identity.

Images