Multi-target tracking method based on hierarchical context guidance

Abstract

The application provides a multi-target tracking method based on hierarchical context guidance, and proposes a new hierarchical context guidance network, which performs detection, bounding box feature extraction and object position refinement through hierarchical global processing, local processing and object bounding box processing of video frames. The tracker learns the temporal and spatial context features of the video frames in a global and local manner respectively, guides multi-scale feature aggregation, locates the region of interest and extracts rich bounding box features. In this way, each detected bounding box contains not only semantic information but also context relationship information, reducing the loss of important information of incomplete or unclear objects. Based on the learned context features, a context-guided object position refinement module is designed, which learns the representative features of the target object trajectory segment by propagating the bounding box features of the object in each frame, and generates the object after position refinement according to the representative features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a multi-target tracking method based on hierarchical context guidance. BACKGROUND

[0002] With the development of deep learning, convolutional neural networks are applied in more and more scenarios, and multi-target tracking is receiving more and more attention in the field of computer vision due to its wide application in video monitoring, human-computer interaction and virtual reality. Multi-target tracking aims to locate multiple target objects in a given video sequence, assign different identities ID to different objects and record the trajectory of each ID in the video. At present, with the continuous development of target detection technology based on convolutional neural networks, detection-based tracking algorithms have become the mainstream direction of multi-target tracking. Detection-based tracking algorithms first need to perform target detection on each video frame to obtain the detection results of each frame, and then perform data association according to the detection results to create the trajectory of each object in the video.

[0003] The core idea of one-stage multi-target tracking method is to simultaneously perform target detection and tracking in a deep learning network, and to reduce inference time by sharing most of the computational load. Recently, a one-stage multi-target tracking method that simultaneously compatible with tracking efficiency and accuracy has been proposed, which is an anchor-free method using ResNet-34 structure as the backbone network, which can well compatible with accuracy and speed, and adding Deep Layer Aggregation (DLA) structure on the backbone network to realize detection of different scales. This network design can dynamically adapt to the receptive field according to the size and posture of the target, and at the same time help to alleviate the alignment problem.

[0004] The framework of the method is shown in Figure 2 The main parts are feature extraction network for extracting features, detection part and recognition part for embedding ID. Among them, the detection branch uses a heat map to predict the position of the object center, obtains the center offset of the target and estimates the height and width of the target bounding box, and the purpose of the recognition branch is to generate features that can distinguish objects. The detection part and the recognition part share the features extracted by the feature extraction network and are trained and learned at the same time. These one-time methods save some computational load, but the performance is still limited in the case of complex tracking scenarios. SUMMARY

[0005] Embodiments of the present application provide a multi-target tracking method based on hierarchical context guidance to solve the technical problems existing in the prior art.

[0006] In order to achieve the above purpose, the present application adopts the following technical solutions.

[0007] Hierarchical context-guided multi-target tracking methods include:

[0008] S1 is based on the original video sequence, let t represent the t-th frame of the original video sequence;

[0009] S2 When t=1, the original video sequence is processed by the context-guided multi-scale aggregation module to obtain the detection value of the target object in the current frame. , bounding box features corresponding to the detected values and features of the current video frame and use the detection value Initialize target object Use the bounding box features corresponding to the detected values. Initialize the bounding box feature of the object Use the bounding box feature of the target object. Representative features of the trajectory fragment of the initialized object When t>1, the t-th and (t-1)-th frames of the original video sequence are processed by a context-guided multi-scale aggregation module to mine and obtain video frame features with spatiotemporal context information for the current frame. And obtain the detection value in the t-th frame image based on the context information. The bounding box features corresponding to the detected values and features of the t-th frame image ;

[0010] S3 uses a context-guided object position refinement module, which uses the bounding box features of the target object at frame t-1. Initialize representative features of the trajectory segment of the target object at frame t-1. Obtain representative features of the updated trajectory segment of the target object at frame t. ;

[0011] S4 Representative features of the updated trajectory segment obtained in step S3 The video frame features with spatiotemporal context information obtained in step S2 The target object is obtained after location refinement through a context-guided object location refinement module. ;

[0012] S5. Detection values ​​in the t-th frame image obtained in step S2 , bounding box features corresponding to the detected values The representative features of the updated trajectory segment of the target object at frame t obtained in step S3. The target object with refined position obtained in step S4 Construct a bipartite graph;

[0013] S6 matching based on the bipartite graph by the Hungarian algorithm;

[0014] S7 obtaining the tracking trajectory result of the target based on the matching result of step S6, comprising: taking the position coordinates of the detection value as the position of the target object matched with the detection value in the current frame, and taking the bounding box feature corresponding to the target object as the feature of the target object in the current frame;

[0015] S8 judging whether the processing of the original video sequence is finished, if yes, outputting all the tracking trajectory results of the targets, otherwise, performing the processing operation on the next frame of the original video sequence.

[0016] Preferably, the context-guided multi-scale aggregation module comprises a global temporal context mining submodule and a patch-based spatial context mining submodule;

[0017] The global temporal context learning submodule is used to process the temporal context relationship between different frames, generate connection features by concatenating the bottom features of the previous frame and the bottom features of the current frame, then perform global temporal context learning on the connection features, calculate the relationship between each pixel of the current frame features and all pixels of the previous frame, and deliver the calculation results to the local spatial context learning submodule;

[0018] The local spatial context learning submodule is used to: based on the calculation results of the global temporal context learning submodule, use ResNet34 to perform progressive coding to obtain a multi-scale feature pyramid, cross- segment the features of each layer of the multi-scale feature pyramid to generate multiple local features, then independently perform spatial context learning on each local feature to obtain the spatial context relationship in a specific range, splice all local features with spatial context information in the same layer of features to obtain the complete feature map with spatial context in each layer of the feature pyramid, and finally use a deep aggregation method to perform aggregation operation on the feature map of each layer of the multi-scale feature pyramid to obtain the multi-scale aggregated feature of the current video, based on the multi-scale aggregated feature to locate the region of interest, obtain the detection value and the bounding box feature of the detection value.

[0019] Preferably, the local spatial context learning submodule is performed by formula

[0020] (1)

[0021] (2) ​

[0022] Calculate the features of the current video obtained by multi-scale aggregation. In the formula, Y represents the context-informed features learned by the module, and X represents the input features. When calculating the temporal context, X is the result of concatenating the low-level features of the current frame and the previous frame. When calculating the spatial context, X is the local feature after cross-segmentation of each layer in the multi-scale pyramid. This indicates four convolutional layers, where Softmax represents the activation function and Maxpool is the max pooling operation. It is a coefficient, specifically a feature relationship calculated through a self-attention mechanism;

[0023] The local spatial context learning submodule will compute the features of the current video obtained by multi-scale aggregation. The data is fed into the detection branch and the bounding box feature extraction branch to obtain the detection values ​​of the current frame. , bounding box features corresponding to the detected values .

[0024] Preferably, step S3 includes: through formula

[0025] (3)

[0026] Calculate the representative features of the updated trajectory segment of the target object at frame t. In the formula, j represents the j-th target object. This represents the bounding box feature of the target object at frame t-1. This represents a representative feature of the trajectory segment of the target object at frame t-1.

[0027] Preferably, step S4 includes:

[0028] S41 will use the frame features of the current video frame with spatiotemporal context information. Features of the previous video frame The input is then fed into the encoder for encoding.

[0029] S42 Through-type

[0030] (4)

[0031] The result obtained in step S3 The encoded result obtained in sub-step S41 is input into the decoder for object probing, and the refined object is calculated. In the formula, TransD and TransE are the encoder and decoder of the Transformer structure. It is a function composed of convolutional layers used to predict the center point position, offset, and bounding box of a target object.

[0032] As can be seen from the technical solutions provided by the above-mentioned embodiments of the present application, the present application provides a multi-target tracking method based on hierarchical context guidance, and proposes a new hierarchical context guidance network, which performs detection, bounding box feature extraction and object position refinement through hierarchical bottom-up global processing, local processing and object bounding box processing. Specifically, the multi-scale aggregation module of the network learns temporal and spatial context features in a global and local manner respectively, guides multi-scale feature aggregation, thereby locating the region of interest and extracting rich bounding box features. In this way, in addition to semantic information, the bounding box features of each detection value also have context relationships, reducing the loss of important information of incomplete or unclear objects. Finally, based on the learned context features, a context-guided object position refinement module is designed, which learns the representative features of the object trajectory segment by propagating the bounding box features of the object in the historical frames, and generates the position-refined object in each frame using the representative features and the context features of the multi-scale aggregation module. This way can alleviate the false matching between objects and detections. In this way, the performance of the multi-target tracking task model in complex tracking scenarios is improved.

[0033] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0035] Figure 1 The processing flow chart of the multi-target tracking method based on hierarchical context guidance provided by the present application is shown in the following figure.

[0036] Figure 2 The multi-target tracking processing process chart of the prior art is shown in the following figure. DETAILED DESCRIPTION

[0037] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present application, and cannot be explained as a limitation of the present application.

[0038] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprise" (or comprise), "comprises" (or comprises) and "comprising" (or comprising) when used in this specification, specify the presence of stated features, integers, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or groups thereof. It is further understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements can be present. In addition, the use of "connection" or "coupling" herein also includes wireless connection or coupling. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0039] It is to be understood that the terms so used are intended to encompass the meanings commonly associated with the terms in addition to their meanings as defined in the present disclosure. The meaning of "a," "an," and "the" as used herein shall be interpreted as including the plural forms of the terms so that the application includes one or more channels, networks, and the like, unless otherwise indicated.

[0040] For the purpose of promoting an understanding of the principles of the application, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the application. It will, nevertheless, be understood that no limitation of the scope of the application is intended by this disclosure.

[0041] The present application provides a multi-target tracking method based on hierarchical context guidance, which is used to solve the following technical problems in the prior art:

[0042] Many existing multi-target tracking techniques have realized a one-stage method, but these one-stage methods greatly improve the efficiency of multi-target tracking, but ignore the performance of the method. In practical applications, very complex situations are often encountered: the number of objects often reaches dozens or even hundreds, and frequent interactions and occlusions between objects can easily lead to incomplete or unclear appearance of objects. However, many existing one-stage methods still have limited performance in complex tracking scenarios due to the lack of comprehensive consideration of context clues.

[0043] In view of this, this invention proposes a novel hierarchical context-guided network for one-stage multi-target tracking tasks. This network consists of a context-guided multi-scale aggregation module and a context-guided object location refinement module. Specifically, to reduce performance limitations caused by interference from similar objects or objects with incomplete or unclear appearances, a context-guided multi-scale aggregation module is designed. This module learns the spatiotemporal context information of video frames in a global and local manner to guide multi-scale feature aggregation for detection and bounding box feature extraction. To effectively alleviate mismatches and handle long-term occlusion problems, a context-guided object location refinement module is designed based on the multi-scale aggregation module to adjust the object's position.

[0044] See Figure 1 This invention provides a multi-target tracking method based on hierarchical context guidance, comprising the following steps:

[0045] S1 is based on the original video sequence, let t represent the t-th frame of the original video sequence;

[0046] S2 When t=1, the original video sequence is processed by the context-guided multi-scale aggregation module to obtain the detection value of the target object in the current frame. , bounding box features corresponding to the detected values and features of the current video frame and use the detection value Initialize target object Use the bounding box features corresponding to the detected values. Initialize the bounding box feature of the object Use the bounding box feature of the target object. Representative features of the trajectory fragment of the initialized object When t>1, the t-th and (t-1)-th frames of the original video sequence are processed by a context-guided multi-scale aggregation module to mine and obtain video frame features with spatiotemporal context information for the current frame. And obtain the detection value in the t-th frame image based on the context information. The bounding box features corresponding to the detected values and features of the t-th frame image ;

[0047] S3 uses a context-guided object position refinement module, which uses the bounding box features of the target object at frame t-1. Initialize representative features of the trajectory segment of the target object at frame t-1. Obtain representative features of the updated trajectory segment of the target object at frame t. ;

[0048] S4 representative features of the updated trajectory segment obtained based on step S3 and the video frame features with spatio-temporal context information obtained in step S2 , and the position-refined target object obtained in step S4 are processed by a context-guided object position refinement module to obtain a position-refined target object ;

[0049] S5 detection values in the t-th frame image obtained in step S2 , bounding box features corresponding to the detection values , representative features of the trajectory segment of the updated target object obtained in step S3 at the t-th frame and the position-refined target object obtained in step S4 , a bipartite graph is constructed

[0050] S6 matching is performed based on the bipartite graph by using a Hungarian algorithm

[0051] S7 a tracking trajectory result of the target is obtained based on the matching result in step S6, including: taking the position coordinates of the detection value as the position of the target object matched with the detection value in the current frame, and taking the bounding box features corresponding to the target object as the features of the target object in the current frame ;

[0052] S8 it is determined whether the processing of the original video sequence is completed, if yes, all tracking trajectory results of the target are output, otherwise, a processing operation for the next frame of the original video sequence is performed

[0053] The method provided by the application first needs to perform target detection on the video sequence, then extracts features of the target in the target detection frame, calculates the matching degree between the target in the front and back frames, and obtains the target trajectory through data correlation.

[0054] In the actual application scene of multi-target tracking, it is often necessary to obtain the motion trajectory of the target in the video sequence according to a video sequence without any annotation information, so it is necessary to know the possible position of the target first. With the continuous improvement of target detection technology, the method of obtaining the possible position of the target in the video through target detection has become the mainstream of multi-target tracking.

[0055] After the possible positions of the target are grasped, how to associate the positions of the same target between adjacent video frames to obtain the trajectory of the target in the entire video sequence, i.e., data association, is a key problem to be solved in multi-target tracking. In the embodiment of step S2, first, the target needs to be initialized according to the target detection box in the first frame, and then data association of subsequent frames is performed. After all video frames are processed, the trajectory corresponding to each target is output, i.e., which frame and which position each target appears in.

[0056] The overall framework of the model of the application is composed of two modules, i.e., a context-guided multi-scale aggregation module and a context-guided object refinement module. In the application, each frame of the video sequence is sequentially input into the multi-scale aggregation module and the object position refinement module to obtain a detection value in the video frame, a feature of a bounding box corresponding to the detection value, a representative feature of a target object trajectory segment, and a target object after position refinement. According to the obtained results, a bipartite graph is constructed, and the Hungarian algorithm is used for data association to obtain a matching result between the detection value and the target object after refinement. Finally, the tracking result is obtained according to the matching result.

[0057] In the method provided in the application, a new hierarchical context-guided network for a one-stage multi-target tracking task is proposed, which includes a new context-guided multi-scale aggregation module and a context-guided object position refinement module.

[0058] It is found through analysis that the existing one-stage multi-target tracking model is limited in performance due to the interference of similar target objects, incomplete or unclear appearance, and therefore a context-guided multi-scale aggregation module is designed, which uses ResNet34 as a backbone network. Since the spatial topological structures of different detections are different and change over time, the context learning in the multi-scale aggregation module helps to understand and locate the detections. In addition, the video frame features with spatio-temporal context information provided by the module lay a solid foundation for subsequent object position refinement. The module learns the context relationship in the video frame through two steps of global temporal context learning and local spatial context learning to guide multi-scale feature aggregation, reduce mixed individuals, and generate distinctive embeddings with clearer boundaries.

[0059] The context-guided multi-scale aggregation is designed to reduce the performance limitations caused by similar target interference, incomplete or unclear appearance of target objects, and improve the effectiveness of aggregated features. The module includes two steps of global temporal context learning and local spatial context mining to learn the context relationship to guide multi-scale feature aggregation.

[0060] Regarding the context-guided multi-scale aggregation module, a preferred embodiment is specifically provided, which proposes a method for hierarchical learning of context spatio-temporal features. Unlike existing methods, the method is based on the principle of learning the context relationship of the video frame from the global temporal perspective and the local spatial perspective, and then guiding the multi-scale feature aggregation.Figure 2 ), considering that in a video sequence, each pixel point of adjacent data frames can change, and the change between pixel points in the same data frame is often closely related to its near points, therefore, a global-to-local way is used to learn the spatial context. Specifically, a context learning module is proposed, taking Resnet50 as a feature extractor, using the context learning module in a global way to learn the temporal context in the bottom layer features obtained by Resnet50, and learning the spatial context information of the target and its surrounding environment in a local way in the multi-scale feature pyramid. The calculation formula of the context learning module is shown in formula (1) (2), Y is the feature learned by the module with context information. X is the input feature, when using the context learning module in a global way to learn the temporal context, X is the result of concatenating the bottom layer features of the current frame and the bottom layer features of the previous frame; when learning the spatial context in a local way, X is the result of uniformly dividing the features in each layer of the multi-scale pyramid (local features after cross partitioning). represents 4 convolutional layers, Softmax represents an activation function, Maxpool is a maximum pooling operation, is a coefficient, which is a feature relationship obtained by known self-attention mechanism calculation.

[0061] (1)

[0062] (2)

[0063] In the context-guided multi-scale aggregation module, the embodiment uses a deep aggregation method to aggregate the features obtained in each layer with spatio-temporal context information to obtain the feature of the current frame , and then the feature is fed into the detection branch and the bounding box feature extraction branch to obtain the detection value , the bounding box feature corresponding to the detection value .

[0064] Global temporal context learning takes different frames as the research object to study the temporal context relationship between different frames, and the obtained output is transmitted to the spatial context learning. Local spatial context learning takes the obtained temporal context feature as a premise, and then uses a deep aggregation method to obtain a multi-scale aggregation feature, so as to locate the region of interest and extract rich bounding box features. The output obtained by the module is transmitted to the context-guided object position refinement module for processing.

[0065] Specifically, the context-guided multi-scale aggregation module uses ResNet34 as the backbone network. The global temporal context learning takes different frames as the research object to study the temporal context relationship between different frames. It concatenates the bottom-level features of the previous frame with the bottom-level features of the current frame to generate connection features, and then performs global temporal context learning on the connection features through the context learning module to calculate the relationship between each pixel of the current frame feature and all pixels of the previous frame. The output obtained is passed to the spatial context learning. The local spatial context mining takes the temporal context features as the premise and uses ResNet34 for progressive coding to obtain a multi-scale feature pyramid. The same operation is performed on each layer of features: cross-division is performed on the features to generate multiple local features, then spatial context learning is independently performed on each local feature to obtain the spatial context relationship of a specific range, all local features with spatial context information in the same layer of features are spliced to obtain the complete feature map of each layer of features with spatial context in the feature pyramid, and finally a deep aggregation method is used to aggregate the feature maps of each layer of the multi-scale feature pyramid to obtain the multi-scale aggregation features of the video frame Based on the multi-scale aggregation features, the region of interest is located, and a detection value is obtained The bounding box features of the detection value .

[0066] If there is long-term occlusion, it will cause the target prediction to fail, therefore, how to reduce the problem of false matching and detection of the tracked object in tracking becomes a problem that cannot be ignored, and thus a context-guided object position refinement module is designed, which explores the possible position of the object in each frame by means of the learned video frame features with spatio-temporal context information and the updated representative features of the trajectory segment, to obtain a refined target as close as possible to the real target in each frame. The module consists of two parts: representative feature updating of the trajectory segment and object position refinement. It learns the representative features of the trajectory segment of the object by propagating the bounding box features of the object in the historical frames, and generates the object after position refinement in each frame by using the representative features and the context features of the multi-scale aggregation module. Using the representative features of the trajectory segment instead of the bounding box features of the target object in a single frame can provide more reliable feature information for object position refinement and suppress refinement failure caused by some noisy queries. The representative features of the trajectory segment convey the semantic, context and position information of the previously seen object, and the position refinement module can achieve good tracking object position refinement effect. Under the action of this module, the refined target is closer to the real position, and the association performance between the target and the detection is improved.

[0067] Specifically, the context-guided object location refinement module introduces two concepts: bounding box features of the target object and representative features of its trajectory segments. The bounding box features refer to the features of the target within a specific frame, while the representative features of the trajectory segments refer to the representative features of the target over a given time period. The global temporal context learning submodule handles the temporal context relationships between different frames. It generates connection features by concatenating the low-level features of the previous frame with those of the current frame. Then, it performs global temporal context learning on these connection features, calculating the relationship between each pixel of the current frame's features and all pixels of the previous frame. The calculation results are then fed into the local spatial context learning submodule.

[0068] The local spatial context learning submodule is used to: obtain a multi-scale feature pyramid by progressive encoding using ResNet34 based on the computation results of the global temporal context learning submodule; perform cross-segmentation on the features of each layer of the multi-scale feature pyramid to generate multiple local features; then independently perform spatial context learning on each local feature to obtain spatial context relationships within a specific range; concatenate all local features with spatial context information in the same layer to obtain a complete feature map with spatial context for each layer of the feature pyramid; finally, use a deep aggregation method to aggregate the feature maps of each layer of the multi-scale feature pyramid to obtain the multi-scale aggregated features of the current video. Based on multi-scale aggregated features, the region of interest is located and the detection value is obtained. bounding box features of the detected values

[0069] Specifically, the representative features of the trajectory segments are updated using the bounding box features of all target objects tracked in the previous frame, as shown in formula (3). Here, j represents the j-th target object. This represents the bounding box feature at frame t-1. This represents the representative features of the trajectory segment at frame t-1. For a target that has just appeared in frame t-1, we directly initialize the representative features of the trajectory segment using its bounding box features in frame t-1.

[0070] (3)

[0071] In a preferred embodiment provided by the present invention, step S4 specifically includes:

[0072] S41 will use the frame features of the current video frame with spatiotemporal context information. Features of the previous video frame The input is then fed into the encoder for encoding.

[0073] S42 Through-type

[0074] (4)

[0075] The object detection result obtained in step S3 is input into the decoder to perform object detection, and the refined object is calculated and obtained .In the formula, TransD and TransE are the encoder and decoder of the Transformer structure, is a function composed of a convolutional layer for predicting the target object center point position, offset and bounding box.

[0076] As shown in the model flowchart, the overall flow steps of the present application are: Figure 1 As shown in the model flowchart, the overall flow steps of the present application are:

[0077] 1. Given a video sequence that needs to be tracked, input it into the model.

[0078] 2. Use t to represent the t-th frame of the input video, starting from t=1 until all video frames of the video sequence are processed. When t=1, the first frame is input into the context-guided multi-object aggregation module to obtain the detection value , the bounding box feature corresponding to the detection value , and the feature of the current video frame , and use to initialize the target object , use to initialize the bounding box feature of the target object , and use to initialize the representative feature of the track segment ;

[0079] 3. When t>1, the bottom features of the current frame and the previous frame are input into the context-guided multi-feature aggregation module to obtain the detection value , the bounding box feature corresponding to the detection value , and the feature of the current video frame by learning the temporal and spatial context.

[0080] 4. The representative feature of the track segment is updated, and the object bounding box feature of the (t-1)th frame is propagated to the historical track segment representative feature to update the track segment representative in the t-th frame.

[0081] 5. The object after position refinement is generated by using the context-guided object position refinement module .

[0082] 6. According to , , and Construct a bipartite graph, where the affinity values ​​in the bipartite graph include and Euclidean distance between them and The Euclidean distance between the center points;

[0083] 7. The Hungarian algorithm is used for result matching on the bipartite graph. In data association, the position coordinates of the detected value are used as the position of the matched object in the current frame, and the corresponding bounding box features are used as the features of the object in the current frame. Furthermore, a dual verification strategy is proposed to handle long-term occluded objects. Specifically, the Hungarian algorithm is used to obtain the matching results of the bipartite graph. Detections that do not match the refined object generate a new object, and refined objects that do not match the detection will remain dormant for up to 50 frames before being discarded. The new object is compared with the dormant object using the calculated affinity value to determine whether the new object belongs to the dormant object. By storing dormant objects online and comparing them with new objects, tracking failures of long-term occluded objects can be effectively suppressed. For new objects that do not belong to any dormant object, we treat them as truly new target objects.

[0084] 8. Root association results generate the target trajectory;

[0085] 9. Determine if the entire video sequence has ended. If it has ended, output the tracking trajectories of all targets; otherwise, perform the tracking operation for the next frame.

[0086] In summary, this invention provides a hierarchical context-guided multi-object tracking method and proposes a novel hierarchical context-guided network. This network performs detection, feature extraction, and object location refinement through hierarchical global processing, local processing, and object processing. Specifically, the tracker learns temporal and spatial context features globally and locally, guiding multi-scale aggregation to locate regions of interest and extract rich bounding box features. Thus, the embedding of each detection possesses contextual relationships in addition to semantic information, reducing the loss of important information about incomplete or unclear objects. Finally, based on the learned context features, a context-guided object location refinement module is designed. By propagating the bounding box features of the object in each frame, representative features of the target object's trajectory segment are learned, and objects with refined locations are generated based on these representative features, mitigating mismatches between objects and detections. This improves the performance of multi-object tracking models in complex tracking scenarios.

[0087] Those skilled in the art can understand that the modules or flows in the drawings are not necessarily required for implementing the present application.

[0088] From the above description of the embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and necessary universal hardware platforms. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0089] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, the device or system embodiments are described more simply because they are basically similar to the method embodiments, and the relevant parts can be referred to the part of the method embodiments. The above-described device and system embodiments are merely illustrative, and the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to the actual needs. Those skilled in the art can understand and implement it without creative labor.

[0090] The above description is only the preferred embodiment of the present application, but the protection scope of the present application is not limited to this. Any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, and should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-target tracking method based on hierarchical context guidance, characterized in that, Comprising: S1 based on the original video sequence, let t represents the t frame of the original video sequence; S2 When t=1, the original video sequence is processed by the context-guided multi-scale aggregation module to obtain the detection value of the target object in the current frame. , bounding box features corresponding to the detected values and features of the current video frame and use the detection value Initialize target object Use the bounding box features corresponding to the detected values. Initialize the bounding box feature of the object Use the bounding box feature of the target object. Representative features of the trajectory fragment of the initialized object When t>1, the t-th and (t-1)-th frames of the original video sequence are processed by a context-guided multi-scale aggregation module to mine and obtain video frame features with spatiotemporal context information for the current frame. And obtain the detection value in the t-th frame image based on the context information. The bounding box features corresponding to the detected values and features of the t-th frame image ; The context-guided multi-scale aggregation module comprises a global temporal context learning sub-module and a local spatial context learning sub-module; S3 an object position refinement module by context guidance, obtaining the bounding box feature of the target object at the t-1th frame initializing the representative feature of the trajectory segment of the target object at the t-1th frame , obtaining the updated representative feature of the trajectory segment of the target object at the tth frame ; S4 representative features of the updated trajectory segment obtained based on step S3 and the video frame features with spatio-temporal context information obtained in step S2 , are processed by a context-guided object position refinement module to obtain a target object after position refinement ; S5 a detection value in the t-th frame image obtained based on step S2 , a bounding box feature corresponding to the detection value , a representative feature of the updated trajectory segment of the target object at the t-th frame obtained in step S3 , and the target object after position refinement obtained in step S4 , constructing a bipartite graph; S6 based on the bipartite graph is matched through the Hungarian algorithm; S7 obtaining a tracking trajectory result of the target based on the matching result of step S6, including: taking the position coordinates of the detection value as the position of the target object matched with the detection value in the current frame, and taking the bounding box feature corresponding to the target object as the feature of the target object in the current frame; S8 judges whether the processing of the original video sequence is finished, if yes, outputs all the tracking trajectory results of the target, otherwise, executes the processing operation for the next frame of the original video sequence.

2. The method of claim 1, wherein, The global temporal context learning sub-module is used for processing the temporal context relationship between different frames, generates connection features by concatenating the bottom features of the previous frame and the bottom features of the current frame, then performs global temporal context learning on the connection features, calculates the relationship between each pixel of the current frame features and all pixels of the previous frame, and delivers the calculation result to the local spatial context learning sub-module; The local spatial context learning submodule is configured to: based on the calculation result of the global timing context learning submodule, utilize ResNet34 to perform progressive coding to obtain a multi-scale feature pyramid, cross- segment the features of each layer of the multi-scale feature pyramid, generate a plurality of local features, then independently perform spatial context learning on each local feature to obtain a spatial context relationship in a specific range, splice all local features with spatial context information in the same layer of features to obtain a complete feature map with spatial context for each layer of the feature pyramid, and finally use a deep aggregation method to perform an aggregation operation on the feature map of each layer of the multi-scale feature pyramid to obtain a multi-scale aggregated feature of the current video , based on the multi-scale aggregated feature, locate a region of interest to obtain a detection value and a bounding box feature of the detection value .

3. The method of claim 2, wherein, The local spatial context learning sub-module is through formula (1) (2) Features of the current video are calculated to obtain multi-scale aggregation ; wherein Y is a feature learned by the module with context information, and X is an input feature, wherein X is a result of concatenation of bottom features of a current frame and a previous frame when calculating a temporal context relationship, and X is a local feature after cross-division of each layer in a multi-scale pyramid when calculating a spatial context, denotes four convolutional layers, Softmax denotes an activation function, and Maxpool is a maximum pooling operation, is a coefficient, and is a feature relationship calculated by a self-attention mechanism. The local spatial context learning submodule will calculate the multi-scale aggregated features of the current video obtained The detection value of the current frame is obtained by conveying into the detection branch and the bounding box feature extraction branch The bounding box features corresponding to the detection value .

4. The method of claim 1, wherein, Step S3 comprises: (3) a representative feature of a trajectory segment of the target object at the t-th frame is calculated ; wherein j represents the j-th target object, represents a bounding box feature of the target object at the t-1-th frame, represents a representative feature of a trajectory segment of the target object at the t-1-th frame.

5. The method of claim 1, wherein, Step S4 comprises: S41 inputting the current video frame features with the spatio-temporal context information into an encoder for encoding and the previous video frame features into the encoder S42 through formula (4) The object center point position, offset and bounding box are calculated by inputting the encoding result obtained in step S3 into the decoder. The object center point position, offset and bounding box are calculated by inputting the encoding result obtained in step S3 into the decoder. ; wherein, TransD and TransE are the encoder and decoder of the Transformer structure, is a function composed of a convolutional layer for predicting the target object center point position, offset and bounding box.

Images