Target tracking method based on multi-scale feature sharing, storage medium and device
By using a multi-scale feature sharing method, the feature map set of the target detection network is divided into two groups: shallow and deep layers. Feature vectors are generated and combined with Kalman filtering, which solves the problem of insufficient target tracking efficiency and accuracy in the existing technology and achieves efficient and accurate target tracking results.
Patent Information
- Application Number
- CN202410923501.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2044-07-10
AI Technical Summary
Existing target tracking technologies perform poorly under occlusion and drastic changes in the scene, have high computational costs and insufficient robustness, making it difficult to meet the high efficiency and accuracy requirements for target tracking in autonomous driving.
A multi-scale feature sharing method is adopted to divide the feature map set output by the target detection network into two groups: shallow and deep. Corresponding shallow and deep feature vectors are generated. Combined with the location information predicted by Kalman filter, the feature vector is selected by difference value to calculate target similarity. An overlap threshold is set to match and update the location information predicted by Kalman filter.
It improves the computational efficiency and accuracy of target tracking, reduces the computational load, enhances robustness to occlusion, reduces ID jump frequency, and improves the accuracy of similarity calculation for different targets.
Smart Images

Figure CN118864533B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of intelligent driving technology, and in particular relates to the field of target tracking technology. Background Technology
[0002] With the rapid development of artificial intelligence, the intelligent development of automobiles is also accelerating. With the maturity of Level 1 assisted driving, Level 2 advanced driver assistance is also booming. In particular, advanced driver assistance based on pure vision perception is becoming increasingly popular in the industry. Among them, target tracking in visual perception has always been a challenging task in the industry.
[0003] Currently, in intelligent driving, it is necessary to track detected targets to provide reliable data support for planning, control, and decision-making in downstream tasks. Common target tracking paradigms can be broadly divided into two categories: TBD (Tracking By Detection) and JDE (Joint Detection-and-Embedding).
[0004] TBD primarily separates target detection and target tracking. From a macroscopic perspective, it is a serial process. The performance and quality of tracking depend on the quality of detection. During the tracking process, the target's motion information and appearance information can be applied for tracking.
[0005] When tracking relies solely on motion information, typical algorithms include sorting algorithms. The biggest advantage of these algorithms is their speed and good tracking performance in unobstructed conditions. However, once occlusion occurs or the scene changes drastically, the frequency of target ID jumps increases, which is detrimental to downstream task planning, control, and decision-making.
[0006] When motion and appearance information are used simultaneously, a typical example is the DeepSORT algorithm. The biggest advantage of this type of algorithm is that it can still track targets that reappear after occlusion by relying on their appearance feature vectors and keep their IDs unchanged. The disadvantage is that the appearance feature vectors obtained need to be obtained separately through a neural network, and this neural network needs to be trained. This type of algorithm is equivalent to both target detection and tracking needing to train the neural network separately, and its robustness to occlusion is not very high, and it can only cope with short-term occlusion.
[0007] JDE primarily uses a single network to simultaneously output object detection results and object feature vectors. From a macroscopic perspective, this is equivalent to object detection and object tracking being processed in parallel, which improves efficiency and robustness against occlusion. The biggest advantage of this type of algorithm is its high efficiency, while its disadvantage is its high computational cost. Furthermore, the generation of object feature vectors utilizes high-level semantic information from deeper networks, while the color, shape, and texture information from shallower networks is used less. Summary of the Invention
[0008] This application provides a target tracking method, storage medium, and device based on multi-scale feature sharing, which can improve the computational efficiency of target tracking and the accuracy of similarity calculation for different targets.
[0009] In a first aspect, embodiments of this application provide a target tracking method based on multi-scale feature sharing, comprising: acquiring feature maps of different scales based on a target detection network, and dividing the feature maps into a shallow feature set group and a deep feature set group based on feature types; generating shallow feature vectors and deep feature vectors corresponding to the shallow feature set group and the deep feature set group, respectively, based on the position information output by the target detection network; determining whether to use the shallow feature vector or the deep feature vector to calculate the similarity of targets in two consecutive frames based on the position information output by the target detection network and the position information predicted by the Kalman filter; matching the position information output by the target detection network with the position information predicted by the Kalman filter based on the determined shallow feature vector or the deep feature vector, and updating the position information predicted by the Kalman filter based on the matching result.
[0010] In one implementation of the first aspect, generating shallow feature vectors and deep feature vectors corresponding to the shallow feature set and the deep feature set, respectively, based on the position information output by the target detection network includes: obtaining the target position of the feature map in the shallow feature set and the deep feature set based on the target position and mapping coefficient output by the target detection network; wherein, the target position of the feature map = the target position output by the target detection network / the mapping coefficient; and calculating the corresponding shallow feature vector and deep feature vector for each feature map in the shallow feature set and the deep feature set based on the target position of the feature map in the shallow feature set and the deep feature set, respectively.
[0011] In one implementation of the first aspect, calculating each feature map in the shallow feature set to obtain the corresponding shallow feature vector includes: calculating each target box location region in each feature map of each shallow feature set to obtain a shallow feature matrix corresponding to each shallow feature set; and fusing and splicing the shallow feature matrices to obtain a shallow feature vector.
[0012] In one implementation of the first aspect, calculating the corresponding deep feature vector for each feature map in the deep feature set group includes: upsampling each feature map in the deep feature map set to a uniform size to form feature maps of the same size; and calculating the position regions of each target box in each feature map in each deep feature set group to obtain the deep feature vector.
[0013] In one implementation of the first aspect, determining whether to use the shallow feature vector or the deep feature vector to calculate the similarity of targets in two consecutive frames based on the location information output by the target detection network and the location information predicted by the Kalman filter includes: calculating the difference value of targets in two consecutive frames based on the location information output by the target detection network and the location information predicted by the Kalman filter; comparing the difference value with a difference threshold: when the difference value is greater than or equal to the difference threshold, determining to use the shallow feature vector to calculate the similarity of targets in two consecutive frames; when the difference value is less than the difference threshold, determining to use the deep feature vector to calculate the similarity of targets in two consecutive frames.
[0014] In one implementation of the first aspect, the difference value is any one of the difference hash value, difference mean square error, difference structural similarity index, difference peak signal-to-noise ratio, and difference gradient similarity index.
[0015] In one implementation of the first aspect, matching the location information output by the target detection network with the location information predicted by the Kalman filter based on the determined shallow feature vector or the deep feature vector, and updating the location information predicted by the Kalman filter based on the matching result includes: performing cascade matching based on the determined shallow feature vector or the deep feature vector to obtain the similarity between the target box output by the target detection network and the target box predicted by the Kalman filter, and obtaining a similarity matching result based on the similarity; and updating the location information predicted by the Kalman filter based on the similarity matching result.
[0016] In one implementation of the first aspect, updating the location information predicted by the Kalman filter based on the matching result further includes: calculating the overlap area between the target bounding box output by the target detection network and the target bounding box predicted by the Kalman filter; comparing the overlap area with an overlap threshold: when the overlap area is greater than or equal to the overlap threshold, performing IOU matching between the target bounding box output by the target detection network and the target bounding box predicted by the Kalman filter to obtain an IOU matching result, and updating the location information predicted by the Kalman filter based on the IOU matching result; when the overlap area is less than the overlap threshold, performing Euclidean distance matching between the target bounding box output by the target detection network and the target bounding box predicted by the Kalman filter to obtain an Euclidean distance matching result, and updating the location information predicted by the Kalman filter based on the Euclidean distance matching result.
[0017] In a second aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target tracking method based on multi-scale feature sharing as described in any one aspect of this application.
[0018] Thirdly, embodiments of this application provide an electronic device, the electronic device including a processor and a memory; the memory stores program instructions; the processor is used to run the program instructions to execute the target tracking method based on multi-scale feature sharing as described in any one of the first aspects of this application.
[0019] The target tracking method based on multi-scale feature sharing provided in this application has the following beneficial effects:
[0020] 1. This application utilizes feature sets in the target detection process to obtain feature maps at different scales, performs multi-scale feature upsampling and fusion, and uses them for target feature vector extraction in tracking. Compared with using a single neural network to output the target feature vector, this is more efficient and accurate, and can effectively improve the computational efficiency of target tracking and the accuracy of similarity calculation for different targets.
[0021] 2. In this application, the feature set in the target detection process is divided into two groups: shallow and deep. The corresponding shallow feature vectors and deep feature vectors are constructed. Then, the difference between the target in the two frames is judged based on the difference value of the target in the previous and next frames. Based on this difference, the cosine distance between the feature vectors can be calculated more reasonably, which reduces the amount of calculation and makes reasonable use of the shallow features of the image.
[0022] 3. In this application, an overlap threshold is set to determine whether IOU matching is needed. If the overlap area is greater than the overlap threshold, IOU matching is performed; otherwise, Euclidean distance matching is performed directly. This setting can solve the problem of unreasonable IOU matching in the prior art, thereby improving the accuracy of target tracking. Attached Figure Description
[0023] Figure 1 The flowchart shown is a target tracking method based on multi-scale feature sharing according to an embodiment of this application.
[0024] Figure 2 The flowchart shown is a process for generating shallow and deep feature vectors in a target tracking method based on multi-scale feature sharing according to an embodiment of this application.
[0025] Figure 3 The diagram shown illustrates the principle of generating shallow feature vectors in a target tracking method based on multi-scale feature sharing according to an embodiment of this application.
[0026] Figure 4 The diagram shown illustrates the principle of generating deep feature vectors in a target tracking method based on multi-scale feature sharing according to an embodiment of this application.
[0027] Figure 5The flowchart shown is a process for selecting shallow or deep feature vectors in a target tracking method based on multi-scale feature sharing according to an embodiment of this application.
[0028] Figure 6 The flowchart shown is a target matching process in a target tracking method based on multi-scale feature sharing according to an embodiment of this application.
[0029] Figure 7 The diagram shown illustrates the implementation principle of a target tracking method based on multi-scale feature sharing according to an embodiment of this application.
[0030] Figure 8 The diagram shown is a structural schematic of an electronic device according to an embodiment of this application.
[0031] Component designation explanation
[0032] 100 Electronic devices
[0033] 101 Memory
[0034] 102 processor
[0035] 103 Monitor
[0036] S100~S400 Steps
[0037] Steps S210~S220
[0038] Steps S310~S340
[0039] S410~S460 Steps Detailed Implementation
[0040] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.
[0041] This application provides a target tracking method based on multi-scale feature sharing, which improves the computational efficiency of target tracking and the accuracy of similarity calculation for different targets.
[0042] The following will refer to the appendices in the embodiments of this application. Figure 1 To be continued Figure 8This application provides a detailed description of the technical solutions in its embodiments. This allows those skilled in the art to understand and implement the target tracking method based on multi-scale feature sharing without inventive effort.
[0043] Figure 1 The flowchart shown is a representation of a target tracking method based on multi-scale feature sharing in an embodiment of this application. Figure 1 As shown, the target tracking method based on multi-scale feature sharing provided in this application includes the following steps S100 to S400.
[0044] Step S100: Based on the object detection network, feature maps of different scales are obtained, and the feature maps are divided into shallow feature set group and deep feature set group based on feature type;
[0045] Step S200: Based on the position information output by the target detection network, generate shallow feature vectors and deep feature vectors corresponding to the shallow feature set group and the deep feature set group, respectively.
[0046] Step S300: Based on the position information output by the target detection network and the position information predicted by the Kalman filter, determine whether to use the shallow feature vector or the deep feature vector to calculate the similarity of the targets in the two frames before and after.
[0047] Step S400: Based on the determined shallow feature vector or deep feature vector, match the position information output by the target detection network with the position information predicted by the Kalman filter, and update the position information predicted by the Kalman filter based on the matching result.
[0048] This embodiment utilizes feature sets from the target detection process to acquire feature maps at different scales, performs multi-scale feature upsampling and fusion, and uses them for target feature vector extraction during tracking. Compared to using a single neural network to output the target's feature vector, this is more efficient and accurate, enabling more efficient and stable target tracking and global consistency of target feature vectors. It can effectively improve the computational efficiency of target tracking and the accuracy of similarity calculation for different targets.
[0049] The following provides a detailed description of steps S100 to S400 in the target tracking method based on multi-scale feature sharing in this embodiment.
[0050] Step S100: Based on the object detection network, feature maps of different scales are obtained, and the feature maps are divided into shallow feature sets and deep feature sets based on feature types.
[0051] This embodiment directly utilizes different feature sets output from different layers in the object detection network to establish feature set groups. These feature set groups are then divided into shallow feature set groups and deep feature set groups based on the number of network layers. There is no strict distinction between the number of layers in the shallow and deep feature set groups; in this embodiment, they are distinguished by the type of feature. Shallow features mainly consist of color, shape, and texture. When these features become less prominent or even disappear, deeper, higher-level semantic features emerge. Although shallow features also contain semantic features, they are very subtle.
[0052] For example, based on the feature extraction network in the object detection model, an input image sequence of size 640x640x3 outputs feature sets of five scales: 320x320xC1, 160x160xC2, 80x80xC3, 40x40xC4, and 20x20xC5, where C1-C5 are defined as the number of channels for each scale feature. The size of the output of the feature extraction network is not fixed and can be adjusted according to different convolutional neural networks. Taking the basic framework of YOLOv8 as an example, the feature maps of sizes 320 and 160 still contain shallow features, while the latter three shallow features are no longer present. Therefore, the feature sets of sizes 320x320xC1 and 160x160xC2 can be divided into shallow feature sets, and the feature sets of sizes 80x80xC3, 40x40xC4, and 20x20xC5 can be divided into deep feature sets.
[0053] This embodiment directly utilizes the feature set group in the target detection process, and then divides the feature set group into shallow feature set group and deep feature set group according to the number of network layers. This allows for multi-scale feature upsampling and fusion, which can be used for target feature vector extraction in tracking. This is more efficient than using a single neural network to output the target's feature vector.
[0054] Step S200: Based on the position information output by the target detection network, generate shallow feature vectors and deep feature vectors corresponding to the shallow feature set group and the deep feature set group, respectively.
[0055] In this embodiment, based on the location information in the target detection result, a position mapping relationship of the target position in the image in different size feature sets is established. Based on the position mapping relationship, features of different sizes in the shallow feature set and the deep feature set are upsampled and feature fusion calculated. Finally, features of different sizes are unified to the same size to generate two sets of feature vectors.
[0056] Figure 2 The flowchart shown is a process for generating shallow and deep feature vectors in a target tracking method based on multi-scale feature sharing according to an embodiment of this application. Figure 2As shown, in one embodiment of this example, generating shallow feature vectors and deep feature vectors corresponding to the shallow feature set group and the deep feature set group respectively based on the position information output by the target detection network includes the following steps S210 to S220.
[0057] Step S210: Obtain the target position of the feature map in the shallow feature set group and the deep feature set group based on the target position and mapping coefficient output by the target detection network; wherein, the target position of the feature map = the target position output by the target detection network / mapping coefficient.
[0058] In this embodiment, a positional mapping relationship is established between the target positions of the feature maps in the shallow feature set group and the deep feature set group and the target positions output by the target detection network. The mapping coefficient is the ratio of the size of the input image to the size of the feature map in the feature set group. For example, if the input image size is s1×s1, the feature size is s2×s2, and the mapping coefficient is r, then...
[0059] In one implementation of this embodiment, before establishing the position mapping relationship, the position information output by the target detection network is converted to the top left and bottom right corners of the target box. Then, the mapping relationship between the target position information from the original image to the feature map set is established to obtain the target position information on the feature map. The target position of the feature map = the target position output by the target detection network / the mapping coefficient.
[0060] Step S220: Based on the target positions of the feature maps in the shallow feature set and the deep feature set, calculate the corresponding shallow feature vector and deep feature vector for each feature map in both sets. The calculation method includes any one of the following: standard deviation calculation, variance calculation, Frobenius norm calculation, and mean calculation (reconstructing each target box region into a square matrix and then calculating the mean of the feature values). This embodiment uses variance calculation of each feature map in the shallow feature set and the deep feature set as an example for illustration.
[0061] Figure 3 This diagram illustrates the principle of generating shallow feature vectors in a target tracking method based on multi-scale feature sharing, as described in an embodiment of this application. Figure 3 As shown, in one embodiment of this example, calculating each feature map in the shallow feature set to obtain the corresponding shallow feature vector includes: calculating the location regions of each target box in each feature map in each shallow feature set to obtain a shallow feature matrix corresponding to each shallow feature set; and fusing and splicing the shallow feature matrices to obtain a shallow feature vector.
[0062] For example, in the shallow feature set group, there are C1 feature maps of size 320x320 and C2 feature maps of size 160x160. The standard deviation of the N target box location regions in each feature map is calculated to obtain two matrices Nx1xC1 and Nx1xC2. These two matrices are then fused and concatenated to obtain the shallow feature vector of the target Nx1x(C1+C2).
[0063] It's important to note that in conventional object detection, the neck region of the object detection network can fuse features extracted from the backbone. However, object detection requires simultaneous classification and regression (object location, i.e., bounding box coordinates). Classification is a semantic feature, requiring deep network features, while object coordinates are shallow features, requiring the output of shallower networks. Therefore, feature fusion is necessary. To better preserve shallow feature information and other target feature information, feature fusion network augmentation paths can be used.
[0064] In this embodiment, feature fusion is performed on the feature set output by the backbone in the target detection network, rather than on the features output by the neck.
[0065] Figure 4 This diagram illustrates the principle of generating deep feature vectors in a target tracking method based on multi-scale feature sharing, as described in an embodiment of this application. Figure 4 As shown, in one embodiment of this example, calculating the corresponding deep feature vector for each feature map in the deep feature set includes: upsampling each feature map in the deep feature map set to a uniform size to form feature maps of the same size; and calculating the position regions of each target box in each feature map in each deep feature set to obtain the deep feature vector.
[0066] Since the feature maps of the last two layers in the deep features have very small resolution, the target position error is large when a small target is mapped onto the feature map. In order to better preserve the high-level semantic information in the deep features, in this embodiment, all deep feature maps are upsampled to a uniform size, while the shallow feature map set remains unchanged in size.
[0067] For example, in the deep feature set group, all are 80x80 feature maps, with a total of C3+C4+C5. Similarly, the standard deviation of the N target box location regions in each feature map is calculated to obtain the deep feature vector Nx1x(C3+C4+C5) of the N targets.
[0068] Shallow feature maps primarily describe the color, shape, and texture information of an object, while deep feature maps describe its high-level semantic information. However, sometimes the intra-class differences between different objects of the same category are significant. In such cases, deep high-level semantic information is unnecessary, and shallow features can effectively distinguish between different objects of the same category. When intra-class differences are small, shallow features cannot distinguish between different objects of the same category, and the high-level semantic information from deep features becomes necessary.
[0069] Step S300: Based on the position information output by the target detection network and the position information predicted by the Kalman filter, determine whether to use the shallow feature vector or the deep feature vector to calculate the similarity of the targets in the two frames.
[0070] Figure 5 This is a flowchart illustrating the selection of shallow or deep feature vectors in a target tracking method based on multi-scale feature sharing, as described in an embodiment of this application. Figure 5 As shown, in one embodiment of this example, determining whether to use the shallow feature vector or the deep feature vector to calculate the similarity of targets in two frames based on the location information output by the target detection network and the location information predicted by the Kalman filter includes the following steps S310 to S340.
[0071] Step S310: Calculate the difference value of the target between the two frames based on the position information output by the target detection network and the position information predicted by the Kalman filter;
[0072] Step S320: Compare the difference value with the difference threshold.
[0073] Step S330: When the difference value is greater than or equal to the difference threshold, determine that the shallow feature vector is used to calculate the similarity of the targets in the two frames before and after.
[0074] Step S340: When the difference value is less than the difference threshold, determine that the deep feature vector is used to calculate the similarity of the targets in the two frames before and after.
[0075] Step S400: Based on the determined shallow feature vector or deep feature vector, match the position information output by the target detection network with the position information predicted by the Kalman filter, and update the position information predicted by the Kalman filter based on the matching result.
[0076] In one embodiment of this invention, the difference value is any one of the following: difference hash value, difference mean square error, difference structural similarity index, difference peak signal-to-noise ratio, and difference gradient similarity index. When comparing the difference value with a difference threshold, specifically, the Hamming distance of the difference value is compared with the difference threshold.
[0077] This embodiment uses difference hashing as an example. For instance, based on the target detection results and the location information in the Kalman prediction, the Hamming distance dHash between the target difference hashes is calculated. The magnitude of the difference between the targets is determined based on the dHash value, thereby deciding whether to use shallow or deep feature vectors in calculating the similarity of target appearance features.
[0078] In the specific implementation of this embodiment, when calculating the Hamming distance dHash between the difference hash values, the comparison target is the detection box output by the target detection network and the prediction box output by the Kalman filter. This is a comparison between multiple targets (local image patches) of the same category, rather than the calculation of dHash values between the entire image. This can make full use of shallow and deep features, reduce the amount of computation, and effectively ensure tracking accuracy.
[0079] For example, the Hamming distance dHash between the original image patches of the target bounding box region in two consecutive frames is calculated. When the dHash value is greater than or equal to 10, the intra-class difference of the target can be considered to be large, and shallow feature vectors are used to describe the target. When the dHash value is less than 10, the intra-class difference of the target can be considered to be small, and deep feature vectors are used to describe the target.
[0080] In this embodiment, the feature set is divided into two groups: the shallow feature set and the deep feature set. Based on the position information output by the target detection network, a mapping relationship between the shallow feature set, the deep feature set, and the target is established to form a feature vector set. Then, the difference between targets within the same class is determined based on the Hamming distance (dHash) of the difference hash value. Based on the difference, the cosine distance between feature vectors can be calculated more reasonably, which reduces the amount of computation and also makes reasonable use of the shallow features of the image.
[0081] As can be seen from the above, this embodiment utilizes the target detection network to obtain features of different scales from the outputs of different network layers. These feature sets are then allocated based on the number of network layers, resulting in shallow and deep feature sets. To facilitate subsequent calculation of cosine distance, upsampling and feature fusion are used to unify features of different sizes to the same size. Since the shallow feature set has fewer channels and the deep feature set has more channels, the generated shallow feature vectors are smaller and the deep feature vectors are larger. These feature vectors are then aggregated into shallow and deep feature vector sets. Based on the target location information in the target detection results, the Hamming distance (dHash) between the target's Kalman predicted trajectory and the target detection result is calculated. This dHash value is used to determine the differences between targets within the same class. When the differences are large, the shallow feature vector set is used to calculate the similarity between targets; when the differences are small, the deep feature vector set is used. This separate processing improves computational efficiency, increases the accuracy of similarity calculations for different targets within the same class, and helps reduce the frequency of target tracking ID jumps after occlusion.
[0082] Figure 6 The flowchart shown is a target matching process in a target tracking method based on multi-scale feature sharing according to an embodiment of this application. Figure 6 As shown, in one embodiment of this example, matching the position information output by the target detection network with the position information predicted by the Kalman filter based on the determined shallow feature vector or the deep feature vector, and updating the position information predicted by the Kalman filter based on the matching result, includes at least the following steps S410 and S460.
[0083] Step S410: Perform cascade matching based on the determined shallow feature vector or deep feature vector to obtain the similarity between the target box output by the target detection network and the target box predicted by the Kalman filter, and obtain the similarity matching result based on the similarity.
[0084] Step S460: Update the location information predicted by the Kalman filter based on the matching result. Specifically, update the location information predicted by the Kalman filter based on the similarity matching result.
[0085] In this embodiment, in cascaded matching, the cosine distance between the feature vectors of two targets is calculated, i.e., the similarity. The similarity value is generally between [0,2]. The closer it is to 0, the more similar the two targets are. Conversely, the closer it is to 2, the less similar the two targets are.
[0086] like Figure 6 As shown, in one embodiment of this example, updating the location information predicted by the Kalman filter based on the matching result may further include steps S420 to S460.
[0087] Step S420: Calculate the overlap area between the target bounding box output by the target detection network and the target bounding box predicted by the Kalman filter;
[0088] Step S430: Compare the overlapping area with the overlapping threshold.
[0089] Step S440: When the overlapping area is greater than or equal to the overlapping threshold, the target box output by the target detection network and the target box predicted by the Kalman filter are matched by IOU to obtain the IOU matching result, and the position information predicted by the Kalman filter is updated based on the IOU matching result.
[0090] Step S450: When the overlapping area is less than the overlapping threshold, perform Euclidean distance matching between the target box output by the target detection network and the target box predicted by the Kalman filter to obtain the Euclidean distance matching result, and update the position information predicted by the Kalman filter based on the Euclidean distance matching result.
[0091] When the relative speed between the target and the camera is high, or when there is significant ground jolting, the frame rate may be too low or the target speed too fast. This can result in very small or even no overlap between the target detection box and the Kalman prediction box in adjacent frames, leading to unreasonable IOU matching calculations under normal procedures. In this embodiment, the overlap area between the trajectory and the detection box is calculated, and an overlap threshold is set to determine whether IOU matching is necessary. If the overlap area is greater than the overlap threshold, IOU matching is performed; otherwise, Euclidean distance matching is performed directly. This setting avoids unreasonable IOU matching caused by the aforementioned problem, thereby improving target tracking accuracy.
[0092] Figure 7 The diagram shown illustrates the implementation principle of a target tracking method based on multi-scale feature sharing, according to an embodiment of this application. Figure 7 As shown, firstly, based on the feature extraction network in the object detection model, an image sequence of size 640x640x3 is input, and feature maps of five scales are output: 320x320xC1, 160x160xC2, 80x80xC3, 40x40xC4, and 20x20xC5, where C1-C5 is the number of channels for each scale feature.
[0093] In this embodiment, the features of the multi-scale feature set are fed into the detection head of the target detection model for target classification and localization, obtaining the target's position and category information, which is used as the basic information for subsequent tracking. The target detection information of the first frame is used to initialize the Kalman filter. Considering that the Kalman filter in the tracking algorithm generally only tracks people, in this embodiment, the state space variables are defined as [x,y,a,h,vx,vy,va,vh], where x and y are the coordinates of the center point of the target box, a is the aspect ratio, h is the height of the box, and the last four are the velocity definitions of the corresponding variables in the state space.
[0094] Since multi-class tracking involves not only humans but also other classes, and the motion changes are significant, using aspect ratio and height alone as variables is not reasonable. Therefore, in this embodiment, the target bounding box width and height are defined as variables simultaneously. That is, the initial state space variables are defined as: [x,y,w,h,vx,vy,vw,vh]. Kalman filtering prediction is then performed based on the target detection information from historical frames.
[0095] In this embodiment, for example, features of size 320x320 and 160x160 are defined as shallow feature sets, and features of size 80x80, 40x40, and 20x20 are defined as deep feature sets. Based on the target coordinates and category information output by the target detection model, assuming that the position information of a certain target box is [x0, y0, w, h], where x0 and y0 are the coordinates of the center point of the target box, and w and h are the width and height of the target box, for subsequent calculations, the position information is converted to the top left and bottom right corners of the target box. The converted position information is [x1, y1, x2, y2]. The target position conversion formula is as follows:
[0096]
[0097] Because the feature maps of the last two layers in the deep feature set have very low resolution, the target position error is relatively large when a small target is mapped onto this feature map. Furthermore, to better preserve the high-level semantic information in the deep features, the entire deep feature map set is upsampled to a uniform size of 80x80, while the shallow feature map set remains unchanged. Now, we establish the mapping relationship between the target position information from the original image to the feature map set, obtaining the target position information [x1', y1', x2', y2'] on the feature map. Assuming the input image size is s1xs1, the feature size is s2xs2, and the mapping coefficient is r, the specific mapping formula is as follows:
[0098]
[0099] Based on the target position transformation formula on the feature maps, in the shallow feature set, there are C1 feature maps of size 320x320 and C2 feature maps of size 160x160. The standard deviation of the N target box positions in each feature map is calculated to obtain two matrices, Nx1xC1 and Nx1xC2. These two matrices are then fused and concatenated to obtain the shallow feature vector Nx1x(C1+C2) of the target. In the deep feature set, there are all 80x80 feature maps, totaling C3+C4+C5. Similarly, the standard deviation of the N target box positions in each feature map is calculated to obtain the deep feature vector Nx1x(C3+C4+C5) of the target.
[0100] Therefore, for the shallow feature set, there are C1 feature maps of size 320x320 and C2 feature maps of size 160x160. The standard deviation of each feature map is calculated to obtain two matrices, 1xC1 and 1xC2. These two matrices are then fused and concatenated to obtain the shallow feature vector of the target, 1x(C1+C2). For the deep feature set, there are all feature maps of size 80x80, totaling C3+C4+C5. The standard deviation of each feature map is also calculated to obtain the deep feature vector of the target, 1x(C3+C4+C5).
[0101] To better describe the similarity between targets, this embodiment first calculates the intra-class difference by directly calculating the Hamming distance dHash between the original image blocks of the target bounding box regions in two consecutive frames. When the dHash value is greater than or equal to 10, the intra-class difference of the target can be considered large, and shallow feature vectors are used to describe the target. When the dHash value is less than 10, the intra-class difference of the target can be considered small, and deep feature vectors are used to describe the target.
[0102] Then, based on the determined shallow or deep feature vectors, the position information output by the target detection network is matched with the position information predicted by the Kalman filter. In this embodiment, this step is completed in cascaded matching, calculating the cosine distance between the two target feature vectors, i.e., the similarity, which generally ranges from [0, 2]. The closer to 0, the more similar the two targets are; conversely, the closer to 2, the less similar the two targets are.
[0103] Typically, feature vectors are generated from the Kalman filter prediction and the target detection results, then fed into cascaded matching for similarity calculation, followed by IOU matching. IOU matching calculates the area ratio of two overlapping target boxes. In reality, due to factors such as high target speed, camera lag, and severe ground bumps, the target boxes of the same target may not overlap in the previous and current frames. Calculating IOU matching in such cases would be unreasonable and could cause ID jumps during target tracking. In this embodiment, an overlap threshold β is set, for example, based on 1 / 4 of the area of the target box predicted by the Kalman filter. IOU matching is then performed with the detection results of the current frame. When the overlap area between target boxes is greater than β, IOU matching is used; when the overlap area is less than β, Euclidean distance matching is used. The Euclidean distance matching process is as follows:
[0104] Find the center point and four corner points of the target bounding box. Calculate the coordinates of the top right corner (x3, y3) and bottom left corner (x4, y4) of the target bounding box using the target position transformation formula mentioned above. The specific calculation formula is as follows:
[0105]
[0106] [(x1,y1),(x2,y2),(x3,y3),(x4,y4),(x0,y0)] These are the five points for target detection. The five points predicted by the Kalman filter are [(x1”,y1”),(x2”,y2”),(x3”,y3”),(x4”,y4”),(x0”,y0”)], and the specific calculation formula is as follows:
[0107]
[0108] Finally, taking into account the results of cascade matching, IOU matching, and Euclidean distance matching, the matched and confirmed trajectories are updated using the Kalman filter, while the unmatched and unconfirmed trajectories are deleted.
[0109] The scope of protection of the target tracking method based on multi-scale feature sharing described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.
[0110] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target tracking method based on multi-scale feature sharing provided in any embodiment of this application.
[0111] In the embodiments of this application, any combination of one or more storage media can be used. The storage medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0112] This application also provides an electronic device. Figure 8 The diagram shown is a structural schematic of the electronic device 100 provided in an embodiment of this application. In some embodiments, the electronic device may be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other terminal device. Furthermore, the target tracking method based on multi-scale feature sharing provided in this application can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This application does not impose any limitations on the specific application scenarios of the target tracking method based on multi-scale feature sharing.
[0113] like Figure 8 As shown, the electronic device 100 provided in this application embodiment includes a memory 101 and a processor 102.
[0114] The memory 101 is used to store computer programs; preferably, the memory 101 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.
[0115] Specifically, memory 101 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 101 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.
[0116] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 so that the electronic device 100 executes the target tracking method based on multi-scale feature sharing provided in any embodiment of this application.
[0117] Optionally, the processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0118] Optionally, in this embodiment, the electronic device 100 may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is used to display the relevant GUI interactive interface of the target tracking method based on multi-scale feature sharing.
[0119] In summary, this application utilizes a feature set group in the target detection process to obtain feature maps at different scales, performs multi-scale feature upsampling and fusion, and uses it for target feature vector extraction in tracking. This is more efficient and accurate than using a single neural network to output the target's feature vector, effectively improving the computational efficiency of target tracking and the accuracy of similarity calculation for different targets. In this application, the feature set group in the target detection process is divided into shallow and deep groups, constructing corresponding shallow and deep feature vectors. Then, based on the Hamming distance of the difference values between targets in two consecutive frames, the difference between the two frames is determined. Based on this difference, the cosine distance between feature vectors can be calculated more reasonably, reducing computational load and making reasonable use of the shallow features of the image. This application sets an overlap threshold to determine whether IOU matching is needed. If the overlap area is greater than the overlap threshold, IOU matching is performed; otherwise, Euclidean distance matching is performed directly. This setting solves the problem of unreasonable IOU matching in existing technologies, thereby improving target tracking accuracy. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.
[0120] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.
Claims
1. A target tracking method based on multi-scale feature sharing, characterized in that, include: The image is input into the object detection network to obtain feature maps at different scales, and the feature maps are divided into shallow feature set group and deep feature set group based on feature type; Based on the location information output by the target detection network, shallow feature vectors and deep feature vectors corresponding to the shallow feature set group and the deep feature set group, respectively, are generated; Based on the location information output by the target detection network and the location information predicted by the Kalman filter, it is determined whether to use the shallow feature vector or the deep feature vector to calculate the similarity of targets in the two frames before and after; Based on the determined shallow feature vector or deep feature vector, the position information output by the target detection network is matched with the position information predicted by the Kalman filter, and the position information predicted by the Kalman filter is updated based on the matching result. The step of determining whether to use the shallow feature vector or the deep feature vector to calculate the similarity of targets in two consecutive frames based on the location information output by the target detection network and the location information predicted by the Kalman filter includes: The difference between the target in the two frames is calculated based on the position information output by the target detection network and the position information predicted by the Kalman filter. The difference value is compared with the difference threshold: When the difference value is greater than or equal to the difference threshold, it is determined that the shallow feature vector is used to calculate the similarity of the targets in the two frames before and after. When the difference value is less than the difference threshold, the similarity between the targets in the two frames is determined by using the deep feature vector.
2. The target tracking method based on multi-scale feature sharing according to claim 1, characterized in that, The step of generating shallow feature vectors and deep feature vectors corresponding to the shallow feature set group and the deep feature set group, respectively, based on the location information output by the target detection network, includes: The target positions of the feature maps in the shallow feature set and the deep feature set are obtained based on the target positions and mapping coefficients output by the target detection network; wherein, the target position of the feature map = the target position output by the target detection network / the mapping coefficient; Based on the target positions of the feature maps in the shallow feature set and the deep feature set, the corresponding shallow feature vectors and deep feature vectors are calculated for each feature map in the shallow feature set and the deep feature set, respectively.
3. The target tracking method based on multi-scale feature sharing according to claim 2, characterized in that, Calculating the corresponding shallow feature vectors for each feature map in the shallow feature set includes: The shallow feature matrix corresponding to each shallow feature set group is obtained by calculating the location region of each target box in each feature map of each shallow feature set group. The shallow feature matrices are fused and spliced together to obtain the shallow feature vector.
4. The target tracking method based on multi-scale feature sharing according to claim 2, characterized in that, Calculating the corresponding deep feature vectors for each feature map in the deep feature set includes: The feature maps in the deep feature map set are upsampled to a uniform size to form feature maps of the same size; The deep feature vector is obtained by calculating the location regions of each target box in each feature map of each deep feature set group.
5. The target tracking method based on multi-scale feature sharing according to claim 1, characterized in that, The difference value is any one of the following: difference hash value, difference mean square error, difference structural similarity index, difference peak signal-to-noise ratio, and difference gradient similarity index.
6. The target tracking method based on multi-scale feature sharing according to claim 1, characterized in that, Matching the location information output by the target detection network with the location information predicted by the Kalman filter based on the determined shallow or deep feature vectors, and updating the location information predicted by the Kalman filter based on the matching result includes: Based on the determined shallow feature vector or deep feature vector, cascade matching is performed to obtain the similarity between the target box output by the target detection network and the target box predicted by the Kalman filter, and the similarity matching result is obtained based on the similarity. The location information predicted by the Kalman filter is updated based on the similarity matching results.
7. The target tracking method based on multi-scale feature sharing according to claim 6, characterized in that, The process of updating the Kalman filter-predicted location information based on the matching results also includes: Calculate the overlap area between the target bounding box output by the target detection network and the target bounding box predicted by the Kalman filter; The overlapping area is compared with the overlapping threshold: When the overlapping area is greater than or equal to the overlapping threshold, the target box output by the target detection network and the target box predicted by the Kalman filter are matched by IOU to obtain the IOU matching result, and the position information predicted by the Kalman filter is updated based on the IOU matching result. When the overlapping area is less than the overlapping threshold, the target bounding box output by the target detection network and the target bounding box predicted by the Kalman filter are matched by Euclidean distance to obtain the Euclidean distance matching result, and the position information predicted by the Kalman filter is updated based on the Euclidean distance matching result.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target tracking method based on multi-scale feature sharing as described in any one of claims 1 to 7.
9. An electronic device, characterized in that, The electronic device includes: Processor and memory; The memory stores program instructions; The processor is configured to run the program instructions to execute the target tracking method based on multi-scale feature sharing as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Fingertip tracking method based on deep learning and K-curvature method
CN113608663A
PCB element detection method and system based on deep learning, and storage medium
CN116168017A
