A target tracking method, apparatus and device

Abstract

This application provides a target tracking method, apparatus, and device. The method includes: for a target image in a sequence of images to be detected, determining a search region of the target image and extracting features from the search region to obtain a first feature description of the target image; then filtering out valid feature descriptions associated with the target from the first feature description; calculating the homography matrix from the template image to the target image; further calculating state parameters of the valid feature descriptions; then determining whether the target is identified in the target image, and if the target is identified, determining the target bounding box corresponding to the target in the target image based on the state parameters of the valid features; and updating the state parameters of the target to the state parameters of the valid features. The target tracking method, apparatus, and device provided in this application are used for accurate and reliable target tracking.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a target tracking method, apparatus, and device. Background Technology

[0002] With the development of computer vision technology, people's demand for target tracking is becoming more and more urgent, such as in video surveillance, autonomous driving, and human-computer interaction.

[0003] One of the most pressing problems in target tracking is the accurate tracking of any object. Current target tracking algorithms cannot adapt to changing scenarios. When there are issues such as lighting, background interference, rotation, blurring, occlusion, pose changes, and many other problems, target tracking algorithms cannot guarantee the accuracy of detection. Summary of the Invention

[0004] In view of this, this application provides a target tracking method, apparatus, and device for accurate and reliable target tracking.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] A first aspect of this application provides a target tracking method, the method comprising:

[0007] For the target image to be detected in the image sequence, the search region of the target image is determined, and features are extracted from the search region to obtain the first feature description of the target image;

[0008] Based on the target feature description of the target in the template image and the context feature description of the context associated with the target in the template image, valid feature descriptions associated with the target are filtered from the first feature descriptions; wherein, the valid feature descriptions are feature descriptions that match the target feature descriptions but do not match the context feature descriptions;

[0009] Based on the effective feature description and the target feature description that matches the effective feature description, the homography matrix from the template image to the target image is calculated using the Ransac algorithm;

[0010] Based on the effective feature description, the target feature description matching the effective feature description, and the homography matrix, the state parameters of the effective feature description are calculated using the Mlesac likelihood loss function.

[0011] Based on the state parameters of the effective features and the current state parameters of the target, it is determined whether the target is identified in the target image, and when it is determined that the target is identified, the target bounding box corresponding to the target in the target image is determined based on the state parameters of the effective features;

[0012] The state parameters of the target are updated to the state parameters of the effective feature; wherein the initial value of the state parameters of the target is a preset value.

[0013] A second aspect of this application provides a target tracking device, the device comprising an extraction module, a filtering module, a calculation module, a determination module, and an update module; wherein,

[0014] The extraction module is used to determine the search region of the target image in the image sequence to be detected, and to extract features from the search region to obtain a first feature description of the target image.

[0015] The filtering module is used to filter out valid feature descriptions associated with the target from the first feature description based on the target feature description of the target in the template image and the context feature description of the context associated with the target in the template image; wherein, the valid feature description is a feature description that matches the target feature description but does not match the context feature description;

[0016] The calculation module is used to calculate the homography matrix from the template image to the target image based on the effective feature description and the target feature description that matches the effective feature description, using the Ransac algorithm.

[0017] The calculation module is used to calculate the state parameters of the effective feature description based on the Mlesac likelihood loss function, according to the effective feature description, the target feature description matching the effective feature description, and the homography matrix.

[0018] The determining module is used to determine whether the target is identified in the target image based on the state parameters of the effective features and the current state parameters of the target, and when it is determined that the target is identified, to determine the target bounding box corresponding to the target in the target image based on the state parameters of the effective features;

[0019] The update module is used to update the state parameters of the target to the state parameters of the effective feature; wherein the initial value of the state parameters of the target is a preset value.

[0020] A third aspect of this application provides a target tracking device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0021] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0022] The target tracking method, apparatus, and device provided in this application offer long-term tracking capabilities and can solve complex scene problems encountered in real-world scenarios. They utilize locally invariant features to establish feature descriptions of the target and context, improving robustness. Furthermore, the use of transitive matching enhances accuracy. Moreover, by extracting new features from the target and context and updating the target and context feature descriptions accordingly, the descriptions become consistent with the real-time state, more accurately and fully reflecting the target and context. Subsequent target identification based on this further improves accuracy and robustness. Additionally, by identifying occlusion feature descriptions, accuracy is further improved. Moreover, updating the target and context feature descriptions only when the target is not occluded avoids incorporating invalid features, further enhancing accuracy. Attached Figure Description

[0023] Figure 1 A flowchart of an embodiment of the target tracking method provided in this application;

[0024] Figure 2 A schematic diagram illustrating the target, context, and search area for an exemplary embodiment of this application;

[0025] Figure 3 This is an illustrative embodiment of the present application showing the relationship between the target bounding box in the template image and the target bounding box in each frame of the image to be detected;

[0026] Figure 4 A flowchart of Embodiment 2 of the target tracking method provided in this application;

[0027] Figure 5 A flowchart of Embodiment 3 of the target tracking method provided in this application;

[0028] Figure 6 A flowchart of Embodiment 4 of the target tracking method provided in this application;

[0029] Figure 7This is a hardware structure diagram of the target tracking device in this application.

[0030] Figure 8 This is a schematic diagram of the structure of a target tracking device according to an embodiment of this application. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0032] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0033] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0034] The target tracking method and apparatus provided in this application are applied to a target tracking device, which can be an intelligent forklift AGV, so that the AGV can track and locate the target.

[0035] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0036] Figure 1 A flowchart of an embodiment of the target tracking method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0037] S101. For the target image to be detected in the image sequence, determine the search region of the target image, and extract features from the search region to obtain the first feature description of the target image.

[0038] Specifically, the image sequence to be detected comprises multiple frames. Furthermore, the images to be detected contain one or more targets that need to be tracked; these targets can be people, vehicles, animals, objects, etc. It should be noted that the image sequence to be detected can be an image sequence acquired by imaging devices such as cameras, drones, and sensors.

[0039] Specifically, during target tracking, each frame of the image is detected frame by frame. For example, in one possible implementation, the first frame of the image sequence is set as the template image, and the image sequence consisting of the images after the first frame is set as the image sequence to be detected. During target tracking, the target and its associated context are defined in the template image, and then each frame in the image sequence to be detected is detected frame by frame to identify the target. It should be noted that, for ease of explanation, the current frame to be detected is referred to as the target image.

[0040] Optionally, in one embodiment, the entire area of ​​the target image can be determined as the search area. In another embodiment, the search area of ​​the target image can be determined based on the area where the target is located in the template image, or the area where the context associated with the target in the template image is located. In this embodiment, the method for determining the search area is not limited.

[0041] For example, Figure 2 This is a schematic diagram illustrating the target, context, and search area for an exemplary embodiment of this application. Please refer to... Figure 2 In one embodiment, faces in the image are identified as the targets to be tracked. Figure 2 Figure a in the image illustrates the position of the target within the template image. Figure 2 Figure b in the image illustrates the location of the target-related context within the template image. Furthermore, Figure 2 Figure C in the diagram illustrates the location of the search area in the target image. In this embodiment, the target will be searched within the search area.

[0042] Furthermore, traditional feature extraction methods or neural network-based feature extraction methods can be used to extract features from the search region. In this embodiment, no limitation is imposed.

[0043] Furthermore, the feature descriptions obtained through feature extraction are determined as the first feature description of the target image. It should be noted that each feature description in the first feature description corresponds to a feature point, and this feature description can be represented by its location and features.

[0044] S102. Based on the target feature description of the target in the template image and the context feature description of the context associated with the target in the template image, select valid feature descriptions associated with the target from the first feature descriptions; wherein, the valid feature descriptions are feature descriptions that match the target feature descriptions but do not match the context feature descriptions.

[0045] Specifically, as described above, the template image defines the target and its associated context. It should be noted that the target and its associated context in the template image can be manually annotated or identified using an object detection algorithm; this embodiment does not impose any limitations. The following explanation uses the example where the target and its associated context in the template image are manually annotated.

[0046] Furthermore, it can be understood that the context associated with a target is other objects or entities around the target. This context can provide additional information, which helps to better understand the target's behavior and allows for more accurate identification and tracking of the target.

[0047] In practice, for the template image, features can be extracted from the region where the target is located. The extracted feature description is the feature description of the target. For ease of explanation, the feature description of the target is referred to as the target feature description. Similarly, features can be extracted from the region where the context is located. The extracted feature description is the feature description of the context. For ease of explanation, the feature description of the context is referred to as the context feature description.

[0048] It should be noted that when extracting features from the region where the target is located and / or the region of context, the feature extraction method is selected according to actual needs, and this embodiment does not impose any limitations on it. For example, a feature extraction method based on neural networks can be used.

[0049] Furthermore, it can be understood that the first feature description contains all features of the search area. The first feature description includes feature descriptions that match the target feature and feature descriptions that match the context feature. When a feature description matches both the target feature and the context feature, the feature description is meaningless for identifying the target. Therefore, it is necessary to filter out the feature descriptions that match the target feature description but do not match the context feature description from the first feature description. The filtered feature descriptions are the valid feature descriptions associated with the target. Subsequently, target tracking can be performed based on the valid feature descriptions.

[0050] Optionally, in one embodiment, the specific implementation process of this step may include:

[0051] (1) Using the nearest neighbor search criterion or symmetry criterion of the distance ratio criterion, find the first matching feature description that matches the target feature description from the first feature description.

[0052] Specifically, the distance ratio criterion is a criterion used to determine whether two points are a match. This criterion is based on the distance between two points and can flexibly and efficiently determine whether two points are related.

[0053] Furthermore, the symmetry criterion utilizes the symmetry information between points, making the matching results more reasonable and stable. The symmetry criterion can be applied to different types of tasks.

[0054] In practice, mismatched points are removed using a selection process based on the distance ratio criterion or the symmetry criterion. For example, in one embodiment, the nearest neighbor is defined as the feature point with the smallest Euclidean distance to the target feature description. At the same time, based on the symmetry criterion, the first feature description and the target feature description are detected to determine the points that satisfy the distance ratio criterion or the symmetry criterion. The feature descriptions corresponding to these points are the first matching feature descriptions.

[0055] (2) Using the nearest neighbor search criterion or symmetry criterion of the distance ratio criterion, find a second matching feature description that matches the context feature description from the first feature description.

[0056] The specific implementation process and principle of this step are similar to those of step (1), and will not be repeated here.

[0057] In practice, mismatched points are removed using a selection process based on the distance ratio criterion or the symmetry criterion. For example, in one embodiment, the nearest neighbor is defined as the feature point with the smallest Euclidean distance to the context feature description. Simultaneously, based on the symmetry criterion, the first feature description and the context feature description are detected to determine points that satisfy the distance ratio criterion or the symmetry criterion. The feature descriptions corresponding to these points are the second matching feature descriptions.

[0058] In other words, by matching the first feature description with the target feature description, a first matching feature description can be obtained; similarly, by matching the first feature description with the context feature description, a second matching feature description can be obtained, as shown below:

[0059] T t * =T t -S t

[0060] C t * =C t -S t

[0061] Among them, St T represents the first feature description. t Target feature description, C t T represents a contextual feature description. t * C represents the first matching feature description. t * The second matching feature description is represented by the above formula, which represents the matching relationship between the two features and the first feature description of the target image.

[0062] (3) Delete the second matching feature description from the first matching feature description to obtain the effective feature description.

[0063] In practice, the second feature matching description is deleted from the first matching feature description, and the remaining feature matching description is the valid feature description.

[0064] It is understandable that in the first feature description, there may be some feature descriptions that match both the target's feature description and the context's feature description. Since such feature descriptions cannot distinguish between the target and the context, in this application, these feature descriptions are considered invalid. Therefore, these invalid feature descriptions are removed from the matching results to obtain valid feature descriptions, as shown below:

[0065]

[0066] Wherein, F represents t This represents the final selected valid feature descriptions, which are obtained by removing the second matching feature descriptions that match the context feature description from the first matching feature descriptions that match the target effect description.

[0067] The method in this embodiment utilizes the nearest neighbor search criterion or symmetry criterion based on the distance ratio to find a first matching feature description that matches the target feature description from the first feature description, and then finds a second matching feature description that matches the context feature description from the first feature description. Finally, the second matching feature description is deleted from the first matching feature description to obtain a valid feature description. In this way, feature descriptions that match the target but do not match the context can be obtained, resulting in valid feature descriptions. These valid feature descriptions can effectively distinguish between the target and the context, avoiding interference from the context in target recognition and improving the accuracy of target recognition.

[0068] S103. Based on the effective feature description and the target feature description that matches the effective feature description, the homography matrix from the template image to the target image is calculated using the Ransac algorithm.

[0069] Specifically, the Ransac algorithm is an iterative method for fitting models and identifying outliers in data. It is widely used in the field of homography transformation. The Ransac algorithm can calculate the optimal homography matrix from effective feature descriptions.

[0070] It should be noted that the homography matrix is ​​a matrix that describes perspective transformation relationships. The homography matrix between different image frames can reflect the translation, rotation, and scaling relationships of the target. By using the homography matrix, the position of the target can be matched between different image frames, which helps to improve the accuracy of target tracking.

[0071] Furthermore, the homography from the template image to the target image has 8 degrees of freedom; therefore, only 4 matching feature points are needed to calculate the homography. It should be noted that the more data used in the calculation process, the more reliable the result.

[0072] Specifically, the implementation process of this step may include:

[0073] (1) Randomly select a small sample set (usually 4 feature points) from all feature points corresponding to the effective feature description.

[0074] (2) Calculate the homography matrix based on the feature descriptions corresponding to the above sample sets.

[0075] (3) Project other feature point pairs corresponding to the effective feature descriptions into the current frame, and calculate their projection errors based on the calculated homography matrix.

[0076] (4) Compare the projection error with the threshold to determine which feature point pairs are considered interior points.

[0077] (5) Calculate the number of feature point pairs that were considered interior points in step (4).

[0078] (6) If the number of inliers exceeds the threshold (e.g., greater than a certain percentage of the total number of feature points), these inliers are merged with the initial sample point pairs, and the homography matrix is ​​recalculated.

[0079] (7) If the number of inliers is insufficient, repeat step (1) to select random sample point pairs again.

[0080] (8) Repeat steps (2) to (5) until the number of iterations is met or the stopping condition is met (e.g., a sufficient number of interior points are found).

[0081] (9) Based on the final determined interior points, recalculate the homography matrix.

[0082] S104. Based on the effective feature description, the target feature description matching the effective feature description, and the homography matrix, calculate the state parameters of the effective feature description using the Mlesac likelihood loss function.

[0083] Since the target is in a planar graph, four planar parameters are selected to represent the target's state. Specifically, the target's state parameters are (λ... x , λ y , s, θ), where λ x Used to represent positional change in the x-direction, λ y The y-axis represents positional change, s represents scale change, and θ represents planar rotation. Similarly, the state parameters of the effective features are also (λ). x , λ y ,s,θ).

[0084] The state parameters of effective features can be estimated based on the Mlesac likelihood loss function. The target image needs to be nonlinearly constrained with the template image to ensure the accuracy of the target tracking algorithm. The Mlesac likelihood loss function can be used to optimize the nonlinear constraints.

[0085] Specifically, the Mlesac likelihood loss function is:

[0086]

[0087] e = ||lM -1 ×l * || 2 +||l * -M×l|| 2

[0088]

[0089] Wherein, γ is the mixing parameter, θ is the outlier distribution information, σ is the standard deviation of the error on each matched feature, e is the maximum likelihood estimation error parameter; l is the feature point location described by the effective feature description, l * The location of the feature points described in the target feature description;

[0090] M is the homography matrix, s is the scale change, θ is the planar rotation, and λ is the homography matrix. x For the positional change in the x-direction, the λ y The positional change in the y-direction; (λ) x , λ y , s, θ) are the state parameters describing the effective features.

[0091] Specifically, the state parameters of the effective features are calculated by minimizing the Mlesac likelihood loss function.

[0092] Specifically, the Mlesac likelihood loss function is minimized to calculate the state parameters of the effective features. This step can be characterized by the following formula:

[0093]

[0094] Where P is the state parameter of the effective feature, Ft is the description of the effective feature, and L is the Mlesac likelihood loss function.

[0095] S105. Based on the state parameters of the effective features and the current state parameters of the target, determine whether the target is identified in the target image, and when it is determined that the target is identified, determine the target bounding box corresponding to the target in the target image based on the state parameters of the effective features.

[0096] S106. Update the state parameters of the target to the state parameters of the effective feature; wherein the initial value of the state parameters of the target is a preset value.

[0097] It should be noted that the initial value of the target's state parameters is a preset value. For example, when the target image to be detected is the second frame image in the image to be detected, if the target is identified, the target's state parameters will be determined as the state parameters of the effective features. Furthermore, when the image to be detected is the third frame image, the target is identified based on the state parameters obtained from the second frame image.

[0098] In other words, this application uses a continuous pass-through matching method to detect targets, which compares the state parameters in the current frame with the state parameters in the previous frame to determine whether a target has been identified. Specifically, when calculating the state parameters of valid features, the state parameters of the valid features are compared with the current state parameters of the target to determine whether the target has been identified.

[0099] In practice, the target can be identified when the state parameters of the valid features and the current state parameters of the target satisfy the following conditions:

[0100]

[0101] ||s t -s t-1 ||<thre s

[0102] ||θ t -θ t-1 ||<thre θ

[0103] in, State parameters that are valid features; The current state parameter of the target, thre l The preset position parameter threshold, thre s The preset scale ratio change threshold, thre θ This is a preset planar rotation threshold. If all three conditions are met, then the target is determined to have been identified in the target image.

[0104] Furthermore, after confirming the presence of the target in the target image, the target's position is determined based on its current state parameters, and a bounding box is drawn in the target image. It should be noted that the bounding box needs to completely contain the target and be as small as possible to accurately include the target.

[0105] Furthermore, referring to the preceding description, once a target is identified in the target image, the target's state parameters are updated to the state parameters of the valid features for continuous transmission matching. For example, in one embodiment, the target's state parameters are P1, and the valid feature's state parameters are P2. After determining that a target exists in the target image, the target's state parameters are updated to P2.

[0106] Furthermore, the initial value of the target's state parameters is a preset value. The specific value of this preset value can be set by the configuration personnel according to actual needs. In this embodiment, it is not limited to this.

[0107] Specifically, Figure 3 This diagram illustrates the relationship between the target bounding box in the template image and the target bounding box in each frame of the image to be detected, as shown in an illustrative embodiment of this application. Please refer to... Figure 3 Please refer to Figure 3 The method provided in this embodiment is based on a continuous transmission matching method for target tracking, which can accurately identify the target.

[0108] The target tracking method provided in this embodiment first identifies effective feature descriptions by combining the target feature description with the context feature description. Then, it detects the target based on the effective feature description. This distinguishes the target from the context, improving the accuracy of the identification. Furthermore, the method provided in this embodiment utilizes locally invariant features as feature descriptions. These locally invariant features are highly robust to plane rotation and scale changes, and can handle the influence of illumination to a certain extent, thereby improving the accuracy and reliability of the identification.

[0109] Figure 4 A flowchart of Embodiment 2 of the target tracking method provided in this application. Please refer to... Figure 4Based on the above embodiments, after determining the target bounding box corresponding to the target in the target image according to the state parameters of the effective features, the method further includes:

[0110] S401. Search for a first target feature description belonging to the target box from the first feature description, and when the first target feature description matches the context feature description, determine that the first target feature description is an occlusion feature description corresponding to the occlusion object.

[0111] Specifically, the first target feature description is a feature description belonging to the target bounding box in the first feature description. In a specific implementation, for example, in one embodiment, the first feature description is T1, where the feature description located in the target bounding box is T12, and T12 is the first target feature description.

[0112] Furthermore, the portion of the first target feature description that matches the context features is determined as the occlusion feature description corresponding to the occlusion object. Specifically, in one embodiment, combining the above example, the first target feature description is T12, and the context feature description is T4. T12 and T4 are compared and matched, and the matching result is used as the occlusion feature description corresponding to the occlusion object.

[0113] In a specific implementation, in one embodiment, the occlusion feature description corresponding to the occlusion object can be determined based on the following formula:

[0114] Occ t ={(l,f)|(l,f)∈C t ∩(l,f)∈OBB t}

[0115] Among them, Occ t Here, (l, f) represents the occlusion feature description corresponding to the occluded object, and C represents the first feature description. t For contextual feature description, OBB t The bounding box corresponding to the target in the target image.

[0116] S402. When the number of occlusion feature descriptions is greater than a preset threshold, determine that the target is occluded, and execute the step of determining the search area of ​​the target image again.

[0117] Specifically, a preset threshold is used to determine whether a target is occluded. When the number of occlusion feature descriptions exceeds the preset threshold, the target is determined to be occluded; otherwise, the target is determined not to be occluded. It should be noted that the specific value of the preset threshold is set according to actual needs, and this embodiment does not limit it.

[0118] In a specific implementation, for example, in one embodiment, combined with the example above, if the number of preset thresholds is 30, when the number of occlusion feature descriptions is 45, the number of occlusion feature descriptions is greater than the preset threshold, and it is determined that the target is occluded.

[0119] Furthermore, after confirming that the target is occluded, the currently identified target is determined to be invalid. At this point, the search area of ​​the target image is determined again, and the target is re-identified to improve the accuracy of the detection results.

[0120] Furthermore, after obtaining the occlusion feature descriptions corresponding to the occluded objects, an occlusion feature database can be established. The occlusion feature descriptions corresponding to the occluded objects can be stored in the occlusion feature database for subsequent processing. Specifically, the occlusion feature database can be obtained using the following formula:

[0121] D t =D t-1 ∪Occ t

[0122] Among them, D t For the updated occlusion feature database, D t-1 For the current occlusion feature database, Occ t This describes the occlusion features corresponding to the occluded object.

[0123] S403. When the number of occlusion feature descriptions is not greater than the preset threshold, it is determined that the target is not occluded, and the target box is determined as the tracking result corresponding to the target image.

[0124] In specific implementation, for example, in one embodiment, combined with the above example, when the number of preset thresholds is 50, when the number of occlusion feature descriptions is 45, the number of occlusion feature descriptions is not greater than the preset threshold, it is determined that the target is not occluded, and the target box is directly determined as the tracking result of the target image.

[0125] The target tracking method provided in this embodiment identifies occlusion feature descriptions. If the number of occlusion feature descriptions exceeds a preset threshold, the method determines that the target is occluded and re-executes the step of determining the search area of ​​the target image. Conversely, if the number of occlusion feature descriptions does not exceed the preset threshold, the method determines that the target is not occluded and identifies the target bounding box as the tracking result corresponding to the target image. Based on occlusion feature descriptions, the method confirms target identification only when it determines that the target is not occluded; otherwise, it re-identifies the target, making the target tracking results more accurate and reliable.

[0126] Figure 5 The flowchart for Embodiment 3 of the target tracking method provided in this application is shown below. Please refer to... Figure 5 Based on the above embodiments, after determining that the target is not occluded, the method further includes:

[0127] S501. Find the second target feature description belonging to the target box from the effective feature description, and find the target feature point corresponding to the feature point indicated by the second target feature description in the template image according to the homography matrix.

[0128] Specifically, the second target feature description refers to the feature description that belongs to the target bounding box among the effective feature descriptions. In other words, all the effective feature descriptions contained in the target bounding box constitute the second target feature description, which is formulated as follows:

[0129] T t new =F t ∩OBB t

[0130] in, For the second target feature description, F t As an effective feature, OBB t The bounding box corresponding to the target in the target image.

[0131] Specifically, as described above, the homography matrix describes the homography relationship between the template image and the target image. Through the homography matrix, the target feature points corresponding to the feature points indicated by the second target feature description can be found in the template image.

[0132] Specifically, it is represented as follows:

[0133] l * =H t l

[0134] Among them, H t Let l be the homography matrix between the target image to be detected and the template image, and l represent the position of the feature points indicated by the second target feature description in the target image to be detected. * This indicates the position of the target feature point in the template image, where the target feature point is the feature point in the template image corresponding to the feature point indicated by the second target feature description.

[0135] S502. Add the feature description of the target feature point to the target feature description to update the target feature description.

[0136] The target tracking method provided in this embodiment searches for a second target feature description belonging to the target bounding box from the valid feature descriptions of the current target image to be detected. This allows for the capture of new target features from the target image. These features are then projected onto the target image using a homography matrix, and the target feature description is updated accordingly. This results in a more accurate and comprehensive description of the target, improving the accuracy and robustness of subsequent target identification. Furthermore, the method in this embodiment only updates the target feature description when the target is not occluded, avoiding the inclusion of useless feature descriptions and further improving accuracy.

[0137] Figure 6 The flowchart for Embodiment 4 of the target tracking method provided in this application is shown below. Please refer to... Figure 6 Based on the above embodiments, after determining that the target is not occluded, the method further includes:

[0138] S601. Search for a third target feature description that does not belong to the target box from the first feature description.

[0139] Specifically, feature descriptions in the first feature description that do not belong to the target bounding box are identified as third target feature descriptions. In practice, the position of each feature description in the first feature description can be used to determine whether the feature point indicated by that feature description is within the target bounding box. If it is not within the target bounding box, then that feature description is identified as the third target feature description. Specifically, the process of finding the third target feature description can be represented as follows:

[0140]

[0141] Among them, C t S represents the third feature description of the target image, (l, f) represents the first feature description of the target image, and S represents the third feature description of the target image. i For the search region of the target image, OBB i This refers to the corresponding bounding box in the target image.

[0142] S602. Based on the homography matrix, find the feature points in the template image that correspond to the feature points indicated by the third target feature description and the feature points that correspond to the feature points indicated by the occlusion feature description.

[0143] Specifically, the homography matrix describes the homography relationship between the template image and the target image. Through the homography matrix, feature points corresponding to the feature points indicated by the third target feature description and feature points corresponding to the feature points indicated by the occlusion feature description can be found from the template image.

[0144] S603. Add the feature descriptions of the feature points corresponding to the feature points indicated by the third target feature description and the feature descriptions of the feature points corresponding to the feature points indicated by the occlusion feature description to the context feature description to update the context feature description.

[0145] Specifically, in this step, the feature descriptions of the feature points corresponding to the feature points indicated by the third target feature description, as well as the feature descriptions of the feature points corresponding to the feature points indicated by the occlusion feature description, are added to the context features. This allows for a better distinction between the context features and the target feature descriptions, and enables the updating of the context features.

[0146] It should be noted that, understandably, the feature points corresponding to the third target feature description and the feature points corresponding to the occlusion feature description are feature points that do not belong to the target. In this way, by adding new changes in the context to the context feature description, the context feature description can be updated, resulting in more accurate target tracking.

[0147] Furthermore, context plays a crucial role in occlusion detection. If the number of frames an occlusion remains on the target exceeds the interval, the occlusion may be mistakenly identified as part of the target, failing to detect it and even being updated in the target feature description. However, as described earlier, these occlusions will be included in the occlusion database. This situation can be avoided if the feature descriptions in the occlusion database are accumulated into the context feature description in a timely manner. Therefore, for updating the context feature description, both new changes in the context region and new features in the occlusion database should be added simultaneously. In other words, the method provided in this embodiment, through... Figure 6 The process shown can add new changes in the context region and new features from the occlusion database to the context feature description, enabling more accurate target tracking.

[0148] It should be noted that, in one embodiment, the contextual feature description is obtained by accumulating the features of the context over a historical length across multiple target images. For example, in one embodiment, the contextual feature description can be obtained based on τ images preceding the current target image to be detected.

[0149] The target tracking method provided in this embodiment firstly searches for third target feature descriptions that do not belong to the target bounding box from the first feature description of the target image to be detected. This captures new contextual features from the target image, which are then projected onto the target image using a homography matrix. Based on this, the contextual feature description is updated, resulting in a more accurate and comprehensive description of the context. This improves the accuracy and robustness of subsequent target identification. Secondly, by acquiring occlusion feature descriptions and adding them to the contextual feature description, occlusion is avoided from being mistakenly identified as part of the target, thus preventing its impact on target identification and further improving accuracy. Furthermore, the method provided in this embodiment only updates the target feature description when the target is not occluded, avoiding the updating of useless feature descriptions and further improving accuracy.

[0150] The target tracking method provided in this embodiment offers long-term tracking capabilities and can address complex scene factors encountered in real-world scenarios. It utilizes locally invariant features to establish feature descriptions of the target and context, improving robustness. Furthermore, it employs a transitive matching method to enhance accuracy. Moreover, by extracting new features from the target and context and updating the target and context feature descriptions accordingly, the descriptions become consistent with the real-time state, more accurately and fully reflecting the target and context. Subsequent target identification based on this further improves accuracy and robustness. Additionally, by identifying occlusion feature descriptions, accuracy is further improved. Moreover, by updating the target and context feature descriptions only when the target is not occluded, invalid features are avoided, further enhancing accuracy.

[0151] It should be noted that during long-term tracking or when the target and environment change rapidly, many new features will emerge and be added to the target feature description and context feature description. However, the uncontrollable growth in the number of features will slow down the entire tracking process, and as the target changes, many old features in the target feature description may no longer have corresponding features. Therefore, it is necessary to remove old and invalid features.

[0152] Optionally, in one possible implementation of this application, when the number of target feature descriptions is greater than a first preset number, a uniform random sampling strategy is used to delete some target feature descriptions;

[0153] And / or,

[0154] When the number of context feature descriptions exceeds a second preset number, a uniform random sampling strategy is used to delete some context feature descriptions.

[0155] Specifically, the exact value of the first preset quantity is set by the staff according to actual needs, and this embodiment does not limit it. For example, in one embodiment, the first preset quantity can be set to N based on the machine's memory and processing speed.

[0156] Furthermore, the specific value of the second preset quantity is set by the staff according to actual needs, and this embodiment does not limit it. For example, in one embodiment, the second preset quantity can be set to N1 based on the machine's memory and processing speed.

[0157] The method provided in this embodiment employs a uniform random sampling strategy to delete some target feature descriptions when the number of target feature descriptions exceeds a first preset number, and similarly, employs a uniform random sampling strategy to delete some context feature descriptions when the number of context feature descriptions exceeds a second preset number. This approach prevents excessive data volume caused by prolonged target tracking, keeps the computing device in optimal condition, improves the performance and reliability of target tracking, and avoids the problem of excessively long recognition processes due to large data volumes. Furthermore, it allows for optimization of feature descriptions to quickly and accurately identify targets.

[0158] Corresponding to the aforementioned embodiment of a target tracking method, this application also provides an embodiment of a target tracking device.

[0159] An embodiment of the target tracking device disclosed in this application can be applied to a target tracking device. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the target tracking device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 7 The diagram shown is a hardware structure diagram of a target tracking device in this application, except for... Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, the target tracking device in the embodiment may also include other hardware depending on the actual function of the target tracking device, which will not be described in detail here.

[0160] Figure 8 This is a schematic diagram of the structure of a first embodiment of the target tracking device provided in this application. Please refer to... Figure 8 The apparatus provided in this embodiment includes an extraction module 810, a filtering module 820, a calculation module 830, a determination module 840, and an update module 850; wherein,

[0161] The extraction module 810 is used to determine the search region of the target image in the image sequence to be detected, and to extract features from the search region to obtain a first feature description of the target image.

[0162] The filtering module 820 is used to filter out valid feature descriptions associated with the target from the first feature description based on the target feature description of the target in the template image and the context feature description of the context associated with the target in the template image; wherein, the valid feature description is a feature description that matches the target feature description but does not match the context feature description;

[0163] The calculation module 830 is used to calculate the homography matrix from the template image to the target image based on the effective feature description and the target feature description that matches the effective feature description, using the Ransac algorithm.

[0164] The calculation module 830 is further configured to calculate the state parameters of the effective feature description based on the Mlesac likelihood loss function, according to the effective feature description, the target feature description matching the effective feature description, and the homography matrix.

[0165] The determining module 840 is used to determine whether the target is identified in the target image based on the state parameters of the effective features and the current state parameters of the target, and when it is determined that the target is identified, to determine the target bounding box corresponding to the target in the target image based on the state parameters of the effective features;

[0166] The update module 850 is used to update the state parameters of the target to the state parameters of the effective feature; wherein the initial value of the state parameters of the target is a preset value.

[0167] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0168] Optionally, the Mlesac likelihood loss function is:

[0169]

[0170] e = ||lM -1 ×l * || 2 +||l * -M×l|| 2

[0171]

[0172] Wherein, γ is the mixing parameter, θ is the outlier distribution information, σ is the standard deviation of the error on each matched feature, e is the maximum likelihood estimation error parameter; l is the feature point location described by the effective feature description, l * The location of the feature points described in the target feature description;

[0173] M is the homography matrix, s is the scale change, θ is the planar rotation, and λ is the homography matrix. x For the positional change in the x-direction, the λ y The positional change in the y-direction; (λ) x , λ y , s, θ) are the state parameters describing the effective features;

[0174] The calculation of the state parameters describing the effective features includes:

[0175] The calculation module 830 is specifically used to calculate the state parameters of the effective features by minimizing the Mlesac likelihood loss function.

[0176] Optionally, the determining module 840 is further configured to search for a first target feature description belonging to the target box from the first feature description, and when the first target feature description matches the context feature description, determine that the first target feature description is an occlusion feature description corresponding to the occlusion object;

[0177] The determining module 840 is further configured to determine that the target is occluded when the number of occlusion feature descriptions is greater than a preset threshold, and to perform the step of determining the search area of ​​the target image again;

[0178] The determining module 840 is further configured to determine that the target is not occluded when the number of occlusion feature descriptions is not greater than the preset threshold, and to determine the target box as the tracking result corresponding to the target image.

[0179] Optionally, the filtering module 820 is further configured to search for a second target feature description belonging to the target box from the valid feature descriptions, and find the target feature point corresponding to the feature point indicated by the second target feature description in the template image according to the homography matrix;

[0180] The update module 850 is further configured to add the feature description of the target feature point to the target feature description in order to update the target feature description.

[0181] Optionally, the filtering module 820 is further configured to search for a third target feature description that does not belong to the target box from the first feature description;

[0182] The filtering module 820 is further configured to find, in the template image, feature points corresponding to the feature points indicated by the third target feature description and feature points corresponding to the feature points indicated by the occlusion feature description, based on the homography matrix.

[0183] The update module 850 is further configured to add the feature descriptions of the feature points corresponding to the feature points indicated by the third target feature description and the feature descriptions of the feature points corresponding to the feature points indicated by the occlusion feature description to the context feature description, so as to update the context feature description.

[0184] Optionally, the extraction module 810 is specifically used to use the nearest neighbor search criterion or symmetry criterion of the distance ratio criterion to find a first matching feature description that matches the target feature description from the first feature description;

[0185] The extraction module 810 is further specifically used to use the nearest neighbor search criterion or symmetry criterion of the distance ratio criterion to find a second matching feature description that matches the context feature description from the first feature description;

[0186] The extraction module 810 is further specifically used to delete the second matching feature description from the first matching feature description to obtain the effective feature description.

[0187] Optionally, the updating module 850 is further configured to delete some target feature descriptions using a uniform random sampling strategy when the number of target feature descriptions is greater than a first preset number.

[0188] And / or,

[0189] The update module 850 is further configured to delete some context feature descriptions using a uniform random sampling strategy when the number of context feature descriptions is greater than a second preset number.

[0190] Please continue to refer to Figure 8 This application also provides a target tracking device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0191] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in this application.

[0192] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0193] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0194] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target tracking method, characterized in that, The method includes: For the target image to be detected in the image sequence, the search region of the target image is determined, and features are extracted from the search region to obtain the first feature description of the target image; Based on the target feature description of the target in the template image and the context feature description of the context associated with the target in the template image, valid feature descriptions associated with the target are filtered from the first feature descriptions; wherein, the valid feature descriptions are feature descriptions that match the target feature descriptions but do not match the context feature descriptions; Based on the effective feature description and the target feature description that matches the effective feature description, the homography matrix from the template image to the target image is calculated using the Ransac algorithm; Based on the effective feature description, the target feature description matching the effective feature description, and the homography matrix, the state parameters of the effective feature description are calculated using the Mlesac likelihood loss function. Based on the state parameters of the effective features and the current state parameters of the target, it is determined whether the target is identified in the target image, and when it is determined that the target is identified, the target bounding box corresponding to the target in the target image is determined based on the state parameters of the effective features; The state parameters of the target are updated to the state parameters of the effective feature; wherein the initial value of the state parameters of the target is a preset value.

2. The method according to claim 1, characterized in that, The Mlesac likelihood loss function is: e=||l-M -1 ×l * || 2 +||l * -M×l|| 2 Wherein, γ is a mixing parameter, and the For outlier distribution information, σ is the standard deviation of the error on each matched feature, e is the maximum likelihood estimation error parameter; l is the feature point location described by the effective feature description, l * The location of the feature points described in the target feature description; M is the homography matrix, s is the scale change, θ is the planar rotation, and λ is the homography matrix. x For the positional change in the x-direction, the λ y The positional change in the y-direction; (λ) x , λ y , s, θ) are the state parameters describing the effective features; The calculation of the state parameters describing the effective features includes: The state parameters of the effective features are calculated by minimizing the Mlesac likelihood loss function.

3. The method according to claim 1, characterized in that, After determining the target bounding box corresponding to the target in the target image based on the state parameters of the effective features, the method further includes: Find the first target feature description belonging to the target box from the first feature description, and when the first target feature description matches the context feature description, determine that the first target feature description is the occlusion feature description corresponding to the occlusion object; When the number of occlusion feature descriptions exceeds a preset threshold, it is determined that the target is occluded, and the step of determining the search area of ​​the target image is executed again; When the number of occlusion feature descriptions is not greater than the preset threshold, it is determined that the target is not occluded, and the target bounding box is determined as the tracking result corresponding to the target image.

4. The method according to claim 3, characterized in that, After determining that the target is not occluded, the method further includes: Find the second target feature description belonging to the target box from the effective feature descriptions, and find the target feature point corresponding to the feature point indicated by the second target feature description in the template image according to the homography matrix; The feature description of the target feature point is added to the target feature description to update the target feature description.

5. The method according to claim 3 or 4, characterized in that, After determining that the target is not occluded, the method further includes: Search for a third target feature description that does not belong to the target box from the first feature description; Based on the homography matrix, find the feature points in the template image that correspond to the feature points indicated by the third target feature description, and the feature points that correspond to the feature points indicated by the occlusion feature description; The feature descriptions of the feature points corresponding to the feature points indicated by the third target feature description and the feature points corresponding to the feature points indicated by the occlusion feature description are added to the context feature description to update the context feature description.

6. The method according to claim 1, characterized in that, The step of filtering valid feature descriptions associated with the target from the first feature description based on the target feature description of the target in the template image and the context feature description of the context associated with the target in the template image includes: Using the nearest neighbor search criterion of the distance ratio criterion or the symmetry criterion, find the first matching feature description that matches the target feature description from the first feature description; Using the nearest neighbor search criterion of the distance ratio criterion or the symmetry criterion, find a second matching feature description from the first feature description that matches the context feature description; The second matching feature description is deleted from the first matching feature description to obtain the effective feature description.

7. The method according to claim 6, characterized in that, The method further includes: When the number of target feature descriptions exceeds a first preset number, a uniform random sampling strategy is used to delete some target feature descriptions. And / or, When the number of context feature descriptions exceeds a second preset number, a uniform random sampling strategy is used to delete some context feature descriptions.

8. A target tracking device, characterized in that, The device includes an extraction module, a filtering module, a calculation module, a determination module, and an update module; wherein, The extraction module is used to determine the search region of the target image in the image sequence to be detected, and to extract features from the search region to obtain a first feature description of the target image. The filtering module is used to filter out valid feature descriptions associated with the target from the first feature description based on the target feature description of the target in the template image and the context feature description of the context associated with the target in the template image; wherein, the valid feature description is a feature description that matches the target feature description but does not match the context feature description; The calculation module is used to calculate the homography matrix from the template image to the target image based on the effective feature description and the target feature description that matches the effective feature description, using the Ransac algorithm. The calculation module is further configured to calculate the state parameters of the effective feature description based on the Mlesac likelihood loss function, according to the effective feature description, the target feature description matching the effective feature description, and the homography matrix. The determining module is used to determine whether the target is identified in the target image based on the state parameters of the effective features and the current state parameters of the target, and when it is determined that the target is identified, to determine the target bounding box corresponding to the target in the target image based on the state parameters of the effective features; The update module is used to update the state parameters of the target to the state parameters of the effective feature; wherein the initial value of the state parameters of the target is a preset value.

9. A target tracking device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images