Target tracking methods, devices, systems and storage media

Abstract

A target tracking method, apparatus, system, and storage medium include: acquiring a target image at a first moment; determining the detection position of key points of a target in the target image; acquiring the position of the key points of the target to be tracked at a second moment; the second moment being a moment prior to the first moment; predicting the predicted position of the key points of the target to be tracked at the first moment based on the position of the key points of the target to be tracked at the second moment; determining a matching relationship between the detected target and the target to be tracked based on the detection position and the predicted position; the matching relationship being used to match identical targets between the detected target and the target to be tracked; and tracking the target to be tracked based on the matching relationship.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to a target tracking method, apparatus, system, and storage medium. Background Technology

[0002] In the field of image processing, schemes for target and keypoint detection and tracking have wide applications in behavior recognition, pose analysis, and 3D keypoint estimation. For example, pedestrian tracking can be used to determine the pedestrian's route trajectory, and keypoint tracking of a pedestrian can be used to determine whether the pedestrian has fallen.

[0003] Related techniques typically require separate object detection and keypoint detection for the target object, and then tracking the target object based on the bounding box image obtained from the object detection. This allows for indirect keypoint tracking based on the correlation between the bounding box image and the keypoints. Another related technique involves keypoint detection for the target object, generating a bounding box image based on the keypoints, and then tracking the target object using the generated bounding box image. Summary of the Invention

[0004] On the one hand, a target tracking method is provided, the method comprising: acquiring an image to be detected at a first time moment; determining the detection position of key points of the target in the image to be detected; acquiring the position of the key points of the target to be tracked at a second time moment; the second time moment being a time moment prior to the first time moment; predicting the predicted position of the key points of the target to be tracked at the first time moment based on the position of the key points of the target to be tracked at the second time moment; determining a matching relationship between the detected target and the target to be tracked based on the detection position and the predicted position; the matching relationship being used to match the same targets in the detected target and the target to be tracked; and tracking the target to be tracked based on the matching relationship.

[0005] In some embodiments, the method includes: performing a first target operation on each detected target and each target to be tracked to determine the distance between each detected target and each target to be tracked; the first target operation includes: determining the distance between the i-th detected target and the j-th target to be tracked based on the distance between the detected position of a key point in the i-th detected target and the predicted position of a key point in the j-th target to be tracked; wherein i is a positive integer less than or equal to N; j is a positive integer less than or equal to M; and determining a matching relationship based on the distance between each detected target and each target to be tracked.

[0006] In some embodiments, the method includes: performing a second target operation on each corresponding key point of the i-th detected target and the j-th target to be tracked, to determine the distance between each corresponding key point; the second target operation includes: determining a first key point and a second key point; determining the distance between the first key point and the second key point based on the detection position of the first key point and the predicted position of the second key point; the first key point is any key point in the i-th detected target; the second key point is the key point in the j-th target to be tracked that corresponds to the first key point; and determining the distance between the i-th detected target and the j-th target to be tracked based on the distance between each corresponding key point.

[0007] In some embodiments, the distance between the i-th detected target and the j-th target to be tracked is calculated using the following formula:

[0008]

[0009] Where D(i,j) represents the distance between the i-th detected target and the j-th target to be tracked, p ik Let p′ represent the detection position of the k-th keypoint of the i-th target. jk d(p) represents the predicted location of the k-th keypoint of the j-th target to be tracked. ik ,p′ jk ) represents the distance between the k-th keypoint of the i-th detected target and the k-th keypoint of the j-th target to be tracked, s represents the target size information, and v ik Indicates whether the k-th keypoint of the i-th target has been detected, v′ jk Indicates whether the k-th keypoint of the j-th target to be tracked has been detected, δ(v ik ,v′ jk This represents the detection status of the k-th key point of the i-th detected target and the k-th key point of the j-th target to be tracked.

[0010] In some embodiments, the matching relationship includes at least one of the following: there is at least one target to be tracked that matches the target to be detected, there is at least one target to be tracked that does not match the target to be detected, and there is at least one target to be detected that does not match the target to be tracked.

[0011] In some embodiments, the method includes: determining a weight value for each target to be tracked and a weight value for each detected target; wherein the weight value of any target to be tracked is the minimum distance among the distances between any target to be tracked and each detected target; the weight value of each detected target is a preset weight value; performing a third target operation on each target to be tracked to determine a detected target that matches each target to be tracked; the third target operation includes: determining a first detected target with the shortest distance to the m-th target to be tracked; wherein the sum of the weight values ​​of the first detected target and the m-th target to be tracked is greater than or equal to the distance between the first detected target and the m-th target to be tracked; m is a positive integer; performing a fourth target operation on the m-th target to be tracked and the first detected target to determine a detected target that matches the m-th target to be tracked; the fourth target operation includes: determining whether a first target to be tracked exists; the first target to be tracked is the detected target that matches the first target determined according to the third target operation. The process involves: detecting target matches; determining the m-th target to be tracked if no first target exists; determining the first influence value and second influence value of the m-th target if a first target exists; wherein the first influence value is the difference between the m-th target and the second target; the difference is the difference between the distance between a target and a target and the sum of the weights of the target and the target; the second target is the target with the smallest difference from the m-th target (excluding the first target); the second influence value is the difference between the first target and the third target; the third target is the target with the smallest difference from the first target (excluding the first target); and determining the target that matches the m-th target based on the first and second influence values.

[0012] In some embodiments, the method includes: when the first influence value is less than the second influence value, performing a fourth target operation on the m-th target to be tracked and the second detection target to determine a detection target that matches the m-th target to be tracked; and when the first influence value is greater than the second influence value, determining that the m-th target to be tracked matches the first detection target.

[0013] In some embodiments, the method further includes: if the first influence value is greater than the second influence value, performing a fourth target operation on the first target to be tracked and the third detection target to determine the detection target that matches the first target to be tracked.

[0014] In some embodiments, the method further includes: when the first influence value is less than the second influence value, updating the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detection target according to the first influence value; and when the first influence value is greater than the second influence value, updating the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detection target according to the second influence value.

[0015] In some embodiments, the method includes: increasing the weight value of the m-th target to be tracked by a first influence value, increasing the weight value of the first target to be tracked by a first influence value, and decreasing the weight value of the first detected target by a first influence value; increasing the weight value of the m-th target to be tracked by a second influence value, increasing the weight value of the first target to be tracked by a second influence value, and decreasing the weight value of the first detected target by a second influence value.

[0016] In some embodiments, the method includes: determining a list of targets to be tracked and a list of targets to be detected; wherein the list of targets to be tracked includes each target to be tracked, and the list of targets to be detected includes each target to be detected; iteratively performing a fifth target operation on the targets to be tracked and the targets to be detected in the list of targets to be tracked and the list of targets to be detected until there are no targets to be tracked in the list of targets to be tracked or no targets to be detected in the list of targets to be detected, and determining a matching relationship based on the execution result of the fifth target operation; the fifth target operation includes: determining the target distance between each target to be tracked in the current list of targets to be tracked and each target to be tracked in the current list of targets to be detected; determining the minimum distance among the target distances, and the second target to be tracked and the fourth target to be detected corresponding to the minimum distance; the distance between the second target to be tracked and the fourth target to be detected is the minimum distance; determining the matching relationship between the second target to be tracked and the fourth target to be detected; deleting the second target to be tracked from the current list of targets to be tracked, and deleting the fourth target to be detected from the current list of targets to be detected.

[0017] In some embodiments, the method includes: determining a third target to be tracked; each of the third targets to be tracked has a matching detection target; and updating the position of the key points of the third target to be tracked based on the detection position of the key points of the matching detection target and the predicted position of the key points of the third target to be tracked.

[0018] In some embodiments, the method includes: determining a fifth detection target; each of the fifth detection targets does not have a matching target to be tracked; and adding a fifth detection target to the target to be tracked.

[0019] In some embodiments, the method includes: determining a fourth target to be tracked and the number of mismatches of the fourth target to be tracked; each of the fourth targets to be tracked does not have a matching detection target; the number of mismatches is the number of times the target to be tracked does not have a matching detection target consecutively; if the number of mismatches is greater than a preset tracking threshold, the fourth target to be tracked is deleted from the target to be tracked.

[0020] In some embodiments, the method includes: if the number of mismatches is less than or equal to a preset tracking threshold, using the predicted position of the key point of the fourth target to be tracked or the position of the key point of the fourth target to be tracked at a second time moment as the updated position of the key point of the fourth target to be tracked.

[0021] In some embodiments, the method includes: when a matching detection target exists for the fifth target to be tracked, updating the position of the key point of the fifth target to be tracked based on the detection position of the key point of the matching detection target and the predicted position of the key point of the fifth target to be tracked; the fifth target to be tracked is any one of the targets to be tracked; when no matching detection target exists for the fifth target to be tracked, and the number of mismatches for the fifth target to be tracked is greater than a preset tracking threshold, deleting the fifth target to be tracked from the targets to be tracked; the number of mismatches is the number of times the target to be tracked has no matching detection target consecutively; when no matching target exists for the sixth detection target, adding a sixth detection target to the targets to be tracked; the sixth detection target is any one of the detection targets.

[0022] In some embodiments, the method includes: when there is no matching detection target for the fifth target to be tracked, and the number of mismatches of the fifth target to be tracked is less than or equal to a preset tracking threshold, using the predicted position of the key point of the fifth target to be tracked or the position of the fifth target to be tracked at a second time moment as the updated position of the key point of the fifth target to be tracked.

[0023] In some embodiments, the method includes: determining the motion state of key points of the target to be tracked; and predicting the predicted position of the key points of the target to be tracked at a first time based on the position and motion state of the key points of the target at a second time.

[0024] In some embodiments, the method includes: determining that there is no matching detection target for each target to be tracked when the number of detected targets is 0.

[0025] In some embodiments, the method includes: determining that there is no matching target for each detected target when the number of targets to be tracked is 0.

[0026] On the other hand, a target tracking device is provided, including a processing unit and an acquisition unit. The acquisition unit is configured to acquire a target image at a first time moment; the processing unit is configured to determine the detection position of key points of a target in the target image; the acquisition unit is further configured to acquire the position of the key points of the target to be tracked at a second time moment; the second time moment is a time moment prior to the first time moment; the processing unit is further configured to predict the predicted position of the key points of the target to be tracked at the first time moment based on the position of the key points of the target to be tracked at the second time moment; the processing unit is further configured to determine a matching relationship between the detected target and the target to be tracked based on the detection position and the predicted position; the matching relationship is used to match identical targets between the detected target and the target to be tracked; the processing unit is further configured to track the target to be tracked based on the matching relationship.

[0027] In some embodiments, the processing unit is configured to perform a first target operation on each detected target and each target to be tracked to determine the distance between each detected target and each target to be tracked; the first target operation includes: determining the distance between the i-th detected target and the j-th target to be tracked based on the distance between the detected position of a key point in the i-th detected target and the predicted position of a key point in the j-th target to be tracked; wherein i is a positive integer less than or equal to N; j is a positive integer less than or equal to M; the processing unit is further configured to determine a matching relationship based on the distance between each detected target and each target to be tracked.

[0028] In some embodiments, the processing unit is configured to perform a second target operation on each corresponding key point of the i-th detected target and the j-th target to be tracked, to determine the distance between each corresponding key point; the second target operation includes: determining a first key point and a second key point; determining the distance between the first key point and the second key point based on the detection position of the first key point and the predicted position of the second key point; the first key point is any key point in the i-th detected target; the second key point is the key point in the j-th target to be tracked that corresponds to the first key point; the processing unit is further configured to determine the distance between the i-th detected target and the j-th target to be tracked based on the distance between each corresponding key point.

[0029] In some embodiments, the distance between the i-th detected target and the j-th target to be tracked is calculated using the following formula:

[0030]

[0031] Where D(i,j) represents the distance between the i-th detected target and the j-th target to be tracked, p ik Let p′ represent the detection position of the k-th keypoint of the i-th target. jk d(p) represents the predicted location of the k-th keypoint of the j-th target to be tracked.ik ,p′ jk ) represents the distance between the k-th keypoint of the i-th detected target and the k-th keypoint of the j-th target to be tracked, s represents the target size information, and v ik Indicates whether the k-th keypoint of the i-th target has been detected, v′ jk Indicates whether the k-th keypoint of the j-th target to be tracked has been detected, δ(v ik ,v′ jk This represents the detection status of the k-th key point of the i-th detected target and the k-th key point of the j-th target to be tracked.

[0032] In some embodiments, the matching relationship includes at least one of the following: there is at least one target to be tracked that matches the target to be detected, there is at least one target to be tracked that does not match the target to be detected, and there is at least one target to be detected that does not match the target to be tracked.

[0033] In some embodiments, the processing unit is configured to determine the weight value of each target to be tracked and the weight value of each detected target; wherein the weight value of any target to be tracked is the minimum distance among the distances between any target to be tracked and each detected target; the weight value of each detected target is a preset weight value; perform a third target operation on each target to be tracked to determine a detected target that matches each target to be tracked; the third target operation includes: determining a first detected target with the shortest distance to the m-th target to be tracked; wherein the sum of the weight values ​​of the first detected target and the m-th target to be tracked is greater than or equal to the distance between the first detected target and the m-th target to be tracked; m is a positive integer; perform a fourth target operation on the m-th target to be tracked and the first detected target to determine a detected target that matches the m-th target to be tracked; the fourth target operation includes: determining whether there is a first target to be tracked; the first target to be tracked is the target that matches the m-th target to be tracked as determined by the third target operation. A target to be tracked is detected and matched. If no first target to be tracked exists, the m-th target to be tracked is determined to match the first target. If the first target to be tracked exists, a first influence value and a second influence value of the m-th target to be tracked are determined. The first influence value is the difference between the m-th target to be tracked and the second target to be tracked. The difference is the difference between the distance between a target to be tracked and a target to be tracked and the sum of the weights of the target to be tracked and the target to be tracked. The second target to be tracked is the target to be tracked with the smallest difference from the m-th target to be tracked, excluding the first target to be tracked. The second influence value is the difference between the first target to be tracked and the third target to be tracked. The third target to be tracked is the target to be tracked with the smallest difference from the first target to be tracked, excluding the first target to be tracked. Based on the first influence value and the second influence value, a target to be tracked that matches the m-th target to be tracked is determined.

[0034] In some embodiments, the processing unit is configured to perform a fourth target operation on the m-th target to be tracked and the second detection target when the first influence value is less than the second influence value, and to determine the detection target that matches the m-th target to be tracked; and to determine that the m-th target to be tracked matches the first detection target when the first influence value is greater than the second influence value.

[0035] In some embodiments, the processing unit is configured to perform a fourth target operation on the first target to be tracked and the third detection target when the first influence value is greater than the second influence value, thereby determining the detection target that matches the first target to be tracked.

[0036] In some embodiments, the processing unit is configured to update the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detection target according to the first influence value when the first influence value is less than the second influence value; and to update the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detection target according to the second influence value when the first influence value is greater than the second influence value.

[0037] In some embodiments, the processing unit is configured to increase the weight value of the m-th target to be tracked by a first influence value, increase the weight value of the first target to be tracked by a first influence value, and decrease the weight value of the first detected target by a first influence value; the processing unit is further configured to increase the weight value of the m-th target to be tracked by a second influence value, increase the weight value of the first target to be tracked by a second influence value, and decrease the weight value of the first detected target by a second influence value.

[0038] In some embodiments, the processing unit is configured to determine a list of targets to be tracked and a list of targets to be detected; wherein the list of targets to be tracked includes each target to be tracked, and the list of targets to be detected includes each target to be detected; iteratively performing a fifth target operation on the targets to be tracked and the targets to be detected in the list of targets to be tracked and the list of targets to be detected until there are no targets to be tracked in the list of targets to be tracked or no targets to be detected in the list of targets to be detected, and determining a matching relationship based on the execution result of the fifth target operation; the fifth target operation includes: determining the target distance between each target to be tracked in the current list of targets to be tracked and each target to be tracked in the current list of targets to be detected; determining the minimum distance among the target distances, and the second target to be tracked and the fourth target to be detected corresponding to the minimum distance; the distance between the second target to be tracked and the fourth target to be detected is the minimum distance; determining the matching relationship between the second target to be tracked and the fourth target to be detected; deleting the second target to be tracked from the current list of targets to be tracked, and deleting the fourth target to be detected from the current list of targets to be detected.

[0039] In some embodiments, the processing unit is configured to determine a third target to be tracked; each of the third targets to be tracked has a matching detection target; the processing unit is further configured to update the position of the key point of the third target to be tracked based on the detection position of the key point of the detection target that matches the third target to be tracked and the predicted position of the key point of the third target to be tracked.

[0040] In some embodiments, the processing unit is configured to determine a fifth detection target; each of the fifth detection targets does not have a matching target to be tracked; the processing unit is further configured to add a fifth detection target to the targets to be tracked.

[0041] In some embodiments, the processing unit is configured to determine a fourth target to be tracked and the number of mismatches of the fourth target to be tracked; each of the fourth targets to be tracked has no matching detection target; the number of mismatches is the number of times the target to be tracked has no matching detection target consecutively; the processing unit is further configured to delete the fourth target to be tracked from the target to be tracked if the number of mismatches is greater than a preset tracking threshold.

[0042] In some embodiments, the processing unit is configured to use the predicted position of the key point of the fourth target to be tracked or the position of the key point of the fourth target to be tracked at the second time moment as the updated position of the key point of the fourth target to be tracked when the number of mismatches is less than or equal to a preset tracking threshold.

[0043] In some embodiments, the processing unit is configured to, when a matching detection target exists for the fifth target to be tracked, update the position of the key point of the fifth target to be tracked based on the detection position of the key point of the matching detection target and the predicted position of the key point of the fifth target to be tracked; the fifth target to be tracked is any one of the targets to be tracked; the processing unit is further configured to, when no matching detection target exists for the fifth target to be tracked and the number of mismatches for the fifth target to be tracked is greater than a preset tracking threshold, delete the fifth target to be tracked from the targets to be tracked; the number of mismatches is the number of times the target to be tracked has no matching detection target consecutively; the processing unit is further configured to, when no matching target exists for the sixth detection target, add a sixth detection target to the targets to be tracked; the sixth detection target is any one of the detection targets.

[0044] In some embodiments, the processing unit is configured to use the predicted position of the key point of the fifth target to be tracked or the position of the fifth target to be tracked at the second time moment as the updated position of the key point of the fifth target to be tracked when there is no matching detection target for the fifth target to be tracked and the number of mismatches of the fifth target to be tracked is less than or equal to a preset tracking threshold.

[0045] In some embodiments, the processing unit is configured to determine the motion state of key points of the target to be tracked; the processing unit is further configured to predict the predicted position of the key points of the target to be tracked at the first time based on the position and motion state of the key points of the target at the second time.

[0046] In some embodiments, the processing unit is configured to determine that there is no matching detection target for each target to be tracked when the number of detected targets is 0.

[0047] In some embodiments, the processing unit is configured to determine that there is no matching target for each detected target when the number of targets to be tracked is 0.

[0048] In another aspect, a non-transitory computer-readable storage medium is provided. The computer-readable storage medium stores computer program instructions that, when executed on a computer (e.g., a target tracking device), cause the computer to perform the target tracking method as described in any of the above embodiments.

[0049] In another aspect, a computer program product is provided. The computer program product includes computer program instructions that, when executed on a computer (e.g., a target tracking device), cause the computer to perform the target tracking method as described in any of the above embodiments.

[0050] In another aspect, a computer program is provided. When the computer program is executed on a computer (e.g., a target tracking device), the computer program causes the computer to perform the target tracking method as described in any of the above embodiments.

[0051] In another aspect, a target tracking device is provided, the device comprising: a processor and a memory; the memory for storing computer programs or instructions, and the processor for running the computer programs or instructions to implement the target tracking method as described in any of the above embodiments.

[0052] In another aspect, a chip is provided, which includes a processor and a communication interface coupled to the processor. The processor is used to run computer programs or instructions to implement the target tracking method as described in any of the above embodiments.

[0053] As an example, the chip provided in this disclosure also includes a memory for storing computer programs or instructions.

[0054] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the device, or it may be packaged separately from the processor of the device; this disclosure does not impose any limitation on this.

[0055] In another aspect, a target tracking system is provided, comprising: a target tracking device and an image acquisition device, wherein the image acquisition device is used to acquire an image to be detected, and the target tracking device is used to perform the target tracking method as described in any of the above embodiments.

[0056] In this disclosure, the names of the aforementioned target tracking devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of the claims of this disclosure and their equivalents. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in this disclosure, the accompanying drawings used in some embodiments of this disclosure will be briefly described below. Obviously, the drawings described below are only drawings of some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings. In addition, the drawings described below can be regarded as schematic diagrams and are not intended to limit the actual size of the product, the actual flow of the method, the actual timing of the signals, etc. involved in the embodiments of this disclosure.

[0058] Figure 1 This is an architecture diagram of a target tracking system 10 provided according to some embodiments;

[0059] Figure 2 This is a diagram showing the relationship between targets and key points according to some embodiments;

[0060] Figure 3 This is a flowchart of a target tracking method provided according to some embodiments;

[0061] Figure 4 This is a matching graph of a detected target and a target to be tracked, provided according to some embodiments;

[0062] Figure 5 This is a flowchart of another target tracking method provided according to some embodiments;

[0063] Figure 6 This is a scene diagram of target tracking provided according to some embodiments;

[0064] Figure 7 This is a schematic diagram of a target tracking principle in related technologies;

[0065] Figure 8 This is a schematic diagram of a target tracking method provided according to some embodiments of the present disclosure;

[0066] Figure 9 This is a flowchart of another target tracking method provided according to some embodiments;

[0067] Figure 10 A flowchart of a third target operation provided according to some embodiments;

[0068] Figure 11 A flowchart of a fourth target operation provided according to some embodiments;

[0069] Figure 12 This is a flowchart of another target tracking method provided according to some embodiments;

[0070] Figure 13 A flowchart of the fifth target operation provided according to some embodiments;

[0071] Figure 14 This is a flowchart of another target tracking method provided according to some embodiments;

[0072] Figure 15 This is a diagram showing the relationship between key points of a target to be tracked and a corresponding matched detected target, according to some embodiments.

[0073] Figure 16 This is a flowchart of another target tracking method provided according to some embodiments;

[0074] Figure 17 This is a structural diagram of a target tracking device according to some embodiments;

[0075] Figure 18 This is a structural diagram of another target tracking device provided according to some embodiments. Detailed Implementation

[0076] The technical solutions in some embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments provided in this disclosure are within the scope of protection of this disclosure.

[0077] Unless the context otherwise requires, throughout the specification and claims, the term "comprise" and its other forms, such as the third-person singular "comprises" and the present participle "comprising," are interpreted as open-ended and encompassing, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiments," "example," "specific example," or "some examples," etc., are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples.

[0078] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0079] In describing some embodiments, the terms "coupled" and "connected," and their derivative expressions, may be used. For example, the term "connected" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact with each other. Similarly, the term "coupled" may be used in describing some embodiments to indicate that two or more components have direct physical or electrical contact. However, the terms "coupled" or "communicatively coupled" may also refer to two or more components that do not have direct contact with each other but still cooperate or interact with each other. The embodiments disclosed herein are not necessarily limited to the content of this document.

[0080] "At least one of A, B and C" has the same meaning as "at least one of A, B or C", both including the following combinations of A, B and C: only A, only B, only C, combinations of A and B, combinations of A and C, combinations of B and C, and combinations of A, B and C.

[0081] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0082] As used herein, depending on the context, the term “if” may optionally be interpreted as meaning “when”, “in the event of”, “in response to determination”, or “in response to detection”. Similarly, depending on the context, the phrase “if it is determined that…” or “if [the stated condition or event] is detected” may optionally be interpreted as meaning “in the event of determination that…”, “in response to determination that…”, “when [the stated condition or event] is detected”, or “in response to the detection of [the stated condition or event]”.

[0083] The use of “applies to” or “configured to” in this article implies an open and inclusive language that does not preclude applicability to or configuration to devices that perform additional tasks or steps.

[0084] In addition, the use of “based on” implies openness and inclusivity, because processes, steps, calculations or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0085] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0086] The following explanations of the terms used in the embodiments of this disclosure are provided to facilitate the reader's understanding.

[0087] (1) Key point detection

[0088] Keypoint detection is used to detect specific nodes corresponding to a target, and can be applied to behavior recognition, pose analysis, and other fields. Keypoint detection algorithms include top-down and bottom-up methods. Top-down methods perform target detection on an image and then perform keypoint detection based on the detected targets. Examples of top-down methods include the HRNet algorithm (Deep High-Resolution Representation Learning for Human Pose Estimation). Bottom-up methods detect all keypoints in an image and then cluster them based on the location of each keypoint to obtain the targets corresponding to the associated keypoints. Examples of bottom-up methods include the OpenPose algorithm (Realtime Multi-Person 2D Pose Estimation using Part AffinityFields).

[0089] (2) Hungarian algorithm

[0090] The Hungarian algorithm is used to determine the matching relationship between elements in two sets, that is, to perform data association between elements in two sets in order to obtain matching results with the greatest possible number of combinations.

[0091] (3) KM algorithm (Kuhn-Munkres algorithm)

[0092] The KM algorithm, based on the Hungarian algorithm, uses the degree of association between elements in two sets as a weight to perform data association between elements in the two sets, thereby obtaining the matching result with the optimal matching degree.

[0093] (4) Crossover and union ratio

[0094] Intersection over union (IoU) is the ratio of the area of ​​intersection of two image regions to the area of ​​union of the two image regions. IoU is commonly used to detect the relationship between two image regions, such as tracking a detected target by calculating the IoU of the target in different frames of a video data stream.

[0095] In the field of image processing, schemes for target and keypoint detection and tracking have wide applications in behavior recognition, pose analysis, and 3D keypoint estimation. For example, pedestrian tracking can be used to determine the pedestrian's route trajectory, and keypoint tracking of a pedestrian can be used to determine whether the pedestrian has fallen.

[0096] Related techniques typically require separate object detection and keypoint detection for the target object, and then track the target object based on the bounding box image obtained from object detection, thereby indirectly tracking keypoints based on the correlation between the bounding box image and keypoints. Other techniques also perform keypoint detection on the target object and generate a bounding box image based on the keypoints, then track the target object based on the generated bounding box image. However, these methods are greatly affected by changes in the bounding box image, leading to unstable target object tracking. Furthermore, it is difficult to establish a one-to-one correspondence between the bounding box image and keypoints, resulting in poor keypoint tracking performance. In addition, these methods require object detection to indirectly track keypoints, incurring high computational overhead and thus high computational costs.

[0097] In view of this, this disclosure provides a target tracking method that matches the target to be tracked and the detected target based on the detected keypoints of the detected target and the predicted keypoints of the target to be tracked, and updates the target to be tracked and its keypoints according to the matching relationship. Therefore, this disclosure can directly track targets based on keypoints, improving the stability of target tracking while reducing computational overhead, thereby reducing computational costs.

[0098] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0099] Figure 1 This is an architecture diagram of a target tracking system 10 provided according to some embodiments, such as Figure 1 As shown, the target tracking system 10 includes a target tracking device 101 and an image acquisition device 102. The target tracking device 101 and the image acquisition device 102 are connected via a communication link. This communication link can be a wired communication link or a wireless communication link, and this disclosure does not limit it in this way.

[0100] It should be noted that the target tracking device 101 and the image acquisition device 102 in this disclosure can be one or more. For ease of understanding, Figure 1 Only one target tracking device 101 and one image acquisition device 102 are shown in the diagram.

[0101] The image acquisition device 102 is configured to acquire an image of the target area to be detected and send the image to be detected to the target tracking device 101. Correspondingly, the target tracking device 101 receives the image to be detected sent by the image acquisition device 102.

[0102] In one possible implementation, the image acquisition device 102 can acquire the image to be detected in the target area in real time and send the image to be detected to the target tracking device 101. The image acquisition device 102 can also acquire the image to be detected in the target area at a preset frequency and send video stream data including the image to be detected to the target tracking device 101.

[0103] The image acquisition device 102 in this embodiment is a device that converts image data into analog or digital signals using a photosensor. It can be deployed on land, including indoors or outdoors, handheld or vehicle-mounted. It can also be deployed on water (such as on ships). It can also be deployed in the air (e.g., on airplanes, balloons, and satellites). For example, the image acquisition device 102 includes a camera, video camera, or video camera. The image acquisition device 102 can also be a device with video recording capabilities, such as a mobile phone, tablet computer, laptop computer, PDA, wearable device (e.g., smartwatch, smart bracelet, pedometer, etc.), vehicle-mounted device, or flying device (e.g., intelligent robot, hot air balloon, drone, airplane).

[0104] For example, the image acquisition device 102 in this embodiment of the present disclosure may also be an infrared imager or a night vision device, used to acquire image data of dark areas.

[0105] The target tracking device 101 is configured to acquire the image to be detected at a first moment and determine the detection position of the key points of the target based on the image to be detected.

[0106] The target tracking device 101 is also configured to determine the predicted position of the key points of the target to be tracked at a first moment, and to track the target and the key points of the target to be tracked based on the detected position and the predicted position.

[0107] In this context, the detected target is the target detected by the target tracking device 101 from the image to be detected, and the target to be tracked is the target that the target tracking device 101 tracks. Key points are nodes at specific locations corresponding to the targets (detected target and / or target to be tracked). The targets in this disclosure can be naturally existing tangible objects, such as people, animals, plants, vehicles, and goods.

[0108] Taking people as the target, Figure 2 This is a diagram showing the relationship between targets and key points according to some embodiments. Figure 2 As shown, target 201 can be the detection target or the target to be tracked in this disclosure, and the solid dots 202 are the key points corresponding to target 102. The number and location of key points can be set according to the actual situation, and this disclosure does not limit them.

[0109] It should be noted that the number of detected targets and tracked targets at the first moment can be 0 or one or more.

[0110] For example, when the first moment is the initial moment (e.g., the current image to be detected is the first frame), the number of targets to be tracked is 0. When the target tracking device 101 detects that there is no target or key point of the target in the image to be detected, the number of targets is 0. When the target tracking device 101 detects that there is a target and key point of the target in the image to be detected, the number of targets is one or more.

[0111] The location of key points in this disclosure can be the relative position of key points in the image to be detected relative to the image to be detected, or it can be the actual position determined based on the relative position of the key points in the image to be detected and the region corresponding to the image to be detected.

[0112] The target tracking device 101 of this disclosure can perform real-time target tracking, that is, continuously acquire the image to be detected at the current moment and track the target. The target tracking device 101 can also perform non-real-time target tracking, that is, track the target based on the image data stream over a past period of time.

[0113] The target tracking device 101 in this embodiment of the disclosure can be a server, including:

[0114] The processor can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of programs according to the present disclosure.

[0115] A transceiver can be any type of transceiver used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0116] Memory can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory can exist independently and be connected to the processor via communication lines. Memory can also be integrated with the processor.

[0117] The target tracking device 101 can also be an application program that provides target tracking functionality installed on the server, or the target tracking device 101 can be the central processing unit in the server, or the target tracking device 101 can be a control module in the server configured to execute the target tracking method. The target tracking device 101 can also be coupled to the image acquisition device 102 as the same device.

[0118] It should be noted that the various embodiments of this disclosure can be referenced or learned from each other. For example, the same or similar steps, method embodiments, system embodiments and device embodiments can be referenced from each other without limitation.

[0119] Figure 3 This is a flowchart illustrating a target tracking method according to some embodiments. Figure 3 As shown, the method includes the following steps:

[0120] Step 301: The target tracking device acquires the image to be detected at the first moment.

[0121] The image to be detected is an image acquired for the target area.

[0122] For example, the target area can be an airport waiting area, a road, a scenic area, a building stairwell, or other similar area.

[0123] The first moment can be the current moment when the target tracking device acquires the image to be detected.

[0124] In one possible implementation, the target tracking device can acquire the image to be detected in real time from the image acquisition device and take the current moment as the first moment.

[0125] Alternatively, the first moment can be the acquisition moment of the image to be detected. Here, the acquisition moment is the moment when the image acquisition device acquires the image to be detected.

[0126] In another possible implementation, the target tracking device can acquire video stream data including the image to be detected from the image acquisition device at a preset frequency, and take the acquisition time of the image frame of the detected image as the first moment.

[0127] Step 302: The target tracking device determines the detection location of the key points of the target in the image to be detected.

[0128] The number of detected targets can be zero or one or more. The detection location of key points of the detected targets can be the relative position of the key points in the image to be detected relative to the image itself, or the actual position determined based on the relative position of the key points in the image to be detected and the corresponding region of the image to be detected.

[0129] In one possible implementation, the target tracking device can determine the identifiers of key points of the detected target based on the location of the detected target. That is, in this disclosure, the identifiers of key points at the same location are identical for each detected target. Therefore, the target tracking device can determine the correspondence between key points of different targets based on the identifiers.

[0130] It should be noted that the number of key points in this disclosure can be set according to actual conditions. However, when the target tracking device determines the location information of the key points of the target, some key points may not be detected due to factors such as occlusion. In this case, the target tracking device can set the location information of the undetected key points to a preset location value (e.g., (-1, -1)). This preset location value is used to indicate that the key point has not been detected.

[0131] In one possible implementation, the target tracking device inputs the image to be detected into the key point detection model to obtain the key point detection results.

[0132] The key point detection results include the detection target and the key points corresponding to the detection target.

[0133] For example, the keypoint detection result can be a data vector of size N*K*A. Here, N represents the number of detected targets, K represents the number of keypoints, and A represents the location information of the keypoints. When the location information is in two-dimensional coordinates, it can be represented as (x, y). For undetected keypoints, the target tracking device can set their location information to (-1, -1). The keypoint detection model can be a top-down model, such as the HRNet model, or a bottom-up model, such as the OpenPose model.

[0134] Step 303: The target tracking device acquires the position of the key points of the target to be tracked at the second moment.

[0135] The second moment refers to the moment preceding the first moment.

[0136] It should be noted that the target to be tracked is the target that needs to be tracked in this disclosure. Each target to be tracked has a unique identifier.

[0137] The number of targets to be tracked can be zero or one or more. The number, location, and correspondence of key points of the targets to be tracked can be found in the description of key points for detecting targets in step 302 above, and will not be repeated here.

[0138] In one possible implementation, the target tracking device in this disclosure can perform a target tracking operation at a second moment to obtain the position of the key points of the target to be tracked at the second moment.

[0139] It should be noted that the target tracking method in this disclosure not only needs to identify the target to be tracked and the key points of the target, but also needs to determine the association between the key points of the target to be tracked and the detected target, as well as between the target to be tracked and the detected target, in continuous images (i.e., multiple images acquired continuously), so as to achieve the effect of tracking the target.

[0140] However, if the first moment is the initial moment, there is no second moment, and the number of key points of the target to be tracked acquired by the target tracking device in the second moment is 0. Subsequent target tracking operations by the target tracking device are equivalent to performing an initialization update operation on the target to be tracked. For example, the target tracking device may use the detected target determined in the initial moment as the target to be tracked. Details will be provided later and will not be repeated here.

[0141] Step 304: The target tracking device predicts the predicted position of the key points of the target to be tracked at the first moment based on the position of the key points of the target at the second moment.

[0142] In one possible implementation, the target tracking device can determine the motion state of key points of the target to be tracked.

[0143] The target tracking device can predict the position of the key points of the target at the first moment based on the position and motion state of the key points of the target at the second moment.

[0144] The motion state may include the velocity of the key point, or the velocity and acceleration of the key point, or the speed and direction of the key point.

[0145] Similar to the process described above for determining the position of the key points of the target to be tracked at the second moment, the target tracking device can perform target tracking operations at the second moment to determine the motion state of the key points of the target to be tracked. Details will be provided later and will not be repeated here.

[0146] It should be noted that the target tracking device can set the predicted position of a key point that has not been detected in the target to a preset position value, thereby indicating that the key point has not been detected.

[0147] Step 305: The target tracking device determines the matching relationship between the detected target and the target to be tracked based on the detection position and the predicted position.

[0148] The matching relationship is used to match the same targets between the detected target and the target to be tracked.

[0149] For example, the targets to be tracked include target A and target B. The targets to be detected include target C, target D, and target E. If target A and target C are the same target, then target A and target C match. If target B and target D are not the same target, then target B and target D do not match.

[0150] It should be noted that the detected position of the key points of the target to be detected can characterize the detected position state of the target at the first moment. The predicted position of the key points of the target to be tracked can characterize the predicted position state of the target to be tracked at the first moment. Therefore, the target tracking device can determine the matching relationship between the detected target and the target to be tracked based on the two states of the key point detection position and the predicted position state.

[0151] The matching relationship includes at least one of the following: there is at least one target to be tracked that matches the target to be detected, there is at least one target to be tracked that does not match the target to be detected, and there is at least one target to be detected that does not match the target to be tracked.

[0152] For example, Figure 4 This is a matching graph of a detected target and a target to be tracked, provided according to some embodiments. For example... Figure 4 As shown, target 4011 to be tracked matches target 4021, target 4013 to be tracked matches target 4022, target 4014 to be tracked matches target 4023, target 4012 to be tracked does not match any of the targets detected, target 4015 to be tracked does not match any of the targets detected, and target 4024 to be tracked does not match any of the targets detected.

[0153] Step 306: The target tracking device tracks the target to be tracked according to the matching relationship.

[0154] In one possible implementation, the target tracking device can update the target identifier and the position of key points of the target at the first moment.

[0155] For a target to be tracked that has a matching detection target, the target tracking device can determine the position of the key points of the target to be tracked at the first moment based on the predicted position of the key points of the target to be tracked and the detection position of the key points of the corresponding matching detection target.

[0156] For a detected target that does not have a matching target to be tracked, the target tracking device can add the detected target to the list of targets to be tracked.

[0157] It should be noted that, in the initial moment (the first time step), the number of targets to be tracked is 0, and the target tracking device can determine that there is no matching target for each detected target. Therefore, the target tracking device can add each target from the detected targets to the list of targets to be tracked.

[0158] In one possible implementation, when the first moment is the initial moment, the target tracking device sets the updated motion state of the key points of the target to be tracked as the initial motion state.

[0159] For example, the initial velocity and acceleration can be 0.

[0160] For a target to be tracked that does not have a matching detection target, the target tracking device can determine whether to delete the target from the target to be tracked based on whether the matching status of the target within a certain period of time meets the first preset condition.

[0161] For example, the first preset condition can be that the number of mismatches is less than or equal to a preset tracking threshold. The number of mismatches is the number of times that the target to be tracked does not have a matching detected target consecutively.

[0162] In one possible implementation, when the number of detected targets is 0, the target tracking device determines that there is no matching detected target for each target to be tracked.

[0163] Referring to the example in step 305 above, such as Figure 4 As shown, for targets 4011, 4013, and 4014 to be tracked, the target tracking device can determine the position of the key points of target 4011 at the first moment based on the predicted position of the key points of target 4011 and the detection position of the key points of target 4021. The target tracking device can also determine the position of the key points of target 4013 at the first moment based on the predicted position of the key points of target 4013 and the detection position of the key points of target 4022. Finally, the target tracking device can determine the position of the key points of target 4014 at the first moment based on the predicted position of the key points of target 4014 and the detection position of the key points of target 4023.

[0164] For target 4012 and target 4015 to be tracked, the target tracking device determines that the matching status of target 4012 meets the first preset condition and retains target 4012. If the target tracking device determines that the matching status of target 4015 does not meet the first preset condition, that is, the number of unmatched times is greater than the preset tracking threshold, target 4012 is deleted.

[0165] For the detected target 4024, the target tracking device adds the detected target 4024 to the list of targets to be tracked.

[0166] In one possible implementation, the target tracking device can also send an updated tracking sequence of the target to be tracked to the target device, so that the target device can perform relevant identification and detection based on the tracking sequence.

[0167] The tracking sequence includes the target to be tracked and its key information, such as the target's identifier and the positions of its key points at different times.

[0168] For example, the target device can perform related schemes such as pedestrian action detection and workshop machine status detection based on the tracking sequence.

[0169] In one possible implementation, the target tracking device can determine the motion state of the key points of the target to be tracked based on the positions of the key points at a first time and at a second time.

[0170] Based on the above technical solution, the target tracking device of this disclosure can determine the detection position of the key points of the target based on the image to be detected at the first moment, and predict the predicted position of the key points of the target to be tracked at the first moment based on the position of the key points of the target to be tracked at the second moment. Then, the target tracking device can determine the matching relationship between the detected target and the target to be tracked based on the detection position and the predicted position of the key points of the target to be tracked, and update the position of the target to be tracked and the key points of the target to be tracked based on the matching relationship, thereby achieving target tracking. Therefore, the target tracking device of this disclosure can achieve matching between the detected target and the target to be tracked based on the position information of the key points of the target, and then update the target to be tracked and the key points of the target to be tracked, so that the target to be tracked and the key points of the target to be tracked at different times are correlated. Compared with the related technology that tracks the target based on the target bounding box image determined by target detection, the technical solution of this disclosure first predicts the position of the key points of the target to be tracked at the first moment, and then performs target tracking based on the predicted position and the detection position of the key points of the detected target, which is not affected by changes in the bounding box image, thus improving the stability and accuracy of target tracking. Meanwhile, the technical solution in this disclosure tracks the target directly based on key points, without the need for additional target detection, resulting in low computational overhead and reduced computational costs.

[0171] It should be noted that steps 301-302 and steps 303-304 in the above embodiments are independent of each other. Steps 301-302 can be executed before steps 303-304, after steps 303-304, or in parallel with steps 303-304. Figure 3 This description uses only steps 301-302 executed before steps 303-304 as an example to illustrate the target tracking algorithm provided in this disclosure. This disclosure does not limit the execution order of the above steps.

[0172] The following describes the process by which a target tracking device determines the matching relationship between the detected target and the target to be tracked.

[0173] As one possible embodiment of this disclosure, combined with Figure 3 ,like Figure 5 As shown, step 305 above can also be achieved through the following steps 501-502.

[0174] Step 501: The target tracking device performs a first target operation on each detected target and each target to be tracked to determine the distance between each detected target and each target to be tracked.

[0175] The number of targets to be detected is N, and the number of targets to be tracked is M. M and N are positive integers.

[0176] In one possible implementation, the first target operation includes:

[0177] Step 5011: The target tracking device determines the distance between the i-th detected target and the j-th target to be tracked based on the distance between the detected position of the key point in the i-th detected target and the predicted position of the key point in the j-th target to be tracked.

[0178] Where i is a positive integer less than or equal to N; j is a positive integer less than or equal to M.

[0179] Furthermore, the process by which the target tracking device determines the distance between the i-th detected target and the j-th target to be tracked can be achieved through the following steps 501a-501b:

[0180] Step 501a: The target tracking device can perform a second target operation on each corresponding key point of the i-th detected target and the j-th target to be tracked, and determine the distance between each corresponding key point.

[0181] Among them, the identifiers of the corresponding key points of the i-th detected target and the j-th target to be tracked are the same.

[0182] In one possible implementation, the second target operation includes the following steps 5012-5013:

[0183] Step 5012: The target tracking device determines the first key point and the second key point.

[0184] The first key point is any key point in the i-th detected target. The second key point is the key point in the j-th target to be tracked that corresponds to the first key point.

[0185] Step 5013: The target tracking device determines the distance between the first key point and the second key point based on the detection position of the first key point and the predicted position of the second key point.

[0186] For example, the distance between the first keypoint and the second keypoint is calculated using the following formula 1:

[0187]

[0188] Where p is the first keypoint, p′ is the second keypoint, d(p,p′) represents the distance between the first keypoint and the second keypoint, x and y are the position coordinates of the first keypoint, and x′ and y′ are the position coordinates of the second keypoint.

[0189] It should be noted that among the key points of the detected target and the target to be tracked, there may be undetected key points. In this case, the distance between the undetected key points cannot be determined. Therefore, the target tracking device can set the distance between the undetected key points to a preset distance value. This preset distance value is used to indicate that at least one key point among two corresponding key points has not been detected.

[0190] For example, the preset distance value can be -1.

[0191] Step 501b: The target tracking device determines the distance between the i-th detected target and the j-th target to be tracked based on the distance between each corresponding key point.

[0192] For example, the distance between the i-th detected target and the j-th target to be tracked is calculated using the following formula 2:

[0193]

[0194] Where D(i,j) represents the distance between the i-th detected target and the j-th target to be tracked, p ik Let p′ represent the detection position of the k-th keypoint of the i-th target. jk d(p) represents the predicted location of the k-th keypoint of the j-th target to be tracked. ik ,p′ jk ) represents the distance between the k-th keypoint of the i-th detected target and the k-th keypoint of the j-th target to be tracked, s represents the target size information, and v ik Indicates whether the k-th keypoint of the i-th target has been detected, v′ jk δ(v) indicates whether the k-th keypoint of the j-th target to be tracked has been detected. ik ,v′ jk This represents the detection status of the k-th key point of the i-th detected target and the k-th key point of the j-th target to be tracked.

[0195] It should be noted that 's' can represent the size information of the i-th detected target, the size information of the j-th target to be tracked, or it can be determined by the size information of the i-th detected target and the j-th target to be tracked. For example, 's' can be the average of the size information of the i-th detected target and the j-th target to be tracked.

[0196] Size information can be represented by the target's image area, the positive square root of the target's image area, or the number of pixels occupied by the target.

[0197] For example, if at least one keypoint among the k-th keypoint of the i-th detected target and the k-th keypoint of the j-th target to be tracked is not detected, δ(v ik ,v′ jk The value of δ(v) can be 0. When both the k-th keypoint of the i-th detected target and the k-th keypoint of the j-th tracked target are detected, δ(v) ik ,v′ jk The value of ) can be 1.

[0198] Step 502: The target tracking device determines the matching relationship based on the distance between each detected target and each target to be tracked.

[0199] It should be noted that the target tracking device can determine the matching relationship between the target to be tracked and the detected targets from an overall perspective based on the distance between each detected target and each target to be tracked. Alternatively, the target tracking device can determine the matching relationship between the target to be tracked and the detected targets from the perspective of the individual target to be tracked based on the distance between any target to be tracked and each detected target. Specific implementation methods will be described later and will not be elaborated here.

[0200] In one possible implementation, the target tracking device can determine whether each target to be tracked matches the detected target based on a preset matching algorithm and the distance between each detected target and each target to be tracked.

[0201] For example, the preset matching algorithm can be the KM algorithm.

[0202] In this process, for a matching detection target and a target to be tracked, the target tracking device can further determine whether the distance between the detection target and the target to be tracked meets a second preset condition. If the distance between the detection target and the target to be tracked does not meet the second preset condition, it is determined that the detection target and the target to be tracked do not match. Through the above technical solution, the target tracking device can further filter matching targets to be tracked and detection targets, and eliminate targets to be tracked and detection targets that do not actually match, thereby improving the accuracy of target tracking.

[0203] The second preset condition can be that the distance between the detected target and the target to be tracked is less than or equal to the target distance threshold.

[0204] For example, the target distance threshold is 3.5, the targets to be tracked include A1, A2, A3, and A4, and the detected targets include B1, B2, B3, and B4. A1 matches B1, and the distance between A1 and B1 is 3. A2 matches B2, and the distance between A2 and B2 is 4. A3 matches B4, and the distance between A3 and B4 is 3. A4 matches B3, and the distance between A4 and B3 is 2.

[0205] The distance between A2 and B2 does not meet the second preset condition, therefore, the target tracking device determines that A2 and B2 do not match.

[0206] Figure 6 This is a scene diagram of target tracking according to some embodiments. For ease of understanding, Figure 6 Only one target is shown in the image, but the technical solution in this disclosure is applicable to target tracking scenarios with one or more targets. Target 601 is performing a jumping motion. At a second moment, target 601 in the image to be detected is in a preparatory action. At a first moment, target 601 in the image to be detected is in the process of jumping.

[0207] Figure 7 This is a schematic diagram illustrating the principle of target tracking in related technologies. For example... Figure 7 As shown, in related technologies, according to Figure 6 In the second time step, the outer bounding box image 701 of target 601 is determined from the image a to be detected at the second time step. Figure 6 The bounding box image 702 of target 601 at the first moment is determined from the image b to be detected at the first moment. The IoU between bounding box image 701 and bounding box image 702 is calculated, and it is determined whether the targets corresponding to the two bounding box images are the same target based on the IoU. However, due to the jumping action of target 601, the position and size of the bounding box image change significantly, making it difficult to accurately track target 601 in related technologies.

[0208] Figure 8 This is a schematic diagram of a target tracking principle provided according to some embodiments of this disclosure. For example... Figure 8 As shown, the target tracking device in this disclosure is based on Figure 6 The location of the key point of target 601 determined in the second time step is used to predict the predicted location of the key point of target 601 in the first time step (i.e., Figure 8 (the hollow point in the middle), and according to Figure 6 The detection location of key points of target 601 is determined from the first moment of the image b to be detected (i.e., Figure 8 (The solid points in the image). Then, the target tracking device can determine the target's matching relationship based on the detected and predicted positions. Furthermore, even if some keypoints in the obtained keypoints are not detected (e.g., solid points in the image), the target tracking device can still determine the target's matching relationship. Figure 8 (The predicted location of the key point at the right leg joint was not detected). The target tracking method in this disclosure can still track the target based on the location information of other key points, and will not have too much impact on the target tracking of this disclosure.

[0209] Compared to target tracking methods in related technologies, the technical solution of this disclosure is unaffected by changes in the bounding box image, thus improving the stability and accuracy of target tracking. Furthermore, the technical solution of this disclosure tracks the target directly based on key points, eliminating the need for additional target detection, resulting in low computational overhead and reduced computational costs.

[0210] It should be noted that, Figure 6 This paper uses a single target as an example to illustrate the differences between the tracking principle in this disclosure and the tracking principles in related technologies. The technical solutions in this disclosure are also applicable to multi-target tracking; please refer to the relevant embodiments for details. Figure 6 The scenarios described herein do not limit the scope of the disclosed solution.

[0211] The following describes the process by which the target tracking device determines the matching relationship based on the distance between each detected target and each target to be tracked.

[0212] As one possible embodiment of this disclosure, combined with Figure 5 ,like Figure 9 As shown, step 502 above can also be achieved through the following steps.

[0213] Step 901: The target tracking device determines the weight value of each target to be tracked and the weight value of each detected target.

[0214] The weight of any target to be tracked is the minimum distance between that target and each detected target. The weight of each detected target is a preset weight value.

[0215] It should be noted that the weight value of the target to be tracked is used to characterize the expected value of the target to be tracked in terms of distance, while the weight value of the detected target is used to characterize the expected value of the detected target in terms of distance.

[0216] A lower weight value indicates a higher distance requirement for the target being tracked or detected. Conversely, a higher weight value indicates a lower distance requirement for the target being tracked or detected.

[0217] For example, the preset weight value is 0, but it can be set according to the actual situation. This disclosure does not limit it.

[0218] Step 902: The target tracking device performs a third target operation on each target to be tracked to determine the detection target that matches each target to be tracked.

[0219] like Figure 10 As shown, the third target operation includes the following steps 1001-1002.

[0220] Step 1001: The target tracking device determines the first detection target that is closest to the m-th target to be tracked.

[0221] Wherein, the sum of the weights of the first detected target and the m-th target to be tracked is greater than or equal to the distance between the first detected target and the m-th target to be tracked. m is a positive integer.

[0222] It should be noted that if the sum of the weight values ​​of the target to be tracked and the detected target is greater than or equal to the distance between them, then the distance between them meets their distance requirements. When the first detected target includes multiple detected targets, the target tracking device selects one of them.

[0223] For example, target A to be tracked has a weight value of 4, target B has a weight value of -1, and the distance between target A and target B is 2. Since the sum of the weight values ​​of target A and target B is 3, which is greater than the distance of 2, this distance meets the distance requirement for target A and target B; that is, target A and target B meet the distance matching condition. Afterward, the target tracking device needs to further determine whether this distance is the smallest among the distances that meet the matching condition, and then perform subsequent operations.

[0224] In one possible implementation, if the first detection target is not present among the detection targets, the target tracking device will use the second detection target as the first detection target and update the weight value of the m-th target to be tracked.

[0225] Among them, the difference between the distance between the second detected target and the m-th target to be tracked and the sum of their weight values ​​is the smallest.

[0226] It should be noted that when the detected target includes multiple second detected targets, the target tracking device can randomly select one of the second detected targets as the first detected target. Since there is no detected target whose sum of weight values ​​with the m-th target to be tracked is greater than or equal to the distance, meaning that each detected target does not satisfy the distance matching condition with the m-th target to be tracked, the expected distance value of the m-th target to be tracked needs to be reduced to facilitate matching with a detected target. To ensure optimal matching of the target to be tracked overall, the target tracking device uses the second detected target as the first detected target and updates the weight value of the m-th target to be tracked.

[0227] For example, the target tracking device updates the weight value of the m-th target to be tracked in the following way: the target tracking device adds the difference between the distance between the second detected target and the m-th target to be tracked and the sum of the weight values ​​to the weight value of the m-th target to be tracked.

[0228] Step 1002: The target tracking device performs a fourth target operation on the m-th target to be tracked and the first detection target to determine the detection target that matches the m-th target to be tracked.

[0229] like Figure 11 As shown, the fourth target operation includes the following steps 1101-1104.

[0230] Step 1101: The target tracking device determines whether there is a first target to be tracked.

[0231] The first target to be tracked is the target to be tracked that matches the first detection target, as determined by the third target operation.

[0232] If there is no first target to be tracked, it means that there is currently no matching target to be tracked for the first detection target, so the target tracking device can execute step 1102.

[0233] Step 1102: The target tracking device determines that the m-th target to be tracked matches the first detected target.

[0234] If a first target to be tracked exists, it means that the optimal match between the m-th target to be tracked and the first target to be tracked is the first detection target. That is to say, the target tracking device needs to redetermine the matching relationship between the m-th target to be tracked and one of the first targets to be tracked and the detection targets other than the first detection target, so that the matching relationship between the target to be tracked and the detection targets achieves optimal matching as a whole. Therefore, the target tracking device can execute step 1103.

[0235] Step 1103: The target tracking device determines the first influence value of the m-th target to be tracked and the second influence value of the first target to be tracked.

[0236] The first influence value is the difference between the m-th target to be tracked and the second detection target. This difference is the difference between the distance between a target to be tracked and a detection target and the sum of the weights of the target to be tracked and a detection target. The second detection target is the detection target with the smallest difference between itself and the m-th target to be tracked, excluding the first detection target. The second influence value is the difference between the first target to be tracked and the third detection target. The third detection target is the detection target with the smallest difference between itself and the first target to be tracked, excluding the first detection target.

[0237] Therefore, the first influence value is used to characterize the minimum influence of the target tracking device re-determining the detection target that matches the m-th target to be tracked; the second influence value is used to characterize the minimum influence of the target tracking device re-determining the detection target that matches the first target to be tracked.

[0238] It should be noted that, when the sum of the distances and weight values ​​between multiple detected targets and the m-th target to be tracked is minimized, the target tracking device randomly selects one of these detected targets as the second detected target. Similarly, when the sum of the distances and weight values ​​between multiple detected targets and the first target to be tracked is minimized, the target tracking device randomly selects one of these detected targets as the third detected target.

[0239] Step 1104: The target tracking device determines the detection target that matches the m-th target to be tracked based on the first influence value and the second influence value.

[0240] If the first influence value is less than the second influence value, the target tracking device executes step 1105.

[0241] Step 1105: The target tracking device performs a fourth target operation on the m-th target to be tracked and the second detection target to determine the detection target that matches the m-th target to be tracked.

[0242] It should be noted that, since the first influence value is smaller than the second influence value, the impact of the target tracking device redetermining the matching relationship between the m-th target to be tracked and the second detection target on the overall matching relationship is smaller than the impact of redetermining the matching relationship between the first target to be tracked and the third detection target. To ensure that the target to be tracked and the detection targets achieve optimal matching as a whole, the target tracking device can redetermine the detection target that matches the m-th target to be tracked.

[0243] In one possible implementation, the target tracking device can also update the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detected target based on the first influence value.

[0244] For example, the target tracking device increases the weight value of the m-th target to be tracked by a first influence value, increases the weight value of the first target to be tracked by a first influence value, and decreases the weight value of the first detected target by a first influence value.

[0245] If the first influence value is greater than the second influence value, the target tracking device executes step 1106.

[0246] Step 1106: The target tracking device determines that the m-th target to be tracked matches the first detected target.

[0247] In one possible implementation, the target tracking device can also perform a fourth target operation on the first target to be tracked and the third detection target to determine the detection target that matches the first target to be tracked.

[0248] Similarly, since the first influence value is greater than the second influence value, the impact of redetermining the matching relationship between the m-th target to be tracked and the second detection target on the overall matching relationship is greater than the impact of redetermining the matching relationship between the first target to be tracked and the second detection target. To ensure that the target to be tracked and the detection target achieve optimal matching as a whole, the target tracking device can redetermine the detection target that matches the first target to be tracked.

[0249] In one possible implementation, the target tracking device can also update the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detected target based on the second influence value.

[0250] For example, the target tracking device increases the weight value of the m-th target to be tracked by a second influence value, increases the weight value of the first target to be tracked by a second influence value, and decreases the weight value of the first detected target by a second influence value.

[0251] If the first influence value equals the second influence value, the target tracking device executes step 1105, or the target tracking device executes step 1106.

[0252] It should be noted that in the above scheme, the target tracking device can determine a one-to-one matching relationship between the target to be tracked and the detected targets based on the distance between each detected target and each target to be tracked. In actual implementation, there may be an imbalance between the number of targets to be tracked and the number of detected targets. To obtain a one-to-one matching relationship, some targets can be added to the smaller number of targets to make the number of targets to be tracked equal to the number of detected targets.

[0253] For example, if the number of detected targets is less than the number of targets to be tracked, the target tracking device adds more detected targets to make the number of detected targets the same as the number of targets to be tracked. The distance between the added detected targets and each target to be tracked can be set to a large preset distance to avoid affecting the matching relationship.

[0254] The preset distance can theoretically be infinite. For example, the preset distance could be 1,000,000.

[0255] Furthermore, the above technical solution uses the example of a target tracking device determining a detection target that matches the target to be tracked to illustrate the process of the target tracking device determining the matching relationship in this disclosure. Target tracking can also determine a target to be tracked that matches the detection target through the above technical solution, thereby determining the matching relationship. That is, the execution objects of the target to be tracked and the detection target in the above technical solution are interchanged. The specific implementation method is the same as the above technical solution, and will not be repeated here.

[0256] Based on the above technical solution, the target tracking device in this disclosure can, when determining the matching relationship between the target to be tracked and the detection target, simultaneously consider whether the detection target has already been matched with other targets to be tracked. It adjusts the matching relationship between the target to be tracked and the detection target based on the impact of re-determining the matching relationship between the current target to be tracked and the detection target on the overall matching, as well as the impact of re-determining the matching relationship between already matched targets to be tracked and the detection target on the overall matching, so as to achieve the optimal overall matching between the target to be tracked and the detection target. Therefore, the target tracking device can determine the optimal matching relationship between the target to be tracked and the detection target from an overall perspective, improving the stability and accuracy of target tracking.

[0257] For example, the distance between the target to be tracked and the target to be detected is shown in Table 1 below:

[0258] Table 1: Distance between the target to be tracked and the target to be detected

[0259]

[0260] The targets to be tracked include A1, A2, A3, and A4, and the targets to be detected include B1, B2, B3, and B4. The distance between A1 and B1 is 3, the distance between A1 and B2 is 4, the distance between A1 and B3 is 6, the distance between A1 and B4 is 4, and so on.

[0261] Taking the above scheme as an example, the target tracking device determines the weight value of the target to be tracked, as shown in Table 2 below:

[0262] Table 2: Weight values ​​of the targets to be tracked

[0263]

[0264] The weight values ​​of the detection targets are shown in Table 3 below:

[0265] Table 3. Weight values ​​of the detection targets

[0266]

[0267] 1. The target tracking device determines the matching relationship between A1 and the detected target.

[0268] The sum of the weights of A1 and B1 is 3, which equals the distance between A1 and B1, and the distance between A1 and B1 is the shortest. Therefore, the target tracking device selects B1 as the first detection target. Meanwhile, B1 does not match any other targets to be tracked; therefore, the target tracking device determines that A1 and B1 are a match.

[0269] The matching relationship at this point is: A1 matches B1.

[0270] 2. The target tracking device determines the matching relationship between A2 and the detected target.

[0271] The sum of the weights of A2 and B4 is 3, which equals the distance between A2 and B4, and the distance between A2 and B4 is the shortest. Therefore, the target tracking device selects B4 as the first target to be detected. Meanwhile, B4 does not match any other targets to be tracked; therefore, the target tracking device determines that A2 and B4 are a match.

[0272] The matching relationships at this point are: A1 matches B1, and A2 matches B4.

[0273] 3. The target tracking device determines the matching relationship between A3 and the detected target.

[0274] The sum of the weight values ​​of A3 and B4 is 3, which equals the distance between A3 and B4, and the distance between A3 and B4 is the shortest. Therefore, the target tracking device takes B4 as the first detection target. However, since A2 and B4 have already been matched, the target tracking device needs to determine the difference between the sum of the weight values ​​of A3 and B1 and their distance, the difference between the sum of the weight values ​​of A3 and B2 and their distance, the difference between the sum of the weight values ​​of A3 and B3 and their distance, and the difference between the sum of the weight values ​​of A2 and B1 and their distance, the difference between the sum of the weight values ​​of A2 and B2 and their distance, and the difference between the sum of the weight values ​​of A2 and B3 and their distance.

[0275] The results are shown in Table 4 below:

[0276] Table 4. Difference between the sum of the weight values ​​of A2 and A3 and the detected target and the distance.

[0277]

[0278] In this scenario, both the first and second influence values ​​are 1, the second detection target is B3, and the third detection target is B2. The target tracking device can execute either step 9027 or step 9028; here, we take the execution of step 9028 as an example.

[0279] The target tracking device determines that A3 matches B4 and performs a fourth target operation on A2 and B2.

[0280] In addition, the target tracking device can update the weight values ​​of A2, A3 and B4 based on the second influence value.

[0281] The updated weight values ​​of the targets to be tracked are shown in Table 5 below:

[0282] Table 5: Updated Weights of the Targets to be Tracked

[0283]

[0284] The updated weight values ​​for the detected targets are shown in Table 6 below:

[0285] Table 6: Updated Weights of Detected Targets

[0286]

[0287] Since B2 did not temporarily match any other targets to be tracked, the target tracking device determined that A2 matched B2.

[0288] The matching relationships at this point are: A1 matches B1, A2 matches B2, and A3 matches B4.

[0289] 4. The target tracking device determines the matching relationship between A4 and the detected target.

[0290] The sum of the weights of A4 and B3 equals the distance between A4 and B3, and the distance between A4 and B3 is the shortest. Meanwhile, B3 is not matched with any other target to be tracked; therefore, the target tracking device determines that A4 and B3 are matched.

[0291] The matching relationships at this point are: A1 matches B1, A2 matches B2, A3 matches B4, and A4 matches B3.

[0292] At this point, the target tracking device has identified the detection target that matches each target to be tracked.

[0293] The following describes the process by which the target tracking device determines the matching relationship based on the distance between each detected target and each target to be tracked.

[0294] As another possible embodiment of this disclosure, combined with Figure 5 ,like Figure 12 As shown, step 502 above can also be achieved through the following steps.

[0295] Step 1201: The target tracking device determines the list of targets to be tracked and the list of targets to be detected.

[0296] The list of targets to be tracked includes each target to be tracked, and the list of targets to be detected includes each target to be detected.

[0297] Step 1202: The target tracking device performs the fifth target operation on the target to be tracked and the target to be detected in the target to be tracked list and the target to be detected list until there is no target to be tracked in the target to be tracked list or no target to be detected in the target to be detected list, and determines the target to be detected that matches each target to be tracked.

[0298] It should be noted that in actual execution, there may be a discrepancy between the number of targets to be tracked and the number of targets to be detected. Therefore, there may be some targets to be tracked or some targets to be detected. The target tracking device can determine that there is no matching target for the portion of targets to be tracked, or that there is no matching target for the portion of targets to be detected.

[0299] like Figure 13 As shown, the fifth target operation includes the following steps 1301-1304.

[0300] Step 1301: The target tracking device determines the target distance between each target in the current target tracking list and each target in the current detection target list.

[0301] Step 1302: The target tracking device determines the minimum distance in the target distance, as well as the second target to be tracked and the fourth detection target corresponding to the minimum distance.

[0302] Among them, the distance between the second target to be tracked and the fourth target to be detected is the minimum distance.

[0303] It should be noted that when multiple distance values ​​are at the minimum, the target tracking device can select any one of them as the minimum distance, and designate the target to be tracked corresponding to the minimum distance as the second target to be tracked, and the corresponding detection target as the fourth detection target.

[0304] Step 1303: The target tracking device determines the matching relationship between the second target to be tracked and the fourth target to be detected.

[0305] Since the distance between the second target to be tracked and the fourth detection target is the minimum distance, the target tracking device can preferentially determine that the second target to be tracked and the fourth detection target are matched, thereby obtaining the optimal matching relationship of the second target to be tracked.

[0306] Step 1304: The target tracking device deletes the second target from the current target tracking list and the fourth target from the current target detection list.

[0307] In other words, when determining the matching relationship between the target to be tracked and the detected target, the target tracking device does not need to consider the distance between the target to be tracked with the already determined matching relationship and other detected targets, nor does it need to consider the distance between the detected target with the already determined matching relationship and other target to be tracked. Therefore, the amount of calculation required by the target tracking device when determining the matching relationship is reduced.

[0308] Similarly, the above technical solution uses the example of a target tracking device determining a detection target that matches the target to be tracked to illustrate the process of the target tracking device determining the matching relationship in this disclosure. Target tracking can also determine a target to be tracked that matches the detection target through the above technical solution, thereby determining the matching relationship. That is, the execution objects of the target to be tracked and the detection target in the above technical solution are interchanged. The specific implementation method is the same as the above technical solution, and will not be repeated here.

[0309] Based on the above technical solution, after determining that the second tracked target and the fourth detected target, which have the smallest distance between the target list and the detection target list, match, the target tracking device in this disclosure can remove the second tracked target and the fourth detected target from the list, thereby determining the matching relationship between the remaining tracked targets and the detected targets. Therefore, the target tracking device can determine whether each tracked target matches a detected target from the perspective of the individual tracked target, further reducing the computational cost of determining the matching relationship and thus improving target tracking efficiency.

[0310] Referring to the example in Table 1 above, the target tracking device determines that the list of targets to be tracked includes: A1, A2, A3, and A4, and the list of targets to be detected includes: B1, B2, B3, and B4. The distances between A4 and B3, and between A4 and B4, are both minimum values ​​of 2. Therefore, the target tracking device can select A4 and B3 to match, and then remove A4 and B3 from both the list of targets to be tracked and the list of targets to be detected. At this point, the target tracking device determines that the list of targets to be tracked includes: A1, A2, and A3, and the list of targets to be detected includes: B1, B2, and B4.

[0311] After deletion, the distances between A1 and B1, A2 and B4, and A3 and B4 in the target list and detection list are all 3, which are the minimum values. Therefore, the target tracking device can select A2 and B4 to match and delete A2 and B4 from the target list and detection list. At this time, the target tracking device determines that the target list includes A1 and A3, and the detection list includes B1 and B2.

[0312] After deletion, the distance between A1 and B1 in the target list and the target detection list is the minimum value of 3. Therefore, the target tracking device can select A1 and B1 to match and delete A1 and B1 from the target list and the target detection list. At this time, the target tracking device determines that the target list includes A3 and the target detection list includes B2.

[0313] After deletion, the distance between A3 and B2 in both the target list and the target detection list is the minimum value of 5. Therefore, the target tracking device can select A3 and B2 to match and delete A3 and B2 from both the target list and the target detection list. At this point, the target tracking device determines that there are no targets to be tracked in the target list and no targets to be detected in the target detection list.

[0314] Ultimately, the target tracking device determined the following matching relationships: A1 matches B1, A2 matches B4, A3 matches B2, and A4 matches B3.

[0315] The following describes the process by which the target tracking device updates the position of the target to be tracked and its key points based on the matching relationship.

[0316] As one possible embodiment of this disclosure, combined with Figure 3 ,like Figure 14 As shown, step 306 above can also be achieved through the following steps.

[0317] For a target to be tracked that has a matching detection target, step 306 above can be achieved through the following steps 1401-1402:

[0318] Step 1401: The target tracking device determines the third target to be tracked.

[0319] Among them, each of the third targets to be tracked has a matching detection target.

[0320] It should be noted that the third target to be tracked is the target that successfully matches the detected target among the target targets to be tracked. Therefore, for the third target to be tracked, the target tracking device can associate the third target to be tracked with the matched detected target, thereby achieving the technical effect of target tracking.

[0321] Step 1402: The target tracking device updates the position of the key point of the third target to be tracked based on the detection position of the key point of the detected target that matches the third target to be tracked and the predicted position of the key point of the third target to be tracked.

[0322] For example, the target tracking device can correct the predicted position and the detected position according to the Kalman filtering algorithm to determine the position of the key point of the third target to be tracked at the first moment.

[0323] It should be noted that the third target to be tracked may have undetected key points, and the matching target may also have undetected key points. For example... Figure 15As shown, the left side represents the third target to be tracked, and the right side represents the detected target that matches it. Keypoints 1501 and 1504 are the detected keypoints of the third target to be tracked (i.e.,...). Figure 15 (The solid dots in the image). Key points 1502 and 1503 are undetected key points of the third target to be tracked (i.e., the solid dots). Figure 15 (Hollow points in the image). Key points 1505 and 1506 are the detected key points of the target. Key points 1507 and 1508 are the undetected key points of the target. For ease of understanding, Figure 15 Only one target to be tracked is shown in the image.

[0324] When the key point of the third target to be tracked has been detected, and the corresponding key point of the matched target has been detected (e.g.) Figure 15 (Key points 1501 and 1505 in the data) The target tracking device can update the position of the key points of the third target to be tracked based on the detection position of the key points of the detected target that matches the third target to be tracked and the predicted position of the key points of the third target to be tracked.

[0325] When the key point of the third target to be tracked has been detected, but the corresponding key point of the matching target has not been detected (e.g.) Figure 15 (Key points 1504 and 1508 in the text) The target tracking device can use the predicted position of the key point of the third target to be tracked or the position of the key point at the second time as the position of the key point at the first time.

[0326] If the keypoint of the third target to be tracked is not detected, but the corresponding keypoint of the matching target has been detected (e.g.) Figure 15 (Key points 1502 and 1506 in the target tracking device) can use the detection position of the corresponding key point of the matched target as the position of the undetected key point at the first moment.

[0327] If the keypoint of the third target to be tracked is not detected, and the corresponding keypoint of the matching target is not detected (e.g.) Figure 15 (Key points 1503 and 1507 in the image) The target tracking device can set the position information of undetected key points to preset position values ​​to indicate that the key point has not been detected.

[0328] For a detected target that does not have a matching target, step 306 above can be achieved through the following steps 1403-1404:

[0329] Step 1403: The target tracking device determines the fifth detection target.

[0330] In the fifth detection target, there is no matching target to be tracked for each detection target.

[0331] Step 1404: The target tracking device adds a fifth detection target to the list of targets to be tracked.

[0332] It should be noted that a detected target that does not have a matching target to be tracked may be a newly appeared target within the target area at the first moment. Therefore, the target tracking device needs to add it to the list of targets to be tracked and then track it subsequently.

[0333] In one possible implementation, step 1404 includes: the target tracking device taking the fifth detected target as a newly added target to be tracked at the first moment, and taking the key points of the fifth detected target as the key points of the newly added target to be tracked.

[0334] For a target to be tracked that does not have a matching detection target, step 306 above can be achieved through the following steps 1405-1408:

[0335] Step 1405: The target tracking device determines the fourth target to be tracked and the number of mismatches of the fourth target to be tracked.

[0336] In the fourth group of tracked targets, each tracked target has no matching detection target, and the number of non-matches is the number of times the tracked target has no matching detection target consecutively.

[0337] If no matching detection target exists for the target to be tracked, the target tracking device increments the mismatch count for the target to be tracked by one.

[0338] If a matching detection target exists for the target to be tracked, the target tracking device clears the number of mismatches for the target to be tracked.

[0339] For example, the target tracking device can set a match counter for each target to be tracked. The match counter is used to count the number of non-matches of the target to be tracked.

[0340] Step 1406: The target tracking device determines whether the number of mismatches is greater than the preset tracking threshold.

[0341] The preset tracking threshold can be set according to the actual situation, and this disclosure does not limit it.

[0342] It should be noted that if the number of mismatches exceeds the preset tracking threshold, it indicates that the fourth target to be tracked has failed to be tracked multiple times, and the target tracking device determines that the fourth target to be tracked has disappeared from the target area. The target tracking device can then execute step 1407.

[0343] Step 1407: The target tracking device deletes the fourth target from the list of targets to be tracked.

[0344] If the number of mismatches is less than or equal to a preset tracking threshold, it indicates that the fourth target to be tracked may have failed to match due to other factors. The target tracking device can then execute step 1408.

[0345] Step 1408: The target tracking device uses the predicted position of the key point of the fourth target to be tracked or the position of the key point of the fourth target to be tracked at the second moment as the updated position of the key point of the fourth target to be tracked.

[0346] It should be noted that for key points in the fourth target to be tracked that are not detected, the target tracking device can set the location information of the undetected key points to a preset location value to indicate that the key points have not been detected.

[0347] Based on the above technical solution, the target tracking device in this disclosure can classify the target to be tracked and the detected target into three categories according to the matching relationship: the target to be tracked has a matching detected target, the detected target does not have a matching detected target, and the target to be tracked does not have a matching detected target. Simultaneously, the target tracking device performs corresponding update operations based on the above three categories of targets, effectively improving the stability and accuracy of target tracking.

[0348] It should be noted that steps 1401-1402, 1403-1404, and 1405-1408 in the above embodiments are independent of each other. Steps 1401-1402, 1403-1404, and 1405-1408 can be executed in any combination of execution order, or they can be executed in parallel. This disclosure does not limit the execution order of the above steps.

[0349] The following describes the process by which the target tracking device updates the position of the target to be tracked and its key points based on the matching relationship.

[0350] As another possible embodiment of this disclosure, combined with Figure 3 ,like Figure 16 As shown, in addition to steps 1401-1408 above, step 306 can also be implemented through the following steps.

[0351] Step 1601: The target tracking device determines whether there is a matching detection target for the fifth target to be tracked.

[0352] The fifth target to be tracked is any one of the targets to be tracked.

[0353] If a matching detection target exists for the fifth target to be tracked, the target tracking device executes step 1602.

[0354] Step 1602: The target tracking device updates the position of the key point of the fifth target to be tracked based on the detection position of the key point of the matched target and the predicted position of the key point of the fifth target to be tracked.

[0355] The relevant implementation methods can be found in the description in step 902, and will not be repeated here.

[0356] If no matching detection target exists for the fifth target to be tracked, the target tracking device executes step 1603.

[0357] Step 1603: The target tracking device determines whether the number of mismatches of the fifth target to be tracked is greater than the preset tracking threshold.

[0358] The number of non-matches is the number of times that the target to be tracked does not have a matching detected target consecutively.

[0359] If the number of mismatches of the fifth target to be tracked exceeds the preset tracking threshold, the target tracking device may execute step 1604.

[0360] Step 1604: The target tracking device deletes the fifth target from the list of targets to be tracked.

[0361] If the number of mismatches of the fifth target to be tracked is less than or equal to the preset tracking threshold, the target tracking device may execute step 1605.

[0362] Step 1605: The target tracking device uses the predicted position of the key point of the fifth target to be tracked or the position of the key point of the fifth target to be tracked at the second moment as the updated position of the key point of the fifth target to be tracked.

[0363] It should be noted that for key points in the fifth target to be tracked that are not detected, the target tracking device can set the location information of the undetected key points to a preset location value to indicate that the key points have not been detected.

[0364] Step 1606: The target tracking device determines whether there is a matching target to be tracked for the sixth detection target.

[0365] The sixth detection target is any one of the detection targets.

[0366] If no matching target is found for the sixth detection target, the target tracking device may execute step 1607.

[0367] Step 1607: The target tracking device adds a sixth detection target to the list of targets to be tracked.

[0368] The relevant implementation methods can be found in the description in step 1404, and will not be repeated here.

[0369] Based on the above technical solution, the target tracking device in this disclosure can determine the matching relationship between each target to be tracked and each detected target, and perform corresponding update operations on each target to be tracked and each detected target according to the matching relationship, thereby improving the stability and accuracy of target tracking.

[0370] It should be noted that steps 1601 and 1606 in the above embodiments are independent of each other. This disclosure does not limit the execution order of the above steps.

[0371] This disclosure embodiment can divide the target tracking device into functional modules or functional units according to the above method examples. For example, each function can be divided into its own functional modules or functional units, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module or functional unit. The module or unit division in this disclosure embodiment is illustrative and represents only one logical functional division; other division methods may be used in actual implementation.

[0372] like Figure 17 The diagram shown is a structural diagram of a target tracking device 170 provided according to some embodiments. The device includes a processing unit 1701 and an acquisition unit 1702.

[0373] The acquisition unit 1702 is configured to acquire the image to be detected at the first moment.

[0374] The processing unit 1701 is configured to determine the detection location of key points of the target in the image to be detected.

[0375] Acquisition unit 1702 is also configured to acquire the position of key points of the target to be tracked at a second time point. The second time point is the time point before the first time point.

[0376] The processing unit 1701 is also configured to predict the position of the key points of the target to be tracked at the first time based on the position of the key points of the target at the second time.

[0377] The processing unit 1701 is also configured to determine a matching relationship between the detected target and the target to be tracked based on the detection location and the predicted location. The matching relationship is used to match the same targets between the detected target and the target to be tracked.

[0378] The processing unit 1701 is also configured to track the target to be tracked based on the matching relationship.

[0379] In some embodiments, the processing unit 1701 is configured to perform a first target operation on each detected target and each target to be tracked, determining the distance between each detected target and each target to be tracked; the first target operation includes: determining the distance between the i-th detected target and the j-th target to be tracked based on the distance between the detected position of a key point in the i-th detected target and the predicted position of a key point in the j-th target to be tracked; wherein i is a positive integer less than or equal to N; j is a positive integer less than or equal to M; the processing unit 1701 is further configured to determine a matching relationship based on the distance between each detected target and each target to be tracked.

[0380] In some embodiments, the processing unit 1701 is configured to perform a second target operation on each corresponding key point of the i-th detected target and the j-th target to be tracked, to determine the distance between each corresponding key point; the second target operation includes: determining a first key point and a second key point; determining the distance between the first key point and the second key point based on the detection position of the first key point and the predicted position of the second key point; the first key point is any key point in the i-th detected target; the second key point is the key point in the j-th target to be tracked that corresponds to the first key point; the processing unit 1701 is further configured to determine the distance between the i-th detected target and the j-th target to be tracked based on the distance between each corresponding key point.

[0381] In some embodiments, the distance between the i-th detected target and the j-th target to be tracked is calculated using the following formula:

[0382]

[0383] Where D(i,j) represents the distance between the i-th detected target and the j-th target to be tracked, p ik Let p′ represent the detection position of the k-th keypoint of the i-th target. jk d(p) represents the predicted location of the k-th keypoint of the j-th target to be tracked. ik ,p′ jk ) represents the distance between the k-th keypoint of the i-th detected target and the k-th keypoint of the j-th target to be tracked, s represents the target size information, and v ik Indicates whether the k-th keypoint of the i-th target has been detected, v′ jk Indicates whether the k-th keypoint of the j-th target to be tracked has been detected, δ(v ik ,v′ jk This represents the detection status of the k-th key point of the i-th detected target and the k-th key point of the j-th target to be tracked.

[0384] In some embodiments, the matching relationship includes at least one of the following: at least one target to be tracked matches the target to be detected, at least one target to be tracked does not match the target to be detected, and at least one target to be detected does not match the target to be tracked.

[0385] In some embodiments, the processing unit 1701 is configured to determine the weight value of each target to be tracked and the weight value of each detected target; wherein the weight value of any target to be tracked is the minimum distance among the distances between any target to be tracked and each detected target; the weight value of each detected target is a preset weight value; perform a third target operation on each target to be tracked to determine a detected target that matches each target to be tracked; the third target operation includes: determining a first detected target with the shortest distance to the m-th target to be tracked; wherein the sum of the weight values ​​of the first detected target and the m-th target to be tracked is greater than or equal to the distance between the first detected target and the m-th target to be tracked; m is a positive integer; perform a fourth target operation on the m-th target to be tracked and the first detected target to determine a detected target that matches the m-th target to be tracked; the fourth target operation includes: determining whether there is a first target to be tracked; the first target to be tracked is the one determined according to the third target operation. The process involves identifying a target to be tracked that matches the first detection target; if no first target to be tracked exists, determining the m-th target to be tracked that matches the first detection target; if a first target to be tracked exists, determining the first influence value and the second influence value of the m-th target to be tracked; wherein the first influence value is the difference between the m-th target to be tracked and the second detection target; the difference is the difference between the distance between a target to be tracked and a detection target and the sum of the weights of the target to be tracked and a detection target; the second detection target is the detection target with the smallest difference between itself and the m-th target to be tracked, excluding the first detection target; the second influence value is the difference between itself and the third detection target; the third detection target is the detection target with the smallest difference between itself and the first target to be tracked, excluding the first detection target; and the detection target that matches the m-th target to be tracked is determined based on the first influence value and the second influence value.

[0386] In some embodiments, the processing unit 1701 is configured to perform a fourth target operation on the m-th target to be tracked and the second detection target when the first influence value is less than the second influence value, and to determine the detection target that matches the m-th target to be tracked; and to determine that the m-th target to be tracked matches the first detection target when the first influence value is greater than the second influence value.

[0387] In some embodiments, the processing unit 1701 is configured to perform a fourth target operation on the first target to be tracked and the third detection target when the first influence value is greater than the second influence value, thereby determining the detection target that matches the first target to be tracked.

[0388] In some embodiments, the processing unit 1701 is configured to update the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detection target according to the first influence value when the first influence value is less than the second influence value; and to update the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detection target according to the second influence value when the first influence value is greater than the second influence value.

[0389] In some embodiments, the processing unit 1701 is configured to increase the weight value of the m-th target to be tracked by a first influence value, increase the weight value of the first target to be tracked by a first influence value, and decrease the weight value of the first detected target by a first influence value; the processing unit 1701 is also configured to increase the weight value of the m-th target to be tracked by a second influence value, increase the weight value of the first target to be tracked by a second influence value, and decrease the weight value of the first detected target by a second influence value.

[0390] In some embodiments, the processing unit 1701 is configured to determine a list of targets to be tracked and a list of targets to be detected; wherein the list of targets to be tracked includes each target to be tracked, and the list of targets to be detected includes each target to be detected; iteratively performing a fifth target operation on the targets to be tracked and the targets to be detected in the list of targets to be tracked and the list of targets to be detected until there are no targets to be tracked in the list of targets to be tracked or no targets to be detected in the list of targets to be detected, and determining a matching relationship based on the execution result of the fifth target operation; the fifth target operation includes: determining the target distance between each target to be tracked in the current list of targets to be tracked and each target to be tracked in the current list of targets to be detected; determining the minimum distance among the target distances, and the second target to be tracked and the fourth target to be detected corresponding to the minimum distance; the distance between the second target to be tracked and the fourth target to be detected is the minimum distance; determining the matching relationship between the second target to be tracked and the fourth target to be detected; deleting the second target to be tracked from the current list of targets to be tracked, and deleting the fourth target to be detected from the current list of targets to be detected.

[0391] In some embodiments, the processing unit 1701 is configured to determine a third target to be tracked; each of the third targets to be tracked has a matching detection target; the processing unit 1701 is further configured to update the position of the key point of the third target to be tracked based on the detection position of the key point of the detection target that matches the third target to be tracked and the predicted position of the key point of the third target to be tracked.

[0392] In some embodiments, the processing unit 1701 is configured to determine a fifth detection target; each of the fifth detection targets does not have a matching target to be tracked; the processing unit 1701 is further configured to add a fifth detection target to the targets to be tracked.

[0393] In some embodiments, the processing unit 1701 is configured to determine a fourth target to be tracked and the number of mismatches of the fourth target to be tracked; each of the fourth targets to be tracked does not have a matching detection target; the number of mismatches is the number of times the target to be tracked does not have a matching detection target consecutively; the processing unit 1701 is further configured to delete the fourth target to be tracked from the target to be tracked if the number of mismatches is greater than a preset tracking threshold.

[0394] In some embodiments, the processing unit 1701 is configured to use the predicted position of the key point of the fourth target to be tracked or the position of the key point of the fourth target to be tracked at the second time moment as the updated position of the key point of the fourth target to be tracked when the number of mismatches is less than or equal to a preset tracking threshold.

[0395] In some embodiments, the processing unit 1701 is configured to update the position of the key point of the fifth target to be tracked based on the detection position of the key point of the matching detection target and the predicted position of the key point of the fifth target to be tracked when a matching detection target exists for the fifth target to be tracked; the fifth target to be tracked is any one of the targets to be tracked; the processing unit 1701 is further configured to delete the fifth target to be tracked from the targets to be tracked when no matching detection target exists for the fifth target to be tracked and the number of mismatches for the fifth target to be tracked is greater than a preset tracking threshold; the number of mismatches is the number of times the target to be tracked has no matching detection target; the processing unit 1701 is further configured to add a sixth detection target to the targets to be tracked when no matching target exists for the sixth detection target; the sixth detection target is any one of the detection targets.

[0396] In some embodiments, the processing unit 1701 is configured to, when there is no matching detection target for the fifth target to be tracked and the number of mismatches of the fifth target to be tracked is less than or equal to a preset tracking threshold, use the predicted position of the key point of the fifth target to be tracked or the position of the fifth target to be tracked at the second time moment as the updated position of the key point of the fifth target to be tracked.

[0397] In some embodiments, the processing unit 1701 is configured to determine the motion state of key points of the target to be tracked; the processing unit 1701 is also configured to predict the predicted position of the key points of the target to be tracked at the first time based on the position and motion state of the key points of the target at the second time.

[0398] In some embodiments, the processing unit 1701 is configured to determine that there is no matching detection target for each target to be tracked when the number of detected targets is 0.

[0399] In some embodiments, the processing unit 1701 is configured to determine that there is no matching target for each detected target when the number of targets to be tracked is 0.

[0400] When implemented in hardware, the acquisition unit 1702 in this embodiment can be integrated onto the communication interface, and the processing unit 1701 can be integrated onto the processor. Specific implementation methods are as follows: Figure 18 As shown.

[0401] Figure 18 A schematic diagram of another possible structure of the target tracking device involved in the above embodiments is shown. The target tracking device 180 includes a processor 1802 and a communication interface 1803. The processor 1802 is configured to control and manage the operation of the target tracking device 180, for example, executing the steps performed by the processing unit 1701, and / or being configured to execute other processes of the technology described herein. The communication interface 1803 is configured to support communication between the target tracking device 180 and other network entities, for example, executing the steps performed by the acquisition unit 1702. The target tracking device 180 may also include a memory 1801 and a bus 1804, the memory 1801 being configured to store the program code and data of the target tracking device 180.

[0402] The memory 1801 may be the memory in the target tracking device 180, and the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state drive; the memory may also include a combination of the above types of memory.

[0403] The processor 1802 described above can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. The processor can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. The processor can also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0404] The 1804 bus can be an Extended Industry Standard Architecture (EISA) bus, etc. The 1804 bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 18 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0405] Figure 18 The target tracking device 180 can also be a chip. The chip includes one or more processors 1802 and a communication interface 1803.

[0406] In some embodiments, the chip further includes a memory 1801, which may include read-only memory and random access memory, and provides operation instructions and data to the processor 1802. A portion of the memory 1801 may also include non-volatile random access memory (NVRAM).

[0407] In some implementations, memory 1801 stores elements such as execution modules or data structures, or subsets thereof, or extended sets thereof.

[0408] In this embodiment of the disclosure, the corresponding operation is executed by calling the operation instructions stored in the memory 1801 (the operation instructions may be stored in the operating system).

[0409] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0410] Some embodiments of this disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) storing computer program instructions that, when executed on a computer (e.g., a target tracking device), cause the computer to perform a target tracking method as described in any of the embodiments above.

[0411] For example, the aforementioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., CDs (Compact Disks), DVDs (Digital Versatile Disks), etc.), smart cards, and flash memory devices (e.g., EPROMs (Erasable Programmable Read-Only Memory), cards, sticks, or key drives, etc.). The various computer-readable storage media described in this disclosure may represent one or more devices for storing information and / or other machine-readable storage media. The term "machine-readable storage media" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0412] Some embodiments of this disclosure also provide a computer program product, for example, stored on a non-transitory computer-readable storage medium. The computer program product includes computer program instructions that, when executed on a computer (e.g., a target tracking device), cause the computer to perform the target tracking method as described in the above embodiments.

[0413] Some embodiments of this disclosure also provide a computer program. When the computer program is executed on a computer (e.g., a target tracking device), the computer program causes the computer to perform the target tracking method as described in the above embodiments.

[0414] The beneficial effects of the aforementioned computer-readable storage medium, computer program product, and computer program are the same as the beneficial effects of the target tracking method described in some of the above embodiments, and will not be repeated here.

[0415] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0416] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0417] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0418] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A target tracking method, comprising: Acquire the image to be detected at the first moment; Determine the detection location of key points of the target in the image to be detected; The location of the key points of the target to be tracked is obtained at a second time point; the second time point is the time before the first time point. Based on the position of the key points of the target to be tracked at the second time, predict the predicted position of the key points of the target to be tracked at the first time. Based on the detection location and the predicted location, a matching relationship between the detected target and the target to be tracked is determined; The matching relationship is used to match the same targets among the detected target and the target to be tracked; Based on the matching relationship, the target to be tracked is tracked; The number of the detection targets is N The number of targets to be tracked is [number]. M indivual; M , N It is a positive integer; Determining the matching relationship between the detected target and the target to be tracked based on the detected location and the predicted location includes: For each detected target and each target to be tracked, a first target operation is performed to determine the distance between each detected target and each target to be tracked; the first target operation includes: according to the... i The detection location of key points in the first detection target, and the first j The distance between the predicted locations of key points of the target to be tracked is used to determine the distance between the predicted locations of the first target. i The first detection target and the first j The distance between the targets to be tracked; where i Less than or equal to N Positive integers; j Less than or equal to M Positive integers; Determining the matching relationship based on the distance between each detected target and each target to be tracked includes: Determine the weight value of each target to be tracked and the weight value of each detected target; wherein, the weight value of any target to be tracked is the minimum distance among the distances between any target to be tracked and each detected target; the weight value of each detected target is a preset weight value; For each target to be tracked, a third target operation is performed to determine a detection target that matches each target to be tracked; The third target operation includes: determining a first detected target that is closest to the m-th target to be tracked; wherein the sum of the weight values ​​of the first detected target and the m-th target to be tracked is greater than or equal to the distance between the first detected target and the m-th target to be tracked; m is a positive integer; Perform a fourth target operation on the m-th target to be tracked and the first detection target to determine the detection target that matches the m-th target to be tracked; The fourth target operation includes: determining whether a first target to be tracked exists; the first target to be tracked is a target to be tracked that matches the first detected target as determined by the third target operation; If the first target to be tracked does not exist, the m-th target to be tracked is determined to match the first detected target; In the presence of the first target to be tracked, a first influence value and a second influence value of the m-th target to be tracked are determined; wherein, the first influence value is the difference determined based on the m-th target to be tracked and the second detection target; the difference is the difference between the distance between a target to be tracked and a detection target and the sum of the weights of the target to be tracked and the detection target; the second detection target is the detection target other than the first detection target that has the smallest difference with the m-th target to be tracked; the second influence value is the difference determined based on the first target to be tracked and the third detection target; the third detection target is the detection target other than the first detection target that has the smallest difference with the first target to be tracked; Based on the first influence value and the second influence value, a detection target matching the m-th target to be tracked is determined.

2. The method according to claim 1, wherein, According to the first i The detection location of key points in the first detection target, and the first j The distance between the predicted locations of key points of the target to be tracked is used to determine the distance between the predicted locations of the first target. i The first detection target and the first j The distance between the targets to be tracked includes: For the first i The detection target and the first j For each corresponding key point of a target to be tracked, a second target operation is performed to determine the distance between each corresponding key point; the second target operation includes: determining a first key point and a second key point; determining the distance between the first key point and the second key point based on the detection position of the first key point and the predicted position of the second key point; the first key point is the... i Any key point among the detection targets; the second key point is the first... j The key point corresponding to the first key point among the targets to be tracked; Based on the distance between each corresponding key point, determine the first... i The first detection target and the first j The distance between the targets to be tracked.

3. The method according to claim 1, wherein, The first i The first detection target and the first j The distance between the targets to be tracked is calculated using the following formula: in, Indicates the first i The first detection target and the first j The distance between the targets to be tracked Indicates the first i The first detection target k Detection location of key points Indicates the first j The first target to be tracked k Predicted location of key points Indicates the first i The first detection target k The key point and the first j The first target to be tracked k The distance between key points s This indicates the size information of the target. Indicates the first i The first detection target k Whether the key points have been detected Indicates the first j The first target to be tracked k Whether the key points have been detected Characterizing the first i The first detection target k The key point and the first j The first target to be tracked k The detection status of key points.

4. The method according to claim 1, wherein, The matching relationship includes at least one of the following: there is at least one target to be tracked that matches the target to be detected, there is at least one target to be tracked that does not match the target to be detected, and there is at least one target to be detected that does not match the target to be tracked.

5. The method according to claim 1, wherein, The step of determining the detection target matching the m-th target to be tracked based on the first influence value and the second influence value includes: If the first influence value is less than the second influence value, the fourth target operation is performed on the m-th target to be tracked and the second detection target to determine the detection target that matches the m-th target to be tracked; If the first influence value is greater than the second influence value, the m-th target to be tracked is determined to match the first detection target.

6. The method according to claim 5, wherein, The method further includes: If the first influence value is greater than the second influence value, the fourth target operation is performed on the first target to be tracked and the third detection target to determine the detection target that matches the first target to be tracked.

7. The method according to claim 5, wherein, The method further includes: If the first influence value is less than the second influence value, the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detected target are updated according to the first influence value. If the first influence value is greater than the second influence value, the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detected target are updated according to the second influence value.

8. The method according to claim 7, wherein, The step of updating the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detected target based on the first influence value includes: Increase the weight value of the m-th target to be tracked by the first influence value, increase the weight value of the first target to be tracked by the first influence value, and decrease the weight value of the first detected target by the first influence value. The step of updating the weight values ​​of the m-th target to be tracked, the first target to be tracked, and the first detected target according to the second influence value includes: The weight value of the m-th target to be tracked is increased by the second influence value, the weight value of the first target to be tracked is increased by the second influence value, and the weight value of the first detected target is decreased by the second influence value.

9. The method according to any one of claims 1-8, wherein, The step of tracking the target to be tracked based on the matching relationship includes: A third target to be tracked is identified; each of the third targets to be tracked has a matching detection target. The position of the key point of the third target to be tracked is updated based on the detection position of the key point of the target matched with the third target to be tracked and the predicted position of the key point of the third target to be tracked.

10. The method according to any one of claims 1-8, wherein, The step of tracking the target to be tracked based on the matching relationship includes: A fifth detection target is identified; each of the fifth detection targets has no matching target to be tracked. Add the fifth detection target to the list of targets to be tracked.

11. The method according to any one of claims 1-8, wherein, The step of tracking the target to be tracked based on the matching relationship includes: A fourth target to be tracked and the number of mismatches for the fourth target to be tracked are determined; each of the fourth targets to be tracked has no matching detection target; the number of mismatches is the number of times the target to be tracked has no consecutive matching detection target. If the number of mismatches exceeds a preset tracking threshold, the fourth target to be tracked is deleted from the list of targets to be tracked.

12. The method according to any one of claims 1-8, wherein, The step of predicting the predicted position of the key points of the target to be tracked at the first time point based on the position of the key points of the target at the second time point includes: Determine the motion state of the key points of the target to be tracked; Based on the position and motion state of the key points of the target to be tracked at the second time, predict the predicted position of the key points of the target to be tracked at the first time.

13. A target tracking device, comprising: The acquisition unit is configured to acquire the image to be detected at the first moment; The processing unit is configured to determine the detection location of key points of the target in the image to be detected; The acquisition unit is further configured to acquire the position of key points of the target to be tracked at a second time point; the second time point is the time before the first time point. The processing unit is further configured to predict the predicted position of the key points of the target to be tracked at the first time moment based on the position of the key points of the target to be tracked at the second time moment. The processing unit is further configured to determine a matching relationship between the detected target and the target to be tracked based on the detection location and the predicted location; the matching relationship is used to match the same targets among the detected target and the target to be tracked; the number of detected targets is... N The number of targets to be tracked is [number]. M indivual; M , N It is a positive integer; Determining the matching relationship between the detected target and the target to be tracked based on the detected location and the predicted location includes: For each detected target and each target to be tracked, a first target operation is performed to determine the distance between each detected target and each target to be tracked; the first target operation includes: according to the... i The detection location of key points in the first detection target, and the first j The distance between the predicted locations of key points of the target to be tracked is used to determine the distance between the predicted locations of the first target. i The first detection target and the first j The distance between the targets to be tracked; where i Less than or equal to N Positive integers; j Less than or equal to M Positive integers; Determining the matching relationship based on the distance between each detected target and each target to be tracked includes: Determine the weight value of each target to be tracked and the weight value of each detected target; wherein, the weight value of any target to be tracked is the minimum distance among the distances between any target to be tracked and each detected target; the weight value of each detected target is a preset weight value; For each target to be tracked, a third target operation is performed to determine a detection target that matches each target to be tracked; The third target operation includes: determining a first detected target that is closest to the m-th target to be tracked; wherein the sum of the weight values ​​of the first detected target and the m-th target to be tracked is greater than or equal to the distance between the first detected target and the m-th target to be tracked; m is a positive integer; Perform a fourth target operation on the m-th target to be tracked and the first detection target to determine the detection target that matches the m-th target to be tracked; The fourth target operation includes: determining whether a first target to be tracked exists; the first target to be tracked is a target to be tracked that matches the first detected target as determined by the third target operation; If the first target to be tracked does not exist, the m-th target to be tracked is determined to match the first detected target; In the presence of the first target to be tracked, a first influence value and a second influence value of the m-th target to be tracked are determined; wherein, the first influence value is the difference determined based on the m-th target to be tracked and the second detection target; the difference is the difference between the distance between a target to be tracked and a detection target and the sum of the weights of the target to be tracked and the detection target; the second detection target is the detection target other than the first detection target that has the smallest difference with the m-th target to be tracked; the second influence value is the difference determined based on the first target to be tracked and the third detection target; the third detection target is the detection target other than the first detection target that has the smallest difference with the first target to be tracked; Based on the first influence value and the second influence value, determine the detection target that matches the m-th target to be tracked; The processing unit is also configured to track the target to be tracked based on the matching relationship.

14. A target tracking device, comprising: Processor and memory; The memory is used to store computer programs or instructions, and the processor is used to run the computer programs or instructions to implement the target tracking method as described in any one of claims 1-12.

15. A target tracking system, comprising a target tracking device and an image acquisition device, wherein the image acquisition device is used to acquire an image to be detected, and the target tracking device is used to perform the target tracking method as described in any one of claims 1-12.

16. A non-transitory computer-readable storage medium, wherein, The non-transitory computer-readable storage medium stores instructions that, when executed by a computer, perform the target tracking method as described in any one of claims 1-12.

17. A computer program product comprising computer program instructions, wherein when executed on a computer, the computer program instructions cause the computer to perform the target tracking method as described in any one of claims 1-12.

Images