Target tracking method and device, electronic equipment and readable storage medium

Abstract

The present disclosure provides a target tracking method and device, electronic equipment and readable storage medium, relates to the technical field of artificial intelligence, specifically to the fields of computer vision, image processing, deep learning, etc., and can be applied to smart city scenarios. The method comprises: performing target detection on a current image frame to obtain at least one detected target; transforming the coordinate values of pixels in the at least one detected target according to at least one first feature point in the current image frame and at least one second feature point in a historical image frame to obtain at least one updated detected target; performing first matching on the at least one updated detected target and at least one confirmed tracked target according to a first preset feature to obtain at least one first target pair; and updating the confirmed tracked target in the at least one first target pair according to the updated detected target in the at least one first target pair. The present disclosure fully considers the influence of camera motion on target tracking, and can improve the accuracy and robustness of target tracking.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically computer vision, image processing, deep learning, and other technical fields, and can be applied to scenarios such as smart cities. It provides a target tracking method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the continuous increase in the total number of vehicles and the total mileage of highways in my country, road traffic management faces new challenges. Conventional deployment of highway cameras has limited spacing, failing to guarantee full coverage of road areas, and suffers from fixed field of view and high manpower and material costs. Drones, vehicles, and motorcycles, with their high mobility, wide field of view, and flexible deployment, offer solutions. Installing surveillance cameras on these vehicles compensates for the shortcomings of traditional video surveillance, playing a positive role in establishing a comprehensive, three-dimensional, and intuitive monitoring system, realizing intelligent traffic management, and improving the response speed to emergencies. Summary of the Invention

[0003] According to a first aspect of this disclosure, a target tracking method is provided, comprising: performing target detection on a current image frame to obtain at least one detected target; transforming the coordinate values ​​of pixels in the at least one detected target based on at least one first feature point in the current image frame and at least one second feature point in a historical image frame to obtain at least one updated detected target; performing a first match between the at least one updated detected target and at least one confirmed tracking target based on a first preset feature to obtain at least one first target pair; and updating the confirmed tracking target in the at least one first target pair based on the updated detected target in the at least one first target pair.

[0004] According to a second aspect of this disclosure, a target tracking apparatus is provided, comprising: a detection unit for detecting a target in a current image frame to obtain at least one detected target; an alignment unit for transforming the coordinate values ​​of pixels in the at least one detected target based on at least one first feature point in the current image frame and at least one second feature point in a historical image frame to obtain at least one updated detected target; a first matching unit for performing a first matching between the at least one updated detected target and at least one confirmed tracking target based on a first preset feature to obtain at least one first target pair; and a tracking unit for updating the confirmed tracking target in the at least one first target pair based on the updated detected target in the at least one first target pair.

[0005] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0006] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0007] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0008] As can be seen from the above technical solutions, this disclosure combines the method of aligning the coordinate systems of two frames of images to achieve the purpose of motion compensation, and fully considers the impact of camera motion on target tracking. Therefore, in tracking scenarios with large scene changes and fast relative motion between the camera and the target, it can improve the accuracy and robustness of target tracking.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0012] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0013] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0015] Figure 5 This is a block diagram of an electronic device used to implement the target tracking method of the embodiments of this disclosure. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and mechanisms are omitted in the following description.

[0017] Figure 1 This is a schematic diagram based on the first embodiment of this disclosure. (See diagram below.) Figure 1 As shown, the target tracking method in this embodiment specifically includes the following steps:

[0018] S101. Perform target detection on the current image frame to obtain at least one detected target;

[0019] S102. Based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame, transform the coordinate values ​​of pixels in the at least one detection target to obtain at least one updated detection target.

[0020] S103. Based on the first preset feature, perform a first match between the at least one updated detection target and the at least one confirmed tracking target to obtain at least one first target pair;

[0021] S104. Update the confirmed tracked target in the at least one first target pair according to the updated detected target in the at least one first target pair.

[0022] The target tracking method in this embodiment, after obtaining at least one detected target in the current image frame, first transforms the coordinate values ​​of pixels in the detected target based on feature points extracted from the current image frame and historical image frames to obtain an updated detected target, thereby aligning the coordinate systems of the two frames. Then, the updated detected target is matched with the confirmed tracking target, and target tracking is completed based on the obtained at least one first target pair. Since this embodiment combines the method of aligning the coordinate systems of the two frames to achieve motion compensation, it fully considers the influence of camera motion on target tracking. Therefore, in tracking scenarios with large scene changes and fast relative motion between the camera and the target, it can improve the accuracy and robustness of target tracking.

[0023] In this embodiment, when executing S101, an image frame captured by an image acquisition device (e.g., an airborne camera) on the drone can be acquired as the current image frame. In other words, this embodiment uses a drone with an image acquisition device to perform target tracking, which can better cover areas that are difficult to capture with a fixed image acquisition device, thereby making it more convenient to perform target tracking tasks in different scenarios such as smart security, disaster relief, smart transportation, and smart cities.

[0024] In this embodiment, the detection target obtained by executing S101 can be a vehicle; that is, the target tracking method in this embodiment can be applied to the scenario of UAV road inspection, and by tracking vehicles on the road, the purpose of using UAV to monitor vehicle anomalies or illegal behaviors can be achieved.

[0025] In this embodiment, when executing S101, a pre-trained target detection model can be used to detect targets in the current image frame, thereby obtaining at least one detected target. Different detected targets correspond to different detection boxes.

[0026] In this embodiment, after obtaining at least one detection target by executing S101, S102 is executed to transform the coordinate values ​​of pixels in at least one detection target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame to obtain at least one updated detection target; wherein, the historical image frame is the previous frame image of the current image frame.

[0027] Specifically, in this embodiment, when performing S102 to transform the coordinate values ​​of pixels in at least one detection target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame to obtain at least one updated detection target, the optional implementation method can be as follows: matching at least one first feature point with at least one second feature point to obtain at least one feature point pair. In this embodiment, the matching between feature points can be performed based on the feature descriptors of the feature points; obtaining the coordinate transformation matrix between the current image frame and the historical image frame based on at least one feature point pair. In this embodiment, the RANSAC method can be used to obtain the coordinate transformation matrix based on the feature point pair; and transforming the coordinate values ​​of pixels in at least one detection target based on the obtained coordinate transformation matrix to obtain at least one updated detection target.

[0028] In this embodiment, when performing S102 to extract at least one first feature point from the current image frame, the optional implementation method is as follows: perform feature point detection on the current image frame to obtain at least one feature point. In this embodiment, the feature point obtained can be an ORB (Oriented Fast and Rotated BRIEF) feature point. In this embodiment, the feature descriptor of the feature point can be further obtained. The feature descriptor is used for matching between feature points. The feature point located outside the detection target (e.g., the bounding box of the detection target) among the obtained at least one feature point is taken as at least one first feature point.

[0029] In this embodiment, when performing S102 to extract at least one second feature point from a historical image frame, an optional implementation method can be adopted as follows: perform feature point detection on the historical image frame to obtain at least one feature point; and take the feature point located outside the tracking target (e.g., the bounding box of the tracking target) among the obtained at least one feature point as at least one second feature point.

[0030] In other words, since feature points located inside the detection or tracking target can affect the accuracy of coordinate transformation, this embodiment selects only feature points located outside the target as the first or second feature points, ensuring that the first and second feature points for matching are both located on the background of the image frame, thereby improving the accuracy of coordinate transformation.

[0031] In this embodiment, after obtaining at least one updated detection target by executing S102, S103 is executed to perform a first match between the at least one updated detection target and at least one confirmed tracking target according to the first preset feature, thereby obtaining at least one first target pair; wherein, each first target pair obtained by executing S103 in this embodiment includes an updated detection target and a confirmed tracking target that matches the updated detection target.

[0032] In this embodiment, the tracking targets located in the historical image frames are divided into confirmed tracking targets and unconfirmed tracking targets. Unconfirmed tracking targets are tracking targets that appear consecutively less than or equal to a preset number of times in multiple historical image frames, while confirmed tracking targets are tracking targets that appear consecutively more than a preset number of times in multiple historical image frames. In this embodiment, confirmed and unconfirmed tracking targets can be determined by setting target labels for the tracking targets (e.g., the target label for confirmed tracking targets is 1, and the target label for unconfirmed tracking targets is 0).

[0033] Specifically, in this embodiment, when performing S103, according to the first preset feature, performing a first match between at least one updated detection target and at least one confirmed tracking target to obtain at least one first target pair, the optional implementation method can be: obtaining the color histogram features of at least one updated detection target; obtaining the color histogram features of at least one confirmed tracking target; and performing a first match between at least one updated detection target and at least one confirmed tracking target according to the obtained color histogram features to obtain at least one first target pair.

[0034] In other words, this embodiment uses the color histogram feature of the target as the first preset feature for matching between targets. Since the color histogram feature has a low computational cost and the advantage of invariance to translation, rotation and scaling, it can improve the matching efficiency between targets while ensuring a high accuracy rate, thereby improving the efficiency of target tracking.

[0035] In this embodiment, when executing S103, the color histogram features of the updated detected target or the confirmed tracked target can be obtained using the following calculation formula:

[0036] P = (p0, p1, ..., p 23 )

[0037]

[0038] In the above formula: P represents the target's color histogram feature; p i c represents the percentage of pixels of color i; i This represents the number of pixels of color i.

[0039] In this embodiment, when executing S103, the 24-dimensional features obtained from the three channels R, G, and B, as well as the eight intervals (0-31, 32-63, 64-95, 96-127, 128-159, 160-191, 192-223, 224-255) divided according to the pixel grayscale value (0-255), can be used as color histogram features.

[0040] In this embodiment, when performing S103 to perform a first match between at least one updated detection target and at least one confirmed tracking target based on the acquired color histogram features to obtain at least one first target pair, an optional implementation method can be adopted as follows: calculate the cosine distance cost matrix between at least one updated detection target and at least one confirmed tracking target based on the color histogram features; obtain at least one first target pair based on the cosine distance cost matrix, for example, first determine the elements in the cosine distance cost matrix that are less than a preset threshold, and then obtain a first target pair based on the updated detection target and the confirmed tracking target corresponding to the element.

[0041] In other words, this embodiment determines the matching relationship between the updated detection target and the confirmed tracking target based on the cost matrix calculated from the color histogram features of at least one updated detection target and at least one confirmed tracking target, thereby obtaining the first target pair. This simplifies the matching steps between targets, improves the matching efficiency between targets, and thus meets the performance requirements of target tracking in UAV road inspection scenarios.

[0042] In this embodiment, when executing S103, the appearance features of the target can also be used as the first preset feature, or the target can be input into a pre-trained feature extraction model, and the output of the model can be used as the first preset feature.

[0043] In this embodiment, after obtaining at least one first target pair by executing S103, S104 is executed to update the confirmed tracked target in at least one first target pair according to the updated detection target in the at least one first target pair.

[0044] In this embodiment, when executing S104 to update the confirmed tracked target in the first target pair according to the updated detection target in the first target pair, the ReID of the confirmed tracked target can be updated to the ReID of the updated detection target. Alternatively, the position of the bounding box of the confirmed tracked target in the current image frame, the size of the bounding box, the direction of movement of the target, the speed of movement of the target, and other parameters can be updated to the relevant parameters of the updated detection target.

[0045] In this embodiment, after executing S104 to update the confirmed tracking target according to the updated detection target, the Kalman filter method can be used to predict the parameters of the confirmed tracking target in the next frame image based on the updated parameters of the confirmed tracking target (such as the position of the bounding box, the size of the bounding box, the direction of movement of the target, the speed of movement of the target, etc.). In other words, this embodiment can use the predicted relevant parameters of the confirmed tracking target to perform the first matching of the next frame image.

[0046] Figure 2 This is a schematic diagram according to the second embodiment of this disclosure. (See diagram below.) Figure 2 As shown, after executing S103 "obtaining at least one first target pair", the target tracking method of this embodiment may further include the following steps:

[0047] S201. Obtain at least one unassociated tracking target;

[0048] S202. Obtain updated detection targets that are not included in the at least one first target pair, as at least one unassociated detection target;

[0049] S203. Based on the second preset feature, perform a second matching between the at least one unassociated detected target and the at least one unassociated tracked target to obtain at least one second target pair;

[0050] S204. Update the unassociated tracked targets in the at least one second target pair according to the unassociated detected targets in the at least one second target pair.

[0051] In other words, since there may be updated detection targets that are not included in the first target pair after the first matching of the updated detection target and the confirmed tracking target is completed, this embodiment avoids the omission of updated detection targets during matching by obtaining at least one unassociated tracking target and performing a second matching with the updated detection target, thereby further improving the accuracy and robustness of target tracking.

[0052] In this embodiment, when executing S201 to obtain at least one unassociated tracking target, a confirmed tracking target that is not included in at least one first target pair can be obtained as at least one unassociated tracking target.

[0053] In other words, this embodiment performs a second matching on confirmed tracking targets that did not match the updated detection target in the first matching, ensuring that confirmed tracking targets and updated detection target pairs that did not match successfully in the first stage can go through the second stage of matching, thereby improving the accuracy of matching tracking targets and detection targets.

[0054] In this embodiment, when executing S201 to obtain at least one unassociated tracking target, at least one unconfirmed tracking target can also be obtained as at least one unassociated tracking target.

[0055] In other words, this embodiment only performs one match on unconfirmed tracking targets, avoiding the problem of unconfirmed targets consuming computing resources in the first match, further improving the efficiency of target tracking, and thus meeting the performance requirements of target tracking in UAV road inspection scenarios.

[0056] It is understood that, when performing S201 in this embodiment, unconfirmed tracking targets and confirmed tracking targets that are not included in at least one first target pair can be obtained simultaneously as at least one unassociated tracking target.

[0057] Specifically, in this embodiment, when performing S203 to perform a second matching of at least one unassociated detection target and at least one unassociated tracking target according to the second preset feature to obtain at least one second target pair, the optional implementation method is as follows: based on the movement direction of the unassociated detection target, the center point of the unassociated detection target, the center point of the unassociated tracking target, and the trajectory center point of the unassociated tracking target, the weighted distance between the center points of at least one unassociated detection target and at least one unassociated tracking target is obtained; based on the obtained weighted distance between the center points, at least one second target pair is obtained, for example, obtaining unassociated detection targets and unassociated tracking targets whose weighted distance between the center points is less than a preset distance to obtain a second target pair.

[0058] In other words, this embodiment uses the weighted distance between the center points of unassociated tracked targets and unassociated detected targets as the second preset feature to perform the second matching between targets. It fully considers the impact of camera movement on target tracking and has a certain tracking capability even when the target is completely occluded in a short period of time, thereby improving the accuracy and robustness of target tracking.

[0059] In this embodiment, when performing S203 to obtain the center point weighted distance between at least one unassociated detected target and at least one unassociated tracked target, the following calculation formula can be used:

[0060] d=(w0+cosθ)d0

[0061] In the above formula: d is the weighted distance between the center points of the unassociated detection targets; w0 is the preset weight coefficient; θ is the angle between the motion direction of the unassociated detection target and the line connecting the center point of the trajectory of the unassociated tracking target; d0 is the Euclidean distance between the center point of the unassociated detection target and the center point of the unassociated tracking target.

[0062] Each second target pair obtained by executing S203 in this embodiment includes an unassociated detection target and an unassociated tracking target that matches the unassociated detection target.

[0063] In this embodiment, the process of updating the unassociated tracked target according to the unassociated detected target in the second target pair in step S204 is similar to the process of updating the confirmed tracked target according to the updated detected target in the first target pair in step S104, and will not be described in detail here.

[0064] Since this embodiment treats unconfirmed tracking targets as unassociated tracking targets when executing S201, this embodiment may also include the following when executing S204 to update unassociated tracking targets in at least one second target pair: if it is determined that an unassociated tracking target in at least one second target pair is an unconfirmed tracking target, obtain the number of consecutive occurrences of the unassociated tracking target; if it is determined that the obtained number of consecutive occurrences is greater than a preset number, update the unassociated tracking target to a confirmed tracking target.

[0065] After executing S204, this embodiment may further include the following: obtaining unassociated detected targets not included in at least one second target pair as unconfirmed tracking targets; this embodiment may also update the obtained unconfirmed tracking targets.

[0066] In other words, this embodiment uses the updated detection target that has not been matched after two stages of matching as the unconfirmed tracking target for the next second matching, thereby achieving the purpose of real-time updating of the tracking target and ensuring that each updated detection target in the current image frame can be matched in the next round.

[0067] After executing S204, this embodiment may further include the following: obtaining unassociated tracking targets not included in at least one second target pair as tracking targets to be processed; if it is determined that the obtained tracking targets to be processed are confirmed tracking targets, obtaining the unupdated duration of the tracking targets to be processed; if it is determined that the obtained unupdated duration is less than a preset duration, updating the tracking targets to be processed (e.g., using a Kalman filter method for prediction), otherwise deleting the tracking targets to be processed, that is, deleting tracking targets to be processed with an unupdated duration greater than or equal to the preset duration.

[0068] After executing S204, this embodiment may also include the following: if it is determined that the acquired tracking target to be processed is an unconfirmed tracking target, the tracking target to be processed is deleted.

[0069] In other words, after completing the two-stage matching, this embodiment deletes or updates the tracking targets that have not yet been successfully matched based on their type (confirmed or unconfirmed) and the duration of no update, thereby achieving the goal of real-time updating of the tracking targets and further improving the accuracy and robustness of target tracking.

[0070] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure. Figure 3The flowchart of this embodiment during target tracking is shown: S301, target detection is performed on the current image frame to obtain at least one detected target; S302, the coordinate values ​​of pixels in the at least one detected target are transformed to complete motion compensation, thereby obtaining at least one updated detected target; S303, a first match is performed between the at least one updated detected target and at least one confirmed tracking target to obtain at least one first target pair, and updated detected targets and confirmed tracking targets not included in the first target pair; S304, the tracking target is updated based on the detected targets in the first target pair. After obtaining the tracking result (i.e., the updated tracking target), the tracking result is predicted using a Kalman filter method to be used as the confirmed tracking target for the next first match; S305, a second match is performed between the updated detected target not included in the first target pair and the unconfirmed tracking target, and the confirmed tracking target not included in the first target pair, to obtain at least one second target pair, and tracking targets not included in the second target pair. Detecting targets; S306, updating the tracking targets based on the detected targets in the second target pair. After obtaining the tracking results, using Kalman filtering to predict the tracking results, which will be used as confirmed tracking targets for the next first match; S308, taking the detected targets not included in the second target pair as newly created unconfirmed tracking targets, and using Kalman filtering to predict the newly created unconfirmed tracking targets, which will be used as unconfirmed tracking targets for the next second match. Wherein, when it is determined that the number of consecutive occurrences of an unconfirmed tracking target is greater than a preset number, the unconfirmed tracking target is updated to a confirmed tracking target; S307, determining the tracking targets not included in the second target pair, and processing them accordingly based on the type of tracking target and the duration of non-update, such as directly deleting unconfirmed tracking targets, deleting confirmed tracking targets with a non-update duration greater than or equal to a preset duration, or using Kalman filtering to predict confirmed tracking targets with a non-update duration less than a preset duration, which will be used as confirmed tracking targets for the next first match, etc.

[0071] Figure 4 This is a schematic diagram according to the fourth embodiment of this disclosure. (See diagram below.) Figure 4 As shown, the target tracking device 400 of this embodiment includes:

[0072] Detection unit 401 is used to perform target detection on the current image frame to obtain at least one detected target;

[0073] Alignment unit 402 is used to transform the coordinate values ​​of pixels in at least one detection target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame to obtain at least one updated detection target;

[0074] The first matching unit 403 is used to perform a first matching between the at least one updated detection target and the at least one confirmed tracking target according to a first preset feature, so as to obtain at least one first target pair;

[0075] Tracking unit 404 is used to update the confirmed tracked target in the at least one first target pair according to the updated detected target in the at least one first target pair.

[0076] The detection unit 401 can acquire image frames captured by an image acquisition device (e.g., an airborne camera) on the UAV as the current image frame; the detection target obtained by the detection unit 401 can be a vehicle.

[0077] The detection unit 401 can use a pre-trained target detection model to perform target detection on the current image frame, thereby obtaining at least one detected target. Different detected targets correspond to different detection boxes.

[0078] In this embodiment, after the detection unit 401 obtains at least one detection target, the alignment unit 402 transforms the coordinate values ​​of the pixels in the at least one detection target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame to obtain at least one updated detection target; wherein, the historical image frame is the previous frame image of the current image frame.

[0079] Specifically, when the alignment unit 402 transforms the coordinate values ​​of pixels in at least one detection target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame to obtain at least one updated detection target, the optional implementation method may be as follows: matching at least one first feature point with at least one second feature point to obtain at least one feature point pair; obtaining a coordinate transformation matrix between the current image frame and the historical image frame based on the at least one feature point pair; and transforming the coordinate values ​​of pixels in at least one detection target based on the obtained coordinate transformation matrix to obtain at least one updated detection target.

[0080] When the alignment unit 402 extracts at least one first feature point from the current image frame, the optional implementation method may be as follows: perform feature point detection on the current image frame to obtain at least one feature point; and take the feature points located outside the detection target (e.g., the bounding box of the detection target) among the obtained at least one feature points as at least one first feature point.

[0081] When the alignment unit 402 extracts at least one second feature point from the historical image frame, the optional implementation method may be as follows: perform feature point detection on the historical image frame to obtain at least one feature point; and take the feature point located outside the tracking target (e.g., the bounding box of the tracking target) among the obtained at least one feature point as at least one second feature point.

[0082] In other words, since feature points located inside the detection target or tracking target can affect the accuracy of coordinate transformation, the alignment unit 402 selects only feature points located outside the target as the first feature point or the second feature point, ensuring that the first feature point and the second feature point for matching are both located on the background of the image frame, thereby improving the accuracy of coordinate transformation.

[0083] In this embodiment, after the alignment unit 402 obtains at least one updated detection target, the first matching unit 403 performs a first match between the at least one updated detection target and at least one confirmed tracking target according to the first preset feature to obtain at least one first target pair; wherein, each first target pair obtained by the first matching unit 403 includes an updated detection target and a confirmed tracking target that matches the updated detection target.

[0084] In this embodiment, the tracking targets located in the historical image frames are divided into confirmed tracking targets and unconfirmed tracking targets. Unconfirmed tracking targets are tracking targets that appear consecutively less than or equal to a preset number of times in multiple historical image frames, while confirmed tracking targets are tracking targets that appear consecutively more than a preset number of times in multiple historical image frames. In this embodiment, confirmed and unconfirmed tracking targets can be determined by setting target labels for the tracking targets (e.g., the target label for confirmed tracking targets is 1, and the target label for unconfirmed tracking targets is 0).

[0085] Specifically, when the first matching unit 403 performs a first match between at least one updated detection target and at least one confirmed tracking target according to the first preset features to obtain at least one first target pair, the optional implementation method may be as follows: obtain the color histogram features of at least one updated detection target; obtain the color histogram features of at least one confirmed tracking target; and perform a first match between at least one updated detection target and at least one confirmed tracking target according to the obtained color histogram features to obtain at least one first target pair.

[0086] In other words, the first matching unit 403 uses the color histogram feature of the target as the first preset feature to match between targets. Since the color histogram feature has a low computational cost and the advantages of translation, rotation and scaling invariance, it can improve the matching efficiency between targets while ensuring a high accuracy in the matching between targets, thereby improving the efficiency of target tracking.

[0087] The first matching unit 403 can use the following calculation formula to obtain the color histogram features of the updated detected target or the confirmed tracked target:

[0088] P = (p0, p1, ..., p 23 )

[0089]

[0090] In the above formula: P represents the target's color histogram feature; p i c represents the percentage of pixels of color i; i This represents the number of pixels of color i.

[0091] The first matching unit 403 can use the three channels R, G, and B, as well as the 24-dimensional features obtained from the eight intervals (0-31, 32-63, 64-95, 96-127, 128-159, 160-191, 192-223, 224-255) divided according to the pixel grayscale value (0-255), as color histogram features.

[0092] When the first matching unit 403 performs a first match between at least one updated detection target and at least one confirmed tracking target based on the acquired color histogram features to obtain at least one first target pair, the optional implementation method may be as follows: calculate the cosine distance cost matrix between at least one updated detection target and at least one confirmed tracking target based on the color histogram features; and obtain at least one first target pair based on the cosine distance cost matrix.

[0093] In other words, the first matching unit 403 determines the matching relationship between the updated detection target and the confirmed tracking target based on the cost matrix calculated by the color histogram features of at least one updated detection target and at least one confirmed tracking target, thereby obtaining the first target pair. This simplifies the matching steps between targets, improves the matching efficiency between targets, and thus meets the performance requirements of target tracking in UAV road inspection scenarios.

[0094] The first matching unit 403 can also use the appearance features of the target as the first preset feature, or it can input the target into a pre-trained feature extraction model and use the output of the model as the first preset feature.

[0095] In this embodiment, after the first matching unit 403 obtains at least one first target pair, the tracking unit 404 updates the confirmed tracked target in at least one first target pair according to the updated detection target in the at least one first target pair.

[0096] When the tracking unit 404 updates the confirmed tracked target in the first target pair according to the updated detection target in the first target pair, it can update the ReID of the confirmed tracked target to the ReID of the updated detection target, or update the parameters such as the position of the bounding box of the confirmed tracked target in the current image frame, the size of the bounding box, the direction of movement of the target, and the speed of movement of the target to the relevant parameters of the updated detection target.

[0097] After updating the confirmed tracking target according to the updated detection target, the tracking unit 404 can also use the Kalman filter method to predict the parameters of the confirmed tracking target in the next frame image based on the updated parameters of the confirmed tracking target; that is, in this embodiment, the predicted relevant parameters of the confirmed tracking target can be used to perform the first matching of the next frame image.

[0098] The target tracking device 400 of this embodiment may further include a second matching unit 405, which is used to perform the following: acquiring at least one unassociated tracking target; acquiring an updated detection target not included in at least one first target pair as at least one unassociated detection target; performing a second matching between the at least one unassociated detection target and the at least one unassociated tracking target according to a second preset feature to obtain at least one second target pair; and updating the unassociated tracking target in the at least one second target pair according to the unassociated detection target in the at least one second target pair.

[0099] In other words, since there may be updated detection targets that are not included in the first target pair after the first matching of the updated detection target and the confirmed tracking target is completed, the second matching unit 405 obtains at least one unassociated tracking target to perform a second matching with the updated detection target, thereby avoiding the omission of updated detection targets during matching and further improving the accuracy and robustness of target tracking.

[0100] When the second matching unit 405 acquires at least one unassociated tracking target, it may acquire a confirmed tracking target that is not included in at least one first target pair as at least one unassociated tracking target.

[0101] In other words, the second matching unit 405 will perform a second matching on the confirmed tracking targets that did not match the updated detection target in the first matching, so as to ensure that the confirmed tracking targets and updated detection targets that failed to match in the first stage can be matched in the second stage, thereby improving the accuracy of matching the tracking targets and detection targets.

[0102] When the second matching unit 405 acquires at least one unassociated tracking target, it can also acquire at least one unconfirmed tracking target as at least one unassociated tracking target.

[0103] In other words, the second matching unit 405 only performs one match on unconfirmed tracking targets, avoiding the problem of unconfirmed targets consuming computing resources when participating in the first match, further improving the efficiency of target tracking, and thus meeting the performance requirements of target tracking in UAV road inspection scenarios.

[0104] It is understood that the second matching unit 405 can also simultaneously acquire unconfirmed tracking targets and confirmed tracking targets that are not included in at least one first target pair, as at least one unassociated tracking target.

[0105] Specifically, when the second matching unit 405 performs a second matching between at least one unassociated detection target and at least one unassociated tracking target according to the second preset feature to obtain at least one second target pair, the optional implementation method may be as follows: based on the movement direction of the unassociated detection target, the center point of the unassociated detection target, the center point of the unassociated tracking target, and the trajectory center point of the unassociated tracking target, the weighted distance between the center points of at least one unassociated detection target and at least one unassociated tracking target is obtained; based on the obtained weighted distance between the center points, at least one second target pair is obtained.

[0106] In other words, the second matching unit 405 uses the weighted distance between the center points of the unassociated tracked target and the unassociated detected target as the second preset feature to perform the second matching between the targets. This fully considers the impact of camera motion on target tracking and has a certain tracking capability even when the target is completely occluded in a short period of time, thereby improving the accuracy and robustness of target tracking.

[0107] When the second matching unit 405 obtains the center point weighted distance between at least one unassociated detected target and at least one unassociated tracked target, it can use the following calculation formula:

[0108] d=(w0+cosθ)d0

[0109] In the above formula: d is the weighted distance between the center points of the unassociated detection targets; w0 is the preset weight coefficient; θ is the angle between the motion direction of the unassociated detection target and the line connecting the center point of the trajectory of the unassociated tracking target; d0 is the Euclidean distance between the center point of the unassociated detection target and the center point of the unassociated tracking target.

[0110] Each second target pair obtained by the second matching unit 405 includes an unassociated detected target and an unassociated tracked target that matches the unassociated detected target.

[0111] The process by which the second matching unit 405 updates the unassociated tracked target based on the unassociated detected target in the second target pair is similar to the process by which the tracking unit 404 updates the confirmed tracked target based on the updated detected target in the first target pair, and will not be described in detail here.

[0112] Since the second matching unit 405 will treat unconfirmed tracking targets as unassociated tracking targets, when updating unassociated tracking targets in at least one second target pair, the second matching unit 405 may also include the following: if it is determined that an unassociated tracking target in at least one second target pair is an unconfirmed tracking target, obtain the number of consecutive occurrences of the unassociated tracking target; if it is determined that the obtained number of consecutive occurrences is greater than a preset number, update the unassociated tracking target as a confirmed tracking target.

[0113] The second matching unit 405 can also perform the following: obtain unassociated detected targets not included in at least one second target pair as unconfirmed tracking targets; in this embodiment, the obtained unconfirmed tracking targets can also be updated.

[0114] In other words, the second matching unit 405 uses the updated detection target that has not been matched after two stages of matching as the unconfirmed tracking target for the next second matching, thereby achieving the purpose of real-time updating of the tracking target and ensuring that each updated detection target in the current image frame can be matched in the next round.

[0115] The second matching unit 405 may also perform the following: obtain unassociated tracking targets not included in at least one second target pair as tracking targets to be processed; if it is determined that the obtained tracking targets to be processed are confirmed tracking targets, obtain the unupdated duration of the tracking targets to be processed; if it is determined that the obtained unupdated duration is less than a preset duration, update the tracking targets to be processed, otherwise delete the tracking targets to be processed, that is, delete the tracking targets to be processed with an unupdated duration greater than or equal to the preset duration.

[0116] The second matching unit 405 may also perform the following: if it is determined that the acquired target to be tracked is an unconfirmed target, delete the target to be tracked.

[0117] In other words, after completing the two-stage matching, the second matching unit 405 deletes or updates the tracking targets that have not yet been successfully matched, based on the type of the tracking target (confirmed or unconfirmed) and the duration of no update. This achieves the purpose of real-time updating of the tracking targets and further improves the accuracy and robustness of target tracking.

[0118] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0119] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0120] like Figure 5 The diagram shown is a block diagram of an electronic device for a target tracking method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0121] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0122] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of displays, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as target tracking methods. For example, in some embodiments, the target tracking method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508.

[0124] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by computing unit 501, one or more steps of the target tracking method described above may be performed. Alternatively, in other embodiments, computing unit 501 may be configured to perform the target tracking method by any other suitable means (e.g., by means of firmware).

[0125] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0126] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable target tracking device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0127] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0129] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0130] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the management difficulties and weak business scalability inherent in traditional physical hosts and VPS (Virtual Private Server) services. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0131] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0132] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A target tracking method, comprising: Perform target detection on the current image frame to obtain at least one detected target; Based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame, the coordinate values ​​of pixels in the at least one detection target are transformed to obtain at least one updated detection target. The first feature point is a feature point located outside the detection target in the current image frame, and the second feature point is a feature point located outside the tracking target in the historical image frame. Based on the first preset feature, the at least one updated detection target is matched with at least one confirmed tracking target to obtain at least one first target pair; Based on the updated detected target in the at least one first target pair, update the confirmed tracked target in the at least one first target pair, and based on the updated parameters of the confirmed tracked target, predict the parameters of the confirmed tracked target in the next frame image; The step of transforming the coordinate values ​​of pixels in the at least one detected target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame to obtain at least one updated detected target includes: Based on the feature descriptors of the feature points, the at least one first feature point is matched with the at least one second feature point to obtain at least one feature point pair; Based on the at least one pair of feature points, the coordinate transformation matrix between the current image frame and the historical image frame is obtained; The coordinate values ​​of pixels in the at least one detected target are transformed according to the coordinate transformation matrix to obtain the at least one updated detected target; Obtaining at least one first feature point in the current image frame includes: Feature point detection is performed on the current image frame to obtain at least one feature point; The feature point located outside the detection target among the at least one feature points is designated as the at least one first feature point; Obtaining at least one second feature point in the historical image frame includes: Feature point detection is performed on the historical image frames to obtain at least one feature point; The feature point located outside the tracking target among the at least one feature points is designated as the at least one second feature point; The method further includes: After obtaining at least one first target pair, at least one unassociated tracked target is acquired; Obtain updated detection targets that are not included in the at least one first target pair, as at least one unassociated detection target; According to the second preset feature, the at least one unassociated detected target and the at least one unassociated tracked target are matched in the second way to obtain at least one second target pair, including: obtaining the center point weighted distance between the at least one unassociated detected target and the at least one unassociated tracked target by connecting the movement direction of the unassociated detected target, the center point of the unassociated detected target, the center point of the unassociated tracked target and the trajectory center point of the unassociated tracked target; and obtaining the at least one second target pair based on the center point weighted distance. Update the unassociated tracked targets in the at least one second target pair based on the unassociated detected targets in the at least one second target pair.

2. The method according to claim 1, wherein, The first preset feature is used to... The first matching of at least one updated detection target with at least one confirmed tracking target to obtain at least one first target pair includes: Obtain the color histogram features of the at least one updated detection target; Obtain the color histogram features of the at least one confirmed tracked target; Based on the color histogram features, the at least one updated detection target is matched with the at least one confirmed tracking target to obtain the at least one first target pair.

3. The method according to claim 1, wherein, The acquisition of at least one unassociated tracking target includes: Acquire confirmed tracked targets not included in the at least one first target pair as the at least one unassociated tracked target; and / or At least one unconfirmed tracking target is identified as the at least one unassociated tracking target.

4. The method according to claim 1, wherein, Updating the unassociated tracked targets in the at least one second target pair includes: If it is determined that the unassociated tracking target in the at least one second target pair is an unconfirmed tracking target, the number of consecutive occurrences of the unassociated tracking target is obtained; If it is determined that the number of consecutive occurrences is greater than a preset number, the unassociated tracking target is updated to a confirmed tracking target.

5. The method according to claim 1, further comprising: After obtaining at least one second target pair, unassociated detected targets not included in the at least one second target pair are obtained as unconfirmed tracking targets.

6. The method according to claim 1, further comprising: After obtaining at least one second target pair, unassociated tracking targets not included in the at least one second target pair are obtained as tracking targets to be processed; If the target to be tracked is determined to be a confirmed target, the duration of the target's lack of updates is obtained. If the unupdated duration is determined to be less than a preset duration, the target to be tracked is updated; otherwise, the target to be tracked is deleted.

7. A target tracking device, comprising: The detection unit is used to perform target detection on the current image frame to obtain at least one detected target. An alignment unit is used to transform the coordinate values ​​of pixels in at least one detection target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame, so as to obtain at least one updated detection target. The first matching unit is configured to perform a first matching between the at least one updated detection target and the at least one confirmed tracking target based on a first preset feature, so as to obtain at least one first target pair; The tracking unit is configured to update the confirmed tracked target in the at least one first target pair according to the updated detected target in the at least one first target pair, and predict the parameters of the confirmed tracked target in the next frame image according to the updated parameters of the confirmed tracked target. Specifically, when the alignment unit transforms the coordinate values ​​of pixels in the at least one detected target based on at least one first feature point in the current image frame and at least one second feature point in the historical image frame to obtain at least one updated detected target, it performs the following: Based on the feature descriptors of the feature points, the at least one first feature point is matched with the at least one second feature point to obtain at least one feature point pair; Based on the at least one pair of feature points, the coordinate transformation matrix between the current image frame and the historical image frame is obtained; The coordinate values ​​of pixels in the at least one detected target are transformed according to the coordinate transformation matrix to obtain the at least one updated detected target; When the alignment unit acquires at least one first feature point in the current image frame, it specifically performs the following: Feature point detection is performed on the current image frame to obtain at least one feature point; The feature point located outside the detection target among the at least one feature points is designated as the at least one first feature point; When the alignment unit acquires at least one second feature point in the historical image frame, it specifically performs the following: Feature point detection is performed on the historical image frames to obtain at least one feature point; The feature point located outside the tracking target among the at least one feature points is designated as the at least one second feature point; It also includes a second matching unit for execution: After the first matching unit obtains the at least one first target pair, at least one unassociated tracked target is acquired; Obtain updated detection targets that are not included in the at least one first target pair, as at least one unassociated detection target; According to the second preset feature, the at least one unassociated detected target and the at least one unassociated tracked target are matched in the second way to obtain at least one second target pair, including: obtaining the center point weighted distance between the at least one unassociated detected target and the at least one unassociated tracked target by connecting the movement direction of the unassociated detected target, the center point of the unassociated detected target, the center point of the unassociated tracked target and the trajectory center point of the unassociated tracked target; and obtaining the at least one second target pair based on the center point weighted distance. Update the unassociated tracked targets in the at least one second target pair based on the unassociated detected targets in the at least one second target pair.

8. The apparatus according to claim 7, wherein, When the first matching unit performs a first match between the at least one updated detection target and the at least one confirmed tracking target based on a first preset feature, and obtains at least one first target pair, it specifically performs the following: Obtain the color histogram features of the at least one updated detection target; Obtain the color histogram features of the at least one confirmed tracked target; Based on the color histogram features, the at least one updated detection target is matched with the at least one confirmed tracking target to obtain the at least one first target pair.

9. The apparatus according to claim 7, wherein, When the second matching unit acquires at least one unassociated tracking target, it specifically performs the following: Acquire confirmed tracked targets not included in the at least one first target pair as the at least one unassociated tracked target; and / or At least one unconfirmed tracking target is identified as the at least one unassociated tracking target.

10. The apparatus according to claim 7, wherein, When updating the unassociated tracked targets in the at least one second target pair, the first matching unit specifically performs the following: If it is determined that the unassociated tracking target in the at least one second target pair is an unconfirmed tracking target, the number of consecutive occurrences of the unassociated tracking target is obtained; If it is determined that the number of consecutive occurrences is greater than a preset number, the unassociated tracking target is updated to a confirmed tracking target.

11. The apparatus according to claim 7, wherein, The second matching unit is also used to perform: After obtaining at least one second target pair, unassociated detected targets not included in the at least one second target pair are obtained as unconfirmed tracking targets.

12. The apparatus according to claim 7, wherein, The second matching unit is also used to perform: After obtaining at least one second target pair, unassociated tracking targets not included in the at least one second target pair are obtained as tracking targets to be processed; If the target to be tracked is determined to be a confirmed target, the duration of the target's lack of updates is obtained. If the unupdated duration is determined to be less than a preset duration, the target to be tracked is updated; otherwise, the target to be tracked is deleted.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images