Trajectory Determination Method, Apparatus, Computer-Readable Storage Medium, and Electronic Device

Abstract

An embodiment of the present application provides a trajectory determination method, apparatus, computer-readable storage medium, and electronic device. The method includes: determining a target detection box in a first image frame of a target video sequence and a confidence level corresponding to the target detection box; determining a target prediction box of a target object in the first image frame according to a historical trajectory corresponding to a second image frame; determining a target association parameter according to the target detection box and the target prediction box; and when the target association parameter is greater than a preset parameter threshold, determining a trajectory update method corresponding to the confidence level and updating the historical trajectory according to the trajectory update method to obtain a target trajectory. Through the present application, the problem that in the process of multi-target tracking, when the target moves too fast or the video frame rate is too low, due to a large offset between target frames, the accuracy of trajectory determination is poor is solved, and thus the effect of real-time tracking is achieved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer vision, and more specifically, to a method and device for trajectory determination, a computer-readable storage medium, and an electronic device. Background Art

[0002] Multiple object tracking (MOT) is a key technology in the field of computer vision and is widely applied in directions such as autonomous driving, video surveillance, and behavior recognition. In multiple object tracking, factors such as scale changes, deformations, occlusions, rapid movements, and illumination changes of the tracked objects will all affect the tracking results of multiple objects.

[0003] In the related art during the process of multiple object tracking, when the object moves too fast or the video frame rate is too low, due to the large offset between target frames, there is a problem that the trajectory determination is inaccurate, that is, the object tracking fails.

[0004] In view of the above problems, no effective solution has been proposed yet. Summary of the Invention

[0005] Embodiments of the present application provide a method and device for trajectory determination, a computer-readable storage medium, and an electronic device, so as to at least solve the problem in the related art that during the process of multiple object tracking, when the object moves too fast or the video frame rate is too low, due to the large offset between target frames, the accuracy of trajectory determination is poor.

[0006] According to an embodiment of the present application, a method for trajectory determination is provided, including: determining a target detection box in a first image frame of a target video sequence and a confidence level corresponding to the target detection box, where the target detection box is used to frame a target object in the first image frame; determining a target prediction box of the target object in the first image frame according to a historical trajectory corresponding to a second image frame, where the second image frame is the previous image frame adjacent to the first image frame in the target video sequence, and the historical trajectory is used to represent the movement trajectory of the target object before the first image frame; determining a target association parameter according to the target detection box and the target prediction box, where the target association parameter is determined at least by an overlap degree, a displacement distance, and a movement direction between the target detection box and the target prediction box, and the target association parameter is used to represent the similarity degree between the target detection box and the target prediction box; and when the target association parameter is greater than a preset parameter threshold, determining a trajectory update method corresponding to the confidence level, and updating the historical trajectory according to the trajectory update method to obtain a target trajectory.

[0007] In an exemplary embodiment, determining the target association parameter based on the target detection box and the target prediction box includes: determining the target overlap degree cost value according to the overlap degree between the target detection box and the target prediction box; determining the target distance cost value according to the displacement distance between the target detection box and the target prediction box; determining the target direction cost value according to the motion direction corresponding to the target detection box and the motion direction corresponding to the target prediction box; and determining the target association parameter according to the target overlap degree cost value, the target distance cost value, and the target direction cost value.

[0008] In an exemplary embodiment, determining the target association parameter based on the target overlap degree cost value, the target distance cost value, and the target direction cost value includes: determining the first weight value corresponding to the target overlap degree cost value; determining the second weight value corresponding to the target distance cost value; determining the third weight value corresponding to the target direction cost value; and calculating according to the target overlap degree cost value, the first weight value, the target distance cost value, the second weight value, the target direction cost value, and the third weight value to obtain the target association parameter.

[0009] In an exemplary embodiment, determining the target overlap degree cost value according to the overlap degree between the target detection box and the target prediction box includes: adjusting the border sizes of the target detection box and the target prediction box, where the difference in the border sizes between the adjusted target detection box and the target prediction box is not greater than the difference in the border sizes between the unadjusted target detection box and the target prediction box; determining the overlap degree between the adjusted target detection box and the target prediction box; determining the area similarity between the adjusted target detection box and the target prediction box; and determining the target overlap degree cost value according to the overlap degree and the area similarity.

[0010] In an exemplary embodiment, determining the overlap degree between the adjusted target detection box and the target prediction box includes: determining the first area selected by the adjusted target detection box in the first image frame and the second area selected by the adjusted target prediction box in the first image frame; determining the first area value of the intersection area between the first area and the second area and the second area value of the union area between the first area and the second area; and determining the ratio of the first area value to the second area value as the overlap degree.

[0011] In an exemplary embodiment, determining the area similarity between the adjusted target detection box and the target prediction box includes: determining the third area value of the first area and the fourth area value of the second area; calculating the absolute value of the difference between the third area value and the fourth area value and the larger area value among the third area value and the fourth area value; and determining the area similarity according to the absolute value of the difference and the larger area value.

[0012] In an exemplary embodiment, the border size includes: border height and border width; adjusting the border sizes of the target detection box and the target prediction box includes: determining a first change rate based on the border heights of the target detection box and the target prediction box, where the first change rate is used to characterize the degree of difference between the border height of the target detection box and the border height of the target prediction box; determining a second change rate based on the border widths of the target detection box and the target prediction box, where the second change rate is used to characterize the degree of difference between the border width of the target detection box and the border width of the target prediction box; adjusting the border height or border width of the target detection box and the target prediction box based on the first change rate and the second change rate.

[0013] In an exemplary embodiment, adjusting the border height or border width of the target detection box and the target prediction box based on the first change rate and the second change rate includes: when the first change rate is greater than a first preset threshold and the second change rate is less than a second preset threshold, adjusting the border height of the target detection box and / or the target prediction box, where the first change rate corresponding to the adjusted target detection box and target prediction box is less than the first preset threshold; when the second change rate is greater than the first preset threshold and the first change rate is less than the second preset threshold, adjusting the border width of the target detection box and / or the target prediction box, where the second change rate corresponding to the adjusted target detection box and target prediction box is less than the first preset threshold.

[0014] In an exemplary embodiment, determining the target distance cost value based on the displacement distance between the target detection box and the target prediction box includes: determining a first Euclidean distance between the center point of the target detection box and the center points of each target prediction box; when the first Euclidean distance is not greater than a preset distance threshold, determining the ratio of the first Euclidean distance to a second Euclidean distance as the target distance cost value, where the second Euclidean distance is determined from the first Euclidean distance; when the first Euclidean distance is greater than the preset distance threshold, determining the target preset value as the target distance cost value, where the ratio of the first Euclidean distance to the second Euclidean distance is not greater than the target preset value.

[0015] In an exemplary embodiment, after determining the first Euclidean distance between the center point of the target detection box and the center points of each target prediction box, the method further includes: determining the first Euclidean distances less than the preset distance threshold among the first Euclidean distances as the third Euclidean distance; determining the largest third Euclidean distance among the third Euclidean distances as the second Euclidean distance.

[0016] In an exemplary embodiment, determining the target direction cost value according to the motion direction corresponding to the target detection box and the motion direction corresponding to the target prediction box includes: determining a first motion vector according to the historical trajectory box and the target detection box in the historical trajectory, where the historical trajectory box is used to represent the trajectory position of the target object in the second image frame; determining a second motion vector according to the historical trajectory box and the target prediction box; and determining the target direction cost value according to the first motion vector and the second motion vector.

[0017] In an exemplary embodiment, determining the target direction cost value according to the first motion vector and the second motion vector includes: performing normalization processing on the first motion vector and the second motion vector, where the lengths of the normalized first motion vector and the second motion vector are the same; determining the angular function value of the normalized first motion vector and the second motion vector, where the angular function value is used to represent the direction similarity between the first motion vector and the second motion vector; and determining the target direction cost value according to the angular function value.

[0018] In an exemplary embodiment, the trajectory update method includes: a first update method and a second update method; the target detection box includes: a first detection box and a second detection box, where the first detection box is a target detection box with a confidence level greater than a preset confidence threshold, and the second detection box is a target detection box with a confidence level not greater than the preset confidence threshold; updating the historical trajectory according to the trajectory update method to obtain the target trajectory includes: matching the first detection box with each historical trajectory according to the target association parameter; in the case where the target association parameter corresponding to the first detection box is greater than the preset parameter threshold, determining that the first detection box matches the historical trajectory corresponding to the target association parameter, and updating the historical trajectory according to the first update method to obtain the target trajectory; matching the second detection box with the historical trajectories that have not been successfully matched; and in the case where the target association parameter corresponding to the second detection box is greater than the preset parameter threshold, determining that the second detection box matches the historical trajectory corresponding to the target association parameter, and updating the historical trajectory according to the second update method to obtain the target trajectory.

[0019] In an exemplary embodiment, updating the historical trajectory according to the first update method to obtain the target trajectory includes: determining the target detection box as the target trajectory box, and adding the target trajectory box to the historical trajectory to obtain the target trajectory, where the target trajectory box is used to represent the trajectory position of the target object in the first image frame.

[0020] In an exemplary embodiment, updating the historical trajectory according to the second update method to obtain the target trajectory includes: determining the target trajectory box according to the target detection box and the target prediction box, and adding the target trajectory box to the historical trajectory to obtain the target trajectory.

[0021] In an exemplary embodiment, after determining that the second detection box matches the historical trajectory corresponding to the target association parameter, the method further includes: deleting the historical trajectory when the number of consecutive image frames in which the historical trajectory fails to match is greater than a preset frame number threshold; and establishing a new target trajectory based on the first detection box that fails to match.

[0022] In an exemplary embodiment, determining the target prediction box of the target object in the first image frame according to the historical trajectory corresponding to the second image frame includes: predicting an initial prediction box of the target object in the first image frame using a trajectory tracker according to the trajectory information corresponding to the historical trajectory, where the trajectory information includes at least one of the following: the position and speed of the target object; and updating and adjusting the initial prediction box according to the trajectory tracker to obtain the target prediction box.

[0023] According to another embodiment of the present application, there is provided a trajectory determination device, including: a detection box determination module, configured to determine a target detection box in a first image frame of a target video sequence and a confidence level corresponding to the target detection box, where the target detection box is used to frame a target object in the first image frame; a prediction box determination module, configured to determine a target prediction box of the target object in the first image frame according to the historical trajectory corresponding to a second image frame, where the second image frame is the previous image frame adjacent to the first image frame in the target video sequence, and the historical trajectory is used to represent the movement trajectory of the target object before the first image frame; a parameter determination module, configured to determine a target association parameter according to the target detection box and the target prediction box, where the target association parameter is determined by at least the overlap degree, displacement distance, and movement direction between the target detection box and the target prediction box, and the target association parameter is used to represent the similarity degree between the target detection box and the target prediction box; and a trajectory matching module, configured to determine a trajectory update method corresponding to the confidence level and update the historical trajectory according to the trajectory update method to obtain a target trajectory when the target association parameter is greater than a preset parameter threshold.

[0024] According to still another embodiment of the present application, there is further provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the steps in any of the above-described trajectory determination method embodiments when running.

[0025] According to still another embodiment of the present application, there is further provided an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above-described trajectory determination method embodiments.

[0026] Through this application, it is proposed to calculate the motion similarity between targets based on the three fusion information of IOU (Intersection over Union) value weighted by shape information, motion speed and direction, and center distance. At the same time, in the process of updating the tracker parameters, different target position points are selected for trajectory and parameter update based on different target detection confidence scores. Therefore, it can solve the problem of poor trajectory determination accuracy due to the large offset between target frames in the process of multi-target tracking in the related technology when the target moves too fast or the video frame rate is too low, thereby achieving the effect of improving the robustness of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a hardware structure block diagram of a computer device for implementing a trajectory determination method according to an embodiment of the present application;

[0028] Figure 2 1 is a flow chart of a trajectory determination method provided according to an embodiment of the present application;

[0029] Figure 3 is a schematic diagram of multi-target tracking provided according to an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of an object detection bounding box of the same object between adjacent frames provided according to an embodiment of the present application;

[0031] Figure 5 1 is a flow chart of a method for determining target association parameters according to an embodiment of the present application;

[0032] Figure 6 1 is a flow chart of a method for determining an overlap cost value provided in accordance with an embodiment of the present application;

[0033] Figure 7 This is a schematic diagram of a specific process of a multi-target tracking stage provided according to an embodiment of the present application;

[0034] Figure 8 is a schematic diagram of a process of multi-target tracking based on target detection provided according to an embodiment of the present application;

[0035] Figure 9a This is a schematic diagram of the effect of applying a trajectory determination method for multi-target tracking according to an embodiment of the present application;

[0036] Figure 9b This is a schematic diagram showing the effect of multi-target tracking using another trajectory determination method provided in an embodiment of the present application;

[0037] Figure 10It is a schematic structural diagram of a trajectory determination device provided according to an embodiment of the present application. Detailed implementation manners

[0038] In the following, embodiments of the present application will be described in detail with reference to the accompanying drawings and in conjunction with the embodiments.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence.

[0040] In the field of multi-object tracking, the two currently commonly used tracking modes are: tracking-by-detection (TBD) and joint-detection-and-tracking (JDT). TBD means that in each frame of the video, first, the target object is detected by a target detector, and then the detected target object is associated with the target object in the previous frame to achieve the purpose of tracking. When using this method, it is necessary to focus on the performance of the detector and the data association of the target (target feature representation), while the JDT method is an end-to-end trainable tracking mode, which completes tracking by jointly learning detection and appearance features.

[0041] With the excellent performance brought by deep learning to object detection algorithms, the detection-based tracking method (TBD) has become the mainstream tracking framework. Among them, some algorithms, for example, the byte-track tracking method, adopt a kalman (Kalman) tracking mode centered on estimation in terms of tracking trajectory update. In addition, since the current object detection algorithms have made great progress, in terms of object localization accuracy, especially in the case of no occlusion, the localization accuracy is quite accurate. Therefore, there are also some algorithms, for example, the oc-sort algorithm, which completely builds a tracker centered on observation (fully trusting the detection and localization results). However, if a tracker is completely built centered on observation, when the target object is occluded or affected by light, at this time, due to the influence of the detection ability of the target detector, there may be inaccurate localization and relatively large tracking noise may be introduced.

[0042] In terms of tracking matching, data association is mainly based on the result of motion similarity. Here, the calculation of motion similarity is mainly obtained by calculating the IOU between the detection target bounding box in the current frame and the kalman filter prediction bounding box. For example, algorithms such as the sort algorithm and the byte-track algorithm all adopt this motion similarity calculation model. Therefore, when the tracking target moves too fast between adjacent video frames, or the frame rate of the collected video is too low, it is easy for the tracking to fail due to the lack of overlapping area in the IOU. Figure 3Shows an example of target tracking failure occurring between consecutive video frames. For example, in the first frame, there are two targets, target 1 and target 2. In the second frame, due to the excessive movement speed of the targets, the predicted target bounding box in the second frame and the actual detected target bounding box have no IOU overlapping area, resulting in tracking failure. In the third frame, target 1 is associated with target 2 because there is an IOU overlapping area with the lost target 2 (in the retained state), which leads to an ID swap, while the actual target 2 remains in the lost state.

[0043] Analysis Figure 3 The direct reasons for tracking loss and ID swap in [analysis] are that a single IOU metric is used to measure motion similarity. When the video frame rate is too low or the target movement speed is too large, it causes a large displacement of the target between two consecutive frames, resulting in tracking failure.

[0044] In addition, during the tracking process, as Figure 4 shown, due to the deformation of the target, the detected bounding box of the same target changes significantly between adjacent frames, resulting in a large error in the IOU calculation result between two consecutive frames. Therefore, the single IOU based on the detected bounding box cannot well reflect the displacement change between two consecutive frames.

[0045] To solve the above problems, the embodiments of the present application provide corresponding solutions, which will be described in detail below.

[0046] The method embodiments provided in the embodiments of the present application can be executed on a mobile terminal, a computer device, or a similar computing device. Taking running on a computer device as an example, Figure 1 is a hardware structure block diagram of a computer device for a trajectory determination method according to an embodiment of the present application. As Figure 1 shown, the computer device may include one or more ( Figure 1 only one is shown in [figure]) processors 102 (the processors 102 may include, but are not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Among them, the above computer device may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 the structure shown in [figure] is only schematic and does not limit the structure of the above computer device. For example, the computer device may further include more or fewer components than Figure 1 shown in [figure], or have a different configuration from Figure 1 shown in [figure].

[0047] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the trajectory determination method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories can be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, intranet, local area network, mobile communication network, and combinations thereof.

[0048] The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the computer device. In one instance, the transmission device 106 includes a network adapter (Network Interface Controller, abbreviated as NIC), which can be connected to other network devices through a base station and thus communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0049] In this embodiment, a trajectory determination method running on the above computer device is provided. Figure 2 It is a schematic flowchart of a trajectory determination method provided according to an embodiment of the present application, as Figure 2 shown, and the process includes the following steps:

[0050] Step S202, determine the target detection box in the first image frame of the target video sequence and the confidence level corresponding to the target detection box, where the target detection box is used to frame the target object in the first image frame;

[0051] Specifically, in the target detection stage, when a new video image is input, the target detector is used to perform target detection on the first image frame (current frame), determine the target detection box in the first image frame, and classify the detected targets into two categories according to the confidence score of the target detection box. One category is the detection targets with high confidence (high score) (i.e., the first detection box), and the other category is the detection targets with low confidence (low score) (i.e., the second detection box).

[0052] Step S204: Determine the target prediction box of the target object in the first image frame according to the historical trajectory corresponding to the second image frame, where the second image frame is the previous image frame adjacent to the first image frame in the target video sequence, and the historical trajectory is used to characterize the movement trajectory of the target object before the first image frame;

[0053] In some embodiments of the present application, determining the target prediction box of the target object in the first image frame according to the historical trajectory corresponding to the second image frame includes the following steps: Predict the initial prediction box of the target object in the first image frame using a trajectory tracker according to the trajectory information corresponding to the historical trajectory, where the trajectory information includes at least one of the following: the position and speed of the target object; Update and adjust the initial prediction box according to the trajectory tracker to obtain the target prediction box.

[0054] In this embodiment, the above trajectory tracker may be a Kalman filter.

[0055] Specifically, in the tracking trajectory prediction link, based on Kalman filtering, according to the historical tracking trajectory (i.e., the above historical trajectory), predict the position of each trajectory in the current frame (i.e., the above first image frame), and adjust the predicted position according to the observation situation of the first image frame to obtain the target prediction box. At the same time, feature extraction is performed according to the prediction result for the trajectory matching stage.

[0056] Step S206: Determine the target association parameter according to the target detection box and the target prediction box, where the target association parameter is at least determined by the overlap degree, displacement distance, and movement direction between the target detection box and the target prediction box, and the target association parameter is used to characterize the similarity between the target detection box and the target prediction box;

[0057] In the technical solution provided in step S206, the above target association parameter is the parameter value of the motion cost matrix cost(det,track).

[0058] Figure 5 It is a schematic flowchart of a method for determining a target association parameter according to an embodiment of the present application. As Figure 5 shown, determining the target association parameter according to the target detection box and the target prediction box includes the following steps:

[0059] Step S502: Determine the target overlap degree cost value according to the overlap degree between the target detection box and the target prediction box;

[0060] The above target overlap degree cost value is cost iou (det,track), which is used to represent the IoU distance between the target detection box and the target prediction box;

[0061] The method for determining the target overlap degree cost value will be further introduced below.

[0062] Figure 6 It is a schematic flowchart of a method for determining the overlap degree cost value provided by an embodiment of the present application. As Figure 6 shown, based on the overlap degree between the target detection box and the target prediction box, the target overlap degree cost value cost iou (det, track) includes the following steps:

[0063] Step S602: Adjust the frame sizes of the target detection box and the target prediction box. Among them, the difference in the frame sizes between the adjusted target detection box and the target prediction box is not greater than the difference in the frame sizes between the unadjusted target detection box and the target prediction box.

[0064] The above frame sizes include: frame height and frame width.

[0065] In some embodiments of the present application, adjusting the frame sizes of the target detection box and the target prediction box includes the following steps: determining a first change rate based on the frame heights of the target detection box and the target prediction box, where the first change rate is used to characterize the degree of difference between the frame height of the target detection box and the frame height of the target prediction box; determining a second change rate based on the frame widths of the target detection box and the target prediction box, where the second change rate is used to characterize the degree of difference between the frame width of the target detection box and the frame width of the target prediction box; adjusting the frame height or frame width of the target detection box and the target prediction box based on the first change rate and the second change rate.

[0066] In some embodiments of the present application, adjusting the frame height or frame width of the target detection box and the target prediction box based on the first change rate and the second change rate includes the following steps: when the first change rate is greater than a first preset threshold and the second change rate is less than a second preset threshold, adjusting the frame height of the target detection box and / or the target prediction box, where the first change rate corresponding to the adjusted target detection box and the target prediction box is less than the first preset threshold; when the second change rate is greater than the first preset threshold and the first change rate is less than the second preset threshold, adjusting the frame width of the target detection box and / or the target prediction box, where the second change rate corresponding to the adjusted target detection box and the target prediction box is less than the first preset threshold.

[0067] Specifically, the bounding box is adaptively adjusted according to the shape similarity between the detection target (equivalent to the above-mentioned target detection box) and the tracking target (equivalent to the above-mentioned target prediction box). For example, taking a person as an example, during the tracking process, due to the posture changes of pedestrians (such as striding, retracting the legs, sitting, squatting), the bounding boxes between two consecutive frames often vary greatly. Therefore, before calculating the IOU, the bounding box will be adaptively adjusted according to the actual situation, and the calculation process is shown in the following formula.

[0068]

[0069] Among them, r h ,r w respectively represent the height-width change rates between the target detection box and the target prediction box (i.e., the above-mentioned first change rate and the above-mentioned second change rate), h d ,w d represent the bounding box height and bounding box width of the detection target detection box, h t ,w t represent the bounding box height and bounding box width of the target prediction box. When r h (i.e., the above-mentioned first change rate) is greater than a certain threshold or r w (i.e., the above-mentioned second change rate) is greater than a certain threshold, the bounding box will be adjusted.

[0070] According to prior experience, during the movement of the same pedestrian target, r h ,r w rarely change greatly at the same time. Therefore, in actual use, when r h is greater than a certain threshold (i.e., the above-mentioned first preset threshold) and r w is close to 0 (equivalent to less than the above-mentioned second preset threshold), the height of the bounding box will be adjusted based on the width (w) (corresponding to the sudden squatting, occlusion, etc. of pedestrians); when r w is greater than a certain threshold (i.e., the above-mentioned first preset threshold) and r h is close to 0 (equivalent to less than the above-mentioned second preset threshold), the width of the bounding box will be adjusted based on the height (corresponding to actions such as striding and retracting the legs of pedestrians); when r w and r h are both greater than a certain threshold, it is highly likely that the tracking bounding box and the detection bounding box are not the same target, so no bounding box adjustment will be made.

[0071] Step S604, determine the overlap degree between the adjusted target detection box and the target prediction box;

[0072] In some embodiments of the present application, determining the overlap degree between the adjusted target detection box and the target prediction box includes the following steps: determining the first region selected by the adjusted target detection box in the first image frame, and the second region selected by the adjusted target prediction box in the first image frame; determining the first area value of the intersection region between the first region and the second region, and the second area value of the union region between the first region and the second region; determining the ratio of the first area value to the second area value as the overlap degree.

[0073] The above overlap degree is iou(det ′ ,track ′ ), which represents the iou value between the adjusted target detection box and the adjusted target prediction box, and the calculation formula is as shown in the following formula.

[0074]

[0075] Among them, rect() represents the region selected by the rectangular bounding box, rect(det ′ ) is the above-mentioned first region, and rect(track′) is the above-mentioned second region.

[0076] Step S606, determining the area similarity between the adjusted target detection box and the target prediction box;

[0077] In some embodiments of the present application, determining the area similarity between the adjusted target detection box and the target prediction box includes the following steps: determining the third area value of the first region and the fourth area value of the second region; calculating the absolute value of the difference between the third area value and the fourth area value, and the larger area value among the third area value and the fourth area value; determining the area similarity based on the absolute value of the difference and the larger area value.

[0078] The above area similarity is shape(det ′ ,track ′ ), which represents the area similarity between the target detection box and the target prediction box, and is specifically as shown in the following formula:

[0079]

[0080] Among them, size() represents the area size of the rectangular bounding box, generally calculated as size() = h*w, where h is the height of the bounding box and w is the width of the bounding box.

[0081] Specifically, the greater the area difference, the smaller the similarity, and the smaller the area difference, the greater the similarity.

[0082] Step S608, determining the target overlap degree cost value according to the overlap degree and the area similarity.

[0083] The target overlap degree cost value is cost iou (det, track)'s calculation formula is as follows:

[0084]

[0085] Among them, α is an adjustable parameter value, which can be determined according to actual needs. For example, α = 2.

[0086] When the area difference is smaller, cost iou (det, track) depends more on the iou value. When the area difference is larger, or when there is no overlap in iou, cost iou (det, track) depends more on the area similarity shape value.

[0087] In the target overlap degree cost value calculation of the solution of this application, it replaces the conventional IOU calculation method. First, it adaptively adjusts the size of the target bounding box, and then calculates the target overlap degree cost value based on the adaptively adjusted bounding box. In the target overlap degree cost value calculation, in order to prevent the IOU value from being 0 due to too fast movement speed, the present invention introduces the shape similarity (i.e., the above-mentioned area similarity) ratio to perform weighted calculation on the IOU overlapping area.

[0088] Step S504, determine the target distance cost value according to the displacement distance between the target detection box and the target prediction box;

[0089] The above target distance cost value is cost dis (det, track), which represents the central displacement distance between the target detection box and the target prediction box.

[0090] In some embodiments of this application, determining the target distance cost value according to the displacement distance between the target detection box and the target prediction box includes the following steps: determining the first Euclidean distance between the center point of the target detection box and the center points of each target prediction box; when the first Euclidean distance is not greater than the preset distance threshold, determining the ratio of the first Euclidean distance to the second Euclidean distance as the target distance cost value, where the second Euclidean distance is determined by the first Euclidean distance; when the first Euclidean distance is greater than the preset distance threshold, determining the target preset value as the target distance cost value, where the ratio of the first Euclidean distance to the second Euclidean distance is not greater than the target preset value.

[0091] In some embodiments of this application, after determining the first Euclidean distance between the center point of the target detection box and the center points of each target prediction box, the method further includes the following steps: determining the first Euclidean distances less than the preset distance threshold in the first Euclidean distances as the third Euclidean distances; determining the largest third Euclidean distance among the third Euclidean distances as the second Euclidean distance.

[0092] In the related art, when calculating the displacement distance between a tracking target and a detection target, generally, the center point of the tracking target (equivalent to the above-mentioned target prediction box) is used as a reference point, and the displacement distances between the center points of all detection targets (such as n detection targets) (equivalent to the above-mentioned target detection boxes) and this point are calculated to generate a set of displacement distance values with a length of n. Then, based on the maximum value in this set of distance values, the displacement distance values are normalized. This global normalization method is easily affected by the farthest displacement distance, weakening the cost dis (det, track) in the effectiveness of motion association calculation.

[0093] However, in the actual matching process, the tracking target can only be matched with the detection targets within a local range. Therefore, the present application proposes a displacement distance calculation method based on distance constraint, which is specifically shown in the following formula:

[0094]

[0095] Among them, dis(det, track) represents the Euclidean distance between the centers of n detection targets and the tracking center point, and is an array of displacement distances with a length of n. When the Euclidean distance between the detection point and the tracking point is less than the thre value (i.e., the above-mentioned preset distance threshold), it is marked as a valid displacement distance, and the displacement cost value is calculated by where m is less than or equal to n, indicating that there are m Euclidean distances (i.e., the above-mentioned first Euclidean distance) that meet the requirement of being less than thre, and max(dis(det, track) i＝1…m )(i.e., the above-mentioned second Euclidean distance) represents the maximum value in this array (the maximum value of m values). When the Euclidean distance between the detection point and the tracking point is greater than the thre value, it is marked as an invalid displacement distance, and the displacement cost value is set to the maximum cost value 1 (i.e., the above-mentioned target preset value).

[0096] The displacement cost value calculation method based on distance constraint can not only effectively remove the influence of global distance, but also can maximize the distinction of the displacement difference values between each target detection point and the tracking point within a local range. The setting of thre can be based on the size of the tracking target bounding box (i.e., the target prediction box), for example, the effective range is 5 times the size of the tracking target bounding box, or it can be set according to the predicted speed of the tracking target. The specific situation can be obtained through actual project tests.

[0097] The solution of the present application proposes a central displacement distance calculation method based on distance constraint in the calculation of the target distance cost value. This method can effectively reduce the introduction of errors and improve the distinguishability of the central displacement distance.

[0098] Step S506: Determine the target direction cost value based on the motion direction corresponding to the target detection box and the motion direction corresponding to the target prediction box.

[0099] The above target direction cost value is cost v (det, track) represents the correlation between the motion direction of the detected target (target detection box) and the motion direction of the target trajectory (target prediction box).

[0100] In some embodiments of the present application, determining the target direction cost value based on the motion direction corresponding to the target detection box and the motion direction corresponding to the target prediction box includes the following steps: Determine the first motion vector based on the historical trajectory box in the historical trajectory and the target detection box, where the historical trajectory box is used to represent the trajectory position of the target object in the second image frame; Determine the second motion vector based on the historical trajectory box and the target prediction box; Determine the target direction cost value based on the first motion vector and the second motion vector.

[0101] In some embodiments of the present application, determining the target direction cost value based on the first motion vector and the second motion vector includes the following steps: Perform normalization processing on the first motion vector and the second motion vector, where the lengths of the normalized first motion vector and the second motion vector are the same; Determine the angular function value of the normalized first motion vector and the second motion vector, where the angular function value is used to represent the direction similarity between the first motion vector and the second motion vector; Determine the target direction cost value based on the angular function value.

[0102] The target direction cost value cost v (det, track) represents the cost value of the correlation between the detected target and the motion direction of the historical tracking trajectory. The greater the correlation (i.e., the more consistent the motion direction), the smaller the cost v (det, track) cost value, and vice versa, the greater the cost value.

[0103] Assume that the current is the t-th frame (the first image frame), v track-t (i.e., the above second motion vector) represents the motion vector of the historical tracking trajectory at the t-th frame, obtained through kalman estimation; v det-t (i.e., the above first motion vector) represents the motion vector of the detected target at the t-th frame, calculated through the detected target at the t-th frame and the tracked target at the (t - 1)-th frame (the second image frame), and then the motion direction similarity between the two is calculated through the cosine angle formula.

[0104] Step S508: Determine the target association parameter based on the target overlap cost value, the target distance cost value, and the target direction cost value.

[0105] In some embodiments of the present application, determining the target association parameter based on the target overlap cost value, the target distance cost value, and the target direction cost value includes the following steps: determining a first weight value corresponding to the target overlap cost value; determining a second weight value corresponding to the target distance cost value; determining a third weight value corresponding to the target direction cost value; calculating based on the target overlap cost value, the first weight value, the target distance cost value, the second weight value, the target direction cost value, and the third weight value to obtain the target association parameter.

[0106] Specifically, the calculation formula for the above target association parameter is as follows:

[0107] cost(det,track)=λ1cost iou (det,track)+λ2cost dis (det,track)

[0108] +λ3cost v (det,track)

[0109] Wherein, λ1, λ2, and λ3 respectively represent the above first weight value, second weight value, and third weight value, and can be calculated based on prior knowledge.

[0110] The solution of the present application integrates the IOU cost value (i.e., the above target overlap cost value), the center displacement cost value (i.e., the above target distance cost value), and the motion speed correlation cost value (i.e., the above target direction cost value) through weight weighting, and calculates the motion similarity (i.e., the above target association parameter) between targets to improve the robustness of multi-target tracking. cost(det,track) represents the similarity cost value between the target detection box and the target prediction box. The larger this value, the smaller the similarity between the two, and vice versa.

[0111] Step S208, when the target association parameter is greater than the preset parameter threshold, determine the trajectory update method corresponding to the confidence level, and update the historical trajectory according to the trajectory update method to obtain the target trajectory.

[0112] The above trajectory update methods include: a first update method and a second update method; the target detection boxes include: a first detection box and a second detection box, wherein the first detection box is a target detection box with a confidence level greater than the preset confidence threshold, and the second detection box is a target detection box with a confidence level not greater than the preset confidence threshold;

[0113] In some embodiments of the present application, updating the historical trajectory according to the trajectory update method to obtain the target trajectory includes the following steps: matching the first detection box with each historical trajectory according to the target association parameter; when the target association parameter corresponding to the first detection box is greater than the preset parameter threshold, determining that the first detection box successfully matches the historical trajectory corresponding to the target association parameter, and updating the historical trajectory according to the first update method to obtain the target trajectory; matching the second detection box with the historical trajectories that have not been successfully matched; when the target association parameter corresponding to the second detection box is greater than the preset parameter threshold, determining that the second detection box successfully matches the historical trajectory corresponding to the target association parameter, and updating the historical trajectory according to the second update method to obtain the target trajectory.

[0114] Specifically, as Figure 7 shown, in the tracking trajectory matching stage, according to the confidence score of the detected target, first use the detection target with a high confidence score (i.e., the above-mentioned first detection box) to perform multi-target matching calculation with the predicted tracking trajectory (i.e., the above-mentioned target prediction box). By calculating the motion cost matrix (i.e., the above-mentioned target association parameter), the Hungarian matching method is used for trajectory matching. Then, according to the matching result, for the successfully tracked trajectories, use the detection result to update the tracking trajectory, and perform tracking parameter update processing centered on the observation (i.e., update the historical trajectory according to the first update method above).

[0115] In some embodiments of the present application, updating the historical trajectory according to the first update method to obtain the target trajectory includes the following steps: determining the target detection box as the target trajectory box, and adding the target trajectory box to the historical trajectory to obtain the target trajectory, where the target trajectory box is used to represent the trajectory position of the target object in the first image frame.

[0116] Specifically, during the trajectory update process, when the tracking trajectory successfully matches the detection bounding box with a high confidence score (i.e., the above-mentioned first detection box), trust the detection result, directly use the target detection box as the true position of the tracking trajectory (historical trajectory) in the current frame (equivalent to the above-mentioned target trajectory box), and update the tracker parameters;

[0117] For the trajectories that fail to be tracked (i.e., the historical trajectories that have not been successfully matched above), multi-target tracking matching will be performed using the target detection box with a low confidence score (i.e., the above-mentioned second detection box). The same as the multi-target tracking matching process with a high confidence score, feature extraction, calculation of the motion cost function, and trajectory matching operations still need to be performed. According to the matching conditions, for the successfully tracked trajectories, because the confidence of the detection box is low, the estimated center of the target will be used for trajectory update and tracking parameter update (i.e., update the historical trajectory according to the second update method above).

[0118] In some embodiments of the present application, updating the historical trajectory according to the second update method to obtain the target trajectory includes the following steps: determining the target trajectory box based on the target detection box and the target prediction box, and adding the target trajectory box to the historical trajectory to obtain the target trajectory.

[0119] Specifically, when the tracking trajectory successfully matches the detection bounding box with a low confidence score (i.e., the above-mentioned second detection box), the credibility of the detection result at this time is not high. Then, the target trajectory box estimated by the target prediction box and the target detection box of the tracker is still used for trajectory update, and the tracker parameters are updated simultaneously.

[0120] In some embodiments of the present application, after determining that the second detection box successfully matches the historical trajectory corresponding to the target association parameter, the method further includes the following steps: deleting the historical trajectory when the number of consecutive unmatched image frames in the historical trajectory is greater than the preset frame number threshold; and establishing a new target trajectory based on the unmatched first detection box.

[0121] Specifically, for the unmatched tracking trajectory (historical trajectory), the trajectory is temporarily placed in the cache. If the tracking is successful next time, the trajectory is reactivated; if it has not been activated within a certain time range, the network is deleted. For example, when the target loss time is greater than 30 frames (i.e., the above-mentioned preset frame number threshold), it is considered that the target is in a lost state and the trajectory is deleted. For the unmatched target detection box, it is determined whether a new tracking trajectory needs to be established according to the level of its confidence score.

[0122] It should be noted that although appearance information is not used in the present application, it does not mean that the present application cannot use appearance information. In actual usage scenarios, the present application can be combined with any appearance information for multi-target trajectory association judgment calculation.

[0123] In addition, the solution of the present application can also be used in single-target tracking and can be combined with various other tracking methods.

[0124] Next, the trajectory determination method in steps S202 to S208 of the embodiments of the present application is further introduced.

[0125] Figure 8 It is a schematic diagram of the process of multi-target tracking based on target detection provided by the embodiments of the present application, as Figure 8 shown. The process includes the following steps:

[0126] Step 1, first input a video sequence, perform target detection on the basis of the first frame image using a target detector, and establish a tracking sequence according to the target detection result;

[0127] In this solution, the target detector is not limited to a specific detector, and a suitable target detector can be selected according to the tracking object and the requirements of the tracking device.

[0128] Step 2: Continuously input video sequence images (the second frame and subsequent images), perform target detection using the target detector, and then perform target association calculation based on the target detection results (i.e., the above-mentioned target detection boxes) and the video trajectory sequence of the previous frame (i.e., the above-mentioned historical trajectory) to complete the multi-target tracking task.

[0129] In the calculation of the multi-target tracking task, it mainly includes four parts of work: feature extraction, cost matrix (i.e., the above-mentioned target association parameters) calculation, multi-target trajectory association, and tracker parameter update.

[0130] First, based on the target detection results of the current frame (equivalent to the above-mentioned target detection boxes) and the historical trajectory information, feature extraction is performed. In the solution of this application, the features mainly include the IOU value, the center displacement value, and the motion direction correlation between the target detection box and the tracking trajectory box of the previous frame. Of course, it can also be the appearance feature. Among them, during the calculation of the IOU value, the target box will be adaptively adjusted according to the prior features of the target object to reduce the IOU calculation error; during the calculation of the center displacement, the center displacement distance value is obtained by using the center displacement calculation mode based on distance constraint.

[0131] Then, based on the extracted features, the cost matrix is calculated to calculate the cost distance (i.e., the above-mentioned target association parameters) between the targets detected in the current frame and each historical trajectory; and based on the calculated cost matrix, data association calculation is performed to complete the multi-target trajectory association, and common methods such as the Hungarian matching method are used.

[0132] Finally, according to the tracking results and the target detection box score values, the adaptive update calculation of the tracker parameters corresponding to the trajectories is performed. When the confidence score of the detected target is relatively high, the tracking update mode centered on observation (i.e., the above-mentioned first update method) will be selected; when the confidence score of the detected target is relatively low, the tracking update mode centered on estimation (i.e., the above-mentioned second update method) will be selected. In the solution of this application, by combining the confidence level of the detected target, the update strategy of the tracker is adjusted in real time to minimize the influence of tracking noise.

[0133] Step 4: Update the multi-target historical trajectory according to the target tracking results.

[0134] This application calculates the motion similarity through multiple features to complete the association judgment of multiple targets. During the calculation of motion similarity, in order to effectively prevent problems such as inaccurate bounding boxes caused by target pose changes or occlusions, too fast target motion speed, and too low video frame rate, this application proposes to calculate the motion similarity between targets based on three types of fusion information: the IOU value weighted by shape information, motion speed and direction, and center distance, so as to improve the robustness of multi-target tracking. At the same time, during the process of updating the tracker parameters, this application selects different target position points for trajectory and parameter update based on different target detection confidence scores, which can effectively prevent the introduction of errors during the tracking process and further improve the robustness of tracking. At the same time, this application only performs trajectory association based on motion information, with fast tracking speed and can effectively perform real-time tracking. The schematic diagram of the effect of multi-target tracking using the above trajectory determination method of this application is as Figure 9a 、 Figure 9b shown.

[0135] Among them, the execution subject of the above steps can be a server, a terminal, etc., but is not limited thereto.

[0136] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of this application.

[0137] In this embodiment, a trajectory determination device is also provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0138] Figure 10 is a schematic structural diagram of a trajectory determination device provided according to an embodiment of this application. As Figure 10 shown, this device includes:

[0139] A detection box determination module 10, configured to determine the target detection box in the first image frame of the target video sequence, and the confidence corresponding to the target detection box, where the target detection box is used to frame the target object in the first image frame;

[0140] A prediction box determination module 12, configured to determine a target prediction box of a target object in a first image frame according to a historical trajectory corresponding to a second image frame, where the second image frame is the previous image frame adjacent to the first image frame in a target video sequence, and the historical trajectory is used to characterize the motion trajectory of the target object before the first image frame;

[0141] In some embodiments of the present application, determining a target prediction box of a target object in a first image frame according to a historical trajectory corresponding to a second image frame includes: predicting an initial prediction box of the target object in the first image frame using a trajectory tracker according to trajectory information corresponding to the historical trajectory, where the trajectory information includes at least one of the following: the position and speed of the target object; and updating and adjusting the initial prediction box according to the trajectory tracker to obtain the target prediction box.

[0142] A parameter determination module 14, configured to determine a target association parameter according to a target detection box and a target prediction box, where the target association parameter is determined by at least an overlap degree, a displacement distance, and a motion direction between the target detection box and the target prediction box, and the target association parameter is used to characterize the similarity degree between the target detection box and the target prediction box;

[0143] In some embodiments of the present application, determining a target association parameter according to a target detection box and a target prediction box includes: determining a target overlap degree cost value according to the overlap degree between the target detection box and the target prediction box; determining a target distance cost value according to the displacement distance between the target detection box and the target prediction box; determining a target direction cost value according to the motion direction corresponding to the target detection box and the motion direction corresponding to the target prediction box; and determining the target association parameter according to the target overlap degree cost value, the target distance cost value, and the target direction cost value.

[0144] In some embodiments of the present application, determining a target association parameter according to a target overlap degree cost value, a target distance cost value, and a target direction cost value includes: determining a first weight value corresponding to the target overlap degree cost value; determining a second weight value corresponding to the target distance cost value; determining a third weight value corresponding to the target direction cost value; and calculating according to the target overlap degree cost value, the first weight value, the target distance cost value, the second weight value, the target direction cost value, and the third weight value to obtain the target association parameter.

[0145] In some embodiments of the present application, determining the target overlap degree cost value based on the overlap degree between the target detection box and the target prediction box includes: adjusting the border sizes of the target detection box and the target prediction box, wherein the difference in border sizes between the adjusted target detection box and the target prediction box is not greater than the difference in border sizes between the target detection box and the target prediction box before adjustment; determining the overlap degree between the adjusted target detection box and the target prediction box; determining the area similarity between the adjusted target detection box and the target prediction box; and determining the target overlap degree cost value based on the overlap degree and the area similarity.

[0146] In some embodiments of the present application, determining the overlap degree between the adjusted target detection box and the target prediction box includes: determining the first region selected by the adjusted target detection box in the first image frame, and the second region selected by the adjusted target prediction box in the first image frame; determining the first area value of the intersection region of the first region and the second region, and the second area value of the union region of the first region and the second region; and determining the ratio of the first area value to the second area value as the overlap degree.

[0147] In some embodiments of the present application, determining the area similarity between the adjusted target detection box and the target prediction box includes: determining the third area value of the first region, and the fourth area value of the second region; calculating the absolute value of the difference between the third area value and the fourth area value, and the larger area value of the third area value and the fourth area value; and determining the area similarity based on the absolute value of the difference and the larger area value.

[0148] In some embodiments of the present application, the border size includes: border height and border width; adjusting the border sizes of the target detection box and the target prediction box includes: determining a first change rate based on the border heights of the target detection box and the target prediction box, wherein the first change rate is used to characterize the degree of difference between the border height of the target detection box and the border height of the target prediction box; determining a second change rate based on the border widths of the target detection box and the target prediction box, wherein the second change rate is used to characterize the degree of difference between the border width of the target detection box and the border width of the target prediction box; and adjusting the border height or border width of the target detection box and the target prediction box based on the first change rate and the second change rate.

[0149] In some embodiments of the present application, adjusting the border height or border width of the target detection box and the target prediction box according to the first change rate and the second change rate includes: when the first change rate is greater than the first preset threshold and the second change rate is less than the second preset threshold, adjusting the border height of the target detection box and / or the target prediction box, wherein the first change rate corresponding to the adjusted target detection box and the target prediction box is less than the first preset threshold; when the second change rate is greater than the first preset threshold and the first change rate is less than the second preset threshold, adjusting the border width of the target detection box and / or the target prediction box, wherein the second change rate corresponding to the adjusted target detection box and the target prediction box is less than the first preset threshold.

[0150] In some embodiments of the present application, determining the target distance cost value according to the displacement distance between the target detection box and the target prediction box includes: determining the first Euclidean distance between the center point of the target detection box and the center points of each target prediction box; when the first Euclidean distance is not greater than the preset distance threshold, determining the ratio of the first Euclidean distance to the second Euclidean distance as the target distance cost value, wherein the second Euclidean distance is determined by the first Euclidean distance; when the first Euclidean distance is greater than the preset distance threshold, determining the target preset value as the target distance cost value, wherein the ratio of the first Euclidean distance to the second Euclidean distance is not greater than the target preset value.

[0151] In some embodiments of the present application, after determining the first Euclidean distance between the center point of the target detection box and the center points of each target prediction box, the parameter determination module 14 is further configured to: determine the first Euclidean distances less than the preset distance threshold in the first Euclidean distances as the third Euclidean distances; determine the largest third Euclidean distance among the third Euclidean distances as the second Euclidean distance.

[0152] In some embodiments of the present application, determining the target direction cost value according to the movement direction corresponding to the target detection box and the movement direction corresponding to the target prediction box includes: determining a first motion vector according to the historical trajectory box and the target detection box in the historical trajectory, wherein the historical trajectory box is used to represent the trajectory position of the target object in the second image frame; determining a second motion vector according to the historical trajectory box and the target prediction box; determining the target direction cost value according to the first motion vector and the second motion vector.

[0153] In some embodiments of the present application, determining the target direction cost value based on the first motion vector and the second motion vector includes: performing a normalization process on the first motion vector and the second motion vector, where the lengths of the normalized first motion vector and the second motion vector are the same; determining the angular function values of the normalized first motion vector and the second motion vector, where the angular function values are used to characterize the direction similarity between the first motion vector and the second motion vector; and determining the target direction cost value based on the angular function values.

[0154] The trajectory matching module 16 is configured to determine a trajectory update method corresponding to the confidence level and update the historical trajectory according to the trajectory update method to obtain the target trajectory when the target association parameter is greater than the preset parameter threshold.

[0155] In some embodiments of the present application, the trajectory update methods include: a first update method and a second update method; the target detection frames include: a first detection frame and a second detection frame, where the first detection frame is a target detection frame with a confidence level greater than the preset confidence threshold, and the second detection frame is a target detection frame with a confidence level not greater than the preset confidence threshold; updating the historical trajectory according to the trajectory update method to obtain the target trajectory includes: matching the first detection frame with each historical trajectory according to the target association parameter; when the target association parameter corresponding to the first detection frame is greater than the preset parameter threshold, determining that the first detection frame and the historical trajectory corresponding to the target association parameter are successfully matched, and updating the historical trajectory according to the first update method to obtain the target trajectory; matching the second detection frame with the historical trajectories that have not been successfully matched; when the target association parameter corresponding to the second detection frame is greater than the preset parameter threshold, determining that the second detection frame and the historical trajectory corresponding to the target association parameter are successfully matched, and updating the historical trajectory according to the second update method to obtain the target trajectory.

[0156] In some embodiments of the present application, updating the historical trajectory according to the first update method to obtain the target trajectory includes: determining the target detection frame as the target trajectory frame and adding the target trajectory frame to the historical trajectory to obtain the target trajectory, where the target trajectory frame is used to characterize the trajectory position of the target object in the first image frame.

[0157] In some embodiments of the present application, updating the historical trajectory according to the second update method to obtain the target trajectory includes: determining the target trajectory frame based on the target detection frame and the target prediction frame, and adding the target trajectory frame to the historical trajectory to obtain the target trajectory.

[0158] In some embodiments of the present application, after it is determined that the second detection box matches the historical trajectory corresponding to the target association parameter, the trajectory matching module 16 is further configured to: delete the historical trajectory when the number of consecutive image frames in which the historical trajectory fails to match successfully is greater than a preset frame number threshold; and establish a new target trajectory based on the first detection box that fails to match successfully.

[0159] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited thereto: all the above-mentioned modules are located in the same processor; or, the above-mentioned various modules are separately located in different processors in any combination form.

[0160] Embodiments of the present application further provide a computer-readable storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any one of the above method embodiments for trajectory determination when running: determining a target detection box in a first image frame of a target video sequence and a confidence level corresponding to the target detection box, where the target detection box is used to frame a target object in the first image frame; determining a target prediction box of the target object in the first image frame according to the historical trajectory corresponding to a second image frame, where the second image frame is the previous image frame immediately adjacent to the first image frame in the target video sequence, and the historical trajectory is used to characterize the movement trajectory of the target object before the first image frame; determining a target association parameter according to the target detection box and the target prediction box, where the target association parameter is at least determined by the overlap degree, displacement distance, and movement direction between the target detection box and the target prediction box, and the target association parameter is used to characterize the similarity degree between the target detection box and the target prediction box; when the target association parameter is greater than a preset parameter threshold, determining a trajectory update method corresponding to the confidence level, and updating the historical trajectory according to the trajectory update method to obtain a target trajectory.

[0161] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as a USB flash drive, a read-only memory (ROM for short), a random access memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc that can store a computer program.

[0162] Embodiments of the present application further provide an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments for trajectory determination.

[0163] In an exemplary embodiment, the above electronic device may further include a transmission device and an input / output device. Wherein, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0164] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be elaborated herein again.

[0165] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to implement. In this way, the present application is not limited to any specific combination of hardware and software.

[0166] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the principle of the present application shall be included in the protection scope of the present application.

Claims

1. A trajectory determination method, characterized in that, Including: Determine the target detection box in the first image frame of the target video sequence and the confidence corresponding to the target detection box, where the target detection box is used to frame the target object in the first image frame; Determine the target prediction box of the target object in the first image frame according to the historical trajectory corresponding to the second image frame, where the second image frame is the previous image frame adjacent to the first image frame in the target video sequence, and the historical trajectory is used to represent the movement trajectory of the target object before the first image frame; Determine the target association parameter according to the target detection box and the target prediction box, where the target association parameter is determined by at least the overlap degree, displacement distance, and movement direction between the target detection box and the target prediction box, and the target association parameter is used to represent the similarity degree between the target detection box and the target prediction box; When the target association parameter is greater than the preset parameter threshold, determine the trajectory update method corresponding to the confidence, and update the historical trajectory according to the trajectory update method to obtain the target trajectory; Wherein, the trajectory update method includes: a first update method and a second update method; the target detection box includes: a first detection box and a second detection box, where the first detection box is a target detection box with a confidence greater than the preset confidence threshold, and the second detection box is a target detection box with a confidence not greater than the preset confidence threshold; updating the historical trajectory according to the trajectory update method to obtain the target trajectory includes: matching the first detection box with each historical trajectory according to the target association parameter; when the target association parameter corresponding to the first detection box is greater than the preset parameter threshold, determining that the first detection box and the historical trajectory corresponding to the target association parameter match successfully, and updating the historical trajectory according to the first update method to obtain the target trajectory; matching the second detection box with the historical trajectories that have not been successfully matched; when the target association parameter corresponding to the second detection box is greater than the preset parameter threshold, determining that the second detection box and the historical trajectory corresponding to the target association parameter match successfully, and updating the historical trajectory according to the second update method to obtain the target trajectory; Wherein, updating the historical trajectory according to the first update method to obtain the target trajectory includes: determining the target detection box as the target trajectory box, and adding the target trajectory box to the historical trajectory to obtain the target trajectory, where the target trajectory box is used to represent the trajectory position of the target object in the first image frame; Wherein, updating the historical trajectory according to the second update method to obtain the target trajectory includes: determining the target trajectory box according to the target detection box and the target prediction box, and adding the target trajectory box to the historical trajectory to obtain the target trajectory.

2. The method according to claim 1, characterized in that Determining the target association parameter according to the target detection box and the target prediction box includes: Determine the target overlap cost value according to the overlap degree between the target detection box and the target prediction box; Determine the target distance cost value according to the displacement distance between the target detection box and the target prediction box; Determine the target direction cost value according to the movement direction corresponding to the target detection box and the movement direction corresponding to the target prediction box; Determine the target association parameter according to the target overlap cost value, the target distance cost value, and the target direction cost value.

3. The method according to claim 2, characterized in that Determining the target association parameter based on the target overlap degree cost value, the target distance cost value, and the target direction cost value includes: Determining a first weight value corresponding to the target overlap degree cost value; Determining a second weight value corresponding to the target distance cost value; Determining a third weight value corresponding to the target direction cost value; Calculating according to the target overlap degree cost value, the first weight value, the target distance cost value, the second weight value, the target direction cost value, and the third weight value to obtain the target association parameter.

4. The method according to claim 2, wherein Determining the target overlap degree cost value according to the overlap degree between the target detection box and the target prediction box includes: Adjusting the border sizes of the target detection box and the target prediction box, wherein the difference in the border sizes between the adjusted target detection box and the target prediction box is not greater than the difference in the border sizes between the unadjusted target detection box and the target prediction box; Determining the overlap degree between the adjusted target detection box and the target prediction box; Determining the area similarity between the adjusted target detection box and the target prediction box; Determining the target overlap degree cost value according to the overlap degree and the area similarity.

5. The method according to claim 4, wherein Determining the overlap degree between the adjusted target detection box and the target prediction box includes: Determining a first region selected by the adjusted target detection box in the first image frame, and a second region selected by the adjusted target prediction box in the first image frame; Determining a first area value of the intersection region of the first region and the second region, and a second area value of the union region of the first region and the second region; Determining the ratio of the first area value to the second area value as the overlap degree.

6. The method according to claim 5, wherein Determining the area similarity between the adjusted target detection box and the target prediction box includes: Determining a third area value of the first region and a fourth area value of the second region; Calculating the absolute value of the difference between the third area value and the fourth area value, and the larger area value among the third area value and the fourth area value; Determining the area similarity according to the absolute value of the difference and the larger area value.

7. The method according to claim 4, wherein The border size includes: border height and border width; adjusting the border sizes of the target detection box and the target prediction box includes: Determining a first change rate according to the border heights of the target detection box and the target prediction box, where the first change rate is used to characterize the degree of difference between the border height of the target detection box and the border height of the target prediction box; Determining a second change rate according to the border widths of the target detection box and the target prediction box, where the second change rate is used to characterize the degree of difference between the border width of the target detection box and the border width of the target prediction box; Adjusting the border height or border width of the target detection box and the target prediction box according to the first change rate and the second change rate.

8. The method according to claim 7, wherein Adjusting the border height or border width of the target detection box and the target prediction box according to the first change rate and the second change rate includes: When the first change rate is greater than a first preset threshold and the second change rate is less than a second preset threshold, adjusting the border height of the target detection box and / or the target prediction box, where the first change rate corresponding to the adjusted target detection box and the target prediction box is less than the first preset threshold; When the second change rate is greater than a first preset threshold and the first change rate is less than a second preset threshold, adjusting the border width of the target detection box and / or the target prediction box, where the second change rate corresponding to the adjusted target detection box and the target prediction box is less than the first preset threshold.

9. The method according to claim 2, characterized in that, Determining a target distance cost value according to the displacement distance between the target detection box and the target prediction box includes: Determining a first Euclidean distance between the center point of the target detection box and the center points of each of the target prediction boxes; When the first Euclidean distance is not greater than a preset distance threshold, determining the ratio of the first Euclidean distance to a second Euclidean distance as the target distance cost value, where the second Euclidean distance is determined from the first Euclidean distance; When the first Euclidean distance is greater than the preset distance threshold, determining a target preset value as the target distance cost value, where the ratio of the first Euclidean distance to the second Euclidean distance is not greater than the target preset value.

10. The method according to claim 9, wherein After determining the first Euclidean distance between the center point of the target detection box and the center points of each of the target prediction boxes, the method further includes: Determining the first Euclidean distances less than the preset distance threshold among the first Euclidean distances as third Euclidean distances; Determining the largest of the third Euclidean distances as the second Euclidean distance.

11. The method according to claim 2, wherein Determining a target direction cost value according to the movement direction corresponding to the target detection box and the movement direction corresponding to the target prediction box includes: Determining a first motion vector according to a historical trajectory box in the historical trajectory and the target detection box, where the historical trajectory box is used to represent the trajectory position of the target object in the second image frame; Determining a second motion vector according to the historical trajectory box and the target prediction box; Determining the target direction cost value according to the first motion vector and the second motion vector.

12. The method according to claim 11, characterized in that Determining the target direction cost value according to the first motion vector and the second motion vector includes: Performing a normalization process on the first motion vector and the second motion vector, where the lengths of the first motion vector and the second motion vector after the normalization process are the same; Determining an angular function value of the first motion vector and the second motion vector after the normalization process, where the angular function value is used to represent the direction similarity between the first motion vector and the second motion vector; Determining the target direction cost value according to the angular function value.

13. The method according to claim 1, characterized in that, After determining that the second detection box matches the historical trajectory corresponding to the target association parameter, the method further includes: Deleting the historical trajectory when the number of consecutive image frames in which the historical trajectory fails to match successfully is greater than a preset frame number threshold; and Establishing a new target trajectory based on the first detection box that fails to match successfully.

14. The method according to claim 1, characterized in that, Determining the target prediction box of the target object in the first image frame based on the historical trajectory corresponding to the second image frame includes: Predicting an initial prediction box of the target object in the first image frame using a trajectory tracker according to the trajectory information corresponding to the historical trajectory, where the trajectory information includes at least one of the following: the position and speed of the target object; Updating and adjusting the initial prediction box according to the trajectory tracker to obtain the target prediction box.

15. A target tracking device, characterized in that, Including: A detection box determination module, configured to determine a target detection box in a first image frame of a target video sequence and the confidence corresponding to the target detection box, where the target detection box is used to frame a target object in the first image frame; A prediction box determination module, configured to determine a target prediction box of the target object in the first image frame according to the historical trajectory corresponding to a second image frame, where the second image frame is the previous image frame adjacent to the first image frame in the target video sequence, and the historical trajectory is used to represent the motion trajectory of the target object before the first image frame; A parameter determination module, configured to determine a target association parameter according to the target detection box and the target prediction box, where the target association parameter is determined by at least the overlap degree, displacement distance, and motion direction between the target detection box and the target prediction box, and the target association parameter is used to represent the similarity degree between the target detection box and the target prediction box; A trajectory matching module, configured to determine a trajectory update method corresponding to the confidence when the target association parameter is greater than a preset parameter threshold, and update the historical trajectory according to the trajectory update method to obtain a target trajectory; Wherein, the trajectory update method includes: a first update method and a second update method; the target detection box includes: a first detection box and a second detection box, where the first detection box is a target detection box with a confidence greater than a preset confidence threshold, and the second detection box is a target detection box with a confidence not greater than the preset confidence threshold; the trajectory matching module is further configured to: match the first detection box with each historical trajectory according to the target association parameter; when the target association parameter corresponding to the first detection box is greater than the preset parameter threshold, determine that the first detection box matches the historical trajectory corresponding to the target association parameter successfully, update the historical trajectory according to the first update method to obtain a target trajectory; match the second detection box with the historical trajectories that fail to match successfully; when the target association parameter corresponding to the second detection box is greater than the preset parameter threshold, determine that the second detection box matches the historical trajectory corresponding to the target association parameter successfully, and update the historical trajectory according to the second update method to obtain a target trajectory; Among them, the trajectory matching module is further configured to: determine the target detection box as the target trajectory box, and add the target trajectory box to the historical trajectory to obtain the target trajectory, where the target trajectory box is used to represent the trajectory position of the target object in the first image frame; Among them, the trajectory matching module is further configured to: determine the target trajectory box according to the target detection box and the target prediction box, and add the target trajectory box to the historical trajectory to obtain the target trajectory.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the trajectory determination method described in any one of claims 1 to 14 are implemented.

17. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the trajectory determination method described in any one of claims 1 to 14 are implemented.

Images