Target tracking detection method and device

Abstract

The application discloses a target tracking detection method and device. The method comprises the following steps: acquiring multiple frames of images of a video, and determining a target frame set of each frame of image; for any frame of image, taking a first tracking frame set of a previous frame of image of the frame of image, determining the similarity between the first target frame set of the frame of image and each frame in the first tracking frame set, determining a first parameter according to the position offset of each group of frames corresponding to the similarity greater than a threshold, and compensating the position of all first tracking frames to obtain a second tracking frame set; determining the similarity between the first target frame set and each frame in the second tracking frame set by using a translation intersection ratio, determining a second parameter according to the position offset of each group of frames corresponding to the similarity greater than a threshold, and compensating the position of each second tracking frame to obtain a third tracking frame set; and tracking and matching the first target frame set and the third tracking frame set to obtain a tracking detection result. The application solves the technical problem that the performance of a target tracking detection algorithm in related technologies cannot meet the requirements in a camera continuous motion scene.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a target tracking and detection method and apparatus. Background Technology

[0002] Currently, the application of drones equipped with AI video analytics in target tracking is becoming increasingly widespread. For example, drones can be used to continuously monitor pedestrian and vehicle traffic in real time, or to analyze crowd trajectories. There are two common methods for setting up cameras on drones for target tracking: one is to send the drone camera video data to a remote server via a 5G network, but this method may suffer severe image quality degradation due to unstable 5G signals; the other is to mount a lightweight mobile computing terminal device with low computing power to directly connect to the camera and obtain the video stream. However, this method suffers from low computing efficiency due to the device's limited computing power, thus requiring frame-by-frame analysis to achieve real-time monitoring. However, because the time span between frames is large, and the drone continues to move during this period, the camera image will undergo translation, tilting, and rotation, which will bring some problems to target tracking.

[0003] Traditional motion compensation methods for target tracking typically use feature point matching to detect feature points in adjacent frames and obtain matching point pairs to derive the transformation matrix between the two frames for motion compensation. However, in real-world scenarios, video stream resolutions often exceed 1080p, and the speed of feature point extraction slows down as image size increases. Therefore, this method is unsuitable for applications with high real-time requirements.

[0004] Currently, mainstream tracking algorithms such as ByteTrack, DeepSort, and Sort perform well in security scenarios with fixed camera positions. However, in real-world scenarios such as drones, vehicles, and ships, when the camera position and angle change, even if the object itself is stationary, camera shake and offset will cause changes in the object's position and angle within the frame. Because the movement of both the camera and the target overlaps, the target appears to move too quickly in the frame, causing mainstream tracking algorithms to frequently lose track of the original target, resulting in decreased tracking accuracy.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This application provides a target tracking and detection method and apparatus to at least solve the technical problem that the performance of target tracking and detection algorithms in related technologies cannot meet the requirements in scenes with continuous camera movement.

[0007] According to one aspect of the embodiments of this application, a target tracking and detection method is provided, comprising: acquiring multiple frames of images from a video to be analyzed, and determining a set of target detection boxes in each frame of images using a target detector; for any frame of a first image other than a first frame image, acquiring a set of first tracking detection boxes corresponding to the previous frame of the first image, and determining a first similarity between each first target detection box in the first target detection box set corresponding to the first image and each first tracking detection box in the first tracking detection box set, wherein if the second image is the first frame image, the first tracking detection box set is the set of second target detection boxes corresponding to the second image; and determining a first position offset between each set of first target detection boxes and first tracking detection boxes based on the first similarity greater than a first preset threshold. First compensation parameters are determined for all first tracking detection boxes, and position compensation is performed on all first tracking detection boxes based on the first compensation parameters to obtain a second tracking detection box set. A translation-intersection-union algorithm is used to determine the second similarity between each first target detection box and each second tracking detection box in the second tracking detection box set. Second compensation parameters are determined for each second tracking detection box based on the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to the second similarity greater than a second preset threshold, and position compensation is performed on each second tracking detection box based on the second compensation parameters to obtain a third tracking detection box set. A target tracker is used to track and match the first target detection box set and the third tracking detection box set to obtain a fourth tracking detection box set corresponding to the first image.

[0008] Optionally, determining the first similarity between each first target detection box in the first target detection box set corresponding to the first image and each first tracking detection box in the first tracking detection box set includes: grouping each first target detection box in the first target detection box set and each first tracking detection box in the first tracking detection box set into pairs to obtain multiple sets of first detection box pairs; for each set of first detection box pairs, determining a first ratio of the smaller box length to the larger box length, a second ratio of the smaller box width to the larger box width, and a third ratio of the smaller box area to the larger box area between the first target detection box and the first tracking detection box in the first detection box pair; determining the first product of the first ratio, the second ratio, and the third ratio as the first similarity of the first detection box pair; and forming a first similarity set from the first similarities of each first detection box pair.

[0009] Optionally, determining the first compensation parameter for all first tracking detection boxes based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to a first similarity greater than a first preset threshold includes: forming a third similarity set from all first similarities in the first similarity set that are greater than the first preset threshold, and sorting the first similarities in the third similarity set from largest to smallest; determining the first position offset between the first target detection box and the first tracking detection box in the first detection box pair by traversing each first similarity pair in the third similarity set, performing position compensation on all first tracking detection boxes based on the first position offset to obtain a fifth tracking detection box set, determining the intersection-union ratio (IUU) between each first target detection box and each fifth tracking detection box in the fifth tracking detection box set, and determining the average of multiple IUU; and determining the first position offset between the first target detection box and the first tracking detection box in the group of first detection box pairs with the largest average as the first compensation parameter for all first tracking detection boxes.

[0010] Optionally, the second similarity between each first target detection box and each second tracking detection box in the second tracking detection box set is determined using the translation, intersection, union, and comparison algorithm. This includes: grouping each first target detection box in the first target detection box set and each second tracking detection box in the second tracking detection box set into pairs to obtain multiple sets of second detection box pairs; for each set of second detection box pairs, determining the fourth ratio between the intersection of the box lengths and the union of the box lengths of the first target detection box and the second tracking detection box in the second detection box pair, the fifth ratio between the intersection of the box widths and the union of the box widths, and the third ratio between the smaller box area and the larger box area; determining the second product of the fourth ratio and the third ratio, and the third product of the fifth ratio and the third ratio; determining the larger product of the second product and the third product as the second similarity of the second detection box pair; and forming a second similarity set from the second similarity of each second detection box pair.

[0011] Optionally, determining the second compensation parameter for each second tracking detection box based on the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to a second similarity greater than a second preset threshold includes: sorting the second similarities in the second similarity set from largest to smallest; sequentially traversing each second similarity in the second similarity set; if the second similarity is greater than the second preset threshold, using the second position offset between the first target detection box and the second tracking detection box in the second detection box pair corresponding to the second similarity as the second compensation parameter for the second tracking detection box, recording the second position offset in the offset set, and deleting the second similarities in the second similarity set that are associated with the first target detection box and the second tracking detection box; if the second similarity is not greater than the second preset threshold, stopping the traversal; after the traversal is completed, determining the average offset of all second position offsets in the offset set, and using the average offset as the second compensation parameter for all second tracking detection boxes whose second compensation parameters have not yet been determined.

[0012] Optionally, when there is no first similarity greater than a first preset threshold, or no second similarity greater than a second preset threshold, the target tracker is used to track and match the first target detection box set and the first tracking detection box set to obtain the fourth tracking detection box set corresponding to the first image.

[0013] Optionally, obtaining multiple frames of images from the video to be analyzed includes: extracting multiple frames of images from the video to be analyzed according to a preset frame interval.

[0014] According to another aspect of the embodiments of this application, a target tracking and detection apparatus is also provided, comprising: an acquisition module, configured to acquire multiple frames of images from a video to be analyzed, and determine a set of target detection boxes in each frame of images using a target detector; a first determination module, configured to, for any frame of a first image other than a first frame of images, acquire a set of first tracking detection boxes corresponding to the previous frame of a second image of the first image, and determine a first similarity between each first target detection box in the set of first target detection boxes corresponding to the first image and each first tracking detection box in the set of first tracking detection boxes, wherein if the second image is the first frame of images, the set of first tracking detection boxes is the set of second target detection boxes corresponding to the second image; and a first compensation module, configured to determine the first position offset between each set of first target detection boxes and first tracking detection boxes based on the first similarity greater than a first preset threshold. A first compensation parameter is defined for all first tracking detection boxes, and position compensation is performed on all first tracking detection boxes according to the first compensation parameter to obtain a second tracking detection box set; a second determination module is used to determine the second similarity between each first target detection box and each second tracking detection box in the second tracking detection box set using a translation-intersection-union algorithm; a second compensation module is used to determine the second compensation parameter of each second tracking detection box according to the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to the second similarity greater than a second preset threshold, and position compensation is performed on each second tracking detection box according to the second compensation parameter to obtain a third tracking detection box set; a tracking detection module is used to track and match the first target detection box set and the third tracking detection box set using a target tracker to obtain a fourth tracking detection box set corresponding to the first image.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described target tracking and detection method by running the computer program.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described target tracking and detection method through the computer program.

[0017] In this embodiment, a two-stage processing method of coarse compensation and fine compensation is adopted. First, the overall motion offset is roughly estimated, and then the motion offset of each target between consecutive frames is predicted independently. Then, the offset is applied to each target, and tracking and matching are performed after offset compensation. Without increasing the time consumption and complex computation, the motion compensation method can reduce the target offset in the picture caused by the camera's own movement, which can track the target more stably and improve the target tracking effect in complex scenes. It effectively solves the technical problem that the performance of target tracking and detection algorithms in related technologies cannot meet the requirements in scenes with continuous camera movement. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a schematic diagram of the structure of an optional computer terminal according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional target tracking and detection method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram comparing an optional target detection box in the current image with an uncompensated tracking detection box in the previous frame image, according to an embodiment of this application.

[0022] Figure 4 This is a schematic diagram comparing an optional target detection box in the current image with a tracking detection box in the previous frame image after one compensation, according to an embodiment of this application.

[0023] Figure 5 This is a schematic diagram comparing an optional target detection box in the current image with a tracking detection box in the previous frame image after secondary compensation, according to an embodiment of this application.

[0024] Figure 6 This is a schematic diagram of an optional target tracking and detection device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] To better understand the embodiments of this application, the following is a translation and explanation of some nouns or terms that appear in the description of the embodiments of this application:

[0028] Intersection Over Union (IOU) is a commonly used method for calculating the accuracy of object detection algorithms. It evaluates object detection accuracy by calculating the ratio between the intersection and union of the predicted and ground truth bounding boxes. Specifically, the IOU algorithm calculates the intersection area and union area of ​​the two boxes, and divides the intersection area by the union area to obtain the IOU value. The IOU value ranges from 0 to 1; a higher value indicates a larger overlap between the predicted and ground truth bounding boxes, resulting in more accurate detection.

[0029] The Shift Intersection Over Union (Shift IOU) algorithm is an improved version based on the IOU algorithm. It translates two bounding boxes by a certain distance and then calculates the intersection-over-union ratio (IOU) of the translated bounding boxes. By continuously trying different translation distances, the Shift IOU algorithm can find the maximum IOU, thus more accurately evaluating the overlap between bounding boxes.

[0030] Object detectors are computer vision techniques used to detect and identify the location and category of specific objects in images or videos. These objects can be people, cars, animals, furniture, etc. Object detectors typically use deep learning algorithms, such as convolutional neural networks, to extract features and classify the input image to achieve object detection.

[0031] Target tracker: A computer vision technology used to track the position and movement of a specific target in a continuous sequence of images or videos. It is widely used in video surveillance, autonomous driving, robot navigation, and other fields.

[0032] Example 1

[0033] According to the embodiments of this application, a target tracking and detection method is first provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0034] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a target tracking and detection method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the target tracking and detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the vulnerability detection method of the application described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0037] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0038] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0039] Under the above operating environment, embodiments of this application provide a target tracking and detection method, such as... Figure 2 As shown, the method includes at least the following steps:

[0040] Step S202: Obtain multiple frames of images from the video to be analyzed, and use the object detector to determine the set of object detection boxes in each frame of the image;

[0041] Step S204: For any first image other than the first frame image, obtain the first tracking detection box set corresponding to the previous second image of the first image, and determine the first similarity between each first target detection box in the first target detection box set corresponding to the first image and each first tracking detection box in the first tracking detection box set. If the second image is the first frame image, the first tracking detection box set is the second target detection box set corresponding to the second image.

[0042] Step S206: Determine the first compensation parameter of all first tracking detection boxes based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to the first similarity greater than the first preset threshold, and perform position compensation on all first tracking detection boxes based on the first compensation parameter to obtain the second tracking detection box set.

[0043] Step S208: Use the translation, intersection, union and comparison algorithm to determine the second similarity between each first target detection box and each second tracking detection box in the second tracking detection box set;

[0044] Step S210: Determine the second compensation parameter of each second tracking detection box based on the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to the second similarity greater than the second preset threshold, and perform position compensation on each second tracking detection box based on the second compensation parameter to obtain the third tracking detection box set.

[0045] Step S212: Use the target tracker to track and match the first target detection box set and the third tracking detection box set to obtain the fourth tracking detection box set corresponding to the first image.

[0046] The following describes each step of the target tracking and detection method of this application embodiment in conjunction with a specific implementation process.

[0047] Considering the limited computing power of the device, when acquiring images from the video to be analyzed, a video frame extraction technique can be used to extract multiple frames from the video according to a preset frame interval. For example, if the frame interval is set to 0.2 seconds, one frame is extracted from every five frames for object detection and tracking, and the images of the remaining video frames can be skipped. Then, an object detector (such as the ByteTrack tracker) can be used to detect the set of object detection boxes in each frame.

[0048] Taking vehicle tracking and detection as an example, the target detection box set mainly consists of the detection boxes of multiple vehicles in the image. The specific position coordinate format can be represented as (x1,x2,y1,y2), where (x1,y1) and (x2,y2) represent the coordinate positions of the detection box at the lower left and lower right corners of the image, respectively.

[0049] Figure 3This diagram illustrates a comparison between the first target detection bounding box (BVR) of the current frame's first image and the first tracking detection bounding box of the previous frame's second image (without compensation) in a vehicle tracking detection scenario. BVRs with "vehicle" in the bounding box represent the first target detection bounding box of the current frame (n²), while BVRs without "vehicle" represent the first target tracking detection bounding box of the previous frame (n¹). Tables 1 and 2 show the coordinates and dimensions of the first tracking and target detection bounding boxes, respectively. To simplify the tables, only four sets of data are shown here, which does not match the number of boxes in the image example. It can be seen that the drone is moving to the right, so the vehicles in the image are generally skewed to the left. Furthermore, the distance difference between the first tracking and target detection bounding boxes of many vehicles exceeds the length of a vehicle body, drastically increasing the difficulty of tracking and matching.

[0050] Table 1

[0051]

[0052] Table 2

[0053]

[0054] To address the aforementioned issues, this application proposes a motion compensation mechanism. First, the motion offset of the second image as a whole is roughly estimated, and then the first tracking detection box corresponding to the second image is initially compensated.

[0055] As an optional implementation, the first similarity between each first target detection box in the first target detection box set corresponding to the first image and each first tracking detection box in the first tracking detection box set corresponding to the second image can be determined firstly. Specifically, each first target detection box in the first target detection box set and each first tracking detection box in the first tracking detection box set are grouped into pairs to obtain multiple sets of first detection box pairs, specifically a total of n1×n2 sets.

[0056] Specifically, for each pair of first detection boxes, a first ratio is determined between the smaller and larger box lengths of the first target detection box and the first tracking detection box in that pair. Determine a second ratio between the smaller and larger bounding box widths of the first target detection box and the first tracking detection box in the first detection box pair. Determine a third ratio between the smaller and larger bounding box areas of the first target detection box and the first tracking detection box in the first detection box pair. Determine the first ratio w s The second ratio h s The third ratio a s The first product is the first similarity s = w of the first detection box pair. s ×hs ×a s Finally, the first similarity scores of each first detection box pair are combined to form a first similarity set.

[0057] Taking the data in Tables 1 and 2 as examples, the first similarity s between each first tracking detection box and the first target detection box can be calculated according to the above formula, forming the first similarity set S1. The specific data is shown in Table 3.

[0058] Table 3

[0059]

[0060] Subsequently, the first compensation parameter for all first tracking detection boxes can be determined based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to the first similarity greater than the first preset threshold. Specifically, this can be achieved as follows: All first similarities in the first similarity set greater than the first preset threshold are grouped into a third similarity set, and the first similarities in the third similarity set are sorted from largest to smallest; the first detection box pairs corresponding to each first similarity in the third similarity set are sequentially traversed, and the first position offset between the first target detection box and the first tracking detection box in the first detection box pair is determined; position compensation is performed on all first tracking detection boxes based on the first position offset to obtain a fifth tracking detection box set; the intersection-union ratio (IUU) between each first target detection box and each fifth tracking detection box in the fifth tracking detection box set is determined, and the average value of multiple IUUs is determined; the first position offset between the first target detection box and the first tracking detection box in the group of first detection box pairs with the largest average value is determined as the first compensation parameter for all first tracking detection boxes.

[0061] Taking a first preset threshold of 0.7 as an example, similarity data greater than 0.7 are retained from the first similarity set S1 in Table 3 to form the third similarity set S3, and the data in the third similarity set S3 are sorted from largest to smallest. The specific data in the sorted third similarity set S3 is shown in Table 4.

[0062] Table 4

[0063] The first detection frame that meets the conditions <![CDATA[D 13 -D 23 ]]> <![CDATA[D 13 -D 22 ]]> <![CDATA[D 11 -D 22 ]]> <![CDATA[D 14 -D 24 ]]> Morphological similarity s 0.917 0.857 0.810 0.710

[0064] For each pair of first detection boxes in S3, the first position offsets Δx and Δy of the corresponding box center are calculated. The first position offsets of each pair of first detection boxes are shown in Table 5 below.

[0065] Table 5

[0066]

[0067] Position compensation is performed based on the first position offsets (-40, 8), (-161, -31.5), (-34.5, 20.5), and (-15.5, 8.5) of each first detection box pair to obtain four sets of fifth tracking detection boxes D5. The average value of the intersection-union ratio (IUU) between the first target detection box and the fifth tracking detection boxes in each set of fifth tracking detection boxes is calculated in turn. The first position offset of the first detection box pair with the largest IUU is selected as the first compensation parameter.

[0068] The set of fifth tracking detection boxes after position compensation of all first tracking detection boxes with the first position offset (Δx, Δy) = (-40, 8) is shown in Table 6. The cross-union ratio between the first target detection box and the fifth tracking detection box in the fifth tracking detection box set is shown in Table 7.

[0069] Table 6

[0070]

[0071] Table 7

[0072]

[0073] Finally, the average cross-union ratio (CUI) data of all first detection boxes that meet the conditions after position compensation is shown in Table 8. The first position offset (Δx, Δy) = (-40, 8) of the first detection box pair with the largest CUI is used as the first compensation parameter for all first tracking detection boxes.

[0074] Table 8

[0075]

[0076]

[0077] Based on the aforementioned first compensation parameters, coarse motion compensation is performed on all first tracking detection boxes to obtain the second tracking detection box set D3. The positions of the second tracking detection boxes are the same as those of the fifth tracking detection boxes in Table 6, as detailed below. Figure 4 As shown, the boxes with "vehicle" represent the first target detection boxes in the current frame, while the boxes without "vehicle" represent the second target tracking detection boxes in the previous frame. (Refer to...) Figure 4 It can be seen that the second tracking detection box after rough motion compensation still has some deviation from the first target detection box on the vertical axis, so further motion compensation is needed.

[0078] Optionally, the second similarity between each first target detection box in the first target detection box set and each second tracking detection box in the second tracking detection box set can be determined firstly using the translation, intersection, and comparison algorithm. The specific process is as follows: group each first target detection box in the first target detection box set and each second tracking detection box in the second tracking detection box set into pairs to obtain multiple pairs of second detection boxes.

[0079] For each pair of second detection boxes, the following ratios can be determined: a fourth ratio between the intersection and union of the box lengths of the first target detection box and the second tracking detection box; a fifth ratio between the intersection and union of the box widths; and a third ratio between the smaller and larger box areas. The second product of the fourth and third ratios, and the third product of the fifth and third ratios are then determined. The larger of the second and third products is taken as the second similarity of the second detection box pair. The specific formula is as follows:

[0080]

[0081] The second similarity scores of each second detection box pair can be combined to form a second similarity set. The specific data is shown in Table 9.

[0082] Table 9

[0083]

[0084] Subsequently, the second compensation parameter for each second tracking detection box can be determined based on the second position offset between each pair of first target detection boxes and second tracking detection boxes corresponding to the second similarity that is greater than the second preset threshold. Specifically, this is achieved as follows: the second similarities in the second similarity set are sorted from largest to smallest, and each second similarity in the second similarity set is traversed sequentially. If the second similarity is greater than the second preset threshold, the second position offset between the first target detection box and the second tracking detection box in the second detection box pair corresponding to the second similarity is used as the second compensation parameter for the second tracking detection box. This second position offset is recorded in the offset set, and the second similarities in the second similarity set that are associated with the first target detection box and the second tracking detection box are deleted. If the second similarity is not greater than the second preset threshold, the traversal is stopped. After the traversal is completed, the average offset of all second position offsets in the offset set is determined, and the average offset is used as the second compensation parameter for all second tracking detection boxes whose second compensation parameters have not yet been determined.

[0085] Taking the second preset threshold set to 0.7 as an example, when the traversal is completed, the second compensation parameters corresponding to each second tracking detection box are determined as shown in Table 10.

[0086] Table 10

[0087]

[0088] Based on the aforementioned second compensation parameters, fine motion compensation is performed on each of the second tracking detection boxes to obtain the third tracking detection box set D4, as detailed below. Figure 5 As shown, the boxes with "vehicle" represent the first target detection box in the current frame, while the boxes without "vehicle" represent the third target tracking detection box in the previous frame. (Refer to...) Figure 5 As can be seen, the third tracking detection box after fine motion compensation has only minor differences from the first target detection box, and its impact on target tracking and matching is negligible. Therefore, the combination of coarse motion compensation of the first compensation parameter and fine motion compensation of the second compensation parameter in the embodiments of this application can improve the target tracking and detection effect.

[0089] It should be noted that in the above compensation process, when there is no first similarity greater than the first preset threshold or no second similarity greater than the second preset threshold, the target tracker can be directly used to track and match the first target detection box set and the first tracking detection box set to obtain the fourth tracking detection box set corresponding to the first image.

[0090] In this embodiment, a two-stage processing method of coarse compensation and fine compensation is adopted. First, the overall motion offset is roughly estimated, and then the motion offset of each target between consecutive frames is predicted independently. Then, the offset is applied to each target, and tracking and matching are performed after offset compensation. Without increasing the time consumption and complex computation, the motion compensation method can reduce the target offset in the picture caused by the camera's own movement, which can track the target more stably and improve the target tracking effect in complex scenes. It effectively solves the technical problem that the performance of target tracking and detection algorithms in related technologies cannot meet the requirements in scenes with continuous camera movement.

[0091] Example 2

[0092] According to an embodiment of this application, a target tracking and detection apparatus for implementing the target tracking and detection method in Embodiment 1 is also provided, such as... Figure 6 As shown, the target tracking and detection device includes at least: an acquisition module 61, a first determination module 62, a first compensation module 63, a second determination module 64, a second compensation module 65, and a tracking and detection module 66, wherein:

[0093] The acquisition module 61 is used to acquire multiple frames of images from the video to be analyzed and to use the object detector to determine the set of object detection boxes in each frame of the image;

[0094] The first determining module 62 is used to, for any first image other than the first frame image, obtain the first tracking detection box set corresponding to the previous frame second image of the first image, and determine the first similarity between each first target detection box in the first target detection box set corresponding to the first image and each first tracking detection box in the first tracking detection box set, wherein if the second image is the first frame image, the first tracking detection box set is the second target detection box set corresponding to the second image.

[0095] The first compensation module 63 is used to determine the first compensation parameters of all first tracking detection boxes based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to the first similarity greater than the first preset threshold, and to perform position compensation on all first tracking detection boxes based on the first compensation parameters to obtain a second tracking detection box set.

[0096] The second determining module 64 is used to determine the second similarity between each first target detection box and each second tracking detection box in the second tracking detection box set using the translation, intersection and comparison algorithm;

[0097] The second compensation module 65 is used to determine the second compensation parameters of each second tracking detection box based on the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to the second similarity greater than the second preset threshold, and to perform position compensation on each second tracking detection box based on the second compensation parameters to obtain a third tracking detection box set.

[0098] The tracking and detection module 66 is used to track and match the first target detection box set and the third tracking and detection box set using a target tracker to obtain the fourth tracking and detection box set corresponding to the first image.

[0099] Optionally, the acquisition module can extract multiple frames of images from the video to be analyzed based on a preset frame interval.

[0100] Optionally, when determining the first similarity, the first determining module can group each first target detection box in the first target detection box set with each first tracking detection box in the first tracking detection box set to obtain multiple sets of first detection box pairs; for each set of first detection box pairs, determine the first ratio of the smaller box length to the larger box length, the second ratio of the smaller box width to the larger box width, and the third ratio of the smaller box area to the larger box area between the first target detection box and the first tracking detection box in the first detection box pair; determine the first product of the first ratio, the second ratio, and the third ratio as the first similarity of the first detection box pair; and form a first similarity set from the first similarity of each first detection box pair.

[0101] Optionally, when determining the first compensation parameter, the first compensation module can form a third similarity set by combining all first similarities in the first similarity set that are greater than a first preset threshold, and sort the first similarities in the third similarity set from largest to smallest; sequentially traverse the first detection box pairs corresponding to each first similarity in the third similarity set, determine the first position offset between the first target detection box and the first tracking detection box in the first detection box pair, perform position compensation on all first tracking detection boxes based on the first position offset, obtain a fifth tracking detection box set, determine the intersection-union ratio (IUU) between each first target detection box and each fifth tracking detection box in the fifth tracking detection box set, and determine the average value of multiple IUU; determine the first position offset between the first target detection box and the first tracking detection box in the first detection box pair with the largest average value as the first compensation parameter for all first tracking detection boxes.

[0102] Optionally, when determining the second similarity, the second determining module can group each first target detection box in the first target detection box set with each second tracking detection box in the second tracking detection box set to obtain multiple sets of second detection box pairs; for each set of second detection box pairs, determine the fourth ratio between the intersection of the box lengths and the union of the box lengths of the first target detection box and the second tracking detection box in the second detection box pair, the fifth ratio between the intersection of the box widths and the union of the box widths, and the third ratio between the smaller box area and the larger box area; determine the second product of the fourth ratio and the third ratio, and the third product of the fifth ratio and the third ratio; determine the larger product of the second product and the third product as the second similarity of the second detection box pair; and form a second similarity set from the second similarity of each second detection box pair.

[0103] Optionally, when determining the second compensation parameter, the second compensation module can sort the second similarities in the second similarity set from largest to smallest, and sequentially traverse each second similarity in the second similarity set. If the second similarity is greater than the second preset threshold, the second position offset between the first target detection box and the second tracking detection box in the second detection box pair corresponding to the second similarity is used as the second compensation parameter of the second tracking detection box. The second position offset is recorded in the offset set, and the second similarities in the second similarity set that are associated with the first target detection box and the second tracking detection box are deleted. If the second similarity is not greater than the second preset threshold, the traversal stops. After the traversal is completed, the average offset of all second position offsets in the offset set is determined, and the average offset is used as the second compensation parameter of all second tracking detection boxes whose second compensation parameters have not yet been determined.

[0104] Optionally, when there is no first similarity greater than the first preset threshold, or no second similarity greater than the second preset threshold, the tracking and detection module can directly use the target tracker to track and match the first target detection box set and the first tracking detection box set to obtain the fourth tracking detection box set corresponding to the first image.

[0105] It should be noted that each module in the target tracking and detection device in this application embodiment corresponds one-to-one with each implementation step of the target tracking and detection method in embodiment 1. Since embodiment 1 has been described in detail, some details not shown in this embodiment can be referred to embodiment 1, and will not be elaborated further here.

[0106] Example 3

[0107] According to an embodiment of this application, a non-volatile storage medium is also provided, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the target tracking and detection method in Embodiment 1 by running the computer program.

[0108] According to an embodiment of this application, a processor is also provided for running a computer program, wherein the computer program executes the target tracking and detection method in embodiment 1 during runtime.

[0109] According to an embodiment of this application, an electronic device is also provided, comprising: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the target tracking and detection method of embodiment 1 through the computer program.

[0110] Specifically, the computer program executes the following steps: acquiring multiple frames of images from the video to be analyzed, and using an object detector to determine the set of target detection boxes in each frame; for any frame of the first image other than the first frame, acquiring the set of first tracking detection boxes corresponding to the previous frame of the first image, and determining the first similarity between each first target detection box in the set of first target detection boxes corresponding to the first image and each first tracking detection box in the set of first tracking detection boxes, wherein if the second image is the first frame, the set of first tracking detection boxes is the set of second target detection boxes corresponding to the second image; determining all first tracking detection boxes based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to the first similarity greater than a first preset threshold. The first compensation parameter of the tracking detection box is used, and the position compensation of all first tracking detection boxes is performed according to the first compensation parameter to obtain a second tracking detection box set; the second similarity between each first target detection box and each second tracking detection box in the second tracking detection box set is determined using the translation-intersection-union algorithm; the second compensation parameter of each second tracking detection box is determined according to the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to the second similarity greater than a second preset threshold, and the position compensation of each second tracking detection box is performed according to the second compensation parameter to obtain a third tracking detection box set; the first target detection box set and the third tracking detection box set are tracked and matched using a target tracker to obtain a fourth tracking detection box set corresponding to the first image.

[0111] The sequence numbers of the above embodiments are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0117] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A target tracking and detection method, characterized in that, include: Multiple frames of images are acquired from the video to be analyzed, and the set of target detection boxes in each frame is determined using an object detector; For any first image other than the first frame image, obtain the first tracking detection box set corresponding to the previous second image of the first image, and determine the first similarity between each first target detection box in the first target detection box set corresponding to the first image and each first tracking detection box in the first tracking detection box set, wherein if the second image is the first frame image, the first tracking detection box set is the second target detection box set corresponding to the second image. Based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to a first similarity greater than a first preset threshold, the first compensation parameter of all first tracking detection boxes is determined, and the position compensation of all first tracking detection boxes is performed based on the first compensation parameter to obtain a second tracking detection box set. The translation, intersection, and comparison algorithm is used to determine the second similarity between each first target detection box and each second tracking detection box in the second set of tracking detection boxes; The second compensation parameter of each second tracking detection box is determined based on the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to the second similarity greater than the second preset threshold, and the position compensation of each second tracking detection box is performed based on the second compensation parameter to obtain the third tracking detection box set. The first set of target detection boxes and the third set of tracking detection boxes are tracked and matched using a target tracker to obtain a fourth set of tracking detection boxes corresponding to the first image.

2. The method according to claim 1, characterized in that, Determining the first similarity between each first target detection box in the first target detection box set corresponding to the first image and each first tracking detection box in the first tracking detection box set includes: Each first target detection box in the first target detection box set is paired with each first tracking detection box in the first tracking detection box set to obtain multiple pairs of first detection boxes; For each pair of the first detection boxes, a first ratio of the smaller box length to the larger box length, a second ratio of the smaller box width to the larger box width, and a third ratio of the smaller box area to the larger box area are determined between the first target detection box and the first tracking detection box in the first detection box pair. The first product of the first ratio, the second ratio, and the third ratio is determined as the first similarity of the first detection box pair. The first similarity scores of each of the first detection box pairs are combined to form a first similarity set.

3. The method according to claim 2, characterized in that, The first compensation parameters for all first tracking detection boxes are determined based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to a first similarity greater than a first preset threshold, including: All first similarities in the first similarity set that are greater than the first preset threshold are combined into a third similarity set, and the first similarities in the third similarity set are sorted from largest to smallest; The first detection box pairs corresponding to each first similarity in the third similarity set are sequentially traversed. The first position offset between the first target detection box and the first tracking detection box in the first detection box pair is determined. The position compensation of all first tracking detection boxes is performed based on the first position offset to obtain the fifth tracking detection box set. The intersection-union ratio between each first target detection box and each fifth tracking detection box in the fifth tracking detection box set is determined, and the average value of multiple intersection-union ratios is determined. The first position offset between the first target detection box and the first tracking detection box in the first detection box pair with the largest average value is determined as the first compensation parameter for all first tracking detection boxes.

4. The method according to claim 1, characterized in that, The second similarity between each first target detection box and each second tracking detection box in the second set of tracking detection boxes is determined using the translation-intersection-union algorithm, including: Each first target detection box in the first target detection box set is paired with each second tracking detection box in the second tracking detection box set to obtain multiple pairs of second detection boxes; For each pair of second detection boxes, determine the fourth ratio between the intersection of the box lengths and the union of the box lengths of the first target detection box and the second tracking detection box in the second detection box pair, the fifth ratio between the intersection of the box widths and the union of the box widths, and the third ratio between the smaller box area and the larger box area. Determine the second product of the fourth ratio and the third ratio, and the third product of the fifth ratio and the third ratio. Determine the larger of the second product and the third product as the second similarity of the second detection box pair. The second similarity scores of each of the second detection box pairs are combined to form a second similarity set.

5. The method according to claim 4, characterized in that, The second compensation parameters for each second tracking detection box are determined based on the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to a second similarity greater than a second preset threshold, including: Sort the second similarity scores in the second similarity set from largest to smallest; The process iterates through each second similarity in the second similarity set. If the second similarity is greater than the second preset threshold, the second position offset between the first target detection box and the second tracking detection box in the second detection box pair corresponding to the second similarity is used as the second compensation parameter of the second tracking detection box. The second position offset is recorded in the offset set, and the second similarity in the second similarity set that is associated with the first target detection box and the second tracking detection box is deleted. If the second similarity is not greater than the second preset threshold, the iteration stops. After traversal is completed, the average offset of all second position offsets in the offset set is determined, and the average offset is used as the second compensation parameter for all second tracking detection boxes whose second compensation parameters have not yet been determined.

6. The method according to claim 1, characterized in that, The method further includes: When there is no first similarity greater than the first preset threshold, or no second similarity greater than the second preset threshold, the target tracker is used to track and match the first target detection box set and the first tracking detection box set to obtain the fourth tracking detection box set corresponding to the first image.

7. The method according to claim 1, characterized in that, Obtain multiple frames of images from the video to be analyzed, including: Multiple frames are extracted from the video to be analyzed based on a preset frame interval.

8. A target tracking and detection device, characterized in that, include: The acquisition module is used to acquire multiple frames of images from the video to be analyzed and to use the object detector to determine the set of object detection boxes in each frame of the image; The first determining module is used to, for any first image other than the first frame image, obtain a set of first tracking detection boxes corresponding to the previous frame of the first image, and determine a first similarity between each first target detection box in the set of first target detection boxes corresponding to the first image and each first tracking detection box in the set of first tracking detection boxes, wherein if the second image is the first frame image, the first tracking detection box set is the set of second target detection boxes corresponding to the second image. The first compensation module is used to determine the first compensation parameters of all first tracking detection boxes based on the first position offset between each group of first target detection boxes and first tracking detection boxes corresponding to a first similarity greater than a first preset threshold, and to perform position compensation on all first tracking detection boxes based on the first compensation parameters to obtain a second tracking detection box set. The second determining module is used to determine the second similarity between each first target detection box and each second tracking detection box in the second tracking detection box set using the translation, intersection and comparison algorithm; The second compensation module is used to determine the second compensation parameters of each second tracking detection box based on the second position offset between each group of first target detection boxes and second tracking detection boxes corresponding to the second similarity greater than the second preset threshold, and to perform position compensation on each second tracking detection box based on the second compensation parameters to obtain a third tracking detection box set. The tracking and detection module is used to track and match the first target detection box set and the third tracking and detection box set using a target tracker to obtain a fourth tracking and detection box set corresponding to the first image.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the target tracking and detection method according to any one of claims 1 to 7 by running the computer program.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the target tracking and detection method according to any one of claims 1 to 7 through the computer program.

Images