Multi-target tracking method and device for unmanned view angle and electronic equipment

Abstract

The invention relates to a multi-target tracking method and device for an unmanned view angle and electronic equipment. The method comprises the following steps: predicting second position information of a target object in a current video frame based on first position information of the target object in a previous video frame; tracking the positions of a plurality of target objects in the video by using a single-target tracking network, and judging the tracking effectiveness; if the tracking is effective, performing multi-target real-time tracking by using a single-target tracking network; and if the tracking is invalid, performing initialization processing on the single-target tracking network, and continuing to perform multi-target real-time tracking on the plurality of target objects by using the processed single-target tracking network. Multi-target tracking is performed by adopting a single-target tracking mode, and the single-target tracker is restarted to continue to track the target when the target object is lost, so that challenges caused by picture instability are overcome, efficient and stable tracking of multiple targets in the flight process of the unmanned aerial vehicle is realized, and more flexible and convenient operation experience is provided for a user.

Description

Technical Field

[0001] The present application relates to the field of target tracking, and in particular to a method, device and electronic equipment for multi-target tracking from an unmanned perspective. Background Art

[0002] Most multi-target tracking solutions on the market are based on the principles of machine learning. They perform target detection on each video frame, use the position and size of the target in adjacent video frames to estimate the target's motion posture, and complete target tracking.

[0003] The commonly used multi-target tracking methods currently require that the video screen is still or the video screen changes relatively smoothly, because fast-moving video screens will cause the linear motion state estimation of the target to fail. If the tracking method is only based on appearance, the target to be tracked in the video must have more details, otherwise it will be confused when homogeneous targets appear. However, in the scenario of multi-target tracking of drones, in most cases the picture cannot change smoothly and stably, and the size of the target presented in the video is not ideal. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for multi-target tracking from an unmanned perspective, so as to at least solve the problem in the related art that multi-target tracking cannot be achieved in an unstable state.

[0005] In a first aspect, an embodiment of the present application provides a multi-target tracking method for an unmanned perspective, characterized by comprising:

[0006] Extracting a target object in a video, and predicting second position information of the target object in a current video frame based on first position information of the target object in a previous video frame;

[0007] Tracking the positions of the plurality of target objects in the video using a single target tracking network, and judging the effectiveness of the tracking based on the second position information;

[0008] If the tracking is effective, the single target tracking network is used to perform multi-target real-time tracking;

[0009] If the tracking is invalid, the single target tracking network is initialized and the processed single target tracking network is used to continue to perform multi-target real-time tracking on the multiple target objects.

[0010] In one embodiment, extracting the target object in the video and predicting the second position information of the target object in the current video frame based on the first position information of the target object in the previous video frame includes:

[0011] Selecting the target object in the video, and acquiring the first position information of the target object in the previous video frame;

[0012] A feature representation of the target object is extracted, and second position information of the target object in the current video frame is predicted based on the feature representation.

[0013] In one embodiment, extracting a feature representation of the target object, and predicting second position information of the target object in the current video frame based on the feature representation, includes:

[0014] Extracting a target feature network of the target object using a Siamese network structure to obtain a feature representation of the target object;

[0015] The feature representation is input into the RPN network to obtain output data, and the output data is translated and scaled to obtain the second position information.

[0016] In one embodiment, the using a single target tracking network to track the positions of the plurality of target objects in the video, and judging the validity of the tracking based on the second position information, includes:

[0017] Obtain a target detection result queue storing the second position information, compare the tracking result of the single target tracking network with the content in the target detection result queue, and if a preset threshold number of consecutive tracking results do not overlap with the content in the target detection result queue, determine that the tracking is invalid.

[0018] In one embodiment, it further includes:

[0019] When using the single target tracking network to track multiple targets in real time, different pixel distances of different target objects are processed differently;

[0020] If the pixel distance is greater than the distance threshold, the target object is determined to be a separation target, and a Kalman filter is used to smooth and predict the target trajectory of the separation target;

[0021] If the pixel distance is less than or equal to the distance threshold, the target object is determined to be an adjacent target, and the actual position of the adjacent target is corrected.

[0022] In one embodiment, if the pixel distance is greater than a distance threshold, the target object is determined to be a separation target, and a Kalman filter is used to perform smoothing and trajectory prediction on the target trajectory of the separation target, including:

[0023] If the pixel distance is greater than the distance threshold, the target object is determined to be a separation target;

[0024] Acquire the motion trajectory of the separated target in continuous video frames, and perform smoothing and trajectory prediction on the target trajectory by using Kalman filtering;

[0025] When the single target tracking network loses the target, the surrounding area of ​​the trajectory prediction is obtained, and the matrix similarity calculation is performed using the feature representation of the surrounding area and the feature representation of the historical frame;

[0026] If the calculated result is greater than the threshold, it is considered that the target object is found again, the regional features of the target object are used to initialize the single target tracking network, and the processed single target tracking network is used to continue multi-target real-time tracking of multiple target objects.

[0027] In one embodiment, if the pixel distance is less than or equal to a distance threshold, determining that the target object is an adjacent target, and correcting the actual position of the adjacent target include:

[0028] If the pixel distance is less than or equal to the distance threshold, the target object is determined to be an adjacent target;

[0029] Obtaining the average speed and instantaneous speed of the adjacent target movement in the historical frame;

[0030] The actual position of the adjacent target in the historical frame is corrected based on the average speed and the instantaneous speed.

[0031] In a second aspect, an embodiment of the present application provides a multi-target tracking device for unmanned viewing angles, comprising:

[0032] An information prediction module, used to extract a target object in a video, and predict second position information of the target object in a current video frame based on first position information of the target object in a previous video frame;

[0033] A validity judgment module, used for tracking the positions of the plurality of target objects in the video using a single target tracking network, and judging the validity of the tracking based on the second position information;

[0034] A first execution module, configured to use the single target tracking network to perform multi-target real-time tracking if the tracking is effective;

[0035] The second execution module is used to initialize the single target tracking network if the tracking is invalid, and use the processed single target tracking network to continue to perform multi-target real-time tracking on the multiple target objects.

[0036] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-target tracking method for an unmanned perspective as described in the first aspect above is implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-target tracking method for an unmanned perspective as described in the first aspect above.

[0038] The embodiments of the present application provide a method, device and electronic device for multi-target tracking from an unmanned perspective, which have at least the following technical effects.

[0039] By adopting a single target tracking method for multi-target tracking, the single target tracker is restarted to continue tracking the target when the target object is lost, overcoming the challenges brought by image instability, achieving efficient and stable tracking of multiple targets during drone flight, and providing users with a more flexible and convenient operating experience.

[0040] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 is a flow chart of a multi-target tracking method for an unmanned perspective provided in this embodiment;

[0043] Figure 2 is a schematic flow chart of step S10 provided in this embodiment;

[0044] Figure 3 is an additional flow chart of the multi-target tracking method for an unmanned perspective provided in this embodiment;

[0045] Figure 4 is a schematic structural diagram of a multi-target tracking device for unmanned viewing provided by this embodiment;

[0046] Figure 5 It is a structural schematic diagram of an electronic device provided by this embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] Based on the above situation, the embodiments of the present application provide a multi-target tracking method, device and electronic device for unmanned perspective.

[0052] In a first aspect, an embodiment of the present application provides a multi-target tracking method for an unmanned perspective. Figure 1 is a flowchart of a multi-target tracking method for unmanned viewing, such as Figure 1 As shown, the method includes:

[0053] Step S10: extracting a target object in a video, and predicting second position information of the target object in a current video frame based on first position information of the target object in a previous video frame.

[0054] The video here is a video shot from the perspective of a drone, and the drone needs to track multiple target objects at the same time during flight shooting. In this application, for each target object to be tracked, it is necessary to extract the position information in different video frames, so that the position where the target object may appear in the video frame can be predicted based on the position information in the subsequent steps, and then the tracking operation can be implemented according to the prediction results.

[0055] Step S20: Track the positions of multiple target objects in the video using a single target tracking network, and determine the effectiveness of the tracking based on the second position information.

[0056] Among them, after determining the position of each target object in different video frames, a single target tracking network is used to track each target object in the multi-target objects. The single target tracking network here refers to DaSiamRPN (Distractor-aware SlamRPN, a twin network for visual tracking). In the process of tracking a single target object, DaSiamRPN needs to periodically compare the position of the tracked target object with the second position information of the target object obtained in the current frame to ensure the effectiveness of the tracking performed by the single target tracking network.

[0057] Step S30: If the tracking is effective, a single target tracking network is used to perform multi-target real-time tracking.

[0058] Among them, if it is judged according to the previous step that the action of tracking the target object using a single target tracking network is effective, no modification is required at this time, and the tracking action is continued. By tracking a single target object through multiple single target tracking networks, multi-target real-time tracking is achieved.

[0059] Step S40: If the tracking is invalid, the single target tracking network is initialized and the processed single target tracking network is used to continue to perform multi-target real-time tracking on multiple target objects.

[0060] Among them, if it is judged according to the previous step that the action of using the single target tracking network to track the target object is invalid, that is, the single target tracker loses tracking. At this time, the target position and size of the tracking result of the previous frame are obtained and compared with the target detection result of the current frame. If the target detection result exists in the sliding window, that is, the overlap degree of the target frame information of the two in the picture is greater than or equal to 60%, the single target tracker is reinitialized using the target detection result, and the processed single target tracking network is used to continue to perform multi-target real-time tracking of multiple target objects.

[0061] This application uses a single target tracking method to track multiple targets. When the target object is lost, the single target tracker is restarted to continue tracking the target. This overcomes the challenges brought by image instability and achieves efficient and stable tracking of multiple targets during drone flight, providing users with a more flexible and convenient operating experience.

[0062] In one embodiment, if Figure 2 As shown, step S10 includes:

[0063] Step S11, selecting a target object in the video, and obtaining first position information of the target object in the previous video frame.

[0064] Step S12: extracting a feature representation of the target object, and predicting second position information of the target object in the current video frame based on the feature representation.

[0065] The basis for determining the position of the target object in the current video frame is the position information of the target object in the previous video frame. Specifically, assuming that the target object has first position information in the previous video frame, the second position of the target object in the current video frame is determined based on the first position information, and the path from the first position information to the second position information is the feature representation of the target object.

[0066] First, a target object is selected in the video based on a deep learning target detection method, and the first position information of the target object in the previous video frame is determined.

[0067] Secondly, the target feature network of the target object is extracted using the Siamese network structure to obtain the feature representation of the target object, the feature representation is input into the RPN network to obtain output data, and the output data is translated and scaled to obtain the second position information. The RPN (Region Proposal Network) network here is usually used to extract candidate boxes, and here it is used to obtain the second position information of the target object in the current video frame according to the input feature representation.

[0068] Specifically, the target feature representation is passed through the RPN network, and the feature representation of the target is used to search for the feature representation of the target image on the image to predict the target location. The RPN network will output a bounding box indicating the location and size of the target search in the image.

[0069] Furthermore, the location information of the target object can be updated according to the output of the RPN network, and the search position and size of the current RPN network prediction result can be saved. The anchor (a reference frame of different sizes and different aspect ratios preset on the image) obtained by the RPN network prediction result is post-processed. The search position and size closest to the actual position of the target can be obtained by translation and scaling.

[0070] The post-processing of the RPN network output data here is different from the traditional RPN network processing. It does not need to predict the target position very accurately. Instead, it uses linear regression to model and fine-tune the search window. By reducing the target search size and expanding the coefficient of the target search position, the search range and the feature representation of a smaller area can be expanded. Because the target feature representation may change significantly during the drastic change of the video, it is necessary to find the possible target feature representation through smaller target features and a larger search range.

[0071] Get the position and size of the target tracking result of the previous video frame, process the current video frame, and execute the calculation formula of the search range CropRange as shown in Formula 1:

[0072] CropRange = O(ScaleZ)*Φ(SearchSize) Formula 1;

[0073] Where, ScaleZ = ExemplarSize / (ContextAmount * f (box) + β),

[0074] SearchSize=(InstanceSize-ExemplarSize) / 2);

[0075] In the formula, InstanceSize is the instance size, which is output by the actual detection result; ExemplarSize is the sample size; ContextAmount is a custom coefficient. The lower the ContextAmount value, the smaller the final CropRange. This means that if there is a large change in the video frame, the CropRange value will shrink rapidly, and the final target will be in this reduced area.

[0076] Through the above steps, if you find that the CropRange shrinks rapidly, it means that the image motion is too large. At this time, the target is likely to become blurred, and the motion trajectory is unpredictable. It is likely to lose the target, and wait for the corresponding processing solution if the subsequent tracking is invalid.

[0077] In one embodiment, step S20 includes:

[0078] Obtain a target detection result queue storing the second position information, compare the tracking result of the single target tracking network with the content in the target detection result queue, and if a preset threshold number of consecutive tracking results do not overlap with the content in the target detection result queue, determine that the tracking is invalid.

[0079] In order to improve the continuity of tracking, the target detection results will be used to assist the single target tracker. However, if the target detection result is a false detection, it may cause the single target tracker to initialize the wrong target. The target detection will store a target detection result queue with a sliding window of 10. Each time, the single target tracker result will be compared with the detection result of the corresponding frame. If the single target tracker result does not overlap with the target detection result for more than 10 consecutive frames, the single target tracker tracking will be cancelled.

[0080] When multiple single-target tracking networks are used simultaneously to track multiple targets, the pixel distances of different target objects will affect the tracking results. Here, two cases are explained based on the relationship between the pixel distance and the distance threshold.

[0081] In one embodiment, if Figure 3 As shown, it also includes:

[0082] Step S50, when using a single target tracking network to perform real-time tracking of multiple targets, pixel distances of different target objects are processed differently.

[0083] Step S60: If the pixel distance is greater than the distance threshold, the target object is determined to be a separated target, and a Kalman filter is used to smooth and predict the target trajectory of the separated target.

[0084] Specifically, if the pixel distance is greater than the distance threshold, the target object is determined to be a separated target; the motion trajectory of the separated target continuous video frames is obtained, and the target trajectory is smoothed and predicted by using Kalman filtering; when the single target tracking network loses the target, the surrounding area of ​​the trajectory prediction is obtained, and the matrix similarity is calculated using the feature representation of the surrounding area and the feature representation of the historical frame; if the calculation result is greater than the threshold, it is considered that the target object is found again, and the single target tracking network is initialized using the regional features of the target object, and the processed single target tracking network is used to continue to perform multi-target real-time tracking of multiple target objects.

[0085] For example, the target motion is acquired continuously for n frames, and the Kalman filter is used to smooth the target trajectory and predict the trajectory for the next m frames. When the tracker loses the target, the Siamese network is used to re-acquire the feature representation of the region using the pixels near the trajectory position predicted for the next m frames, and the matrix similarity is calculated with the past n real image feature representations. If it is greater than a threshold, it is considered that the target is found again and a new regional feature is obtained. The tracker is initialized using the new regional feature.

[0086] Step S70: If the pixel distance is less than or equal to the distance threshold, the target object is determined to be an adjacent target, and the actual position of the adjacent target is corrected.

[0087] Specifically, if the pixel distance is less than or equal to the distance threshold, the target object is determined to be an adjacent target; the average speed and instantaneous speed of the adjacent target in the historical frame are obtained; and the actual position of the adjacent target in the historical frame is corrected based on the average speed and the instantaneous speed.

[0088] For example, take two targets as an example. Let the bottom corner of the screen be the origin O (0, 0), and the two target center points be A and B, and get Assume the current frame number is x frames, and go forward to get the historical coordinate point A of frames A and B. x-n and B x-n Calculate the average speed of A and B in the past n frames ΔA n and ΔB n .

[0089] Get the instantaneous speed ΔA of the historical coordinate points A and B when the frame number is x-1 x-1 and ΔBx-1 , by changing the speed, correct the center position of A and B to prevent confusion among multiple targets that are too close, and execute formula 2:

[0090]

[0091] Use formula 3 to correct the actual position of targets A and B:

[0092]

[0093] The above formula can be used to prevent the tracker from tracking targets that are interlaced, merged, or interchanged according to the rate of change when the target is close. The essence is to dynamically reduce the speed of multi-target approach, thereby reducing the problem of losing the tracking target when there are adjacent targets in the picture and the picture is rotated.

[0094] This application aims to improve the accuracy of multi-target tracking and the stability of continuous tracking time from the perspective of drones. The method provided by this application has the following advantages for multi-target tracking in drone videos:

[0095] In drone video missions, multiple targets can be tracked simultaneously in the video. In the video, the drone turntable and the drone's own flight posture have less impact on target tracking in the video, reducing the chance of target loss. Combined with multiple deep learning network models (target detection, RPN and Siamese network), multiple detection results are merged to improve target detection accuracy. When the target requires recognition pixels of 40px×40px or more, the original video resolution is at 1080P resolution, and the analysis speed can meet the real-time requirements of the video. It can be accelerated on general GPUs on the market and track more than 4 targets at the same time. In the state where the target is temporarily lost in the video, if the target returns to the picture, the target can still be found.

[0096] Example 1

[0097] A multi-target tracking method based on deep learning target detection and single target tracking. Take the example of a drone performing a mission in a city and transmitting ground video:

[0098] 1) The drone video is transmitted back to the ground, and the detection model of the corresponding target is selected, such as people, non-motor vehicles, motor vehicles, etc. for target detection, and the selectable target box boundaries are displayed on the front end.

[0099] 2) Click the target box position. Send a tracking request to the background server, and initialize the single target tracker. If you click multiple targets, initialize multiple single target trackers.

[0100] 3) If three vehicles are driving in the video at the same time, click on the three target boxes

[0101] 4) The three target boxes all present their own unique IDs (such as A, B, and C) and move with the target.

[0102] 5) When target A leaves the screen from the left side of the video and then re-enters the screen for 2 seconds, the ID of target A will reappear instead of generating a new ID.

[0103] Example 2

[0104] A multi-target tracking method based on deep learning target detection and single target tracking. Take a gimbaled drone in a city performing a mission and transmitting ground video as an example:

[0105] 1) Repeat steps 1 to 4 of Example 1

[0106] 2) When the vehicles (A, B, C) move at a constant speed along the road to the right side of the screen, the drone gimbal is controlled to move to the left side of the screen. The traditional tracking algorithm cannot continue to track the target trajectory according to the law of motion, which will cause ID changes. The technical solution of the present application can resist the disturbance caused by the large rotation of the picture. Even if the target position in the picture shifts with the background, it can still stabilize the (A, B, C) ID at their respective corresponding target positions. Specifically, when the entire picture starts to move due to the rotation of the turntable, the position of the target frame closely follows the target, but the size of the target frame shrinks rapidly.

[0107] This application combines the single target tracking target frame size threshold and target detection results to sense the large changes in the picture, thereby relocating the target's true position and reinitializing the single target tracker. The video screen shows that the target frame position follows closely, but the size shrinks rapidly and then enlarges to the target size. This process is repeated when the gimbal is rotated later.

[0108] 3) After target B leaves the screen from the right side and does not return, the frame will eventually shrink at the border. After the sliding window of 10 frames of target detection is set, it will be determined that the target has completely disappeared, and the target frame displayed in the video will disappear.

[0109] Example 3

[0110] A multi-target tracking method based on deep learning target detection and single target tracking. Take a gimbal drone in a city performing a mission, but the weather visibility is low and the ground video is transmitted back as an example:

[0111] 1) Repeat steps 1) to 4) of Example 1

[0112] 2) When the drone passes through a dense fog area (an area with low visibility), the detection frames of the targets (A, B, C) will stay at the original position for 10 frames.

[0113] 3) If the target can be guaranteed to reappear in the picture after 10 frames, and the pixel offset from the original position does not exceed 100px, the single target tracker combined with the target detection result of deep learning can be relocated to the current position of the original ID, and re-follow the target to draw the trajectory.

[0114] 4) Finally, the video shows stable tracking of multiple targets. You can change the target ID label according to your own cognition, such as changing A to jeep.

[0115] In a second aspect, an embodiment of the present application provides a multi-target tracking device for unmanned perspective. Figure 4 This is a block diagram of a multi-target tracking device for unmanned viewing. Figure 4 As shown, the multi-target tracking device for unmanned viewing angle includes:

[0116] The information prediction module 410 is used to extract the target object in the video, and predict the second position information of the target object in the current video frame based on the first position information of the target object in the previous video frame.

[0117] The video here is a video shot from the perspective of a drone, and the drone needs to track multiple target objects at the same time during flight shooting. In this application, for each target object to be tracked, it is necessary to extract the position information in different video frames, so that the position where the target object may appear in the video frame can be predicted based on the position information in the subsequent steps, and then the tracking operation can be implemented according to the prediction results.

[0118] The validity judgment module 420 is used to track the positions of the multiple target objects in the video using a single target tracking network, and judge the validity of the tracking based on the second position information.

[0119] Among them, after determining the position of each target object in different video frames, a single target tracking network is used to track each target object in the multi-target objects. The single target tracking network here refers to DaSiamRPN (Distractor-aware SlamRPN, a twin network for visual tracking). In the process of tracking a single target object, DaSiamRPN needs to periodically compare the position of the tracked target object with the second position information of the target object obtained in the current frame to ensure the effectiveness of the tracking performed by the single target tracking network.

[0120] The first execution module 430 is configured to use the single target tracking network to perform multi-target real-time tracking if the tracking is effective.

[0121] Among them, if it is judged according to the previous step that the action of tracking the target object using a single target tracking network is effective, no modification is required at this time, and the tracking action is continued. By tracking a single target object through multiple single target tracking networks, multi-target real-time tracking is achieved.

[0122] The second execution module 440 is used to initialize the single target tracking network if the tracking is invalid, and use the processed single target tracking network to continue to perform multi-target real-time tracking on the multiple target objects.

[0123] Among them, if it is judged according to the previous step that the action of using the single target tracking network to track the target object is invalid, that is, the single target tracker loses tracking. At this time, the target position and size of the tracking result of the previous frame are obtained and compared with the target detection result of the current frame. If the target detection result exists in the sliding window, that is, the overlap degree of the target frame information of the two in the picture is greater than or equal to 60%, the single target tracker is reinitialized using the target detection result, and the processed single target tracking network is used to continue to perform multi-target real-time tracking of multiple target objects.

[0124] This application uses a single target tracking method to track multiple targets. When the target object is lost, the single target tracker is restarted to continue tracking the target. This overcomes the challenges brought by image instability and achieves efficient and stable tracking of multiple targets during drone flight, providing users with a more flexible and convenient operating experience.

[0125] It should be noted that the multi-target tracking device for unmanned perspective provided in this embodiment is used to implement the above-mentioned implementation mode, and the descriptions that have been made will not be repeated. As used above, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the above embodiment is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0126] In a third aspect, an embodiment of the present application provides an electronic device, Figure 5 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Figure 5 As shown, the electronic device may include a processor 51 and a memory 52 storing computer program instructions.

[0127] Specifically, the processor 51 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0128] Among them, the memory 52 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 52 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 52 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 52 may be inside or outside the data processing device. In a specific embodiment, the memory 52 is a non-volatile memory. In a specific embodiment, the memory 52 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0129] The memory 52 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 51 .

[0130] The processor 51 reads and executes computer program instructions stored in the memory 52 to implement any one of the multi-target tracking methods for unmanned viewing angles in the above embodiments.

[0131] In one embodiment, the electronic device may further include a communication interface 53 and a bus 50. Figure 5 As shown, the processor 51, the memory 52, and the communication interface 53 are connected via a bus 50 and communicate with each other.

[0132] The communication interface 53 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 53 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0133] The bus 50 includes hardware, software or both, and couples the components of the electronic device to each other. The bus 50 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 50 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 50 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0134] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a program stored thereon, and when the program is executed by a processor, the multi-target tracking method for unmanned perspective provided in the first aspect is implemented.

[0135] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0136] In a possible implementation, the present application can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the multi-target tracking method for unmanned perspective provided in the first aspect.

[0137] Among them, the program code for executing the present application can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or completely on the remote device.

[0138] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-target tracking method for unmanned viewing angles, characterized in that: include: Extracting a target object in a video, and predicting second position information of the target object in a current video frame based on first position information of the target object in a previous video frame; Tracking the positions of the plurality of target objects in the video using a single target tracking network, and judging the effectiveness of the tracking based on the second position information; If the tracking is effective, the single target tracking network is used to perform multi-target real-time tracking; If the tracking is invalid, the single target tracking network is initialized and the processed single target tracking network is used to continue to perform multi-target real-time tracking on the multiple target objects.

2. The multi-target tracking method for unmanned viewing angle according to claim 1, characterized in that: The step of extracting a target object from a video and predicting second position information of the target object in a current video frame based on first position information of the target object in a previous video frame includes: Selecting the target object in the video, and acquiring the first position information of the target object in the previous video frame; A feature representation of the target object is extracted, and second position information of the target object in the current video frame is predicted based on the feature representation.

3. The multi-target tracking method for unmanned viewing angle according to claim 2, characterized in that: The step of extracting a feature representation of the target object and predicting second position information of the target object in the current video frame based on the feature representation includes: Extracting a target feature network of the target object using a Siamese network structure to obtain a feature representation of the target object; The feature representation is input into the RPN network to obtain output data, and the output data is translated and scaled to obtain the second position information.

4. The multi-target tracking method for unmanned viewing angle according to claim 1, characterized in that: Tracking the positions of the plurality of target objects in the video using a single target tracking network, and judging the effectiveness of the tracking based on the second position information, includes: Obtain a target detection result queue storing the second position information, compare the tracking result of the single target tracking network with the content in the target detection result queue, and if a preset threshold number of consecutive tracking results do not overlap with the content in the target detection result queue, determine that the tracking is invalid.

5. The multi-target tracking method for unmanned viewing angle according to claim 1, characterized in that: Also includes: When using the single target tracking network to track multiple targets in real time, different pixel distances of different target objects are processed differently; If the pixel distance is greater than the distance threshold, the target object is determined to be a separation target, and a Kalman filter is used to smooth and predict the target trajectory of the separation target; If the pixel distance is less than or equal to the distance threshold, the target object is determined to be an adjacent target, and the actual position of the adjacent target is corrected.

6. The multi-target tracking method for unmanned viewing angle according to claim 5, characterized in that: If the pixel distance is greater than a distance threshold, the target object is determined to be a separation target, and a Kalman filter is used to perform smoothing and trajectory prediction on the target trajectory of the separation target, including: If the pixel distance is greater than the distance threshold, the target object is determined to be a separation target; Acquire the motion trajectory of the separated target in continuous video frames, and perform smoothing and trajectory prediction on the target trajectory by using Kalman filtering; When the single target tracking network loses the target, the surrounding area of ​​the trajectory prediction is obtained, and the matrix similarity calculation is performed using the feature representation of the surrounding area and the feature representation of the historical frame; If the calculated result is greater than the threshold, it is considered that the target object is found again, the regional features of the target object are used to initialize the single target tracking network, and the processed single target tracking network is used to continue multi-target real-time tracking of multiple target objects.

7. The multi-target tracking method for unmanned viewing angle according to claim 5, characterized in that: If the pixel distance is less than or equal to a distance threshold, determining that the target object is an adjacent target, and correcting the actual position of the adjacent target, comprises: If the pixel distance is less than or equal to the distance threshold, the target object is determined to be an adjacent target; Obtaining the average speed and instantaneous speed of the adjacent target movement in the historical frame; The actual position of the adjacent target in the historical frame is corrected based on the average speed and the instantaneous speed.

8. A multi-target tracking device for unmanned viewing angle, characterized in that: include: An information prediction module, used to extract a target object in a video, and predict second position information of the target object in a current video frame based on first position information of the target object in a previous video frame; A validity judgment module, used for tracking the positions of the plurality of target objects in the video using a single target tracking network, and judging the validity of the tracking based on the second position information; A first execution module, configured to use the single target tracking network to perform multi-target real-time tracking if the tracking is effective; The second execution module is used to initialize the single target tracking network if the tracking is invalid, and use the processed single target tracking network to continue to perform multi-target real-time tracking on the multiple target objects.

9. An electronic device, characterized in that: include Memory, processor, and A computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-target tracking method for unmanned perspective as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-target tracking method for unmanned perspective is implemented as described in any one of claims 1 to 7.

Images