A tracking method, device, equipment and medium

Abstract

The present application discloses a tracking method, apparatus, device and medium. On the one hand, the present application does not need to save a template image, but only needs to save the central position information of the target object in the image to achieve tracking of the target object. Compared with the image, digital data such as position information occupies less memory. On the other hand, since the present application does not need to track based on the saved template image, there will be no problem of affecting the tracking quality due to the low quality of the template image. On the other hand, the present application tracks based on a deep learning detection algorithm, that is, it tracks based on a pre-trained tracking model. Compared with the non-deep learning detection algorithm in the prior art, it has stronger generalization and higher tracking quality.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a tracking method, device, equipment and medium. Background Art

[0002] Multi-target tracking is an important part of intelligent analysis of surveillance videos. The current mainstream technology for multi-target tracking is detection-based tracking, which means that the tracking of objects can be achieved through target detection algorithms. However, when tracking objects based on target detection algorithms, it is possible that a certain object may fail to be tracked in a certain image, resulting in the object being missed. The state of the missed object is the so-called "lost" state. If the object is in the Lost state in multiple images collected continuously, even if the object is detected again in other images later, it is difficult to associate it with the previous tracking trajectory of the object, commonly known as "lost".

[0003] In order to avoid the impact of missed detection on object tracking, in the multi-target tracking algorithm based on target detection, after determining that an object in a certain image is in the Lost state, it is necessary to further track the object in the Lost state through other tracking algorithms. In the prior art, common tracking algorithms include Kalman filter prediction, single target tracking algorithm based on correlation filtering, and single target tracking algorithm based on deep learning, etc., wherein the single target tracking algorithm based on correlation filtering can be a minimum output sum of squared error filter (Minimum Output Sum of Squared Error, MOSSE) algorithm or CSK algorithm, etc., wherein the single target tracking algorithm based on deep learning can be a tracking algorithm based on a Siamese region proposal network (Siamese-RPN), etc.

[0004] In the related art, the widely used tracking algorithm is the single target tracking algorithm, which first uses the sub-image of the target object in the Lost state in the image whose acquisition time is before the current image and adjacent to the current image as the template image, and then determines the maximum response area corresponding to the template image in the current image (that is, the area in the current image with the greatest similarity to the initial template), and then updates the sub-image corresponding to the maximum response area as the template image, and repeats this process to achieve tracking. On the one hand, in order to further track the object, for each object in the Lost state, a template image corresponding to the object needs to be saved in the electronic device, and since the images captured by the existing image acquisition device are generally high-resolution images, the template image corresponding to the object occupies a relatively large amount of memory. If too many template images are subsequently saved, it will occupy too much memory space of the electronic device. On the other hand, for each object, if the image quality of the template image corresponding to the object is relatively low, it will also seriously affect the tracking quality of the object. Summary of the invention

[0005] The present application provides a tracking method, apparatus, device and medium to solve the problem in the prior art that when tracking a target object in a Lost state, a high-resolution template image occupies a large amount of memory, or the quality of the template image is not high, resulting in low tracking quality for the object.

[0006] The present application provides a tracking method, the method comprising:

[0007] Determine the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image; and determine each first target frame in the current image where the target predicted center position is located;

[0008] For each first target frame, input the sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output the first target category corresponding to the target object in the sub-image enclosed by the first target frame and the first target confidence corresponding to the target object;

[0009] Determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target object stored in advance as the second target confidence;

[0010] The intersection-and-union ratio of the first target frame and the second target frame is determined based on the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image that is captured before the current image and adjacent to the current image; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold and the second target confidence is greater than a preset confidence threshold, it is determined that the object at the target prediction center position in the current image is the tracked target object.

[0011] Furthermore, the method further comprises: determining that the target object is before the target predicted center position in the current image based on the pre-saved center position information of the target object to be tracked in other images collected before the current image;

[0012] Based on the target detection algorithm, it is identified whether there is an object in the first target image that has not been successfully matched with the current image; if so, the unsuccessfully matched object is determined as the target object.

[0013] Furthermore, the method further comprises:

[0014] Determine whether the first identification information corresponding to the target object in the first target image stored in advance represents whether the target object is tracked according to the target detection algorithm, and determine whether the target object is located in the central area of ​​the first target image;

[0015] If the first identification information indicates that the target object is not tracked according to the target detection algorithm, or the target object is located in the central area of ​​the first target image, retaining the target object;

[0016] Otherwise, the target item is deleted.

[0017] Furthermore, determining the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image includes:

[0018] The predicted center position of the target is determined according to the center position information of the target object in the first target image and the center position information of a second target image acquired before the first target image and adjacent to the first target image.

[0019] Furthermore, before determining each first target frame in which the target prediction center position is located in the current image, the method further includes:

[0020] According to the width and height of the third target frame corresponding to the target object in the other image, the target width and target height of each first target frame corresponding to the target object in the current image are determined.

[0021] Further, determining the target width and target height of each first target frame corresponding to the target object in the current image according to the width and height of the third target frame corresponding to the target object in the other image includes:

[0022] The second identification information corresponding to the target object in other images indicates that the target object is tracked according to the target detection algorithm, and other images that are most recently acquired in time with the current image are determined as third target images;

[0023] determining whether the third target image and the first target image are the same image;

[0024] If yes, determining the height and width of the third target frame corresponding to the target object in the third target image as the target height and the target width;

[0025] If not, the height and width of the third target frame corresponding to the target object in the third target image, and the height and width of the fourth target frame corresponding to the target object in the first target image, are respectively determined as the target height and the target width corresponding to the first target frame of the target object in the current image.

[0026] Furthermore, the step of inputting the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model and outputting the first target category corresponding to the target object and the first target confidence corresponding to the target object includes:

[0027] Inputting the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model, and outputting a first target category corresponding to the target object, a first target confidence corresponding to the target object, and a predicted target frame corresponding to the target object;

[0028] Before determining the intersection-over-combination ratio of the first target frame and the second target frame according to the first target frame and pre-saved coordinate information of the second target frame corresponding to the target object in the first target image acquired before the current image and adjacent to the current image, the method further includes:

[0029] The predicted target frame is updated to the first target frame.

[0030] Furthermore, after determining the intersection-over-combination ratio of the first target frame and the second target frame according to the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image acquired before the current image and adjacent to the current image, the method further includes:

[0031] If the intersection-over-union ratio is not greater than a preset intersection-over-union ratio threshold, or the second target confidence is not greater than a preset second confidence threshold, it is determined that the target object tracking fails.

[0032] The present application also provides a tracking device, which includes:

[0033] A determination module, configured to determine the target predicted center position of the target object in the current image based on pre-saved center position information of the target object to be tracked in other images collected before the current image; and determine each first target frame in the current image where the target predicted center position is located;

[0034] An output module is used to input, for each first target frame, a sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output a first target category corresponding to the target object in the sub-image enclosed by the first target frame and a first target confidence corresponding to the target object;

[0035] The determination module is also used to determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target item saved in advance as the second target confidence; determine the intersection-and-union ratio of the first target frame and the second target frame according to the coordinate information of the first target frame and the second target frame corresponding to the target item in the first target image collected before the current image and adjacent to the current image in advance; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, and the second target confidence is greater than a preset confidence threshold, then determine that the object at the target prediction center position in the current image is the tracked target item.

[0036] Furthermore, the determination module is also used to identify whether there is an object in the first target image that has not been successfully matched with the current image based on a target detection algorithm; if so, determining the unsuccessfully matched object as the target object.

[0037] Furthermore, the device also includes:

[0038] A processing module is used to determine whether the first identification information corresponding to the target object in the first target image stored in advance represents that the target object is tracked according to the target detection algorithm, and determine whether the target object is located in the central area of ​​the first target image; if the first identification information represents that the target object is not tracked according to the target detection algorithm, or the target object is located in the central area of ​​the first target image, the target object is retained; otherwise, the target object is deleted.

[0039] Furthermore, the determination module is specifically used to determine the predicted center position of the target based on the center position information of the target object in the first target image and the center position information of a second target image acquired before the first target image and adjacent to the first target image.

[0040] Furthermore, the determination module is specifically configured to determine a target width and a target height of each first target frame corresponding to the target object in the current image according to a width and a height of a third target frame corresponding to the target object in the other images.

[0041] Furthermore, the output module is specifically used to determine the other image that is tracked according to the target detection algorithm and is closest to the current image in terms of the second identification information corresponding to the target object in the other image as a third target image; determine whether the third target image is the same image as the first target image; if so, determine the height and width of the third target box corresponding to the target object in the third target image as the target height and the target width; if not, determine the height and width of the third target box corresponding to the target object in the third target image and the height and width of the fourth target box corresponding to the target object in the first target image as the target height and the target width corresponding to the first target box of the target object in the current image, respectively.

[0042] Furthermore, the output module is specifically used to input the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model, and output the first target category corresponding to the target object, the first target confidence corresponding to the target object, and the predicted target frame corresponding to the target object;

[0043] The processing module is further used to update the predicted target frame to the first target frame.

[0044] Furthermore, the determination module is further configured to determine that the target object tracking fails if the intersection-over-union ratio is not greater than a preset intersection-over-union ratio threshold, or the second target confidence is not greater than a preset confidence threshold.

[0045] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the tracking methods described above when executing a computer program stored in a memory.

[0046] The present application also provides a computer-readable storage medium, which stores a computer program executable by a terminal. When the program runs on the terminal, the terminal executes the steps of any one of the above-mentioned tracking methods.

[0047] In the present application, the target predicted center position of the target object in the current image is determined according to the pre-saved center position information of the target object to be tracked in other images collected before the current image, and each first target frame in which the target predicted center position is located in the current image is determined. For each first target frame, the sub-image enclosed by the first target frame in the current image is input into a pre-trained tracking model, and the first target category corresponding to the target object and the first target confidence corresponding to the target object are output. Among the first target confidences corresponding to the first target category consistent with the second target category of the pre-saved target object, the largest first target confidence is determined as the second target confidence. According to the coordinate information of the first target frame and the second target frame corresponding to the target object in the pre-saved first target image collected before the current image and adjacent to the current image, the intersection-over-union ratio of the first target frame and the second target frame is determined. If the intersection-over-union ratio is greater than a preset intersection-over-union ratio threshold and the second target confidence is greater than a preset confidence threshold, the object at the target predicted center position in the current image is determined to be the tracked target object. On the one hand, the present application does not need to save the template image, but only needs to save the central position information of the target object in the image to achieve tracking of the target object. Compared with the image, digital data such as position information occupies less memory; on the other hand, since the present application does not need to track based on the saved template image, there will be no problem of affecting the tracking quality due to the low quality of the template image. On the other hand, the present application tracks based on a deep learning detection algorithm, that is, it tracks based on a pre-trained tracking model. Compared with the non-deep learning detection algorithm in the prior art, it has stronger generalization and higher tracking quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 A schematic diagram of a tracking process provided by some embodiments of the present application;

[0050] Figure 2 A schematic diagram of a target object tracking process provided in some embodiments of the present application;

[0051] Figure 3a A schematic diagram of displaying a target object in a first target image provided by some embodiments of the present application;

[0052] Figure 3b A display intent after screening target items provided in some embodiments of the present application;

[0053] Figure 4 A schematic diagram of displaying a first target frame expansion provided in some embodiments of the present application;

[0054] Figure 5 A schematic diagram of a tracking process provided for some embodiments of the present application;

[0055] Figure 6 A schematic diagram of the structure of a tracking device provided in some embodiments of the present application;

[0056] Figure 7 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0058] In the present application, the target predicted center position of the target object in the current image is determined according to the pre-saved center position information of the target object to be tracked in other images collected before the current image, and each first target frame in which the target predicted center position is located in the current image is determined. For each first target frame, the sub-image enclosed by the first target frame in the current image is input into a pre-trained tracking model, and the first target category corresponding to the target object and the first target confidence corresponding to the target object are output. Among the first target confidences corresponding to the first target category consistent with the second target category of the pre-saved target object, the largest first target confidence is determined as the second target confidence. According to the coordinate information of the first target frame and the second target frame corresponding to the target object in the pre-saved first target image collected before the current image and adjacent to the current image, the intersection-over-union ratio of the first target frame and the second target frame is determined. If the intersection-over-union ratio is greater than a preset intersection-over-union ratio threshold and the second target confidence is greater than a preset confidence threshold, the object at the target predicted center position in the current image is determined to be the tracked target object.

[0059] In order to avoid the problem of storing too many template images and occupying a large amount of memory, and to improve the quality of tracking objects, the embodiments of the present application provide a tracking method, apparatus, device and medium.

[0060] Embodiment 1:

[0061] Figure 1 A schematic diagram of a tracking process provided in some embodiments of the present application, the process includes the following steps:

[0062] S101: Determine the target predicted center position of the target object in the current image based on pre-saved center position information of the target object to be tracked in other images captured before the current image; and determine each first target frame in the current image where the target predicted center position is located.

[0063] The tracking method provided in the embodiment of the present application is applied to electronic devices, which may be smart terminals, PCs, servers, image acquisition devices, etc. The tracking method provided in the present application can be applied in various tracking scenarios, such as intelligent traffic monitoring scenarios, high-altitude panoramic video monitoring and tracking scenarios, etc.

[0064] The current image and other images involved in the embodiments of the present application may be captured by the electronic device itself, or may be captured by other devices and then sent to the electronic device.

[0065] The target items to be tracked involved in the embodiments of the present application may be all items contained in an image (first target image) that is adjacent to the current image and acquired before the current image, or may be items in the first target image that are determined based on a target detection algorithm and that are not successfully matched with items in the current image. In other words, all items in the first target image may be tracked based on the tracking method provided by the present application, or items in the first target image that are successfully matched with items in the current image may be detected based on a target detection algorithm, and then other items in the first target image other than the successfully matched items may be determined as target items to be tracked, and the target items may be tracked based on the tracking method provided by the present application. For ease of description, the target items are described below by taking the example of other items in the first target image other than the successfully matched items.

[0066] Since general objects move at a limited speed, and for each object within the acquisition range of the image acquisition device, within a set time length, each image continuously acquired by the image acquisition device may contain the object, and the difference between the corresponding position information of the object in any two images with adjacent acquisition times will not be too large. Therefore, in order to track the target object to be tracked, in an embodiment of the present application, the electronic device can pre-save the center position information of the target object in other images acquired before the current image, and then determine the target predicted center position of the target object in the current image based on the center position information of the target object in other images acquired before the current image. Among them, the other images can be 2, 3, etc. Specifically, the number corresponding to the other images can be set according to needs. Among them, the center position information includes the horizontal coordinate of the center position and the vertical coordinate of the center position.

[0067] In order to determine the target predicted center position of the target object in the current image, in a possible implementation, every two adjacent other images of the target object to be tracked in the other images collected before the current image that are pre-saved can be obtained, and according to the center position information corresponding to the target object in the two adjacent other images, the difference of the horizontal coordinate corresponding to the center position of the target object in the two adjacent other images and the difference of the vertical coordinate corresponding to the center position are determined. Then, according to the difference of the horizontal coordinate corresponding to the center position of every two adjacent other images in the other images, the average horizontal coordinate difference is determined, and according to the difference of the vertical coordinate corresponding to the center position of every two adjacent other images, the average vertical coordinate difference is determined, and finally, the sum of the horizontal coordinate of the center position in the center position information corresponding to the target object in the first target image and the difference of the average horizontal coordinate is determined as the target predicted center horizontal coordinate of the target predicted center position of the target object in the current image, and the sum of the vertical coordinate of the center position in the center position information corresponding to the target object in the first target image and the difference of the average vertical coordinate is determined as the target predicted center vertical coordinate of the target predicted center position of the target object in the current image.

[0068] Since the size of objects is generally limited, for each object, the corresponding size of the object in the image captured by any image acquisition device is also limited. In an embodiment of the present application, after determining the target predicted center position of the target object in the current image, each first target frame where the target predicted center position is located in the current image can be determined, wherein the first target frame surrounds the target object.

[0069] For each first target frame, the first target frame can be a square frame, a matrix frame, etc. Specifically, the shape of the first target frame can be set according to requirements. The size of the first target frame is related to the installation position of the image acquisition device, the distance and angle of the target object from the image acquisition device, etc. Specifically, the size of the first target frame can be set according to actual experience.

[0070] For example, if it is determined based on actual experience that the width of the target box corresponding to most objects in the image is a and the height is b, or the width is c and the height is d, then two first target boxes of different sizes can be determined. The two first target boxes are respectively a target box with a width of a and a height of b centered on the target prediction center position, and a target box with a width of c and a height of d centered on the target prediction center position.

[0071] S102: For each first target frame, the sub-image enclosed by the first target frame in the current image is input into a pre-trained tracking model, and the first target category corresponding to the target object in the sub-image enclosed by the first target frame and the first target confidence corresponding to the target object are output.

[0072] In order to determine whether the object at the target prediction center position in the current image is the tracked target object, in the embodiment of the present application, a tracking model is pre-trained to detect the first target category corresponding to the target object to be detected and the first target confidence corresponding to the target object. The pre-trained tracking model can be a regional convolutional neural network (Region-CNN, RCNN) model, a real-time target detection (You Only Look Once, YOLO) model, an SSD model, etc.

[0073] In order to detect the first target category corresponding to the target item to be detected and the first target confidence corresponding to the target item, for each first target frame, the sub-image surrounded by the first target frame in the current image is input into a pre-trained tracking model, and the pre-trained tracking model outputs the first target category corresponding to the target item in the sub-image surrounded by the first target frame and the first target confidence corresponding to the target item. The first target category can be a person, a car, etc. The higher the first target confidence, the more accurate the predicted first target category corresponding to the target item.

[0074] Among them, for each first target frame, the pre-trained tracking model outputs that the first target category corresponding to the target object in the sub-image surrounded by the first target frame can be one or more, and the first target confidence corresponding to the target object can be one or more.

[0075] For example, the target item is item A, and the number of first target frames determined is 2. For the first first target frame, after the sub-image enclosed by the first first target frame is input into the pre-trained tracking model, the pre-trained tracking model outputs that the predicted category corresponding to item A in the sub-image enclosed by the first target frame is apple, and the corresponding confidence level of apple is 0.9, and the predicted category corresponding to the target item is pear, and the corresponding confidence level of pear can be 0.2. After the sub-image enclosed by the second first target frame is input into the pre-trained tracking model, the pre-trained tracking model outputs that the predicted category corresponding to item A in the sub-image enclosed by the second target frame is apple, and the corresponding confidence level of apple is 0.85, and the predicted category corresponding to the target item is orange, and the corresponding confidence level of pear can be 0.1.

[0076] S103: Determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target object stored in advance as the second target confidence.

[0077] In an embodiment of the present application, in order to facilitate tracking of the target object, after obtaining the first target category corresponding to the target object in each sub-image surrounded by the sub-target box output by the pre-trained tracking model and the first target confidence corresponding to the target object, the largest first target confidence among the first target confidences corresponding to the first target category that is consistent with the pre-saved second target category of the target object is determined as the second target confidence.

[0078] For example, the target item is item A, and the pre-saved second target type of item A is apple. The pre-trained tracking model outputs four first target categories and four first target confidences, namely: the first target category is apple, and the first target confidence is 0.9; the first target category is pear, and the first target confidence is 0.1; the first target category is apple, and the first target confidence is 0.95; the first target category is orange, and the first target confidence is 0.2. Then, the first target confidences corresponding to the first target category consistent with the pre-saved second target category of the target item are determined to be 0.9 and 0.95, and the second target confidence is determined to be 0.95.

[0079] S104: Determine an intersection-and-union ratio (IoU) of the first target frame and the second target frame based on coordinate information of the first target frame and a pre-saved second target frame corresponding to the target object in a first target image that is captured before the current image and adjacent to the current image; if the IoU is greater than a preset IoU threshold and the second target confidence is greater than a preset confidence threshold, determine that the object at the predicted center position of the target in the current image is the tracked target object.

[0080] In order to determine whether the object at the target prediction center position in the current image is the target object to be tracked, in an embodiment of the present application, the intersection-and-union ratio of the first target frame and the second target frame can be determined based on the coordinate information of the first target frame and the coordinate information of the second target frame corresponding to the target object in the pre-saved first target image to characterize the similarity between the first target frame and the second target frame. The larger the intersection-and-union ratio, the greater the similarity between the sub-image surrounded by the first target frame and the sub-image surrounded by the second target frame, and the greater the possibility that the object at the target prediction center position is the target object to be tracked.

[0081] Since the first target frame is located in the current image, the second target frame is located in the first target image, and the current image and the first target image are both acquired by the same image acquisition device, it can be considered that each corresponding pixel point in the current image and the first target image should correspond to the same, and the coordinate system of the current image and the first target image is the same. In order to more easily determine the intersection-and-union ratio between the first target frame and the second target frame, in an embodiment of the present application, the first target frame and the second target frame can be placed in the same coordinate system based on the coordinate information of the first target frame and the coordinate information of the second target frame, and then the first area corresponding to the intersection of the first target frame and the second target frame is determined, and the second area corresponding to the union of the first target frame and the second target frame is determined, and finally the ratio of the first area to the second area is determined as the intersection-and-union ratio of the first target frame and the second target frame.

[0082] In order to determine whether the object at the target prediction center position in the current image is the target object to be tracked, in an embodiment of the present application, a preset intersection-and-union ratio threshold and a preset confidence threshold are pre-saved. After the intersection-and-union ratio of the first target frame and the second target frame is determined, it is determined whether the intersection-and-union ratio is greater than the preset intersection-and-union ratio threshold, and whether the second target confidence output by the pre-trained tracking model is greater than the preset second confidence threshold. If the intersection-and-union ratio is greater than the preset intersection-and-union ratio threshold, and the second target confidence is greater than the preset confidence threshold, it means that the object at the target prediction center position is very likely to be the target object to be tracked, so the object at the target prediction center position in the current image can be determined as the target object to be tracked.

[0083] The preset confidence threshold may be 0.9, 0.85, or other values ​​greater than 0 and less than 1. Specifically, the preset confidence threshold may be set as required. The preset intersection-over-union ratio threshold may be a value greater than 0 and less than 1. Specifically, the preset intersection-over-union ratio threshold may also be set as required.

[0084] S105: If the intersection-over-union ratio is not greater than a preset intersection-over-union ratio threshold, or the second target confidence is not greater than a preset confidence threshold, it is determined that the target object tracking fails.

[0085] If the intersection-and-union ratio of the first target frame and the second target frame is not greater than the preset intersection-and-union ratio threshold, or the second target confidence is not greater than the preset confidence threshold, it means that the possibility that the object at the target prediction center position is the tracked target object is very small, and therefore it is determined that the tracking of the target object has failed.

[0086] Figure 2 A schematic diagram of a target object tracking process provided in some embodiments of the present application is now directed to Figure 2 Provide explanation.

[0087] After determining the second target confidence and the intersection-and-union ratio of the first target frame and the second target frame, determine whether the second target confidence corresponding to the target object is greater than a preset confidence threshold. If it is not greater than the preset confidence threshold, determine that the tracking has failed. If the second target confidence is greater than the preset confidence threshold, determine whether the intersection-and-union ratio is greater than the preset intersection-and-union ratio threshold. If it is greater than the preset intersection-and-union ratio threshold, determine that the tracking is successful. If it is not greater than the preset intersection-and-union ratio threshold, determine that the tracking has failed.

[0088] On the one hand, the present application does not need to save the template image, but only needs to save the central position information of the target object in the image to achieve tracking of the target object. Compared with the image, digital data such as position information occupies less memory; on the other hand, since the present application does not need to track based on the saved template image, there will be no problem of affecting the tracking quality due to the low quality of the template image. On the other hand, the present application tracks based on a deep learning detection algorithm, that is, it tracks based on a pre-trained tracking model. Compared with the non-deep learning detection algorithm in the prior art, it has stronger generalization and higher tracking quality.

[0089] Embodiment 2:

[0090] In order to determine the target object, based on the above embodiment, in the embodiment of the present application, the method further includes: determining that the target object is before the target predicted center position in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image, and:

[0091] Based on the target detection algorithm, it is identified whether there is an object in the first target image that has not been successfully matched with the current image; if so, the unsuccessfully matched object is determined as the target object.

[0092] In an embodiment of the present application, in order to determine the target object, the objects in the first target image are first matched with the objects in the current image based on a target detection algorithm, and the objects in the first target image that have not been successfully matched are determined as target objects to be matched.

[0093] Specifically, based on the target detection algorithm, the category corresponding to each object in the first target image and the current image and the position information of the target frame corresponding to each object are first detected. For the convenience of description, the object in the first target image is called the first object, the object in the current image is called the second object, the target frame corresponding to the first object is called the first frame, and the target frame corresponding to the second object is called the second frame. Then, for each first object in the first target image, it is determined whether there is a second object in the current image that has the same category as the first object and the intersection and union ratio of the corresponding second frame to the first frame is greater than a preset intersection and union ratio threshold. If not, it is determined that the first object is not matched successfully, and the first object is determined as the target object. If it exists, it is determined that the first object is matched successfully, and the first object and the second object that successfully matches the first object are determined to be the same object, and the same identification information is set.

[0094] The process of determining the intersection-over-union ratio of two target frames has been described in the above embodiment and will not be repeated here.

[0095] Embodiment 3:

[0096] In order to screen the target objects to be tracked to reduce the workload of the electronic device, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0097] Determine whether the first identification information corresponding to the target object in the first target image stored in advance represents whether the target object is tracked according to the target detection algorithm, and determine whether the target object is located in the central area of ​​the first target image;

[0098] If the first identification information indicates that the target object is not tracked according to the target detection algorithm, or the target object is located in the central area of ​​the first target image, retaining the target object;

[0099] Otherwise, the target item is deleted.

[0100] In scenarios where the images captured by the image acquisition device contain a large number of objects, for example, in high-altitude panoramic video surveillance and tracking scenarios, since the image acquisition device is installed at a high position, the number of objects contained in the captured panoramic images is generally large, and the corresponding size of each object in the panoramic image is very small. Therefore, in order to reduce the workload of the electronic equipment, the target objects in the first target image can be screened first, and the target objects in the first target image that are not meaningful to track or do not need to be tracked can be deleted, and the target objects that are meaningful to track or need to be tracked can be retained, so as to reduce the number of target objects that need to be tracked, thereby reducing the workload of the electronic equipment.

[0101] On the one hand, if the object has moved to a range that cannot be captured by the image acquisition device when the current image is captured, even if no object that successfully matches the object in the first target image is found in the current image, there is no need to track the target object. Generally, the object that is about to leave the acquisition area of ​​the image acquisition device will not be in the central area of ​​the image. On the other hand, when installing the image acquisition device, when adjusting the installation position and installation angle of the image acquisition device, it is often necessary to consider the sub-image corresponding to the center of the image captured by the image acquisition device as the area of ​​most concern. On the other hand, for the high-altitude panoramic video monitoring and tracking scene, the panoramic image that may be obtained is generated by splicing multiple images, and the appearance of the object in the center is more likely to be deformed, and the object in the center is more likely to be missed during detection. Therefore, the target object located in the central area of ​​the first target image can be retained, and the target image located in the edge area of ​​the image in the first target image can be deleted.

[0102] In a possible implementation manner, in order to determine whether the target object is located in the central area of ​​the first target image, it can be determined in the following manner:

[0103] First, determine the target value according to Dist = abs(obj_center_x–img_center_x)+abs(obj_center_y–img_center_y);

[0104] Wherein, Dist is the target value, abs represents the absolute value operation, obj_center_x is the horizontal coordinate of the center point of the second target frame corresponding to the target object in the first target image, obj_center_y is the vertical coordinate of the center point of the second target frame corresponding to the target object in the first target image, img_center_x is the horizontal coordinate of the center point of the first target image, and img_center_y is the vertical coordinate of the center point of the first target image. The smaller the Dist is, the greater the possibility that the target object is located in the central area of ​​the first target image.

[0105] After the target value is determined, it can be determined whether the target object is located in the central area of ​​the first target image based on a comparison between the target value and a preset value threshold. Specifically, if the target value is less than the preset value threshold, it is determined that the target object is located in the central area of ​​the first target image, and the target object is retained; if the target value is not less than the preset value threshold, it is determined that the target object is not located in the central area of ​​the first target image, and the target object is deleted.

[0106] In the embodiment of the present application, in order to facilitate the determination of the algorithm by which each object in each image is tracked, the electronic device pre-stores the first identification information corresponding to the object in each image, and the first identification information is used to characterize whether the object is tracked according to the target detection algorithm, wherein the first identification information corresponding to the objects tracked using the same algorithm is the same, and the first identification information corresponding to the objects tracked using different algorithms is different. Specifically, the first identification information of the object tracked based on the target detection algorithm can be set to 1, and the first identification information of the object tracked not based on the target detection algorithm can be set to 0, and so on, without specific limitation.

[0107] If the first identification information corresponding to the pre-saved target object in the first target image indicates that the target object is not tracked according to the target detection algorithm, that is, it is tracked based on the tracking method of the present application, in order to ensure the continuity of the tracking trajectory, the target object can also be retained. If the first identification information corresponding to the pre-saved target object in the first target image indicates that the target object is tracked according to the target detection algorithm, the target object is not tracked in the current image because it has moved to a range that the image acquisition device cannot capture, resulting in failure to track. In this case, the target object can be deleted first.

[0108] It should be noted that if the number of target objects in the first target image is small, the target objects in the first target image may not be screened by the method of the above embodiment, and all target objects in the first target image may be directly tracked. Specifically, the electronic device may pre-store a screening threshold, and after determining all target objects in the first target image, determine the number of targets corresponding to all target objects, and compare the target number with the preset screening threshold. If the target number is greater than the preset screening threshold, the target objects in the first target image are screened by the method of the above embodiment. If the target number is not greater than the preset screening threshold, all target objects in the first target image are retained.

[0109] Figure 3a A schematic diagram of displaying a target object in a first target image provided by some embodiments of the present application, Figure 3b A display intention after screening target items is provided in some embodiments of the present application. Figure 3a and Figure 3b Provide explanation.

[0110] The target objects in the first target image include object A, object B, and object C. Figure 3a shown.

[0111] An example is given in which the first identification information of an object tracked based on a target detection algorithm is set to 1, and the first identification information of an object tracked not based on a target detection algorithm is set to 0.

[0112] If the first identification information corresponding to item A is 0, the first identification information corresponding to item B is 1, and the first identification information corresponding to item C is 1, and among items A, B and C, item C is located in the central area of ​​the first target image, and items A and B are located in the non-central area of ​​the first target image, then items A and C are retained, and item B is deleted. For details, the target objects contained in the first target image after the target items are screened are shown in Figure 3b .

[0113] Embodiment 4:

[0114] In order to determine the target predicted center position of the target object in the current image, based on the above embodiments, the method of determining the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image includes:

[0115] The predicted center position of the target is determined according to the center position information of the target object in the first target image and the center position information of a second target image acquired before the first target image and adjacent to the first target image.

[0116] In an embodiment of the present application, in order to determine the target predicted center position of the target object in the current image, the electronic device can determine the target predicted center position based on the center position information of the target object in the first target image and the center position information of the second target image captured before the first target image and adjacent to the first target image.

[0117] Specifically, a first difference between the horizontal coordinate of the center position corresponding to the target object in the first target image and the horizontal coordinate of the center position corresponding to the target object in the second target image can be determined, and a second difference between the vertical coordinate of the center position corresponding to the target object in the first target image and the vertical coordinate of the center position corresponding to the target object in the second target image can be determined. Then, the horizontal coordinate of the center position of the target object in the first target image is added to the first difference to be determined as the horizontal coordinate of the predicted center position of the target, and the vertical coordinate of the center position of the target object in the first target image is added to the second difference to be determined as the vertical coordinate of the predicted center position of the target.

[0118] Specifically, the target prediction center position can be determined by the following methods:

[0119] center_pred_x=center_x_1+(center_x_1–center_x_2)

[0120] center_pred_y=center_y_1+(center_y_1–center_y_2)

[0121] Among them, center_pred_x is the horizontal coordinate of the predicted center position of the target, center_pred_y is the vertical coordinate of the predicted center position of the target, center_x_1 is the horizontal coordinate of the center position of the target item in the first target image, center_y_1 is the vertical coordinate of the center position of the target item in the first target image, center_x_2 is the horizontal coordinate of the center position of the target item in the second target image, and center_y_2 is the vertical coordinate of the center position of the target item in the second target image.

[0122] Embodiment 5:

[0123] In order to more accurately determine the first target frame, based on the above embodiments, before determining each first target frame where the target prediction center position in the current image is located, the method further includes:

[0124] According to the width and height of the third target frame corresponding to the target object in the other image, the target width and target height of each first target frame corresponding to the target object in the current image are determined.

[0125] Since the sizes of the target frames corresponding to objects of different sizes in the current image are different, and for each object within the acquisition range of the image acquisition device, the size of the target frame corresponding to the object contained in each image acquired within the set time length will not change significantly, therefore, in order to more accurately determine each first target frame corresponding to the target object in the current image, in an embodiment of the present application, the target width and target height of each first target frame corresponding to the target object in the current image can be determined based on the width and height of the third target frame corresponding to the target object in other images acquired before the current image.

[0126] If the number of determined first target frames is one, in a possible implementation, the average width and the average height can be determined based on the width and height of the corresponding third target frames in other images, and then the average height is determined as the target height corresponding to the first target frame, and the average width is determined as the target width corresponding to the first target frame.

[0127] In another possible implementation, the width and height of the third target frame corresponding to the target object in the first target image may be determined as the target width and target height corresponding to the first target frame.

[0128] If the number of determined first target frames is multiple, for example, the number of determined first target frames is two, then in a possible implementation, the average width and the average height determined based on the width and height of the corresponding third target frame in the first half image of other images can be determined as the target width and target height of one first target frame, and the average width and the average height determined based on the width and height of the corresponding third target frame in the second half image of other images can be determined as the target width and target height of another first target frame.

[0129] In another possible implementation, the width and height of the third target frame corresponding to the target item in the first target image can be determined as the target width and target height corresponding to a first target frame, and the width and height of the third target frame corresponding to the target item in other images that were acquired before the first target image and are adjacent to the first target image can be determined as the target width and target height corresponding to another first target frame.

[0130] In order to accurately determine the target width and target height corresponding to the first target frame, based on the above embodiment, determining the target width and target height of each first target frame corresponding to the target object in the current image according to the width and height of the third target frame corresponding to the target object in the other image includes:

[0131] The second identification information corresponding to the target object in other images indicates that the target object is tracked according to the target detection algorithm, and other images that are most recently acquired in time with the current image are determined as third target images;

[0132] determining whether the third target image and the first target image are the same image;

[0133] If yes, determining the height and width of the third target frame corresponding to the target object in the third target image as the target height and the target width;

[0134] If not, the height and width of the third target frame corresponding to the target object in the third target image, and the height and width of the fourth target frame corresponding to the target object in the first target image, are respectively determined as the target height and the target width corresponding to the first target frame of the target object in the current image.

[0135] In an embodiment of the present application, in order to more accurately determine the target width and target height of the first target frame, the second identification information corresponding to the target object in other images captured before the current image can also be used to characterize the target object as tracked according to the target detection algorithm, and the image closest to the current image acquisition time can be determined as the third target image, and it is determined whether the third target image is the same image as the first target image. If so, a first target frame can be determined, and the height and width of the third target frame corresponding to the target object in the third target image can be determined as the target height and the target width of the first target frame.

[0136] If not, two first target frames can be determined, and the object can be tracked subsequently based on the two first target frames, and the height and width of the third target frame corresponding to the target object in the third target image, and the height and width of the fourth target frame corresponding to the target object in the first target image, are respectively determined as the target height and the target width corresponding to the first target frame of the target object in the current image. That is to say, of the two first target frames, the target width and target height of one first target frame are the height and width of the third target frame corresponding to the target object in the third target image, and the target width and target height of the other first target frame are the height and width of the fourth target frame corresponding to the target object in the first target image.

[0137] The method for determining the predicted center position of a target provided in the embodiment of the present application is not only applicable to situations where the size of the target object remains basically unchanged during movement, but also to situations where the target object is deformed due to perspective reasons or image splicing, or where the size of the object gradually increases or decreases during movement.

[0138] Embodiment 6:

[0139] In order to more accurately determine the first target frame corresponding to the target object in the current image, based on the above embodiments, in the embodiment of the present application, the sub-image surrounded by the first target frame in the current image is input into a pre-trained tracking model, and the first target category corresponding to the target object and the first target confidence corresponding to the target object are outputted, including:

[0140] Inputting the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model, and outputting a first target category corresponding to the target object, a first target confidence corresponding to the target object, and a predicted target frame corresponding to the target object;

[0141] Before determining the intersection-over-combination ratio of the first target frame and the second target frame according to the first target frame and pre-saved coordinate information of the second target frame corresponding to the target object in the first target image acquired before the current image and adjacent to the current image, the method further includes:

[0142] The predicted target frame is updated to the first target frame.

[0143] In order to more accurately determine the first target frame corresponding to the target object in the current image, in an embodiment of the present application, after determining each first target frame where the target prediction center position of the target object in the current image is located, the predicted target frame corresponding to the target object can be output based on a pre-trained tracking model to further predict the target frame corresponding to the target object in the current image. Specifically, for each first target frame, the sub-image surrounded by the first target frame in the current image is input into the pre-trained tracking model, and the pre-trained tracking model outputs the first target category corresponding to the target object, the first target confidence corresponding to the target object, and the predicted target frame corresponding to the target item, and then the electronic device updates the predicted target frame to the first target frame.

[0144] Specifically, since for each first target frame, the first target frame corresponds to a first target category, a first target confidence and a prediction frame, in order to facilitate the subsequent tracking of the target item, in an embodiment of the present application, the maximum value of the first target confidence corresponding to each prediction frame can be first determined, and the prediction frame corresponding to the maximum value of the first target confidence can be determined as the predicted target frame, and the predicted target frame can be updated to the first target frame.

[0145] It should be noted that since the first target box corresponding to the target object in the determined current image may not contain the complete target object, in order to ensure that the sub-image input into the tracking model to be pre-trained contains the complete target object so as to improve the accuracy of model prediction, in an embodiment of the present application, before the sub-image surrounded by the first target box is input into the pre-trained tracking model, the first target box can be expanded by a certain proportion. Specifically, a corresponding expansion factor can be set in advance, and then the first target box is expanded according to the expansion factor, and the target box obtained after the expansion is updated to the first target box, and then the sub-image surrounded by the first target box in the current image is input into the pre-trained tracking model.

[0146] Specifically, the expanded target frame can be obtained by the following methods:

[0147] patch1=(center_pred_x,center_pred_y,height_s*expand_factor,width_s*expand_factor)

[0148] Among them, patch1 is the information of the target frame after expansion, wherein the expansion factor is expand_factor, center_pred_x is the horizontal coordinate of the center position of the first target frame before expansion, center_pred_y is the vertical coordinate of the center position of the first target frame before expansion, height_s is the height of the first target frame before expansion, and width_s is the width of the first target frame before expansion.

[0149] Figure 4 A schematic diagram of a display of a first target frame expansion provided in some embodiments of the present application is now directed to Figure 4 Provide explanation.

[0150] The first target frame before the expansion is target frame 1, and the first target frame after the expansion is target frame 2.

[0151] On the one hand, since the image input into the pre-trained tracking model is a sub-image surrounded by the first target frame in the current image, compared with directly inputting the current image into the pre-trained tracking model, the sub-image has fewer interference factors, wherein the interference factors include the background area in the current image and other objects other than the target object; on the other hand, since the pre-trained tracking model is based on the input image, when predicting the first target category corresponding to the object in the image, the first target confidence corresponding to the target object, and the predicted target frame, it is generally necessary to reduce the input image to a set input size, which is more likely to cause inaccurate prediction of small-sized objects. In the present application, the sub-image surrounded by the first target frame in the current image is input into the pre-trained tracking model, so that the pixel proportion of small-sized target objects in the sub-image is higher, thereby having a better prediction effect; on the other hand, since there are few objects in the sub-image surrounded by the first target frame in the current image, a lightweight network structure (smaller network input, shallower network depth, etc.) can be used as the pre-trained tracking model, based on which a good prediction effect can be achieved.

[0152] Figure 5 A schematic diagram of a tracking process provided in some embodiments of the present application is now directed to Figure 4 Provide explanation.

[0153] First, determine the target object to be identified and screen the target object. Then, determine the target predicted center position of the target object in the current image based on the pre-saved center position information of the target object in other images collected before the current image, and determine each first target frame in the current image where the target predicted center position is located. Then, for each first target frame, input the sub-image surrounded by the first target frame in the current image into the pre-trained tracking model, and make a second prediction for the target object in the target frame corresponding to the current image. Finally, determine whether the object at the target predicted center position in the current image is the tracked target object based on the prediction result.

[0154] Embodiment 7:

[0155] Figure 6 A schematic diagram of a tracking device structure provided in some embodiments of the present application, the device comprising:

[0156] A determination module 601 is used to determine the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image; and determine each first target frame in the current image where the target predicted center position is located;

[0157] An output module 602 is used to input, for each first target frame, a sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output a first target category corresponding to the target object in the sub-image enclosed by the first target frame and a first target confidence corresponding to the target object;

[0158] The determination module 601 is also used to determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target item saved in advance as the second target confidence; determine the intersection-and-union ratio of the first target frame and the second target frame according to the coordinate information of the first target frame and the second target frame corresponding to the target item in the first target image collected before the current image and adjacent to the current image in advance; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, and the second target confidence is greater than a preset confidence threshold, then determine that the object at the target prediction center position in the current image is the tracked target item.

[0159] In a possible implementation, the determination module 601 is further configured to identify, based on a target detection algorithm, whether there is an object in the first target image that has not been successfully matched with the current image; if so, determining the unsuccessfully matched object as the target object.

[0160] In a possible implementation, the device further includes:

[0161] The processing module 603 is used to determine whether the first identification information corresponding to the target object in the first target image stored in advance represents that the target object is tracked according to the target detection algorithm, and determine whether the target object is located in the central area of ​​the first target image; if the first identification information represents that the target object is not tracked according to the target detection algorithm, or the target object is located in the central area of ​​the first target image, the target object is retained; otherwise, the target object is deleted.

[0162] In a possible implementation, the determination module 601 is specifically used to determine the predicted center position of the target based on the center position information of the target object in the first target image and the center position information of a second target image acquired before the first target image and adjacent to the first target image.

[0163] In a possible implementation, the determination module 601 is specifically configured to determine the target width and target height of each first target frame corresponding to the target object in the current image based on the width and height of the third target frame corresponding to the target object in the other images.

[0164] In a possible implementation, the output module 602 is specifically used to determine other images that are closest to the current image in terms of acquisition time as a third target image, using the second identification information corresponding to the target object in other images to characterize that the target object is tracked according to the target detection algorithm; determine whether the third target image is the same image as the first target image; if so, determine the height and width of a third target frame corresponding to the target object in the third target image as the target height and the target width; if not, determine the height and width of the third target frame corresponding to the target object in the third target image, and the height and width of the fourth target frame corresponding to the target object in the first target image, as the target height and the target width corresponding to the first target frame of the target object in the current image, respectively.

[0165] In a possible implementation, the output module 602 is specifically configured to input the sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output a first target category corresponding to the target object, a first target confidence corresponding to the target object, and a predicted target frame corresponding to the target object;

[0166] The processing module 603 is further configured to update the predicted target frame to the first target frame.

[0167] In a possible implementation, the determination module 601 is further configured to determine that the target object tracking has failed if the intersection-over-union ratio is not greater than a preset intersection-over-union ratio threshold, or the second target confidence is not greater than a preset second confidence threshold.

[0168] Embodiment 8:

[0169] Based on the above embodiments, some embodiments of the present application further provide an electronic device, such as Figure 7 As shown, it includes: a processor 701 , a communication interface 702 , a memory 703 and a communication bus 704 , wherein the processor 701 , the communication interface 702 , and the memory 703 communicate with each other via the communication bus 704 .

[0170] The memory 703 stores a computer program. When the program is executed by the processor 701, the processor 701 performs the following steps:

[0171] Determine the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image; and determine each first target frame in the current image where the target predicted center position is located;

[0172] For each first target frame, input the sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output the first target category corresponding to the target object in the sub-image enclosed by the first target frame and the first target confidence corresponding to the target object;

[0173] Determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target object stored in advance as the second target confidence;

[0174] The intersection-and-union ratio of the first target frame and the second target frame is determined based on the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image that is captured before the current image and adjacent to the current image; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold and the second target confidence is greater than a preset confidence threshold, it is determined that the object at the target prediction center position in the current image is the tracked target object.

[0175] Furthermore, the processor 701 is also used to identify whether there is an object in the first target image that has not been successfully matched with the current image based on a target detection algorithm; if so, determine the unsuccessfully matched object as the target object.

[0176] Furthermore, the processor 701 is also used to determine whether the first identification information corresponding to the target object in the first target image saved in advance, indicating that the target object is tracked according to the target detection algorithm, and determine whether the target object is located in the central area of ​​the first target image; if the first identification information indicates that the target object is not tracked according to the target detection algorithm, or the target object is located in the central area of ​​the first target image, the target object is retained; otherwise, the target object is deleted.

[0177] Furthermore, the processor 701 is specifically used to determine the predicted center position of the target based on the center position information of the target object in the first target image and the center position information of a second target image acquired before the first target image and adjacent to the first target image.

[0178] Furthermore, the processor 701 is further configured to determine a target width and a target height of each first target frame corresponding to the target object in the current image according to a width and a height of a third target frame corresponding to the target object in the other images.

[0179] Further, the processor 701 is specifically used to characterize the target object in other images as a third target image using the second identification information corresponding to the target object in other images to track the target object according to the target detection algorithm, and to determine other images that are closest to the current image acquisition time as the third target image; determine whether the third target image is the same image as the first target image; if so, determine the height and width of the third target frame corresponding to the target object in the third target image as the target height and the target width; if not, determine the height and width of the third target frame corresponding to the target object in the third target image, and the height and width of the fourth target frame corresponding to the target object in the first target image, as the target height and the target width corresponding to the first target frame of the target object in the current image, respectively.

[0180] Further, the processor 701 is specifically configured to input the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model, and output a first target category corresponding to the target object, a first target confidence corresponding to the target object, and a predicted target frame corresponding to the target object;

[0181] The processor 701 is further configured to update the predicted target frame to the first target frame.

[0182] Furthermore, the processor 701 is further configured to determine that the target object tracking fails if the intersection-over-union ratio is not greater than a preset intersection-over-union ratio threshold, or the second target confidence is not greater than a preset second confidence threshold.

[0183] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0184] The communication interface 702 is used for communication between the electronic device and other devices.

[0185] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0186] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0187] Embodiment 9:

[0188] On the basis of the above embodiments, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device, and when the program is run on the electronic device, the electronic device implements the following steps when executing:

[0189] Determine the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image; and determine each first target frame in the current image where the target predicted center position is located;

[0190] For each first target frame, input the sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output the first target category corresponding to the target object in the sub-image enclosed by the first target frame and the first target confidence corresponding to the target object;

[0191] Determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target object stored in advance as the second target confidence;

[0192] The intersection-and-union ratio of the first target frame and the second target frame is determined based on the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image that is captured before the current image and adjacent to the current image; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold and the second target confidence is greater than a preset confidence threshold, it is determined that the object at the target prediction center position in the current image is the tracked target object.

[0193] Furthermore, the method further comprises: determining that the target object is before the target predicted center position in the current image based on the pre-saved center position information of the target object to be tracked in other images collected before the current image;

[0194] Based on the target detection algorithm, it is identified whether there is an object in the first target image that has not been successfully matched with the current image; if so, the unsuccessfully matched object is determined as the target object.

[0195] Furthermore, the method further comprises:

[0196] Determine whether the first identification information corresponding to the target object in the first target image stored in advance represents whether the target object is tracked according to the target detection algorithm, and determine whether the target object is located in the central area of ​​the first target image;

[0197] If the first identification information indicates that the target object is not tracked according to the target detection algorithm, or the target object is located in the central area of ​​the first target image, retaining the target object;

[0198] Otherwise, the target item is deleted.

[0199] Furthermore, determining the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image includes:

[0200] The predicted center position of the target is determined according to the center position information of the target object in the first target image and the center position information of a second target image acquired before the first target image and adjacent to the first target image.

[0201] Furthermore, before determining each first target frame in which the target prediction center position is located in the current image, the method further includes:

[0202] According to the width and height of the third target frame corresponding to the target object in the other image, the target width and target height of each first target frame corresponding to the target object in the current image are determined.

[0203] Further, determining the target width and target height of each first target frame corresponding to the target object in the current image according to the width and height of the third target frame corresponding to the target object in the other image includes:

[0204] The second identification information corresponding to the target object in other images indicates that the target object is tracked according to the target detection algorithm, and other images that are most recently acquired in time with the current image are determined as third target images;

[0205] determining whether the third target image and the first target image are the same image;

[0206] If yes, determining the height and width of the third target frame corresponding to the target object in the third target image as the target height and the target width;

[0207] If not, the height and width of the third target frame corresponding to the target object in the third target image, and the height and width of the fourth target frame corresponding to the target object in the first target image, are respectively determined as the target height and the target width corresponding to the first target frame of the target object in the current image.

[0208] Furthermore, the step of inputting the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model and outputting the first target category corresponding to the target object and the first target confidence corresponding to the target object includes:

[0209] Inputting the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model, and outputting a first target category corresponding to the target object, a first target confidence corresponding to the target object, and a predicted target frame corresponding to the target object;

[0210] Before determining the intersection-over-combination ratio of the first target frame and the second target frame according to the first target frame and pre-saved coordinate information of the second target frame corresponding to the target object in the first target image acquired before the current image and adjacent to the current image, the method further includes:

[0211] The predicted target frame is updated to the first target frame.

[0212] Furthermore, after determining the intersection-over-combination ratio of the first target frame and the second target frame according to the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image acquired before the current image and adjacent to the current image, the method further includes:

[0213] If the intersection-over-union ratio is not greater than a preset intersection-over-union ratio threshold, or the second target confidence is not greater than a preset second confidence threshold, it is determined that the target object tracking fails.

[0214] On the one hand, the present application does not need to save the template image, but only needs to save the central position information of the target object in the image to achieve tracking of the target object. Compared with the image, digital data such as position information occupies less memory; on the other hand, since the present application does not need to track based on the saved template image, there will be no problem of affecting the tracking quality due to the low quality of the template image. On the other hand, the present application tracks based on a deep learning detection algorithm, that is, it tracks based on a pre-trained tracking model. Compared with the non-deep learning detection algorithm in the prior art, it has stronger generalization and higher tracking quality.

[0215] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0216] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0217] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0219] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A tracking method, characterized in that: The method comprises: Based on the target detection algorithm, identify whether there is an object in the first target image that has not been successfully matched with the current image; if so, determine the unsuccessfully matched object as the target object to be tracked; Determine the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image; and determine each first target frame in the current image where the target predicted center position is located; For each first target frame, input the sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output the first target category corresponding to the target object in the sub-image enclosed by the first target frame and the first target confidence corresponding to the target object; Determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target object stored in advance as the second target confidence; The intersection-and-union ratio of the first target frame and the second target frame is determined based on the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image that is captured before the current image and adjacent to the current image; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold and the second target confidence is greater than a preset confidence threshold, it is determined that the object at the target prediction center position in the current image is the tracked target object.

2. The method according to claim 1, characterized in that The method further comprises: Determine whether the first identification information corresponding to the target object in the first target image stored in advance represents whether the target object is tracked according to the target detection algorithm, and determine whether the target object is located in the central area of ​​the first target image; If the first identification information indicates that the target object is not tracked according to the target detection algorithm, or the target object is located in the central area of ​​the first target image, retaining the target object; Otherwise, the target item is deleted.

3. The method according to claim 1, characterized in that The step of determining the target predicted center position of the target object in the current image according to the pre-saved center position information of the target object to be tracked in other images collected before the current image comprises: The predicted center position of the target is determined according to the center position information of the target object in the first target image and the center position information of a second target image acquired before the first target image and adjacent to the first target image.

4. The method according to claim 1, characterized in that: Before determining each first target frame in the current image where the target prediction center position is located, the method further includes: According to the width and height of the third target frame corresponding to the target object in the other image, the target width and target height of each first target frame corresponding to the target object in the current image are determined.

5. The method according to claim 4, characterized in that Determining the target width and target height of each first target frame corresponding to the target object in the current image according to the width and height of the third target frame corresponding to the target object in the other image includes: The second identification information corresponding to the target object in other images indicates that the target object is tracked according to the target detection algorithm, and other images that are most recently acquired in time with the current image are determined as third target images; determining whether the third target image and the first target image are the same image; If yes, determining the height and width of the third target frame corresponding to the target object in the third target image as the target height and the target width; If not, the height and width of the third target frame corresponding to the target item in the third target image, and the height and width of the fourth target frame corresponding to the target item in the first target image, are respectively determined as the target height and the target width corresponding to the first target frame of the target item in the current image.

6. The method according to claim 1, characterized in that The step of inputting the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model and outputting the first target category corresponding to the target object and the first target confidence corresponding to the target object comprises: Inputting the sub-image surrounded by the first target frame in the current image into a pre-trained tracking model, and outputting a first target category corresponding to the target object, a first target confidence corresponding to the target object, and a predicted target frame corresponding to the target object; Before determining the intersection-over-combination ratio of the first target frame and the second target frame according to the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image acquired before the current image and adjacent to the current image, the method further includes: The predicted target frame is updated to the first target frame.

7. The method according to claim 1, characterized in that After determining the intersection-over-combination ratio of the first target frame and the second target frame according to the coordinate information of the first target frame and the pre-saved second target frame corresponding to the target object in the first target image acquired before the current image and adjacent to the current image, the method further includes: If the intersection-over-union ratio is not greater than a preset intersection-over-union ratio threshold, or the second target confidence is not greater than a preset second confidence threshold, it is determined that the target object tracking fails.

8. A tracking device, characterized in that: The device comprises: A determination module is used to identify whether there is an object in the first target image that has not been successfully matched with the current image based on a target detection algorithm; if so, determine the unsuccessfully matched object as a target object to be tracked; determine the target predicted center position of the target object in the current image based on pre-saved center position information of the target object to be tracked in other images collected before the current image; and determine each first target frame in the current image where the target predicted center position is located; An output module is used to input, for each first target frame, a sub-image enclosed by the first target frame in the current image into a pre-trained tracking model, and output a first target category corresponding to the target object in the sub-image enclosed by the first target frame and a first target confidence corresponding to the target object; The determination module is also used to determine the largest first target confidence among the first target confidences corresponding to the first target category consistent with the second target category of the target item saved in advance as the second target confidence; determine the intersection-and-union ratio of the first target frame and the second target frame according to the coordinate information of the first target frame and the second target frame corresponding to the target item in the first target image collected before the current image and adjacent to the current image in advance; if the intersection-and-union ratio is greater than a preset intersection-and-union ratio threshold, and the second target confidence is greater than a preset confidence threshold, then determine that the object at the target prediction center position in the current image is the tracked target item.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to implement the steps of the tracking method described in any one of claims 1 to 7 when executing the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the tracking method described in any one of claims 1 to 7.

Images