Target Tracking Method, Device, Electronic Terminal and Computer Readable Storage Medium

Abstract

The present invention provides a target tracking method, apparatus, electronic terminal, and computer-readable storage medium. The target tracking method includes extracting multi-granularity features of a target object in a current video frame to obtain detection information of the target object; the detection information includes part features of each detected part; matching the detection information of the target object with the historical trajectory data of each corresponding target in historical video frames before the current video frame to determine the target that matches the target object; and associating the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate current trajectory data of the target object. By matching the part features of each detected part of the target object with the historical trajectory data of each target in the historical video frames, this application reduces the influence of occluded parts or undetected parts on the target matching result, thereby improving the matching accuracy of the target object and the tracking stability of the target object.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular, to a target tracking method, device, electronic terminal, and computer-readable storage medium. Background Art

[0002] Target tracking is one of the most important and fundamental tasks in the field of computer vision. Its purpose is to output the position of the target object in each video frame of the video containing the target object. However, in complex scenarios such as target congestion, frequent changes in target poses, small targets in the distance, and inaccurate target bounding boxes, existing tracking algorithms face many problems. For example, when the target is occluded or affected by light and other interferences, the accuracy and stability of target tracking are not good. Summary of the Invention

[0003] The main technical problem to be solved by the present invention is to provide a target tracking method, device, electronic terminal, and computer-readable storage medium, so as to solve the problem of poor accuracy and stability of target tracking in the prior art.

[0004] To solve the above technical problem, the first technical solution adopted by the present invention is: to provide a target tracking method, the target tracking method includes:

[0005] Perform multi-granularity feature extraction on the target object in the current video frame to obtain detection information of the target object; the detection information includes part features of each detection part;

[0006] Match the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object;

[0007] Associate the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate the current trajectory data of the target object.

[0008] Among them, performing multi-granularity feature extraction on the target object in the current video frame to obtain detection information of the target object includes:

[0009] Perform target detection on the current video frame to obtain target detection boxes containing each target object;

[0010] Perform multi-granularity feature extraction on the target object in the target detection box through a multi-granularity feature extraction network to obtain a global feature map corresponding to the target object and part features of each detection part.

[0011] Among them, performing multi-granularity feature extraction on the target object in the current video frame to obtain detection information of the target object includes:

[0012] Perform object detection on the current video frame to obtain object detection bounding boxes containing each object;

[0013] Extract features of the objects in the object detection bounding boxes to obtain the global feature maps of the objects;

[0014] Perform pixel classification on the global feature maps of the objects to obtain the part categories of each pixel point in the global feature maps;

[0015] Aggregate the pixel points with the same part category to obtain the part features of the part category; the part category serves as the part category of the detected part.

[0016] Among them, performing pixel classification on the global feature maps of the objects to obtain the part categories of each pixel point in the global feature maps includes:

[0017] Classify each pixel point in the global feature maps of the objects to obtain the probability values of each pixel point for each preset category;

[0018] Select the preset category corresponding to the maximum probability value of the pixel point as the part category of the pixel point.

[0019] Among them, performing multi-granularity feature extraction on the objects in the current video frame to obtain the detection information of the objects further includes:

[0020] Based on the probability values of the pixel points corresponding to the part category of the detected part, determine whether the detected part is a visible part.

[0021] Among them, based on the probability values of the pixel points corresponding to the category of the detected part, determining whether the detected part is a visible part includes:

[0022] Select the maximum probability value among the probability values of all pixel points corresponding to the part category of the detected part and compare it with a preset value;

[0023] In response to the maximum probability value corresponding to the part category of the detected part being greater than the preset value, determine that the detected part is a visible part.

[0024] Among them, matching the detection information of the object with the historical trajectory data of each corresponding object in the historical video frames before the current video frame to determine the object matching the object includes:

[0025] Match the objects corresponding to the historical trajectory data in the first state with each object to obtain a first matching result; the first matching result includes object-object matching pairs, unmatched object objects, and unmatched historical trajectory data; the number of consecutive frames of the historical video frames containing the object in the historical trajectory data in the first state is not less than a preset number of frames;

[0026] Perform IOU matching on the unmatched target objects in the first matching result respectively with the unmatched historical trajectory data and each piece of historical trajectory data in the second state to obtain a second matching result; the second matching result includes target object-target matching pairs; the number of consecutive frames of the historical video frames containing the target object in the historical trajectory data in the second state is less than a preset number of frames.

[0027] Among them, the detection part has a part category;

[0028] Perform feature matching on the targets corresponding to each piece of historical trajectory data in the first state with each target object to obtain a first matching result, including:

[0029] Based on the part features of the target object and the target corresponding to the same part category, determine the part similarity of the part category; the detection part corresponding to the part category is the visible part.

[0030] Sum and average the part similarities of each part category corresponding between the target object and the target to determine the target similarity between the target object and the target.

[0031] Based on the target similarity between the target object and the target, determine the target matching degree between the target object and the target.

[0032] Based on the target matching degree between the target object and the target, determine whether the target object and the target match.

[0033] Among them, based on the target similarity between the target object and the target, determining the target matching degree between the target object and the target includes:

[0034] In response to the number of part categories of the target object in the current video frame that are the same as the target not exceeding a first preset number, determine the target matching degree between the target object and the target only based on the target similarity between the target object and the target.

[0035] Among them, based on the target similarity between the target object and the target, determining the target matching degree between the target object and the target includes:

[0036] In response to the number of part categories of the target object in the current video frame that are the same as the target exceeding the first preset number, determine the global similarity between the target object and the target based on the global feature map of the target object and the global feature map of the target.

[0037] Based on the corresponding global similarity and target similarity between the target object and the target, determine the target matching degree between the target object and the target.

[0038] Among them, the target tracking method further includes:

[0039] If the current trajectory data of the target object is formed in response to the target object matching the historical trajectory data in the second state, then it is determined whether to update the state of the current trajectory data of the target object based on the number of detected parts of the target object in the current video frame and the number of consecutive occurrences of the target object in the current trajectory data.

[0040] Among them, determining whether to update the state of the current trajectory data of the target object based on the number of detected parts of the target object in the current video frame and the number of consecutive occurrences of the target object in the current trajectory data includes:

[0041] In response to the number of detected parts of the target object in the current video frame exceeding the second preset number and the number of consecutive occurrences of the target object in the current trajectory data exceeding the preset number, update the state of the current trajectory data of the target object to the first state.

[0042] Among them, the historical trajectory data of the target has a first feature library and a second feature library; the first feature library is composed of the part features of each detected part of the target in the first image; the second feature library is composed of the part features of each detected part of the target in the second image; the first image is the historical video frame corresponding to the historical trajectory data with the largest number of detected parts of the target; the second image is the historical video frame closest to the current video frame in the historical trajectory data that contains the target;

[0043] The target tracking method further includes:

[0044] Compare the number of detected parts of the target object in the current video frame with the third preset number to determine the first feature library and the second feature library corresponding to the current trajectory data of the target object.

[0045] Among them, comparing the number of detected parts of the target object in the current video frame with the third preset number to determine the first feature library and the second feature library corresponding to the current trajectory data of the target object includes:

[0046] In response to the number of detected parts of the target object in the current video frame being greater than the total number of detected parts in the first feature library of the historical trajectory data of the target or not less than the first threshold, update the part features of the detected parts of the target object in the current video frame to the first feature library;

[0047] In response to the number of detected parts of the target object in the current video frame not being less than the total number of detected parts in the second feature library of the historical trajectory data of the target or not less than the second threshold, update the part features of the detected parts of the target object in the current video frame to the second feature library.

[0048] Among them, the target tracking method further includes:

[0049] Correct the target detection box of the target object based on the preset size information to determine the effective detection box of the target object;

[0050] Replace the target detection box of the target object in the current trajectory data with the effective detection box of the target object;

[0051] Predict the position information of the target object in the next video frame after the current video frame based on the updated current trajectory data of the target object.

[0052] To solve the above technical problems, the second technical solution adopted by the present invention is: to provide a target tracking device, the target tracking device includes:

[0053] A detection module, configured to perform multi-granularity feature extraction on the target object in the current video frame to obtain the detection information of the target object; the detection information includes the part features of each detected part;

[0054] A matching module, configured to match the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object;

[0055] An association module, configured to associate the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate the current trajectory data of the target object.

[0056] To solve the above technical problems, the third technical solution adopted by the present invention is: to provide an electronic terminal, the electronic terminal includes a memory and a processor coupled to each other, and the processor is configured to execute program instructions stored in the memory, and the processor is configured to execute program data to implement the steps in the target tracking method as described above.

[0057] To solve the above technical problems, the fourth technical solution adopted by the present invention is: to provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the target tracking method as described above.

[0058] The beneficial effects of the present invention are as follows: Different from the prior art, the provided target tracking method, device, electronic terminal and computer-readable storage medium, the target tracking method includes extracting multi-granularity features of the target object in the current video frame to obtain the detection information of the target object; the detection information includes the part features of each detected part; matching the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object; associating the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate the current trajectory data of the target object. By matching the part features of each detected part of the target object with the historical trajectory data of each target in the historical video frames, the present application reduces the influence of the occluded part or the undetected part on the target matching result, thereby improving the matching accuracy of the target object and the tracking stability of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0060] Figure 1 is a schematic flowchart of the target tracking method provided by the present invention;

[0061] Figure 2 is a schematic diagram of a specific embodiment of the key points corresponding to the target provided by the present invention;

[0062] Figure 3 is Figure 1 a schematic diagram of a specific embodiment of step S1 in the target tracking method provided;

[0063] Figure 4 is Figure 1 a schematic diagram of another specific embodiment of step S1 in the target tracking method provided;

[0064] Figure 5 is Figure 1 a schematic diagram of a specific embodiment of step S2 in the target tracking method provided;

[0065] Figure 6 is Figure 5 a schematic diagram of a specific embodiment of step S21 in the target tracking method provided;

[0066] Figure 7 is a schematic framework diagram of an embodiment of the target tracking device provided by the present invention;

[0067] Figure 8 It is a schematic diagram of the framework of an embodiment of the electronic terminal provided by the present invention;

[0068] Figure 9 It is a schematic diagram of the framework of an embodiment of the computer-readable storage medium provided by the present invention. Detailed implementation manners

[0069] The following will combine the accompanying drawings of the specification to elaborate on the solutions of the embodiments of the present application in detail.

[0070] In the following description, specific details such as specific system architectures, interfaces, and technologies are presented for the purpose of illustration rather than limitation, so as to thoroughly understand the present application.

[0071] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after. In addition, "multiple" in this article means two or more than two.

[0072] To enable those skilled in the art to better understand the technical solutions of the present invention, the following will further elaborate on a target tracking method provided by the present invention in combination with the drawings and specific implementation manners.

[0073] Unless otherwise defined, all technical and scientific terms used in this article have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used in this article are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0074] Before further elaborating on the embodiments of the present application, the nouns and terms involved in the embodiments of the present application are described. The nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0075] The DeepSORT algorithm is a target tracking algorithm based on deep learning, which combines the advantages of the SORT algorithm and deep learning feature extraction. The DeepSORT algorithm extracts features in the target box and uses the Kalman filter for target state prediction to achieve target tracking. The DeepSORT algorithm has good robustness in complex situations such as target occlusion and target disappearance.

[0076] BPBREID is a part-based humanoid feature re-identification technology.

[0077] The target tracking method provided by the embodiments of the present application can be implemented by a server or a terminal alone, or by the cooperation of the server and the terminal. In some embodiments, the terminal or the server can implement the target tracking method provided by the embodiments of the present application by running a computer program. For example, the computer program can be a native program or a software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client supporting a virtual scene, such as a game APP; it can also be a small program, that is, a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above computer program can be any form of application program, module or plug-in.

[0078] The following takes the implementation by the server as an example to illustrate the target tracking method provided by the embodiments of the present application.

[0079] Please refer to Figure 1 , Figure 1 which is a schematic flow chart of the target tracking method provided by the present invention.

[0080] In this embodiment, a target tracking method is provided, and the target tracking method includes the following steps.

[0081] S1: Extract multi-granularity features of the target object in the current video frame to obtain detection information of the target object; the detection information includes the part features of each detected part.

[0082] S2: Match the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object.

[0083] S3: Associate the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate the current trajectory data of the target object.

[0084] In one embodiment, a multi-granularity feature extraction network is constructed based on BPBREID, and the constructed multi-granularity feature extraction network is trained according to the actual situation, so that the trained multi-granularity feature extraction network can detect the global feature of the target object, the part features of each preset part, and the part visibility type.

[0085] In one embodiment, according to different task precision requirements, preset schemes with different granularities are determined, that is, the task precision corresponds one-to-one with the preset scheme. The number of preset parts included in each preset scheme is different. For example, the finer the task precision, the more preset parts are included in the preset scheme, and the more part categories the preset parts correspond to. For example, the granularity selection and feature aggregation are carried out according to the classification rules adopted in the target key point detection model based on the openpifpaf library.

[0086] Please refer to Figure 2 , Figure 2 is a schematic diagram of a specific embodiment of the key points corresponding to the target provided by the present invention.

[0087] In one embodiment, all the key points of the target include nose 1, left eye 2, right eye 3, left ear 4, right ear 5, left shoulder 6, right shoulder 7, left elbow 8, right elbow 9, left wrist 10, right wrist 11, left hip 12, right hip 13, left knee 14, right knee 15, left ankle 16, and right ankle 17.

[0088] For example, a preset scheme includes head key points, torso key points, arm and shoulder key points, leg key points, and foot key points. The target is formed by connecting the adjacent key points among the head key points, arm and shoulder key points, torso key points, leg key points, and foot key points in sequence.

[0089] In one embodiment, the training method of the multi-granularity feature extraction network is as follows.

[0090] Obtain a plurality of sample images, where the sample images are images containing the target, and each target has annotation information. The annotation information includes the global annotation features of the target, the part annotation features of each preset part of the target, and the annotation visible type. Among them, the annotation visible type is visible parts and occluded parts. Input the sample images into the multi-granularity feature extraction network to obtain the global prediction features of the target, the part prediction features of each preset part of the target, and the predicted visibility probability. The larger the predicted visibility probability value, the smaller the probability that the preset part is occluded and / or the smaller the occluded area of the preset part. Iteratively train the multi-granularity feature extraction network based on the first loss value between the global annotation features and global prediction features corresponding to the same target, and the second loss value between the part annotation features and part prediction features corresponding to each preset part of the target, so that the trained multi-granularity feature extraction network can realize the feature detection of the target task.

[0091] In a specific embodiment, the multi-granularity feature extraction network is iteratively trained based on the following loss function.

[0092] (Formula 1)

[0093] In the formula: L GiLtRepresents the overall loss value of the target; L id Represents the first loss value of the target; L id Consists of the common cross-entropy loss L CE constitutes, f g Represents the overall embedding, f f Represents the foreground embedding, f c Represents the part feature; L tri Is the second loss value of the target.

[0094] (Formula 2)

[0095] (Formula 3)

[0096] In the formula: dist eucl Represents the Euclidean distance; d ij Represents the average distance corresponding to all part features of the target; d ap Represents the distance between the sample image and the hardest positive sample; d an Represents the distance between the sample image and the hardest negative sample; α represents the triplet loss margin.

[0097] Among them, Mobileone based on the lightweight architecture is used as the multi-granularity feature extraction network. The global prediction feature G of the target = R H×W×C .

[0098] Specifically, the specific implementation of extracting multi-granularity features from the target object in the current video frame in step S1 is as follows.

[0099] In one embodiment, the specific steps for determining the detection information of the target object are as follows.

[0100] Please refer to Figure 3 , Figure 3 is Figure 1 A schematic diagram of a specific embodiment of step S1 in the target tracking method provided.

[0101] S111: Perform target detection on the current video frame to obtain target detection frames containing each target object.

[0102] Specifically, video stream data containing a target object is obtained. The video stream data can consist of multiple consecutive video frames. The video stream data can be real-time captured video stream or offline captured video stream. Target detection is performed on the current video frame through a target detection network model to obtain a target detection box containing the target object in the current video frame. The target detection network model can be RCNN series (RCNN, Fast-RCNN, Faster-RCNN), R-FCN, YOLO, SSD, and FPN. Among them, Faster-RCNN is short for Faster Region-based Convolutional Neural Networks; Fast-RCNN is short for Fast Region-based Convolutional Neural Networks; RCNN is short for Region-based Convolutional Neural Networks.

[0103] S112: Multigranularity feature extraction is performed on the target object in the target detection box through a multigranularity feature extraction network to obtain a global feature map corresponding to the target object and part features of each detected part.

[0104] Specifically, multigranularity feature extraction is performed on the target object in the target detection box through the multigranularity feature extraction network trained in the above embodiment to obtain a global feature map of the target object, part features of each detected part corresponding to the target object, and a visibility index. Among them, the visibility index includes two indices: 0 and 1. Among them, 0 indicates that the detected part is occluded and invisible; 1 indicates that the detected part is not occluded and visible.

[0105] In one embodiment, the detected parts of the target object that the multigranularity feature extraction network detects can include the head, arm shoulders, torso, legs, and feet.

[0106] Please refer to Figure 4 , Figure 4 is Figure 1 a schematic diagram of another specific embodiment of step S1 in the provided target tracking method.

[0107] In one embodiment, the specific steps for determining the detection information of the target object are as follows.

[0108] S121: Target detection is performed on the current video frame to obtain target detection boxes containing each target object.

[0109] Specifically, target detection is performed on the current video frame through a target detection network model to obtain a target detection box containing the target object. Each target detection box contains one target object. Among them, the target object can be an animal, a vehicle, etc.

[0110] S122: Extract features from the target object in the target detection box to obtain the global feature map of the target object.

[0111] Specifically, extract features from the target object in the target detection box to obtain the global feature map of the target object, and the size of the global feature map is H×W×C.

[0112] S123: Perform pixel classification on the global feature map of the target object to obtain the part categories of each pixel point in the global feature map.

[0113] Specifically, classify each pixel point in the global feature map of the target object to obtain the probability values of each pixel point for each preset category; select the preset category corresponding to the maximum probability value of the pixel point as the part category of the pixel point.

[0114] Through the above steps, the part categories of each pixel point in the global feature map can be obtained.

[0115] S124: Aggregate the pixel points of the same part category to obtain the part feature of the part category; the part category is used as the part category of the detected part.

[0116] Specifically, aggregate based on all pixel points corresponding to the same part category to obtain the part feature corresponding to the part category. Use the part feature of the part category as the part category of the detected part.

[0117] In one embodiment, the specific steps for determining the detection information of the target object further include the following.

[0118] S125: Based on the probability values of the pixel points corresponding to the part category of the detected part, determine whether the detected part is a visible part.

[0119] In one specific embodiment, select the maximum probability value among the probability values of all pixel points corresponding to the part category of the detected part and compare it with a preset value. In response to the maximum probability value corresponding to the part category of the detected part being greater than the preset value, determine that the detected part is a visible part. In response to the maximum probability value corresponding to the part category of the detected part not being greater than the preset value, determine that the detected part is an invisible part.

[0120] In another specific embodiment, sum up and average the probability values of all pixel points corresponding to the part category of the detected part to obtain the average probability value of the detected part belonging to the part category, and compare the average probability value of the detected part belonging to the part category with the preset value. In response to the average probability value of the detected part belonging to the part category being greater than the preset value, determine that the detected part is a visible part; in response to the average probability value of the detected part belonging to the part category not being greater than the preset value, determine that the detected part is an invisible part.

[0121] Specifically, in step S2, the detection information of the target object is matched with the historical trajectory data of each corresponding target in the historical video frames before the current video frame, and the specific implementation of determining the target that matches the target object is as follows.

[0122] Please refer to Figure 5 , Figure 5 which Figure 1 is a schematic diagram of a specific embodiment of step S2 in the provided target tracking method.

[0123] S21: Match the targets corresponding to the historical trajectory data in the first state with each target object to obtain a first matching result.

[0124] Among them, the first matching result includes target object - target matching pairs, unmatched target objects, and unmatched historical trajectory data; the number of consecutive frames of the historical video frames containing the target in the historical trajectory data in the first state is not less than a preset number of frames. That is, the historical trajectory data in the first state is confirmed state trajectory.

[0125] Please refer to Figure 6 , Figure 6 which Figure 5 is a schematic diagram of a specific embodiment of step S21 in the provided target tracking method.

[0126] In one embodiment, the specific implementation of obtaining the first matching result is as follows.

[0127] S211: Based on the part features of the target object and the target corresponding to the same part category, determine the part similarity of the part category; the detected part corresponding to the part category is a visible part.

[0128] For example, the head of the target object is a visible part, and the head of the target is a visible part. Determine the head similarity between the target object and the target based on the part features corresponding to the head of the target object and the head of the target.

[0129] For example, if the foot of the target object is an invisible part and the foot of the target is a visible part, then there is no need to calculate the foot similarity between the target object and the target. Or, when the foot of the target object is a visible part and the foot of the target is an invisible part, there is also no need to calculate the foot similarity between the target object and the target.

[0130] Through this step, the part similarity of the part category where both the target object and the target belong to visible parts can be obtained.

[0131] S212: Sum and average the part similarities of each corresponding part category between the target object and the target to determine the target similarity between the target object and the target.

[0132] In one embodiment, the target similarity between the target object and the target can be obtained through the following formula.

[0133] (Formula 4)

[0134] In the formula: represents the part feature distance between the target object and the target; the target similarity between the target object and the target is 1 - , the greater the distance between the target object and the target, the smaller the target similarity between the target object and the target; is used to represent the part similarity between the target object and the target that are mutually visible.

[0135] S213: Based on the target similarity between the target object and the target, determine the target matching degree between the target object and the target.

[0136] Specifically, determine the target matching degree between the target object and the target based on the following formula.

[0137] (Formula 5)

[0138] In the formula: the target matching degree between the target object and the target is 1 - , represents the part difference degree between the target object and the target; represents the global difference degree between the target object and the target; represents a boolean value, taking values of 0 and 1.

[0139] In a specific embodiment, in response to the number of part categories of the target object in the current video frame that are the same as those of the target not exceeding the first preset number, determine the target matching degree between the target object and the target only based on the target similarity between the target object and the target. That is, when the number of part categories of the target object in the current video frame that are the same as those of the target does not exceed the first preset number, the boolean value in Formula 5 takes the value of 0.

[0140] In a specific embodiment, in response to the number of part categories of the target object in the current video frame that are the same as those of the target exceeding the first preset number, determine the global similarity between the target object and the target based on the global feature map of the target object and the global feature map of the target; based on the corresponding global similarity and target similarity between the target object and the target, determine the target matching degree between the target object and the target. That is, when the number of part categories of the target object in the current video frame that are the same as those of the target exceeds the first preset number, the boolean value in Formula 5 takes the value of 1.

[0141] S214: Determine whether the target object and the target match based on the target matching degree between the target object and the target.

[0142] Specifically, in response to the target matching degree between the target object and the target being greater than the preset matching degree threshold, it is determined that the target object and the target match; in response to the target matching degree between the target object and the target not being greater than the matching degree threshold, it is determined that the target object and the target do not match.

[0143] S22: Perform IOU matching on the unmatched target objects in the first matching result with the unmatched historical trajectory data and each piece of historical trajectory data in the second state respectively to obtain a second matching result.

[0144] Among them, the second matching result includes target object - target matching pairs; the number of consecutive frames of the historical video frames containing the target object in the historical trajectory data in the second state is less than the preset number of frames. That is, the historical trajectory data in the second state is non - confirmed trajectory.

[0145] Specifically, in order to improve the matching probability of the target object, perform IOU matching on the unmatched target objects with the unmatched historical trajectory data in the first state and the historical trajectory data in the second state in the first matching result to obtain a second matching result.

[0146] In one embodiment, the target tracking method further includes the following steps.

[0147] In response to the target object matching with the historical trajectory data in the second state to form the current trajectory data of the target object, determine whether to update the state of the current trajectory data of the target object based on the number of detected parts of the target object in the current video frame and the number of consecutive occurrences of the target object in the current trajectory data.

[0148] In response to the number of detected parts of the target object in the current video frame exceeding the second preset number and the number of consecutive occurrences of the target object in the current trajectory data exceeding the preset number, update the state of the current trajectory data of the target object to the first state.

[0149] In one specific embodiment, the historical trajectory data of the target has a first feature library and a second feature library; the first feature library is composed of the part features of each detected part of the target in the first image; the second feature library is composed of the part features of each detected part of the target in the second image; the first image is the historical video frame corresponding to the historical trajectory data with the largest number of detected parts of the target; the second image is the historical video frame closest to the current video frame in the historical trajectory data that contains the target.

[0150] In one embodiment, the target tracking method further includes the following steps.

[0151] Compare the number of detected parts of the target object in the current video frame with a third preset number to determine the first feature library and the second feature library corresponding to the current trajectory data of the target object.

[0152] In a specific embodiment, in response to the number of detected parts of the target object in the current video frame being greater than the total number of detected parts in the first feature library of the target's historical trajectory data or not less than a first threshold, update the part features of the detected parts of the target object in the current video frame to the first feature library.

[0153] In response to the number of detected parts of the target object in the current video frame being not less than the total number of detected parts in the second feature library of the target's historical trajectory data or not less than a second threshold, update the part features of the detected parts of the target object in the current video frame to the second feature library.

[0154] In an embodiment, for the case where the target object is partially occluded, due to the jump of the detection box, even if feature matching realizes the effective association of data, it will also have a greater impact on subsequent Kalman filter updates and predictions. Therefore, the target tracking method further includes the following steps.

[0155] Based on preset size information, correct the target detection box of the target object to determine the effective detection box of the target object. Among them, the target detection box containing the target object is (t, l, w, h), where t represents the x-axis coordinate of the upper left corner of the target detection box; l represents the y-axis coordinate of the upper left corner of the target detection box, w represents the length of the target detection box in the x-axis direction; h represents the length of the target detection box in the y-axis direction.

[0156] In response to the length of the target detection box containing the target object in the x-axis direction and the length in the y-axis direction both conforming to the corresponding preset size, determine the target detection box of the target object as the effective detection box.

[0157] In a specific embodiment, in response to the length w of the target detection box of the target object in the x-axis direction not conforming to the corresponding preset size, correct the length w of the target detection box in the x-axis direction to obtain the effective length w of the target detection box in the x-axis direction new , the effective length w of the target detection box in the x-axis direction new conforms to the corresponding preset size.

[0158] In a specific embodiment, in response to the length h of the target detection box of the target object in the y-axis direction not conforming to the corresponding preset size, correct the length h of the target detection box in the y-axis direction to obtain the effective length h of the target detection box in the y-axis direction new , the effective length h of the target detection box in the y-axis direction new conforms to the corresponding preset size.

[0159] Replace the target detection box of the target object in the current trajectory data with the effective detection box of the target object.

[0160] Predict the position information of the target object in the next video frame after the current video frame based on the updated current trajectory data of the target object.

[0161] In this application, a multi-granularity feature extraction network and a feature matching method are adopted, which can effectively track the target object when the head or other parts are occluded, and do not rely on the detection results of specific parts, so the stability of the algorithm is better.

[0162] In one embodiment, the detection information of the target object in the current video frame is associated with the historical trajectory data of the matching target to generate the current trajectory data of the target object.

[0163] The target tracking method provided in this embodiment matches the part features of each detected part of the target object with the historical trajectory data of each target in the historical video frame, reduces the influence of the occluded part or the undetected part on the target matching result, and thus improves the matching accuracy of the target object and the tracking stability of the target object.

[0164] Please refer to Figure 7 , Figure 7 FIG. is a schematic framework diagram of an embodiment of the target tracking device provided by the present invention. This embodiment provides a target tracking device 60, and the target tracking device 60 includes a detection module 61, a matching module 62, and an association module 63.

[0165] The detection module 61 is used to perform multi-granularity feature extraction on the target object in the current video frame to obtain the detection information of the target object; the detection information includes the part features of each detected part.

[0166] The matching module 62 is used to match the detection information of the target object with the historical trajectory data of the corresponding targets in the historical video frames before the current video frame to determine the target that matches the target object.

[0167] The association module 63 is used to associate the detection information of the target object in the current video frame with the historical trajectory data of the matching target to generate the current trajectory data of the target object.

[0168] The target tracking device provided in this embodiment matches the part features of each detected part of the target object with the historical trajectory data of each target in the historical video frame, reduces the influence of the occluded part or the undetected part on the target matching result, and thus improves the matching accuracy of the target object and the tracking stability of the target object.

[0169] Please refer to Figure 8 ,Figure 8 It is a schematic diagram of the framework of an embodiment of the electronic terminal provided by the present invention. The electronic terminal 80 includes a memory 81 and a processor 82 that are coupled to each other. The processor 82 is configured to execute program instructions stored in the memory 81 to implement the steps of any of the above-described embodiment of the target tracking method. In a specific implementation scenario, the electronic terminal 80 may include, but is not limited to, a microcomputer, a server. In addition, the electronic terminal 80 may also include mobile devices such as a laptop computer, a tablet computer, etc., which are not limited herein.

[0170] Specifically, the processor 82 is configured to control itself and the memory 81 to implement the steps of any of the above-described embodiment of the target tracking method. The processor 82 may also be referred to as a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip having signal processing capabilities. The processor 82 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. Additionally, the processor 82 may be implemented jointly by integrated circuit chips.

[0171] In the above solution, the target tracking method includes performing multi-granularity feature extraction on a target object in a current video frame to obtain detection information of the target object; the detection information includes part features of each detected part; matching the detection information of the target object with historical trajectory data of corresponding targets in historical video frames before the current video frame to determine a target that matches the target object; and associating the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate current trajectory data of the target object.

[0172] Please refer to Figure 9 , Figure 9 It is a schematic diagram of the framework of an embodiment of the computer-readable storage medium provided by the present invention. The computer-readable storage medium 90 stores program instructions 901 that can be run by a processor. The program instructions 901 are used to implement the steps of any of the above-described embodiment of the target tracking method.

[0173] In the above solution, the target tracking method includes extracting multi-granularity features of the target object in the current video frame to obtain detection information of the target object; the detection information includes part features of each detected part; matching the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object; and associating the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate the current trajectory data of the target object.

[0174] In some embodiments, the functions or modules included in the device provided by the present disclosure may be used to execute the methods described in the above method embodiments. The specific implementation may refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0175] The descriptions of the above embodiments tend to emphasize the differences between the embodiments. The same or similar parts can be referred to each other. For the sake of brevity, they will not be repeated in this article.

[0176] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical or other forms.

[0177] In addition, each functional unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0178] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0179] The above are only the embodiments of the present invention, and do not limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A target tracking method, characterized in that, The target tracking method includes: Performing multi-granularity feature extraction on the target object in the current video frame to obtain the detection information of the target object; the detection information includes the part features of each detected part. Matching the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object. Associating the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate the current trajectory data of the target object. The step of matching the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object includes: Performing feature matching between the targets corresponding to the historical trajectory data in the first state and each target object to obtain a first matching result; the first matching result includes target object-target matching pairs, unmatched target objects, and unmatched historical trajectory data; the number of consecutive frames of the historical video frames containing the target in the historical trajectory data in the first state is not less than a preset number of frames. Performing IOU matching between the unmatched target objects in the first matching result and the unmatched historical trajectory data and the historical trajectory data in the second state respectively to obtain a second matching result; the second matching result includes target object-target matching pairs; the number of consecutive frames of the historical video frames containing the target object in the historical trajectory data in the second state is less than the preset number of frames.

2. The target tracking method according to claim 1, wherein: The step of performing multi-granularity feature extraction on the target object in the current video frame to obtain the detection information of the target object includes: Performing target detection on the current video frame to obtain target detection boxes containing each target object. Performing multi-granularity feature extraction on the target object in the target detection box through a multi-granularity feature extraction network to obtain the global feature map corresponding to the target object and the part features of each detected part.

3. The target tracking method according to claim 1, wherein: The step of performing multi-granularity feature extraction on the target object in the current video frame to obtain the detection information of the target object includes: Performing target detection on the current video frame to obtain target detection boxes containing each target object. Performing feature extraction on the target object in the target detection box to obtain the global feature map of the target object. Performing pixel classification on the global feature map of the target object to obtain the part categories of each pixel point in the global feature map. Aggregating the pixel points of the same part category to obtain the part features of the part category; the part category is used as the part category of the detected part.

4. The target tracking method according to claim 3, wherein: The step of performing pixel classification on the global feature map of the target object to obtain the part categories of each pixel point in the global feature map includes: Classify each pixel point in the global feature map of the target object to obtain the probability values of each pixel point for each preset category; Select the preset category corresponding to the maximum probability value of the pixel point as the part category of the pixel point.

5. The target tracking method according to claim 4, wherein: The multi-granularity feature extraction of the target object in the current video frame to obtain the detection information of the target object further includes: Based on the probability value of the pixel point corresponding to the part category of the detected part, determine whether the detected part is a visible part.

6. The target tracking method according to claim 5, wherein: The determining whether the detected part is a visible part based on the probability value of the pixel point corresponding to the category of the detected part includes: Select the maximum probability value among the probability values of all the pixel points corresponding to the part category of the detected part and compare it with a preset value; In response to the maximum probability value corresponding to the part category of the detected part being greater than the preset value, determine that the detected part is a visible part.

7. The target tracking method according to claim 1, wherein The detected part has a part category; The matching the target corresponding to each of the historical trajectory data in the first state with each of the target objects to obtain a first matching result includes: Based on the part features of the target object and the target corresponding to the same part category, determine the part similarity of the part category; the detected part corresponding to the part category is a visible part; Sum and average the part similarities of each part category corresponding between the target object and the target to determine the target similarity between the target object and the target; Based on the target similarity between the target object and the target, determine the target matching degree between the target object and the target; Based on the target matching degree between the target object and the target, determine whether the target object and the target match.

8. The target tracking method according to claim 7, wherein: The determining the target matching degree between the target object and the target based on the target similarity between the target object and the target includes: In response to the number of part categories of the target object in the current video frame that are the same as those of the target not exceeding a first preset number, determine the target matching degree between the target object and the target only based on the target similarity between the target object and the target.

9. The target tracking method according to claim 7, wherein: The determining the target matching degree between the target object and the target based on the target similarity between the target object and the target includes: In response to the number of part categories of the target object in the current video frame that are the same as those of the target exceeding a first preset number, determine the global similarity between the target object and the target based on the global feature map of the target object and the global feature map of the target; Determine the target matching degree between the target object and the target based on the global similarity and the target similarity corresponding between the target object and the target.

10. The target tracking method according to claim 1, wherein The target tracking method further includes: In response to the target object matching the historical trajectory data in the second state to form the current trajectory data of the target object, determine whether to update the state of the current trajectory data of the target object based on the number of detected parts of the target object in the current video frame and the number of consecutive occurrences of the target object in the current trajectory data.

11. The target tracking method according to claim 10, wherein The determining whether to update the state of the current trajectory data of the target object based on the number of detected parts of the target object in the current video frame and the number of consecutive occurrences of the target object in the current trajectory data includes: In response to the number of detected parts of the target object in the current video frame exceeding a second preset number and the number of consecutive occurrences of the target object in the current trajectory data exceeding a preset number, update the state of the current trajectory data of the target object to the first state.

12. The target tracking method according to claim 10, wherein The historical trajectory data of the target has a first feature library and a second feature library; the first feature library is composed of the part features of each detected part of the target in the first image; the second feature library is composed of the part features of each detected part of the target in the second image; the first image is the historical video frame corresponding to the historical trajectory data that contains the largest number of detected parts of the target; the second image is the historical video frame closest to the current video frame in the historical trajectory data that contains the target. The target tracking method further includes: Compare the number of detected parts of the target object in the current video frame with a third preset number to determine the first feature library and the second feature library corresponding to the current trajectory data of the target object.

13. The target tracking method according to claim 12, wherein The comparing the number of detected parts of the target object in the current video frame with a third preset number to determine the first feature library and the second feature library corresponding to the current trajectory data of the target object includes: In response to the number of detected parts of the target object in the current video frame being greater than the total number of detected parts in the first feature library of the historical trajectory data of the target or not less than a first threshold, update the part features of the detected parts of the target object in the current video frame to the first feature library; In response to the number of detected parts of the target object in the current video frame being not less than the total number of detected parts in the second feature library of the historical trajectory data of the target or not less than a second threshold, update the part features of the detected parts of the target object in the current video frame to the second feature library.

14. The target tracking method according to claim 2, wherein The target tracking method further includes: Based on preset size information, correct the target detection box of the target object to determine the effective detection box of the target object; Replace the target detection box of the target object in the current trajectory data with the effective detection box of the target object; Predict the position information of the target object in the next video frame after the current video frame based on the updated current trajectory data of the target object.

15. A target tracking device, characterized in that, The target tracking device includes: A detection module, configured to perform multi-granularity feature extraction on the target object in the current video frame to obtain the detection information of the target object; the detection information includes the part features of each detected part; A matching module, configured to match the detection information of the target object with the historical trajectory data of each corresponding target in the historical video frames before the current video frame to determine the target that matches the target object; configured to perform feature matching between the targets corresponding to the historical trajectory data in the first state and each target object to obtain a first matching result; the first matching result includes target object-target matching pairs, unmatched target objects, and unmatched historical trajectory data; the number of consecutive frames of the historical video frames containing the target in the historical trajectory data in the first state is not less than a preset number of frames; perform IOU matching on the unmatched target objects in the first matching result with the unmatched historical trajectory data and the historical trajectory data in the second state respectively to obtain a second matching result; the second matching result includes target object-target matching pairs; the number of consecutive frames of the historical video frames containing the target object in the historical trajectory data in the second state is less than the preset number of frames; An association module, configured to associate the detection information of the target object in the current video frame with the historical trajectory data of the matched target to generate the current trajectory data of the target object.

16. An electronic terminal, characterized in that, The electronic terminal includes a memory and a processor coupled to each other, the processor is configured to execute program instructions stored in the memory, and the processor is configured to execute program data to implement the steps in the target tracking method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the target tracking method according to any one of claims 1 to 14 are implemented.

Images