An object tracking method, device, apparatus, and computer-readable storage medium

Abstract

Embodiments of the present application disclose an object tracking method, device and equipment and a computer readable storage medium, comprising determining a target frame image after a current frame image from a target video sequence, and the current frame image comprising a plurality of to-be-tracked objects; determining a first mask corresponding to the target frame image based on a target detection model and a target segmentation model, the target detection model being a model with zero-shot recognition capability and language recognition capability, and the mask being a pixel matrix corresponding to the target frame image; determining a second mask corresponding to the target frame image based on a target tracking model and the target frame image; determining a target mask corresponding to the target frame image based on the first mask and the second mask; and obtaining a tracking result of each to-be-tracked object and a target object in the target frame image based on the target tracking model, the target mask and the target frame image, the target object being a newly-added object in the target frame image other than the plurality of to-be-tracked objects.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an object tracking method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Currently, existing target tracking technologies typically employ various small models and often use a detection-then-matching approach. Specifically, target detection algorithms are first used to detect targets in the video, and then motion state matching algorithms are used to determine whether they are the same target object. This involves establishing a correlation between the trajectory and the detection box based on information such as the position and size of the detection box. Then, the detection results of each frame are correlated based on the Hungarian algorithm or target feature information, and the trajectory information is updated based on the correlation results. However, existing detection methods are prone to misjudgment or tracking failure when the target object's shape changes significantly, such as its posture or direction of motion. Furthermore, existing small models are difficult to adapt to infinite scenes and diverse task requirements, ultimately leading to low tracking accuracy. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application aim to provide an object tracking method, apparatus, device, and computer-readable storage medium that can solve the problem of low tracking accuracy in related technologies.

[0004] The technical solution of this application is implemented as follows:

[0005] An object tracking method, the method comprising:

[0006] The target frame image is determined from the target video sequence after the current frame image; wherein the current frame image includes multiple objects to be tracked;

[0007] Based on the object detection model and the object segmentation model, a first mask corresponding to the target frame image is determined; wherein, the object detection model is a model that is trained by using a target sample dataset and model fine-tuning technology to train an initial detection model, and has zero-shot recognition capability and language recognition capability; the mask is the pixel matrix corresponding to the target frame image;

[0008] Based on the target tracking model and the target frame image, determine the second mask corresponding to the target frame image;

[0009] Based on the first mask and the second mask, the target mask corresponding to the target frame image is determined;

[0010] Based on the target tracking model, the target mask, and the target frame image, the tracking results of each of the objects to be tracked and the target objects in the target frame image are obtained; wherein, the target object is a newly added object in the target frame image in addition to the multiple objects to be tracked.

[0011] In the above scheme, determining the target frame image after the current frame image from the target video sequence includes:

[0012] The image frame interval is determined based on the motion parameters of the object to be tracked;

[0013] The target frame image is obtained from the target video sequence based on the current frame image and the frame interval.

[0014] In the above scheme, determining the first mask corresponding to the target frame image based on the target detection model and the target segmentation model includes:

[0015] The target frame image is detected based on the target detection model to obtain a first detection box for each of the objects to be tracked; wherein the first detection box is obtained by filtering multiple candidate detection boxes for each of the objects to be tracked.

[0016] Based on the target segmentation model, the target frame image, and multiple first detection boxes, a mask to be processed is determined for each first detection box, and the first mask is determined based on the multiple masks to be processed.

[0017] In the above scheme, determining the target mask corresponding to the target frame image based on the first mask and the second mask includes:

[0018] A first operation is performed on the first mask and the second mask to obtain an update mask, and the second mask is updated based on the update mask;

[0019] A third mask for the target object is obtained based on the first mask and the second mask;

[0020] A second operation is performed on the updated mask and the third mask to obtain the target mask.

[0021] In the above scheme, the object tracking method further includes:

[0022] Receive a click instruction for the current frame image, and determine the target tracking object from a plurality of objects to be tracked based on the click instruction;

[0023] Obtain the second detection box corresponding to the target tracking object, and determine the fourth mask corresponding to the target tracking object based on the target segmentation model, the second detection box, and the current frame image;

[0024] Based on the target tracking model, the fourth mask, the current frame image, and multiple consecutive frames following the current frame image in the target video sequence, the tracking result of the target object is determined.

[0025] In the above scheme, before determining the target frame image after the current frame image from the target video sequence, it further includes:

[0026] Based on each initial sample image in the initial sample dataset and the content information of each initial sample image, a target sample dataset is constructed; wherein, the target sample dataset includes target images with different scene complexities;

[0027] The target detection model is obtained by training the initial detection model based on the target sample dataset and model fine-tuning techniques.

[0028] In the above scheme, constructing the target sample dataset based on each initial sample image and the content information of each initial sample image in the initial sample dataset includes:

[0029] Based on each of the aforementioned content information, a first target label and a second target label are determined for each of the initial sample images; wherein, the first target label represents the descriptive information of the scene complexity corresponding to the initial sample image; and the second target label represents the quantity information of objects in the initial sample image.

[0030] The target sample dataset is constructed based on multiple initial sample images, multiple first target labels, and multiple second target labels.

[0031] In the above scheme, determining the first target label and the second target label corresponding to each initial sample image based on each piece of content information includes:

[0032] A first prompt word is constructed, and based on the first prompt word, each initial sample image, and the first network model, each first label is determined; wherein, the first prompt word is used to obtain environmental description information of the image;

[0033] A second prompt word is constructed, and based on the second prompt word, each initial sample image, and the second network model, each second label and each second target label are determined; wherein, the second prompt word is used to obtain the number of sample objects of each class in the image;

[0034] Each first target label is determined based on each first label and each second label.

[0035] An object tracking device, the device comprising:

[0036] An acquisition unit is used to determine the target frame image after the current frame image from the target video sequence; wherein the current frame image includes multiple objects to be tracked;

[0037] The detection unit is used to determine the first mask corresponding to the target frame image based on the target detection model and the target segmentation model; wherein, the target detection model is a model that is trained on an initial detection model using a target sample dataset and model fine-tuning technology, and has zero-shot recognition capability and language recognition capability; the mask is the pixel matrix corresponding to the target frame image;

[0038] A segmentation unit is used to determine a second mask corresponding to the target frame image based on the target tracking model and the target frame image;

[0039] The processing unit is configured to determine the target mask corresponding to the target frame image based on the first mask and the second mask;

[0040] The tracking unit is used to obtain the tracking result of each of the objects to be tracked and the target objects in the target frame image based on the target tracking model, the target mask and the target frame image; wherein the target object is a newly added object in the target frame image in addition to the multiple objects to be tracked.

[0041] An object tracking device, the device comprising: a processor, a memory, and a communication bus;

[0042] The communication bus is used to realize the communication connection between the processor and the memory;

[0043] The processor is used to execute the object tracking program stored in the memory to implement the steps of the object tracking method described above.

[0044] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the object tracking method described above.

[0045] The object tracking method, apparatus, device, and computer-readable storage medium provided in this application first determine the target frame image following the current frame image from the target video sequence, wherein the current frame image includes multiple objects to be tracked. Then, based on an object detection model and an object segmentation model, a first mask corresponding to the target frame image is determined. The object detection model is a model trained using a target sample dataset and model fine-tuning techniques, possessing zero-shot recognition capability and language recognition capability. The mask is the pixel matrix corresponding to the target frame image. Then, based on the object tracking model and the target frame image, a second mask corresponding to the target frame image is determined. Then, based on the first mask and the second mask, a target mask corresponding to the target frame image is determined. Finally, based on the object tracking model, the target mask, and the target frame image, the target mask for each object in the target frame image is obtained. The tracking results of the target object and the newly added object in the target frame image, in addition to the multiple target objects, are presented. By adopting a target detection model with zero-shot recognition and language recognition capabilities, the problem of the accuracy of traditional tracking models decreasing over time can be solved. It also has the zero-shot / few-shot generalization capability that is difficult for traditional tracking algorithms to achieve. Then, by combining the segmentation results of a large model such as a target segmentation model, the problem of low tracking efficiency caused by the poor zero-shot generalization and poor domain adaptation of existing small models can be solved. This allows small models to be rapidly iterated, deployed, and applied. Furthermore, by obtaining the annotation frame mask (i.e., the target mask) through the large model, the old target mask can be updated with the segmentation result of the large model with higher accuracy. This solves the problem of the accuracy of the target tracking model (i.e., the small model) decreasing over time, thereby improving the accuracy of object tracking. Attached Figure Description

[0046] Figure 1 A flowchart illustrating an object tracking method provided in an embodiment of this application;

[0047] Figure 2 A flowchart illustrating another object tracking method provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram illustrating the acquisition of a target tracking object in an object tracking method provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram illustrating the visualization results of an object tracking method provided in an embodiment of this application.

[0050] Figure 5 This is a schematic diagram of the structure of an object tracking device provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram of the structure of an object tracking device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0053] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0054] It should be noted that existing target tracking technologies often employ various small models and typically use a detection-then-match approach. Specifically, target detection algorithms are first used to detect targets in the video, and then motion state matching algorithms are used to determine if they belong to the same target. Examples include Simple Online and Realtime Tracking (SORT) and Deep Simple Online and Realtime Tracking (DeepSORT). The core idea is to track targets by associating detection boxes with known trajectories. This involves establishing a correlation between the trajectory and the detection boxes based on their position, size, and other information. Then, the detection results for each frame are correlated using the Hungarian algorithm or target feature information, and the trajectory information is updated based on the correlation results. However, existing methods suffer from several drawbacks. Firstly, when the target object's shape changes significantly, such as its posture or direction of motion, these methods are prone to misjudgment or tracking failure. Furthermore, this method requires detecting targets in the scene for every frame, resulting in low computational efficiency. Secondly, existing small models struggle to adapt to infinite scenes and diverse task requirements. In conventional target tracking tasks, both the detection and tracking models are small, data-fitting models. During tracking, they need to continuously provide masks or bounding boxes of the tracked object as annotations to achieve continuous tracking. Therefore, the accuracy of the annotations is crucial. To meet the accuracy requirements of the masks or bounding boxes and ensure the model's robustness in new scenarios, the small detection and tracking models need to be retrained using a large amount of accumulated scene bounding box or mask annotation data when applied to new scenarios. Therefore, in real-world scenarios or project implementations, small models struggle to achieve rapid iteration, deployment, and application to meet the project's required capabilities.

[0055] Based on this, embodiments of this application provide an object tracking method, which can be applied to an object tracking device, as described above. Figure 1 As shown, the method includes the following steps:

[0056] Step 101: Determine the target frame image following the current frame image from the target video sequence.

[0057] The current frame image contains multiple objects to be tracked.

[0058] In this embodiment, the target video sequence can refer to a video consisting of multiple frames of images in a certain order acquired in a target scene. The target scene can refer to a traffic scene, such as a road scene with vehicles coming and going, such as an intersection, park, highway, or port. The target video sequence includes multiple frames of images in order. The current frame image can refer to the first frame image in the target video sequence, or it can be any frame image specified in the target video sequence. The object to be tracked can refer to the object (such as a vehicle) contained in the current frame image (such as a vehicle image). Of course, the target frame image (denoted as the F-th frame image) also includes the object to be tracked included in the current frame image. In addition, the target frame image can also include newly added objects (i.e., objects not included in the current frame image). The target frame image is one or more frames of images after the current frame image, and specifically, one or more target frame images can be determined from the target video sequence based on the motion parameters of the object to be tracked in the current frame image.

[0059] Step 102: Based on the target detection model and the target segmentation model, determine the first mask corresponding to the target frame image.

[0060] The target detection model is trained using a target sample dataset and model fine-tuning techniques, and it has zero-shot recognition and language recognition capabilities; the mask is the pixel matrix corresponding to the target frame image.

[0061] In this embodiment, the first mask may refer to the first pixel matrix corresponding to the target frame image; the object detection model (i.e., the large model) may be a model used for detecting the object to be tracked, and the object detection model is a model trained on the initial detection model using the target sample dataset and model fine-tuning techniques; the object segmentation model (i.e., the large model) may be a model used for segmenting each object to be tracked; the first mask may refer to a mask obtained based on the large model and the target frame image; the first mask is essentially a pixel matrix, and the first mask may be obtained from the masks corresponding to multiple objects to be tracked included in the target frame image. In a feasible implementation, the object detection model may be an open set object detection large model (Grounding DINO); the object segmentation model may refer to a segmentation anything model (SegmentAnything Model, SAM).

[0062] Step 103: Based on the target tracking model and the target frame image, determine the second mask corresponding to the target frame image.

[0063] In this embodiment, the second mask may refer to the second pixel matrix corresponding to the target frame image; the target tracking model may refer to a small tracking model, and compared with the large tracking model, the small tracking model has fewer parameters and lower computational requirements; the second mask is a prediction mask obtained by predicting the target frame image through the small tracking model, and the second mask is also a pixel matrix; the target frame image can be input into the target tracking model for inference prediction to obtain the prediction mask corresponding to the target frame image.

[0064] Step 104: Based on the first mask and the second mask, determine the target mask corresponding to the target frame image.

[0065] In the embodiments of this application, the target mask can refer to the mask after the target frame image is updated; after obtaining the first mask and the second mask, the first mask and the second mask can be operated on to obtain the target mask of the target frame image object, and the target mask is also a pixel matrix, and the target mask serves as the frame annotation of the target frame image.

[0066] Step 105: Based on the target tracking model, target mask, and target frame image, obtain the tracking results of each object to be tracked and the target object in the target frame image.

[0067] The target object is a newly added object in the target frame image, in addition to the multiple objects to be tracked.

[0068] In this embodiment, the target mask and the target frame image can be input into the target tracking model. When no new object (i.e., target object) is detected in the target frame image, the tracking result of the mask that does not contain the new object in the corresponding F+1 frame can be inferred. When a new object (i.e., target object) is detected in the target frame image, the tracking result of the mask that contains the new object in the corresponding F+1 frame can be inferred.

[0069] The object tracking method provided in this application adopts a target detection model with zero-shot recognition and language recognition capabilities. This not only solves the problem of the accuracy of traditional tracking models decreasing over time, but also has the zero-shot / few-shot generalization capability that is difficult for traditional tracking algorithms to achieve. Then, by combining the segmentation results of a large model such as a target segmentation model, it solves the problem of low tracking efficiency caused by the poor zero-shot generalization and poor domain adaptation of existing small models. This allows small models to be rapidly iterated, deployed, and applied. Furthermore, by obtaining the annotation frame mask (i.e., the target mask) through the large model, the old target mask can be updated with the segmentation results of the large model with higher accuracy. This solves the problem of the accuracy of small target tracking models decreasing over time, thereby improving the accuracy of object tracking.

[0070] Based on the foregoing embodiments, this application provides yet another object tracking method, referring to... Figure 3 As shown, the method may include the following steps:

[0071] It should be noted that although pre-trained large models have good zero-shot generalization ability, fine-tuning large models with a small amount of scene data can improve their inference accuracy in a specific scene. Therefore, when high inference accuracy is required, it is necessary to construct a fine-tuning dataset. This application proposes a data complexity classification method that can efficiently construct a scene fine-tuning dataset without manual selection, as shown in step 201 below:

[0072] Step 201: The object tracking device constructs a target sample dataset based on each initial sample image and the content information of each initial sample image.

[0073] The target sample dataset includes target images with different scene complexities.

[0074] In this embodiment of the application, the initial sample image may refer to the image corresponding to the application scenario of the tracking task. In the traffic scenario, the initial sample image can be obtained through methods such as Real Time Streaming Protocol (RTSP). After obtaining multiple initial sample images, the content information of each initial sample image can be analyzed to obtain the scene complexity label corresponding to each initial sample image. Then, each initial sample image is labeled to obtain the target sample dataset.

[0075] It should be noted that step 201 can be achieved in the following way:

[0076] Step 201A: The object tracking device determines the first target label and the second target label corresponding to each initial sample image based on the content information of each initial sample image.

[0077] The first target label represents the scene complexity description information corresponding to the initial sample image; the second target label represents the number of objects in the initial sample image.

[0078] In this embodiment of the application, the content of the initial sample image includes the environmental description information of the image and the number of objects in the image; for each initial sample image, different network models can be used to analyze the environmental description information of the initial sample image and the number of objects in the image, so as to obtain the first target label and the second target label corresponding to the initial sample image.

[0079] It should be noted that step 201A can be achieved in the following way:

[0080] Step 201a1: The object tracking device constructs a first cue word and determines a first label based on the first cue word, each initial sample image, and the first network model.

[0081] The first prompt word is used to obtain environmental description information of the image; the second prompt word is used to obtain the number of sample objects of each class in the image.

[0082] In this embodiment, setting a standardized prompt word enables the network model to quickly return an answer that matches the intent. The first network model can refer to a large-scale image and text scene understanding model. The first prompt word can be set to return in the format of [weather, light intensity, traffic participant size, occlusion, time]. Then, the large-scale image and text scene understanding model can sequentially read the initial sample images and input the first prompt word into the model to obtain a first label in the specified format (i.e., [weather, light intensity, traffic participant size, occlusion, time]). The dataset with the first label is denoted as: Among them, frame i It is the RGB pixel matrix of the i-th image. It is a frame i The corresponding first label, n is the total number of initial sample images.

[0083] Step 201a2: The object tracking device constructs a second cue word and determines each second label and each second target label based on the second cue word, each initial sample image, and the second network model.

[0084] The second cue word is used to obtain the number of sample objects of each class in the image.

[0085] In this embodiment of the application, the second network model is a model with zero-shot recognition capability and language recognition capability, and can perform zero-shot object detection, such as the GroundingDINO model. Since the number of objects in an image is a range value rather than an exact value in scene level classification, the requirement for object detection accuracy is not high. Therefore, GroundingDINO can be used to perform zero-shot object detection to determine the number of objects in the image.

[0086] Specifically, firstly, a corresponding second prompt word is set according to the actual task and traffic scenario. For example, if the target dataset needs to be used for vehicle and pedestrian detection in an intersection scenario, then the second prompt word can be set to [vehicle.person.]. Then, the second prompt word and the dataset with the first label can be input into the second model for detection. This yields multiple detection boxes for each sample object in the initial sample image, the confidence score (score ∈ (0,1)) corresponding to each detection box, and the category of each pair of sample objects. Next, for each sample object, multiple detection boxes are filtered based on the confidence score. When the score ≥ the confidence threshold (e.g., 0.3), the category detection result of that sample object is determined to be a reliable result. i The corresponding reliable result for the target object is denoted as Y. i ,and in, The label for the reliable result in this initial sample image. m represents the number of reliable results; finally, further refine the frame... i The number of sample objects in the image is used to classify the sample objects and obtain the level label Y of the number of sample objects in the final single frame image. i * And Y i * ∈{Y i s ,Y i m ,Y i h When m∈[3,6), the quantity level label (i.e., the second target label) is Y. i s When m∈[6, 12), the quantity level label is Y. i m When m∈[12,+∞), the quantity level label is Y. i h .

[0087] Step 201a4: The object tracking device determines each first target label based on each first label and each second label.

[0088] In this embodiment of the social situation, for each initial sample image, the first label and the second label can be matched to obtain the first target label, and the first target label is denoted as... For each initial sample image frame i Simultaneously extract tags Weather, light intensity, traffic participant size, occlusion, time, and label Y i *The quantity is used to construct a first target label list [weather, light intensity, traffic participant size, occlusion, time, object quantity level] and establish scene level labels (L). SIMPLE ,L MEDIUM ,L HARD By mapping the frame, you can obtain the frame. i The corresponding scene level label L, where the scene level label L corresponds to L SIMPLE ,L MEDIUM ,L HARD The scene level mapping relationship is as follows:

[0089] ["Sunny day", "No strong light", "Same size", "Unobstructed", "Daytime", "Y"] i s ”], L SIMPLE

[0090] [Sunny / Cloudy / Rainy, No strong light, Inconsistent size, Obstructed, Daytime / Nighttime, Y] i m ”], L MEDIUM

[0091] [Sunny / Cloudy / Rainy / Foggy / Snowy, No strong light, Inconsistent size, Obstructed, Daytime / Nighttime, Y] i h ”], L HARD

[0092] It should be noted that after obtaining the initial sample images, these images can be filtered according to the complexity of the scene environment (i.e., the scene itself), and the scene complexity can be judged and divided into three levels L. SIMPLE L MEDIUM and L HARD Let the scene level label be L, and L∈{L} SIMPLE ,L MEDIUM ,L HARD The scene level label correspondence is described as follows: L SIMPLE Targets are clear, unobstructed, in low light, during the day, in clear weather, and of relatively uniform size; 3-6 target objects are visible within the effective field of view per frame, and the weather is not rainy or foggy; L MEDIUM Targets are clear, obstructed, without strong light, available in all weather conditions, sunny / cloudy / rainy, and of varying sizes (near and far), with 6-12 targets within the effective field of view per frame and in non-rainy / foggy weather; L MEDIUM The following conditions must be met: unclear target, target is obstructed, strong light, all time of day, all weather, target size is not uniform (near or far), more than 12 target objects are within the effective field of view of a single frame, and all weather.

[0093] It should be noted that, in order to ensure the rationality of data distribution, the distribution ratio of the number of scenarios at each level is controlled within L. SIMPLE ,L MEDIUM ,L HARD = 2:5:3. The final constructed initial dataset is denoted as D = {(frame1, L1), ..., (frame...}. i ,L i ),...,(frame n ,L n )}.

[0094] Step 201B: The object tracking device constructs a target sample dataset based on multiple initial sample images, multiple first target labels, and multiple second target labels.

[0095] In this embodiment of the application, the sample objects included in each initial sample image can also be manually labeled to obtain the detection box and the category of each sample object, and the category of the detection box and the object is determined as the second target label; each initial sample image in the target sample dataset has a first target label and a second target label, and each initial sample image can be labeled with its corresponding first target label and second target label, thereby constructing a target sample dataset based on multiple labeled initial sample images.

[0096] It should be noted that the above-mentioned method for constructing scene datasets (i.e., target sample datasets) utilizes the scene recognition and open set detection capabilities of multimodal large models to realize scene classification work that traditionally requires manual judgment and screening during dataset construction, thereby reducing labor costs and achieving automated construction of high-quality datasets.

[0097] Step 202: The object tracking device trains the initial detection model based on the target sample dataset and model fine-tuning technology to obtain the target detection model.

[0098] In this embodiment, the model fine-tuning technique specifically refers to Low Rank Adaptation (LoRA). To enhance the prediction accuracy of the initial detection model in a specific scenario, the initial detection model can be fine-tuned to improve its detection capability in that scenario, allowing it to better learn domain knowledge and thus better adapt to the needs and characteristics of traffic scenarios at the data distribution level. Specifically, the target sample dataset can be input into the initial detection model, and LoRA technique can be used to fine-tune the initial detection model during training. The core of LoRA fine-tuning is to construct a low-rank weight matrix based on the pre-trained model weight matrix. The specific steps are as follows:

[0099] Step 1: Weight matrix initialization: denote the pre-trained weight matrix of the initial detection model as W, and denote the weight matrix of the fine-tuned and updated object detection model as W+W0, wherein W0 is the fine-tuned and updated weight matrix, and W0<<W. Using a low-rank decomposition method, the weight matrix W0 is decomposed into a product of a matrix A and a matrix B, that is: W0=B×A, wherein both the matrix A and the matrix B are low-rank matrices, the matrix A is initialized as a zero matrix, and the matrix B is initialized as a Gaussian matrix.

[0100] Step 2: Weight update: first set the LoRA optimization objective:

[0101]

[0102] The pre-trained weight is constrained and updated by low-rank decomposition W+W0=W+B×A, wherein B∈R d×r , A∈R r×k , and r<<min(d,k). During training, W is not updated, and only the matrix A and matrix B are used to update the trainable parameters. Finally, W+W0 of the object detection model weight is generated after fine-tuning.

[0103] Step 203: The object tracking device determines the image frame number interval based on the motion parameters of the object to be tracked.

[0104] Step 204: The object tracking device acquires a target frame image from a target video sequence based on a current frame image and the image frame number interval.

[0105] In the embodiment of the present application, for target tracking in a global scene, realizing global tracking requires continuously detecting whether a new object appears in the scene. Therefore, it is necessary to determine the target frame image according to the image frame number interval first. Specifically, the image frame number interval (denoted as F) is mainly related to the motion parameters (such as moving speed) of the object to be tracked. When the moving speed of the object to be tracked is relatively fast, F should take a smaller value to capture the motion deformation of the object to be tracked and new objects; when the moving speed of the object to be tracked is relatively slow, F can take a relatively larger value.

[0106] In the embodiment of the present application, after the image frame number interval is determined, with the current frame image as a starting frame, a frame of image, that is, the target frame image, is acquired every the image frame number interval, and the RGB pixel matrix corresponding to the target frame image is denoted as frame rF , frame rF has a size of (h,w,c).

[0107] Step 205: The object tracking device detects the target frame image based on the target detection model to obtain a first detection bounding box of each object to be tracked.

[0108] The first detection box is obtained by filtering multiple candidate detection boxes for each object to be tracked.

[0109] In this embodiment, the target frame image can be input into the target detection model for detection, obtaining multiple detection boxes and the confidence score of each detection box for each tracked object. Then, based on the confidence score, the multiple detection boxes for each tracked object are filtered to obtain the first detection box for each tracked object. Specifically, the trained Grounding DINO model can be used to detect each tracked object in the target frame image and obtain its corresponding detection box. That is, the trained Grounding DINO model is used to detect n tracked objects in the target frame image and generate multiple two-dimensional detection boxes and the confidence score of each detection box for each tracked object, setting a confidence threshold α. To ensure the accuracy of subsequent segmentation, detection boxes with a score > α are selected as the first detection box for each tracked object. The set of first detection boxes obtained after filtering is denoted as: B = {[x1...} min ,y1 min ,x1 max ,y1 max ],...,[xn min ,yn min ,xn max ,yn max ]}, where n is the number of the first detection boxes, (xn min ,yn min (xn) represents the coordinates of the top-left pixel of the detection box. max ,yn max () represents the coordinates of the bottom right corner pixel of the detection box.

[0110] Step 206: The object tracking device determines the mask to be processed corresponding to each first detection box based on the target segmentation model, the target frame image, and multiple first detection boxes, and determines the first mask corresponding to the target frame image based on the multiple masks to be processed.

[0111] In this embodiment, multiple first detection boxes can be used as prompts for the target segmentation model. These first detection boxes and the target frame image are then input into the target segmentation model for segmentation, resulting in multiple masks for each tracked object and a confidence level for each mask. The mask with the highest confidence level is then selected as the mask to be processed for each tracked object, ultimately yielding a set of masks to be processed for the tracked objects in the target frame image. rF ={mask1,...,mask j ,...mask P}, where Masks rFIt is a matrix of dimensions (p, h, w), where h represents the image height, w represents the image width, and p is the number of masks, which is consistent with the number of objects to be tracked in the target frame image; mask j This represents the mask to be processed for a single object to be tracked. It is a Boolean matrix of dimension (h, w). The value of an element is True when its corresponding element is the mask for a specific object to be tracked; otherwise, it is False. Therefore, the mask to be processed for each object to be tracked can be numerically encoded to ensure the uniqueness of each mask. The unique ID is a corresponding non-repeating integer value, and non-tracking pixels are set to 0. The set of encoded masks is denoted as... The first mask corresponding to the final target frame image is It should be noted that after obtaining the mask, the subsequent target tracking model can continuously predict the mask of the same object to be tracked in subsequent frames based on the given mask.

[0112] Step 207: The object tracking device determines the second mask corresponding to the target frame image based on the target tracking model and the target frame image.

[0113] In this embodiment, the second mask is a mask predicted by the target tracking model; the output of the target tracking model (i.e., the second mask) is Pred rF Then Pred rF =T(frame) rF ), where T represents the encoding / decoding structure of the target tracking model, Pred rF This is the inference result of the target tracking model, with dimensions (h, w). It should be noted that when no new objects are detected, for the same object to be tracked, Pred... rF ID and The IDs in Pred are consistent, therefore Pred rF The values ​​of the pixel matrix are represented as integers.

[0114] Step 208: The object tracking device performs a first operation on the first mask and the second mask to obtain an updated mask, and updates the second mask based on the updated mask.

[0115] In this embodiment of the application, the first operation may refer to a matrix intersection operation; an updated mask can be obtained by performing a matrix intersection operation on the first mask and the second mask, that is... Since the target segmentation model has a higher segmentation capability than the predicted mask (i.e., the second mask), the second mask of the target tracking model can be updated with the calculated updated mask refMask. rF .

[0116] Step 209: The object tracking device obtains the third mask of the target object based on the first mask and the second mask.

[0117] In this embodiment, the second mask can be NOTed first, and then the result can be intersected with the first mask to obtain the mask for the newly added object (i.e., the third mask).

[0118] Step 210: The object tracking device performs a second operation on the update mask and the third mask to obtain the target mask.

[0119] In this embodiment, the second operation can refer to a matrix union operation; the target mask is the frame annotation of the target frame image; after obtaining the update mask and the third mask, a matrix union operation (i.e., merging) can be performed on the update mask and the third mask to obtain the target mask, i.e., newMask. rF =refMask rF ∪addMask rF .

[0120] Step 211: The object tracking device obtains the tracking results of each object to be tracked and the target object in the target frame image based on the target tracking model, target mask and target frame image.

[0121] The target object is a newly added object in the target frame image, in addition to the multiple objects to be tracked.

[0122] In this embodiment of the application, the annotation frame mask newMask is used. rF Together with the target frame image, the target tracking model T(·) can be used to infer the tracking result of the corresponding F+1 frame containing the mask of the target object (or not containing the target object), that is, the tracking result of each object to be tracked and the target object.

[0123] It should be noted that for target tracking in a full-scene scenario, the process begins with data collection using tools such as RTSP streaming. Next, the capabilities of a multimodal image-text scene understanding model and a target detection model are leveraged to match image scene complexity, which is then used as the basis for label matching, completing the construction of the scene dataset. Then, the initial detection model is fine-tuned using the constructed scene dataset (i.e., the target sample dataset) to improve the annotation accuracy of the tracking model. Finally, the mask of the tracked target is generated by calling the large model (i.e., the target segmentation model), and this mask is used as the frame annotation for the small tracking model (i.e., the target tracking model). Specifically, based on characteristics such as the target's movement speed within the scene, the target detection model and the segmentation model are called every F frames to obtain the annotated frame mask. The old target's mask annotation is updated with the large model's segmentation result, while some inference results from the small model are cached. The results from the large and small models are compared to obtain the new target's mask annotation, achieving full-scene target tracking.

[0124] It should be noted that the foregoing embodiments may further include the following steps:

[0125] Step 212: The object tracking device receives a click instruction for the current frame image and determines the target tracking object from multiple objects to be tracked based on the click instruction.

[0126] Step 213: The object tracking device acquires the second detection box corresponding to the target tracking object, and determines the fourth mask corresponding to the target tracking object based on the target segmentation model, the second detection box and the current frame image.

[0127] In this embodiment of the application, for tracking a specified target object, the target object can be determined from multiple objects to be tracked in an interactive manner. Specifically, for example... Figure 3 As shown, you can use the mouse to draw a rectangle in the current frame image to select the target object (i.e., the target tracking object). The target tracking object is then enclosed within this rectangle (i.e., the second detection box). The pixel coordinates of the second detection box of the target tracking object are denoted as R = {[x1...}. left ,y1 left ,x1 right ,y1 right ]}, where (x1 left ,y1 left (x1) represents the coordinates of the top-left pixel of the second detection box. right ,y1 right () represents the coordinates of the bottom right pixel of the second detection box. Then, the second detection box and the current frame image are input into the target segmentation model to obtain multiple masks corresponding to the target object and the confidence level of each mask. Finally, the mask with the highest confidence level is determined as the fourth mask corresponding to the target object.

[0128] Step 214: The object tracking device determines the tracking result of the target object based on the target tracking model, the fourth mask, the current frame image, and multiple consecutive frames following the current frame image in the target video sequence.

[0129] In this embodiment, the fourth mask, the current frame image, and multiple consecutive frame images can be input into the target tracking model. The target tracking model infers the mask of the target object in subsequent frames frame by frame. Post-processing then overlays the mask inference result onto the target object in the corresponding frame image to obtain the visualization result output by the target tracking model, such as... Figure 4 The overlay mask shown is the visualization result of tracking the specified vehicle (i.e., the target object).

[0130] It should be noted that for tracking a specific target object, the pixel coordinates of the target object or the coordinates of the selected detection box are obtained by interacting with the front-end interface through the mouse. For tracking a target object in the entire scene, the coordinates of the target object detection box in the entire scene are obtained by using the fine-tuned target detection model. The above coordinates are used as input to the segmentation model to generate the tracking annotation corresponding to the target object.

[0131] It should be noted that this application 1. achieves target tracking through a combination of large and small models: This method not only solves the problem of the accuracy of traditional tracking models decreasing over time, but also possesses the zero-shot / few-shot generalization capability that is difficult for traditional tracking algorithms to achieve. By combining the detection and segmentation results of the large model, it solves the problem of low tracking efficiency caused by the existing small model's inability to generalize to zero shots and poor domain adaptation, enabling the small model to be rapidly iterated, deployed, and applied to meet the required capabilities of the project. By obtaining the annotation frame mask through the large model and updating the old target object mask with the more accurate segmentation result of the large model, it solves the problem of the accuracy of traditional small tracking models decreasing over time. 2. It proposes a scene dataset construction method, that is, by utilizing the scene recognition and open set detection capabilities of the multimodal large model, the scene classification work that traditionally requires manual judgment and screening during dataset construction is realized, reducing manual costs and achieving automated construction of high-quality datasets. 3. A target tracking method based on a large traffic perception model is proposed. This method provides a unified architecture that combines large and small models to achieve tracking tasks. It constructs a complete solution from fine-tuning the dataset to the collaborative tracking of specified targets and targets across the entire scene using both large and small models. This solution can be quickly reused across different projects, and to some extent solves the problem that existing semi-supervised tracking models are unable to adapt to infinite scenarios and diverse task requirements. 4. An interactive target tracking method is provided. Multiple methods for obtaining annotation frames of the target to be tracked are proposed. These methods can interactively obtain the tracking results of a selected target, or obtain annotation frames of objects across the entire scene and generate tracking results. Compared with traditional tracking models that rely on manual annotation, this method can automatically obtain annotation frames and achieve flexible adaptation to application formats.

[0132] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0133] The object tracking method provided in this application adopts a target detection model with zero-shot recognition and language recognition capabilities. This not only solves the problem of the accuracy of traditional tracking models decreasing over time, but also has the zero-shot / few-shot generalization capability that is difficult for traditional tracking algorithms to achieve. Then, by combining the segmentation results of a large model such as a target segmentation model, it solves the problem of low tracking efficiency caused by the poor zero-shot generalization and poor domain adaptation of existing small models. This allows small models to be rapidly iterated, deployed, and applied. Furthermore, by obtaining the annotation frame mask (i.e., the target mask) through the large model, the old target mask can be updated with the segmentation results of the large model with higher accuracy. This solves the problem of the accuracy of small target tracking models decreasing over time, thereby improving the accuracy of object tracking.

[0134] Based on the foregoing embodiments, this application provides an object tracking device that can be applied to... Figure 1 and Figure 2 In the object tracking method provided in the corresponding embodiment, refer to Figure 5 As shown, the object tracking device 3 may include: an acquisition unit 31, a detection unit 32, a segmentation unit 33, a processing unit 34, and a tracking unit 35, wherein:

[0135] The acquisition unit 31 is used to determine the target frame image after the current frame image from the target video sequence; wherein the current frame image includes multiple objects to be tracked;

[0136] The detection unit 32 is used to determine the first mask corresponding to the target frame image based on the target detection model and the target segmentation model; wherein, the target detection model is a model that is trained by using a target sample dataset and model fine-tuning technology to train an initial detection model, and has zero-shot recognition capability and language recognition capability; the mask is the pixel matrix corresponding to the target frame image;

[0137] The segmentation unit 33 is used to determine the second mask corresponding to the target frame image based on the target tracking model and the target frame image;

[0138] Processing unit 34 is used to determine the target mask corresponding to the target frame image based on the first mask and the second mask;

[0139] The tracking unit 35 is used to obtain the tracking results of each object to be tracked and the target object in the target frame image based on the target tracking model, the target mask and the target frame image; wherein, the target object is a newly added object in the target frame image in addition to the multiple objects to be tracked.

[0140] In other embodiments of this application, the acquisition unit 31 is specifically used to perform the following steps:

[0141] The image frame interval is determined based on the motion parameters of the object to be tracked.

[0142] Based on the current frame image and the frame interval, the target frame image is obtained from the target video sequence.

[0143] In other embodiments of this application, the segmentation unit 33 is specifically used to perform the following steps:

[0144] The target frame image is detected based on the target detection model to obtain the first detection box for each object to be tracked; the first detection box is obtained by filtering multiple candidate detection boxes for each object to be tracked.

[0145] Based on the target segmentation model, the target frame image, and multiple first detection boxes, the mask to be processed corresponding to each first detection box is determined, and the first mask is determined based on the multiple masks to be processed.

[0146] In other embodiments of this application, the processing unit 34 is specifically used to perform the following steps:

[0147] Perform a first operation on the first mask and the second mask to obtain an updated mask, and update the second mask based on the updated mask;

[0148] The third mask of the target object is obtained based on the first and second masks;

[0149] Perform a second operation on the updated mask and the third mask to obtain the target mask.

[0150] In other embodiments of this application, the tracking unit 35 is specifically used to perform the following steps:

[0151] Receive a click command for the current frame image, and determine the target object to be tracked from multiple objects to be tracked based on the click command;

[0152] Obtain the second detection box corresponding to the target object being tracked, and determine the fourth mask corresponding to the target object being tracked based on the target segmentation model, the second detection box, and the current frame image;

[0153] Based on the target tracking model, the fourth mask, the current frame image, and multiple consecutive frames following the current frame image in the target video sequence, the tracking result of the target object is determined.

[0154] In other embodiments of this application, the detection unit 32 is specifically used to perform the following steps:

[0155] Based on each initial sample image and its content information in the initial sample dataset, a target sample dataset is constructed; the target sample dataset includes target images with different scene complexities.

[0156] Based on the target sample dataset and model fine-tuning techniques, the initial detection model is trained to obtain the target detection model.

[0157] In other embodiments of this application, the detection unit 32 is specifically used to perform the following steps:

[0158] Based on each piece of content information, a first target label and a second target label are determined for each initial sample image; wherein, the first target label represents the description information of the scene complexity corresponding to the initial sample image; and the second target label represents the number of objects in the initial sample image.

[0159] A target sample dataset is constructed based on multiple initial sample images, multiple first target labels, and multiple second target labels.

[0160] In other embodiments of this application, the detection unit 32 is specifically used to perform the following steps:

[0161] Construct a first prompt word, and determine a first label based on the first prompt word, each initial sample image, and the first network model; wherein, the first prompt word is used to obtain environmental description information of the image;

[0162] Construct a second prompt word, and based on the second prompt word, each initial sample image, and the second network model, determine each second label and each second target label; wherein, the second prompt word is used to obtain the number of sample objects of each class in the image;

[0163] Each first target label is determined based on each first label and each second label.

[0164] It should be noted that a detailed explanation of the steps performed by each unit can be found in [reference needed]. Figure 1 and Figure 2 The object tracking method provided in the corresponding embodiment will not be described in detail here.

[0165] The object tracking device provided in this application adopts a target detection model with zero-shot recognition and language recognition capabilities. This not only solves the problem of the accuracy of traditional tracking models decreasing over time, but also has the zero-shot / few-shot generalization capability that is difficult for traditional tracking algorithms to achieve. Then, by combining the segmentation results of a large model such as a target segmentation model, it solves the problem of low tracking efficiency caused by the existing small models' inability to generalize with zero-shot and poor domain adaptation. This allows the small model to be rapidly iterated, deployed, and applied. Furthermore, by obtaining the annotation frame mask (i.e., the target mask) through the large model, the old target mask can be updated with the segmentation result of the large model with higher accuracy. This solves the problem of the accuracy of small target tracking models decreasing over time, thereby improving the accuracy of object tracking.

[0166] Based on the foregoing embodiments, embodiments of this application provide an object tracking device that can be applied to... Figure 1 and Figure 2 In the object tracking method provided in the corresponding embodiment, refer to Figure 6 As shown, the object tracking device 4 may include: a processor 41, a memory 42, and a communication bus 43, wherein:

[0167] Communication bus 43 is used to realize the communication connection between processor 41 and memory 42;

[0168] Processor 41 is used to execute the object tracking program in memory 42 to perform the following steps:

[0169] Determine the target frame image after the current frame image from the target video sequence; wherein the current frame image includes multiple objects to be tracked;

[0170] Based on the object detection model and the object segmentation model, the first mask corresponding to the target frame image is determined; wherein, the object detection model is a model that is trained by using the target sample dataset and model fine-tuning technology to train the initial detection model, and has zero-shot recognition capability and language recognition capability; the mask is the pixel matrix corresponding to the target frame image;

[0171] Based on the target tracking model and the target frame image, determine the second mask corresponding to the target frame image;

[0172] Based on the first mask and the second mask, determine the target mask corresponding to the target frame image;

[0173] Based on the target tracking model, target mask, and target frame image, the tracking results of each object to be tracked and the target object in the target frame image are obtained; where the target object is a newly added object in the target frame image in addition to the multiple objects to be tracked.

[0174] In other embodiments of this application, processor 41 is configured to execute the object tracking program in memory 42 to determine the target frame image after the current frame image from the target video sequence, in order to perform the following steps:

[0175] The image frame interval is determined based on the motion parameters of the object to be tracked.

[0176] Based on the current frame image and the frame interval, the target frame image is obtained from the target video sequence.

[0177] In other embodiments of this application, processor 41 is used to execute the object tracking program in memory 42 based on the object detection model and the object segmentation model to determine the first mask corresponding to the target frame image, so as to implement the following steps:

[0178] The target frame image is detected based on the target detection model to obtain the first detection box for each object to be tracked; the first detection box is obtained by filtering multiple candidate detection boxes for each object to be tracked.

[0179] Based on the target segmentation model, the target frame image, and multiple first detection boxes, the mask to be processed corresponding to each first detection box is determined, and the first mask is determined based on the multiple masks to be processed.

[0180] In other embodiments of this application, the processor 41 is used to execute the object tracking program in the memory 42 to determine the target mask corresponding to the target frame image based on the first mask and the second mask, in order to implement the following steps:

[0181] Perform a first operation on the first mask and the second mask to obtain an updated mask, and update the second mask based on the updated mask;

[0182] The third mask of the target object is obtained based on the first and second masks;

[0183] Perform a second operation on the updated mask and the third mask to obtain the target mask.

[0184] In other embodiments of this application, processor 41 is configured to execute an object tracking method in an object tracking program stored in memory 42 to perform the following steps:

[0185] Receive a click command for the current frame image, and determine the target object to be tracked from multiple objects to be tracked based on the click command;

[0186] Obtain the second detection box corresponding to the target object being tracked, and determine the fourth mask corresponding to the target object being tracked based on the target segmentation model, the second detection box, and the current frame image;

[0187] Based on the target tracking model, the fourth mask, the current frame image, and multiple consecutive frames following the current frame image in the target video sequence, the tracking result of the target object is determined; where the candidate frame image is multiple consecutive frames following the current frame image in the target video sequence.

[0188] In other embodiments of this application, processor 41 is configured to execute an object tracking method in an object tracking program stored in memory 42 to perform the following steps:

[0189] Based on each initial sample image and its content information in the initial sample dataset, a target sample dataset is constructed; the target sample dataset includes target images with different scene complexities.

[0190] Based on the target sample dataset and model fine-tuning techniques, the initial detection model is trained to obtain the target detection model.

[0191] In other embodiments of this application, processor 41 is used to execute the object tracking program in memory 42 to construct a target sample dataset based on each initial sample image and the content information of each initial sample image in the initial sample dataset, in order to implement the following steps:

[0192] Based on each piece of content information, a first target label and a second target label are determined for each initial sample image; wherein, the first target label represents the description information of the scene complexity corresponding to the initial sample image; and the second target label represents the number of objects in the initial sample image.

[0193] A target sample dataset is constructed based on multiple initial sample images, multiple first target labels, and multiple second target labels.

[0194] In other embodiments of this application, processor 41 is configured to execute a content-based object tracking program in memory 42 to determine a first target label and a second target label corresponding to each initial sample image, in order to perform the following steps:

[0195] Construct a first prompt word, and determine a first label based on the first prompt word, each initial sample image, and the first network model; wherein, the first prompt word is used to obtain environmental description information of the image;

[0196] Construct a second prompt word, and based on the second prompt word, each initial sample image, and the second network model, determine each second label and each second target label; wherein, the second prompt word is used to obtain the number of sample objects of each class in the image;

[0197] Each first target label is determined based on each first label and each second label.

[0198] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 and Figure 2 The object tracking method provided in the corresponding embodiment will not be described in detail here.

[0199] The object tracking device provided in this application adopts a target detection model with zero-shot recognition and language recognition capabilities. This not only solves the problem of the accuracy of traditional tracking models decreasing over time, but also has the zero-shot / few-shot generalization capability that is difficult for traditional tracking algorithms to achieve. Then, by combining the segmentation results of a large model such as a target segmentation model, it solves the problem of low tracking efficiency caused by the existing small models' inability to generalize with zero-shot and poor domain adaptation. This allows the small model to be rapidly iterated, deployed, and applied. Furthermore, by obtaining the annotation frame mask (i.e., the target mask) through the large model, the old target mask can be updated with the segmentation result of the large model with higher accuracy. This solves the problem of the accuracy of small target tracking models decreasing over time, thereby improving the accuracy of object tracking.

[0200] Based on the foregoing embodiments, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to achieve... Figure 1 and Figure 2 The steps in the object tracking method provided in the corresponding embodiment.

[0201] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0202] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0203] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0204] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0205] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0206] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0207] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0208] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An object tracking method, characterized in that, The method includes: The target frame image is determined from the target video sequence after the current frame image; wherein the current frame image includes multiple objects to be tracked; Based on the object detection model and the object segmentation model, a first mask corresponding to the target frame image is determined; wherein, the object detection model is a model trained with an initial detection model using a target sample dataset and model fine-tuning techniques, and possesses zero-shot recognition capability and language recognition capability, and is an open-set object detection large model used for detecting the object to be tracked; the mask is the pixel matrix corresponding to the target frame image; the object segmentation model is a large segmentation model used to segment each of the objects to be tracked; Based on the target tracking model and the target frame image, a second mask corresponding to the target frame image is determined; wherein, the target tracking model is a small tracking model, and compared with the large tracking model, the small tracking model has fewer parameters and lower computational requirements; Based on the first mask and the second mask, the target mask corresponding to the target frame image is determined; Based on the target tracking model, the target mask, and the target frame image, the tracking results of each of the objects to be tracked and the target objects in the target frame image are obtained; wherein, the target object is a newly added object in the target frame image in addition to the multiple objects to be tracked.

2. The method according to claim 1, characterized in that, Determining the target frame image after the current frame image from the target video sequence includes: The image frame interval is determined based on the motion parameters of the object to be tracked; The target frame image is obtained from the target video sequence based on the current frame image and the frame interval.

3. The method according to claim 1, characterized in that, The step of determining the first mask corresponding to the target frame image based on the target detection model and the target segmentation model includes: The target frame image is detected based on the target detection model to obtain a first detection box for each of the objects to be tracked; wherein the first detection box is obtained by filtering multiple candidate detection boxes for each of the objects to be tracked. Based on the target segmentation model, the target frame image, and multiple first detection boxes, a mask to be processed is determined for each first detection box, and the first mask is determined based on the multiple masks to be processed.

4. The method according to claim 1, characterized in that, Determining the target mask corresponding to the target frame image based on the first mask and the second mask includes: A first operation is performed on the first mask and the second mask to obtain an update mask, and the second mask is updated based on the update mask; A third mask for the target object is obtained based on the first mask and the second mask; A second operation is performed on the updated mask and the third mask to obtain the target mask.

5. The method according to claim 1, characterized in that, The method further includes: Receive a click instruction for the current frame image, and determine the target tracking object from a plurality of objects to be tracked based on the click instruction; Obtain the second detection box corresponding to the target tracking object, and determine the fourth mask corresponding to the target tracking object based on the target segmentation model, the second detection box, and the current frame image; Based on the target tracking model, the fourth mask, the current frame image, and multiple consecutive frames following the current frame image in the target video sequence, the tracking result of the target object is determined.

6. The method according to claim 1, characterized in that, Before determining the target frame image after the current frame image from the target video sequence, the process also includes: Based on each initial sample image in the initial sample dataset and the content information of each initial sample image, a target sample dataset is constructed; wherein, the target sample dataset includes target images with different scene complexities; The target detection model is obtained by training the initial detection model based on the target sample dataset and model fine-tuning techniques.

7. The method according to claim 6, characterized in that, The construction of the target sample dataset based on each initial sample image and its content information in the initial sample dataset includes: Based on each of the aforementioned content information, a first target label and a second target label are determined for each of the initial sample images; wherein, the first target label represents the descriptive information of the scene complexity corresponding to the initial sample image; and the second target label represents the quantity information of objects in the initial sample image. The target sample dataset is constructed based on multiple initial sample images, multiple first target labels, and multiple second target labels.

8. The method according to claim 7, characterized in that, The step of determining the first target label and the second target label corresponding to each initial sample image based on each of the content information includes: A first prompt word is constructed, and based on the first prompt word, each initial sample image, and the first network model, each first label is determined; wherein, the first prompt word is used to obtain environmental description information of the image; A second prompt word is constructed, and based on the second prompt word, each initial sample image, and the second network model, each second label and each second target label are determined; wherein, the second prompt word is used to obtain the number of sample objects of each class in the image; Each first target label is determined based on each first label and each second label.

9. An object tracking device, characterized in that, The device includes: An acquisition unit is used to determine the target frame image after the current frame image from the target video sequence; wherein the current frame image includes multiple objects to be tracked; The detection unit is used to determine a first mask corresponding to the target frame image based on the target detection model and the target segmentation model; wherein, the target detection model is a model trained on an initial detection model using a target sample dataset and model fine-tuning techniques, and has zero-shot recognition capability and language recognition capability, and is an open-set object detection large model used for detecting the object to be tracked; the mask is the pixel matrix corresponding to the target frame image; the target segmentation model is a large segmentation model used to segment each of the objects to be tracked; A segmentation unit is used to determine a second mask corresponding to the target frame image based on the target tracking model and the target frame image; wherein the target tracking model is a small tracking model, and compared with the large tracking model, the small tracking model has fewer parameters and lower computational requirements; The processing unit is configured to determine the target mask corresponding to the target frame image based on the first mask and the second mask; The tracking unit is used to obtain the tracking result of each of the objects to be tracked and the target objects in the target frame image based on the target tracking model, the target mask and the target frame image; wherein the target object is a newly added object in the target frame image in addition to the multiple objects to be tracked.

10. An object tracking device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute the object tracking program stored in the memory to implement the steps of the object tracking method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the object tracking method as described in any one of claims 1 to 9.

Images