Object tracking device, object tracking method thereof and model learning method thereof
By using a meta-learning-based object tracking device, combined with visual landing, object detection, and interaction prediction modules, the semantic and positional trajectories of targets are predicted. This solves the problem that existing technologies cannot fully track the semantic trajectories of targets, enabling safe application and improved model generalization capabilities in factory and supply chain management environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HYUNDAI MOTOR CO LTD
- Filing Date
- 2025-05-28
- Publication Date
- 2026-05-22
AI Technical Summary
Existing object tracking methods can only track the location information of the target, which may be insufficient or incomplete in various real-world scenarios. They cannot effectively obtain the semantic trajectory of the target, thus limiting their application in fields such as security, protection, welfare, productivity, sales, and fraud.
A meta-learning-based object tracking device is adopted. By receiving video frames and target initialization statements, it uses a visual landing module, an object detection module, an interaction prediction module, and a multi-object tracking module. Combined with a long short-term memory model and virtual training and testing, it predicts the semantic trajectory and positional trajectory of the target. Meta-optimization is used to improve the generalization ability of the model.
It enables in-depth tracking and understanding of targets, allowing for the understanding of the semantic trajectories of workers or vehicles in factory and supply chain management environments, reducing the risk of accidents and injuries, and improving the model's generalization ability across various data distributions.
Smart Images
Figure CN122072969A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to object tracking, and more specifically, to a meta-learning-based object tracking apparatus, its object tracking method, and its model learning method. Background Technology
[0002] Single object tracking aims to track the positional trajectory of a target in a video. The target should be specified in the first frame of the video and detected and tracked in subsequent frames.
[0003] Object tracking is one of the most important tasks in the field of computer vision, and it is widely used in various fields such as crowd management, robotics, and autonomous vehicle tracking.
[0004] Object tracking methods based on target initialization in the first frame can be divided into two types: bounding box-based object tracking and language-based object tracking.
[0005] Bounding box-based object tracking methods typically initialize the target using the target's coordinates in the first frame, while language-based object tracking methods typically initialize the target using a language description in the first frame.
[0006] Since the above object tracking methods only track the target's position information in the frame, they may be insufficient and / or incomplete in various real-world scenarios.
[0007] Obtaining the semantic trajectory of a target can be useful and beneficial in a variety of real-world scenarios, including security, safety, well-being, productivity, sales, and deceptive practices.
[0008] For example, in a factory or supply chain management environment, understanding the semantic trajectory of workers or vehicles can help minimize the risk of accidents and injuries.
[0009] For example, when the semantic trajectory of a target (person or vehicle) is understood, such as "how safe it is for workers to operate forklifts and transport goods", "how safe it is for drivers to drive to avoid collisions", "how long workers have been working and whether workers need to rest", various measures can be taken, such as adopting systems to prevent accidents and injuries.
[0010] Therefore, a target tracking technology capable of tracking the semantic trajectory of a target (e.g., the positional trajectory of surrounding objects, the interaction between the target and surrounding objects over time, etc.) and the target's positional trajectory would be beneficial.
[0011] The above description of the background technology is only intended to enhance the understanding of the background technology of this disclosure and should not be regarded as an admission that they correspond to the prior art known to those skilled in the art. Summary of the Invention
[0012] This disclosure provides an object tracking technique capable of tracking the semantic trajectory and positional trajectory of a target.
[0013] Another objective of this disclosure is to provide an object tracking technology capable of locating and tracking targets based on input target initialization statements, acquiring the semantic trajectory of targets through tracking, and improving the comprehensive tracking and understanding of targets based on the semantic trajectory of targets.
[0014] Another objective of this disclosure is to provide an object tracking technique that can be applied to various data distributions (i.e., various situations) by applying virtual learning, virtual testing, and meta-optimization.
[0015] Another objective of this disclosure is to provide an object tracking device, an object tracking method, and a model learning method that include the object tracking technology proposed in this disclosure.
[0016] The technical problems to be solved by this disclosure are not limited to those described above, and those skilled in the art will clearly understand from the following description other technical problems not mentioned.
[0017] An object tracking apparatus for achieving the above objectives according to embodiments of the present disclosure may include: a memory for storing a model for object detection and tracking; and a processor for running the model.
[0018] According to embodiments of this disclosure, when running the model, the processor can receive video frames and target initialization statements as input; when the input video frame is the first video frame of the video sequence, the target is specified from the first video frame based on the target initialization statement; when the input video frame is not the first video frame, objects that may interact with the specified target in the input video frame are detected; by predicting the interaction between the specified target and the detected objects, the target and related objects are determined from the corresponding video frames; the target and related objects are determined until the final video frame of the video sequence, and the trajectories of the target and related objects are predicted.
[0019] According to an embodiment, when a target and related objects are determined, the processor can determine the target by matching the specified target with the detected objects, and determine other objects as candidate related objects.
[0020] According to an embodiment, the processor can predict the interaction between the target and each candidate related object based on the characteristics of the target and the characteristics of the candidate related objects, and determine the related objects that interact with the target among the candidate related objects.
[0021] According to an embodiment, the processor can predict the interaction between the target in the current video frame and each candidate related object by concatenating the features of the target in the current video frame with the features of the candidate related objects in each of the previous number of video frames and then inputting these features into a Long Short-Term Memory (LSTM) model.
[0022] According to an embodiment, the processor can add the category of an irrelevant object to the relevant directory of irrelevant objects that do not interact with the target among the candidate relevant objects.
[0023] According to an embodiment, the trajectory may include the location trajectory and semantic trajectory of the target and each related object.
[0024] According to an embodiment, the processor can perform meta-learning on the model based on the training dataset.
[0025] According to an embodiment, the training dataset may include a support dataset and multiple query datasets.
[0026] According to the embodiments, the data distribution of multiple query datasets and supporting datasets differs in terms of object categories and interaction types.
[0027] According to an embodiment, the processor can update the model by reflecting the loss of virtual training performed based on a supporting dataset and the loss of virtual testing performed based on multiple query datasets.
[0028] According to an embodiment, the processor can perform virtual testing on the virtually updated model based on multiple query datasets, reflecting the loss from virtual training.
[0029] According to an embodiment, the vehicle includes the object tracking device.
[0030] According to an embodiment, the object tracking method according to an embodiment of the present disclosure includes the following steps: receiving a video sequence and a target initialization statement as input by a processor; specifying a target from a first video frame of the video sequence based on the target initialization statement by the processor; detecting objects that may interact with the specified target from other input video frames of the video sequence by the processor; predicting the interaction between the specified target and the detected objects by the processor; determining the target and related objects from the corresponding video frames based on the predicted interaction by the processor; and determining the target and related objects up to the final video frame of the video sequence by the processor, and predicting the trajectory of the target and related objects.
[0031] According to an embodiment, the steps of determining the target and related objects may include: determining the target by matching the specified target with the detected objects, and determining other objects as candidate related objects.
[0032] According to an embodiment, the step of determining the target and related objects may include: predicting the interaction between the target and each candidate related object based on the features of the target and the features of the candidate related objects, and determining the related objects among the candidate related objects that interact with the target.
[0033] According to an embodiment, the step of determining the target and related objects may include: predicting the interaction between the target in the current video frame and each candidate related object by concatenating the features of the target in the current video frame with the features of the candidate related objects in each of the previous number of video frames and then inputting these features into a Long Short-Term Memory (LSTM) model.
[0034] According to an embodiment, the step of determining the target and related objects may include: adding categories indicating irrelevant objects to the related directory of irrelevant objects that do not interact with the target among the candidate related objects.
[0035] The object tracking model learning method according to embodiments of the present disclosure is a method for learning an object tracking model using an object tracking device. The method includes the following steps: training an object tracking model based on a support dataset and multiple query datasets using a processor; and performing meta-optimization on the parameters of the object tracking model using a processor based on the loss obtained by performing virtual training computation based on the support dataset and the loss obtained by performing virtual test computation based on the multiple query datasets.
[0036] According to an embodiment, the data distribution of multiple query datasets and supporting datasets may differ in terms of object category and interaction type.
[0037] According to an embodiment, a virtual test can be performed on the updated target tracking model based on the loss calculated during virtual training.
[0038] According to an embodiment, this target tracking model learning method can perform meta-optimization on the parameters of the target tracking model based on Equation 1.
[0039] [Equation 1]
[0040] in
[0041] L(D s ;θ) is the loss calculated for parameter θ during virtual training. It is the loss of the parameters θ′ calculated during virtual testing, and θ′ is the parameter updated through virtual training.
[0042] According to an embodiment, this target tracking model learning method can update the parameters of the target tracking model based on Equation 2.
[0043] [Equation 2]
[0044] in
[0045] α represents the weights, and β represents the learning rate for the overall optimization.
[0046] According to embodiments of this disclosure, an object tracking technology can be provided that can track not only the location trajectory of a target but also its semantic trajectory.
[0047] Therefore, since the semantic trajectory of a target predicted using the object tracking technology according to this embodiment can include the bounding box of the target, the bounding boxes and categories of surrounding related objects, and interactions that change over time, it can provide in-depth tracking and understanding of the target.
[0048] When using this object tracking technology, it is possible to understand the semantic trajectory of workers or vehicles in factories, supply chain management environments, and so on, with the expectation that this can minimize the risk of accidents and injuries.
[0049] Furthermore, according to embodiments of this disclosure, since virtual testing is performed on the object tracking model based on query datasets with various data distributions, and the results of the virtual testing are applied to meta-optimization to update the object tracking model, the generalization ability of the object tracking model can be improved, and it can be effectively applied to various real-world scenarios.
[0050] The effects that can be obtained by this disclosure are not limited to those described above, and other effects not mentioned can be clearly understood by those skilled in the art from the following description. Attached Figure Description
[0051] Figure 1 This is a view showing the configuration of an object tracking device according to an embodiment of the present disclosure.
[0052] Figure 2 This is a functional block diagram illustrating the operation of a processor according to an embodiment of the present disclosure.
[0053] Figure 3 This is a flowchart illustrating an object tracking method according to an embodiment of the present disclosure.
[0054] Figure 4 This is a flowchart illustrating a method for learning an object tracking model according to an embodiment of the present disclosure. Detailed Implementation
[0055] It should be understood that the terms "vehicle" or "automobile," or other similar terms used herein, include general motor vehicles such as passenger cars, which include sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, water vehicles including various boats and ships, aircraft, etc., and also include hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other alternative fuel vehicles (e.g., fuels derived from resources other than petroleum). As described herein, a hybrid vehicle is a vehicle with two or more power sources, such as a vehicle with both gasoline and electric power.
[0056] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when the terms “comprising” and / or “including” are used in the specification, they specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. As used herein, the term “and / or” includes any and all combinations of one or more associated listed items. Throughout the specification, unless explicitly stated otherwise, the word “comprising” and variations such as “including” or “including” will be understood to imply the inclusion of the stated elements rather than the exclusion of any other elements. Additionally, the terms “unit,” “device,” “module,” and “module” described herein refer to a unit for performing at least one function and operation and may be implemented by hardware components or software components and combinations thereof.
[0057] Furthermore, the control logic of this disclosure can be implemented as a non-transitory computer-readable medium containing executable program instructions that run by a processor, controller, etc. Examples of computer-readable media include, but are not limited to, ROM, RAM, optical disc (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable medium can also be distributed across a network-connected computer system, enabling it to be stored and operated in a distributed manner, for example, via a telematics server or a controller area network (CAN).
[0058] In describing the embodiments disclosed in this specification, detailed descriptions of related known technologies will be omitted where it is determined that such detailed descriptions may obscure the spirit of the embodiments disclosed in this specification. Furthermore, the accompanying drawings are provided only to facilitate understanding of the embodiments disclosed in this specification, and the technical spirit disclosed in this specification is not limited to the drawings but should be understood to include all modifications, equivalents, and alternatives within the spirit and scope of this disclosure.
[0059] While terms including ordinal numbers such as first, second, etc., can be used to describe various components, components are not limited to these terms. These terms are only used to distinguish one component from others.
[0060] Unless the context clearly indicates otherwise, a singular expression also indicates a plural expression.
[0061] When a component is referred to as being "connected" or "attached" to another component, it should be understood that although the component may be directly connected or attached to another component, there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly attached" to another component, it should be understood that there are no other components in between.
[0062] In the following description, the embodiments disclosed herein will be described in detail with reference to the accompanying drawings, and the same or similar components will be given the same reference numerals regardless of the reference numerals used, and repeated descriptions thereof will be omitted.
[0063] Figure 1 This is a view showing the configuration of an object tracking device 100 according to an embodiment of the present disclosure.
[0064] Reference Figure 1 According to embodiments of this disclosure, the object tracking device 100 can be a computing device for tracking objects in an input video sequence. For example, the object tracking device 100 can be implemented in a system monitoring a vehicle manufacturing plant and can receive video sequences from cameras capturing the plant's internal environment. For example, the object tracking device 100 can be mounted on a vehicle and receive video sequences from cameras mounted on the vehicle.
[0065] According to an embodiment, the object tracking device 100 may include a processor 110, a memory 130, a storage device 150, a user interface 170, and a bus 190.
[0066] The processor 110 may be a data processing device implemented in hardware having a physical structure for performing the required operations.
[0067] Processor 110 controls the overall operation of each component of object tracking device 100. Processor 110 may be configured to include at least one of a central processing unit (CPU), microprocessor unit (MPU), microcontroller unit (MCU), graphics processing unit (GPU), and any type of processor known in the art of this disclosure. Additionally, processor 110 may perform operations for at least one application or program to run methods / operations according to various embodiments of this disclosure.
[0068] Memory 130 stores various data, commands, and / or information. Memory 130 can load one or more computer programs from storage device 150 to run methods / operations according to various embodiments of this disclosure. For example, memory 130 may be random access memory (RAM) or dynamic random access memory (DRAM), but is not limited thereto, and may be configured to include at least any type of memory known in the art of this disclosure.
[0069] Storage device 150 can non-transitory store one or more computer programs. Storage device 150 can be configured as a non-volatile memory such as flash memory, hard disk, removable disk, or any type of computer-readable recording medium known in the art.
[0070] For example, a computer program may include one or more instructions for implementing methods / operations according to various embodiments of the present disclosure. When the computer program is loaded into memory 130, processor 110 may execute methods / operations according to various embodiments of the present disclosure by running one or more instructions.
[0071] User interface 170 can receive commands, data, information, etc. from outside the object tracking device 100. User interface 170 can output the operation results of object tracking device 100. For example, user interface 170 may include a keyboard, mouse, display, touch screen, etc.
[0072] Bus 190 provides communication functionality between components of object tracking device 100. Bus 190 can be implemented as various types of buses, such as address bus, data bus, control bus, etc.
[0073] Figure 2 This is a functional block diagram illustrating the operation of processor 110 according to an embodiment of the present disclosure.
[0074] According to an embodiment, the processor 110 can receive a target initialization statement and a video sequence as input, and track the semantic trajectory of the target based on the target initialization statement and the video sequence.
[0075] Here, the target initialization statement is a statement that describes the target and is used to spatially specify the target in a video frame and begin tracking. Additionally, the target's semantic trajectory can include the target's bounding box, the bounding boxes and categories of surrounding related objects, and interactions that change over time.
[0076] For this purpose, an object tracking model can be installed on the processor 110. The operation of the processor 110 described below can be performed by the object tracking model installed on the processor 110, and the object tracking function of the object tracking device 100 can be implemented by the object tracking model.
[0077] Reference Figure 2 The processor 110 (or object tracking model) may include a visual grounding module 111, an object detection module 112, an interaction prediction module 113, and a multi-object tracking module 114.
[0078] The visual landing module 111 can receive the target initialization statement and the first video frame of the video sequence, and can specify the target from the first video frame based on the target initialization statement.
[0079] The visual implementation module 111 can use a previously learned language model to generate embedding vectors from the target initialization statement. For example, the language model can be selected from language models well known in the art of this disclosure, such as BERT, Global Vectors (GloVe), fastText, and Language Model Embeddings (ELMo).
[0080] The visual landing module 111 can target a specified object from the first video frame based on the embedding vector.
[0081] The object detection module 112 can receive a video sequence and detect objects that may interact with the target from the video sequence based on a preset object detection algorithm. Here, the object may include the target.
[0082] For example, the object detection module 112 can use a YOLOx-based object detection model to detect objects from a video sequence, and the object detection model or object detection algorithm used by the object detection module 112 is not limited to this.
[0083] According to an embodiment, the object detection module 112 extracts features f_t from the input video frame I_t, and detects the bounding box b_(t,i) and object category c_(t,i) of the object in the video frame based on the extracted features.
[0084] For example, the target detection module 112 is configured to include: a backbone network for extracting features from input video frames; a neck network for collecting the extracted features; and a head network for detecting the bounding box and target category of the target based on the features collected by the neck network.
[0085] To extract high-dimensional features, the backbone network can be configured as a Darknet53 model, the neck network as a Feature Pyramid Network (FPN) model, and the head network as a YOLOx model. However, the types of models constituting the backbone, neck, and head networks are not limited to these.
[0086] For example, ResNet-50 and SpinNet models can be configured with a backbone network, while Path Augmentation Network (PAN), Neural Architecture Search-FPN (NAS-FPN), Fully Connected FPN, and Adaptive Spatial Feature Fusion (ASFF) models can be configured with a neck network, and Single Frame Detector (SSD) and RetinaNet models can be configured with a head network.
[0087] According to an embodiment, the object detection module 112 can further extract intermediate features f_(t,i) corresponding to the bounding box based on the region of interest (RoI) alignment technique to obtain more information about the object.
[0088] The interaction prediction module 113 can predict the interaction between the target and related objects.
[0089] The interactive prediction module 113 can determine the target by matching the target specified by the visual landing module 111 with the objects detected by the object detection module 112 for the input video frame I_t, and can also identify other objects as candidate related objects.
[0090] The interaction prediction module 113 can predict the interaction for each pair based on the features of the target and the features of the candidate related objects. For example, the interaction prediction module 113 can predict the interaction type based on a convolutional neural network (CNN) model, but is not limited to this.
[0091] For example, assuming the target is A and the candidate related objects are B, C and D, the interaction prediction module 113 can predict the interaction between A and B, the interaction between A and C, and the interaction between A and D.
[0092] Information obtained from previous video frames can be used to predict interactions in the next video frame.
[0093] Therefore, embodiments of this disclosure use a Long Short-Term Memory (LSTM) model to predict interactions by using intermediate features from multiple video frames as input.
[0094] The interaction prediction module 113 can predict the interaction between the target and the candidate related objects in the current video frame by connecting the features of the target in the current video frame with the features of the candidate related objects in the previous K video frames and inputting these features into the LSTM model, and determine the candidate related objects that interact with the target from among the candidate related objects.
[0095] In the following text, the candidate related objects that interact with the target will be referred to as "related objects".
[0096] The interaction prediction module 113 can indicate irrelevant objects by adding the "background" category to the relevant directory of candidate relevant objects (irrelevant objects) that have not interacted with the target.
[0097] Therefore, object tracking models can improve the efficiency and performance of object tracking by focusing on tracking relevant objects without performing tracking on unrelated objects that do not interact with the target.
[0098] In this way, the interactive prediction module 113 can filter out irrelevant objects during the interaction between the prediction target and related objects.
[0099] Therefore, since irrelevant objects are excluded and only relevant objects are input into the multi-object tracking module 114, the semantic trajectory of the objects can focus on providing information about the target.
[0100] The multi-object tracking module 114 can predict the trajectory of the target and the trajectories of related objects, and track the target and related objects based on the trajectory of each target and related object. Here, the trajectory can include positional trajectory and semantic trajectory.
[0101] For example, the multi-object tracking module 114 can predict the trajectory of the target and related objects based on the ByteTrack method.
[0102] According to an embodiment, when a target and related objects in two consecutive video frames are input, the multi-object tracking module 114 can predict the trajectory of the target and each related object by matching the target and related objects in the two consecutive video frames.
[0103] For example, the multi-object tracking module 114 can predict the trajectory of a target by matching the target in each of the consecutive first and second video frames, and predict the trajectory of a related object by matching the related object in each of the consecutive first and second video frames.
[0104] On the other hand, semantic tracking is not simply a combination of tasks, but rather a framework that makes it easier for people to track targets using sentences by applying semantic information to object tracking.
[0105] Semantic tracking-based frameworks enable end-to-end learning. During the prediction of the location trajectories and interactions between the target and related objects, the object tracking model can learn interaction information. This interaction information can be used to track the target and related objects, while the location trajectory information can be used to predict the interactions between the target and related objects.
[0106] For example, the interaction "lean" might suggest that two objects (the target and the related object) are moving in the same direction. As another example, a trajectory pattern such as the related object moving away from the target might suggest the interaction "throw".
[0107] These examples demonstrate the possibility of building a joint framework for semantic tracking.
[0108] Simultaneous consideration of location trajectory prediction and interaction prediction can generate new tasks.
[0109] For example, the phrase "an adult drinks water from a bottle" may be used far more frequently than the phrase "an adult cleans a bottle." Therefore, when a trained, phrase-biased model predicts the interaction between the target "adult" and the related object "bottle," it may predict the interaction as "an adult drinks water from a bottle" even if the actual interaction is "an adult cleans a bottle."
[0110] In this way, considering both location trajectory prediction and interaction prediction simultaneously, it is necessary to address the problem of inaccurate model predictions caused by biased training methods.
[0111] As the example illustrates, model performance can be affected by the data distribution used during training. Furthermore, in real-world scenarios, there may be various mismatches between the data distribution and the distribution of the training dataset.
[0112] Semantic tracking models can acquire semantic trajectories that include the object's category and interaction type.
[0113] Embodiments of this disclosure propose an object tracking model that can take into account different data distributions of object categories and interaction types, thereby improving the performance of the model.
[0114] Meta-learning aims to improve the generalization ability of a model by using virtual testing during model learning, and it is very effective in improving the generalization ability to new tasks, new domains, etc.
[0115] To improve object tracking performance, the object tracking apparatus 100 (or object tracking model) according to embodiments of the present disclosure may perform meta-learning.
[0116] According to the implementation example, the training dataset can be divided into a support dataset and N query datasets.
[0117] The support dataset can be used for virtual training, while N query datasets can be used for virtual testing.
[0118] Here, each of the N query datasets has a different data distribution in terms of object category and interaction type than the supporting dataset. N is a plurality of natural numbers and can be selectively determined based on resources, the specifications of the object tracking device, etc.
[0119] The object tracking device 100 can optimize the model by performing virtual tests based on N query datasets, which have different data distributions in terms of object categories and interaction types than the supporting datasets.
[0120] According to the embodiments, the meta-learning of the model can be completed through three steps: virtual training, virtual testing, and meta-optimization.
[0121] The object tracking device 100 can be based on the supporting dataset D s The model is trained virtually, and the loss L(D) is calculated based on the virtual training. s ;θ), and virtually update the model parameters θ based on this loss.
[0122] Here, the loss is L(D) s ;θ) can refer to the support dataset D s The loss of parameter θ when the model is virtually trained.
[0123] Therefore, by conducting virtual training based on the supporting dataset, the model can be updated virtually, and the virtually updated model can be obtained.
[0124] The object tracking device 100 can be based on N query datasets. The model of virtual updates is tested virtually to evaluate the model parameters of virtual updates.
[0125] Here, virtual tests can be performed to evaluate the generalization ability across various data distributions.
[0126] The object tracking device 100 can use a virtual update model based on the query dataset. Perform virtual tests to calculate losses
[0127] Here, the model parameter θ′ represents the model parameters updated through virtual training.
[0128] The loss on the query dataset can be used as feedback on the generalization ability of the virtually updated model.
[0129] The object tracking device 100 can perform meta-optimization to update the model.
[0130] According to an embodiment, the object tracking device 100 can be based on the loss L(D) calculated during virtual training. s ;θ) and the loss calculated during virtual testing To optimize the model parameters θ.
[0131] The object tracking device 100 can perform meta-optimization of the model parameters θ based on Equation 1.
[0132] [Equation 1]
[0133]
[0134] Here, L(D) s ;θ) represents based on the supporting dataset Ds The loss of parameter θ when the model is virtually trained, and Indicates based on query dataset The loss of parameter θ′ when performing virtual testing on a virtually updated model.
[0135] Therefore, the object tracking device 100 can reflect the data based on the supporting dataset D during model updates. s The loss L(D) obtained from the virtual training performed s ;θ) to improve the model's generalization ability, and by reflecting the query dataset during model updates. The loss obtained from the virtual test is used to improve the model's ability to generalize to various data distributions.
[0136] The object tracking device 100 can update the model parameters θ based on Equation 2.
[0137] [Equation 2]
[0138]
[0139] Here, α represents the weight, and β represents the learning rate for the overall optimization.
[0140] Figure 3 This is a flowchart illustrating an object tracking method according to an embodiment of the present disclosure.
[0141] Reference Figure 3 The object tracking device 100 (or object tracking model) can receive video frames and target initialization statements (S300), and determine whether the input video frame is the first video frame of the video sequence (S310).
[0142] When the input video frame is the first video frame of the video sequence (yes in S310), the object tracking device 100 can specify the target from the first video frame according to the target initialization statement (S320).
[0143] In step S320, the object tracking device 100 can use the previously learned language model to generate an embedding vector based on the target initialization statement, and specify the target from the first video frame based on the embedding vector.
[0144] After performing step S320, the object tracking device 100 can perform step S310 to process video frames after the first video frame.
[0145] When the input video frame is not the first video frame of the video sequence (no in S310), the object tracking device 100 can detect objects that may interact with the target from the input video frame (S330).
[0146] In step S330, the object tracking device 100 can extract features from the input video frame and detect the bounding box and object category of the object in the video frame based on the extracted features.
[0147] In step S330, the object tracking device 100 can further extract intermediate features corresponding to the bounding box based on the region of interest (RoI) alignment technique to obtain more information about the object.
[0148] After step S330, the object tracking device 100 can predict the interaction between the target and the object (S340).
[0149] In step S340, the object tracking device 100 can determine the target by matching the target specified in step S320 with the object detected in step S33, and determine other objects as candidate related objects.
[0150] In addition, the object tracking device 100 can predict the interaction between the target and each candidate related object based on the characteristics of the target and the characteristics of the candidate related objects.
[0151] According to an embodiment, the object tracking device 100 can predict the interaction between the target and the candidate related objects in the current video frame by connecting the features of the target in the current video frame with the features of the candidate related objects in the previous K video frames and inputting these features into an LSTM model, and determine the candidate related objects that interact with the target from among the candidate related objects, i.e., the related objects.
[0152] Additionally, the object tracking device 100 can indicate irrelevant objects by adding the "background" category to the relevant directory of candidate relevant objects (irrelevant objects) that have not interacted with the target.
[0153] After step S340, the object tracking device 100 can determine whether the input video frame is the final video frame (S350).
[0154] When the input video frame is not the final video frame (no in S350), the object tracking device 100 can perform steps S330 and S340 for the next video frame.
[0155] When the input video frame is the final video frame (yes in S350), the object tracking device 100 can predict the trajectory of the target and related objects and track the target and related objects (S360).
[0156] Here, the trajectory may include a positional trajectory and a semantic trajectory, and the object tracking device 100 can perform positional tracking and semantic tracking on the target and related objects.
[0157] In step S360, the object tracking device 100 can predict the trajectory of the target and related objects based on the bit tracking method.
[0158] According to an embodiment, the object tracking device 100 can predict the trajectory of the target and each related object by matching the target and related objects in two consecutive video frames.
[0159] For example, the object tracking device 100 can predict the trajectory of a target by matching the target in each of the consecutive first video frames and the second video frames, and predict the trajectory of a related object by matching the related object in each of the consecutive first video frames and the second video frames.
[0160] Figure 4 This is a flowchart illustrating a method for learning an object tracking model according to an embodiment of the present disclosure.
[0161] The object tracking device 100 can perform meta-learning on the object tracking model based on a training dataset, which includes a support dataset and multiple query datasets, wherein the data distribution of the multiple query datasets in terms of object category and interaction type is different from that of the support dataset.
[0162] Reference Figure 4 The object tracking device 100 can perform virtual training on the object tracking model based on the supporting dataset (S400) and obtain a virtually updated object tracking model (S410).
[0163] In step S410, the object tracking device 100 can obtain a virtually updated object tracking model by calculating the loss based on virtual training and virtually updating the model parameters based on the loss.
[0164] In the following, the object tracking device 100 can perform a virtual test on the virtually updated object tracking model based on multiple query datasets (S420), and calculate the loss based on the virtual test (S430).
[0165] In the following text, the object tracking device 100 may perform meta-optimization on the parameters of the object tracking model based on the loss obtained during virtual training and the loss obtained during virtual testing (S440).
[0166] In the following text, the object tracking device 100 can determine whether the learning of the object tracking model is complete (S450), and terminate the learning operation of the object tracking model when the learning is complete (yes in S450) or execute step S400 when the learning is incomplete (no in S450).
[0167] Although embodiments of the present disclosure have been described in more detail with reference to the accompanying drawings, the present disclosure is not necessarily limited to these embodiments, and various modifications can be made without departing from the spirit of the present disclosure. Therefore, the embodiments disclosed in this specification are not intended to limit the spirit of the present disclosure, but rather to explain it, and the scope of the spirit of the present disclosure is not limited by these embodiments. Therefore, the above embodiments should be understood in all respects as illustrative and not restrictive. The scope of protection of this disclosure should be interpreted according to the claims, and all technical spirit within the equivalent scope should be interpreted as included within the scope of the rights of this disclosure.
Claims
1. An object tracking device, comprising: Memory, storing models used for object detection and tracking; as well as The processor runs the model, wherein, when running the model, the processor: Receive video frames and target initialization statements as input; In response to the input video frame being the first video frame of the video sequence, the target is specified from the first video frame based on the target initialization statement; In response that the input video frame is not the first video frame, an object that may interact with a specified target in the input video frame is detected; The target and related objects are determined from the video frame by predicting the interaction between the specified target and the detected objects; The target and related objects are determined up to the final video frame of the video sequence, and Predict the trajectories of the target and related objects.
2. The object tracking device according to claim 1, wherein, In response to determining the target and the related objects, the processor determines the target by matching the specified target with the detected objects, and identifies other objects as candidate related objects.
3. The object tracking device according to claim 2, wherein, The processor predicts the interaction between the target and each candidate related object based on the characteristics of the target and the characteristics of the candidate related objects, and determines the related objects among the candidate related objects that interact with the target.
4. The object tracking device according to claim 2, wherein, The processor predicts the interaction between the target in the current video frame and each candidate related object by concatenating the features of the target in the current video frame with the features of the candidate related objects in each of the previous video frames (a preset number of previous video frames), and then inputting these features into a Long Short-Term Memory (LSTM) model.
5. The object tracking device according to claim 3, wherein, The processor will instruct the category of irrelevant objects to be added to the relevant directory of irrelevant objects that do not interact with the target among the candidate relevant objects.
6. The object tracking device according to claim 1, wherein, The trajectory includes the location trajectory and semantic trajectory of the target and each related object.
7. The object tracking device according to claim 1, wherein, The processor performs meta-learning on the model based on a training dataset, which includes a support dataset and multiple query datasets, and the multiple query datasets have different data distributions in terms of object categories and interaction types compared to the support dataset.
8. The object tracking device according to claim 7, wherein, The processor updates the model by reflecting the loss from virtual training performed based on the supporting dataset and the loss from virtual testing performed based on the plurality of query datasets.
9. The object tracking device according to claim 8, wherein, The processor performs the virtual test on the virtually updated model based on the plurality of query datasets by reflecting the loss of the virtual training.
10. A vehicle comprising the object tracking device according to claim 1.
11. An object tracking method, comprising the following steps: The processor receives video sequences and target initialization statements as input. The processor specifies the target from the first video frame of the video sequence based on the target initialization statement; The processor detects objects that may interact with the specified target from other input video frames of the video sequence. The processor predicts the interaction between a specified target and a detected object, and determines the target and related objects from the corresponding video frames based on the predicted interaction. as well as The processor determines the target and related objects up to the final video frame of the video sequence and predicts the trajectories of the target and related objects.
12. The object tracking method according to claim 11, wherein, The steps of determining the target and the related objects include: determining the target by matching the specified target with the detected objects, and determining other objects as candidate related objects.
13. The object tracking method according to claim 12, wherein, The steps of determining the target and the related objects include: predicting the interaction between the target and each candidate related object based on the features of the target and the features of the candidate related objects, and determining the related objects among the candidate related objects that interact with the target.
14. The object tracking method according to claim 12, wherein, The steps for determining the target and the related objects include: concatenating the features of the target in the current video frame with the features of the candidate related objects in each of the previous number of video frames, and then inputting these features into a long short-term memory model, i.e., an LSTM model, to predict the interaction between the target in the current video frame and each candidate related object.
15. The object tracking method according to claim 12, wherein, The steps of determining the target and the related objects include: adding categories indicating irrelevant objects to the related directories of irrelevant objects that do not interact with the target among the candidate related objects.
16. A method for learning an object tracking model using an object tracking device, the method comprising the following steps: The object tracking model is trained using a processor based on a support dataset and multiple query datasets. as well as The processor performs meta-optimization on the parameters of the object tracking model based on the loss calculated by performing virtual training computation based on the supporting dataset and the loss calculated by performing virtual testing computation based on the multiple query datasets.
17. The object tracking method according to claim 16, wherein, The multiple query datasets differ from the supporting datasets in terms of data distribution across object categories and interaction types.
18. The object tracking method according to claim 16, wherein, The virtual test is performed on the updated target tracking model based on the loss calculated during the virtual training.
19. The object tracking method according to claim 16, wherein, The meta-optimization is performed on the parameters of the target tracking model based on Equation 1. [Equation 1] in, L(D s ;θ) is the loss of parameter θ calculated during the virtual training. It is the loss of parameter θ′ calculated during the virtual test, and θ′ is a parameter updated through the virtual training.
20. The object tracking method according to claim 16, wherein, Based on Equation 2, the parameters of the target tracking model are updated. [Equation 2] Here, α represents the weight, and β represents the learning rate for the overall optimization.