Tracking method and device

Abstract

A tracking method according to one example of the present disclosure may include generating, by a generation device, a first feature based on a first frame through a backbone, generating, by the generation device, first detection information through a first neck for object detection, the first detection information indicating a detection result of a first object based on the first feature, a first feature vector for a visual feature of the first object is generated by the generating device on the basis of the first feature through a second neck for object re-recognition, and / or tracking is performed by a tracking device on the basis of the first detection information and the first feature vector.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from Korean Patent Application No. 10-2024-0032776, filed on Mar. 7, 2024, which is hereby incorporated by reference into this disclosure in its entirety for all purposes. Technical Field

[0003] The present disclosure relates to object detection, and more particularly, to a method and apparatus for tracking the same. Background Art

[0004] With the advancement of deep neural network-based computer vision technology in autonomous driving, various AI models are being researched, such as those for object detection, semantic segmentation, depth map estimation, and lane detection. Research is continuing on methods for autonomous driving that use AI models to track objects. For example, models for object detection and object re-identification can be used for tracking. A method that can increase the performance speed of the algorithm without compromising object detection performance in tracking may be desirable. Summary of the Invention

[0005] The present disclosure can solve the above-mentioned problems existing in the prior art while retaining the advantages of the prior art.

[0006] One aspect of the present disclosure may provide a method and apparatus that improve tracking performance without causing a degradation in object detection performance.

[0007] One aspect of the present disclosure may provide a method and apparatus for training an object re-identification model by applying knowledge distillation technology.

[0008] One aspect of the present disclosure may provide a method and apparatus for training an object re-identification model by using a pre-trained model.

[0009] One aspect of the present disclosure may provide a method and apparatus for simultaneously performing object detection and object re-identification.

[0010] One aspect of the present disclosure may provide a method and apparatus for performing multi-task learning associated with tracking.

[0011] The technical problems to be solved by the present disclosure are not limited to the above problems, and those skilled in the art to which the present disclosure belongs will clearly understand any other technical problems not mentioned in the present disclosure through the following description.

[0012] A device may include: a memory configured to store computer-executable instructions; and at least one processor configured to execute the computer-executable instructions by accessing the memory, wherein the at least one processor is configured to: generate a first feature based on a first frame and via a backbone of a detection model, generate first detection information indicating a detection result of a first object based on the first feature and via a first neck of the detection model, wherein the first neck is configured for object detection, generate a first feature vector for a visual feature of the first object based on the first feature and via a second neck of the detection model, wherein the second neck is configured for object re-identification; and perform tracking on at least one object including the first object based on the first detection information and the first feature vector.

[0013] The at least one processor may be configured to train a second model based on a first model pre-trained for the object re-identification, wherein the second model includes the backbone, the first neck, and the second neck.

[0014] The at least one processor may be configured to train the second neck with parameters of the backbone and the first neck fixed.

[0015] The at least one processor can be configured to: generate a second feature vector based on a first input image and via the first model; generate a third feature vector based on a second input image and via the backbone and the second neck; generate a first loss based on the second feature vector and the third feature vector; and train the second neck based on the first loss.

[0016] The at least one processor may be configured to: generate a second loss based on the third eigenvector and a ground truth (GT); and train the second neck based on the first loss and the second loss.

[0017] The at least one processor may be configured to generate a classification vector based on the third feature vector and via a classification network, and generate the second loss by comparing the classification vector and the GT.

[0018] The GT may include hard-tagged data.

[0019] The second feature vector may include soft label data.

[0020] The first input image may be an image obtained by extracting an object region from the second input image.

[0021] The at least one processor can be configured to: generate a first association vector obtained by combining the first detection information and the first feature vector; generate a second feature based on a second frame and via the backbone, wherein the second frame is a next frame of the first frame; generate a plurality of second detection information about a plurality of second objects based on the second feature and via the first neck; generate a fourth feature vector for visual features of the plurality of second objects based on the second feature and via the second neck; generate a second association vector, wherein the second association vector is obtained by respectively combining the plurality of second detection information and the fourth feature vector; and perform the tracking by associating the first object with an object having an association vector closest to the first association vector in the second association vector.

[0022] One or more methods may be executed by at least one computing device to perform one or more operations and / or implement one or more features described in the present disclosure.

[0023] The features briefly summarized above regarding the present disclosure are merely aspects of the detailed description of the present disclosure described below and do not limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description in conjunction with the accompanying drawings:

[0025] Figure 1 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown;

[0026] Figure 2 An example of a flow chart illustrating a tracking method according to an example of the present disclosure is shown;

[0027] Figure 3 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown;

[0028] Figure 4 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown;

[0029] Figure 5 An example of a flow chart illustrating a tracking method according to an example of the present disclosure is shown;

[0030] Figure 6 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown;

[0031] Figure 7 An example of a flow chart illustrating a tracking method according to an example of the present disclosure is shown;

[0032] Figure 8 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown;

[0033] Figure 9 An example of a block diagram illustrating a tracking device according to one example of the present disclosure; and

[0034] Figure 10 An example of a block diagram of a computing system for performing a tracking method according to one example of the present disclosure is shown. DETAILED DESCRIPTION

[0035] The following will describe various embodiments of the present disclosure in detail in conjunction with the accompanying drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments described in the present disclosure. When describing the embodiments of the present disclosure, the present disclosure will omit detailed descriptions if it is believed that the specific description of the related art will obscure the subject matter of the embodiments of the present disclosure. In addition, in the accompanying drawings, parts that are not relevant to the description of the present disclosure are omitted, and similar parts are given similar reference numerals. In the present disclosure, it should be understood that if an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or indirectly connected to the other element. In addition, if a part "includes" or "has" certain elements, unless explicitly described to the contrary, it means that other elements may be further included but not excluded. In the present disclosure, expressions such as "first" and "second" can represent their elements, regardless of their priority or importance, and can be used to distinguish one element from another, but are not limited to these elements. For example, without departing from the scope of the present disclosure, the first element of one example can be referred to as the second element of another example. Similarly, the second element of one example can be referred to as the first element of another example. In the present disclosure, the elements that are distinguished from each other are only used to clearly describe the features and do not mean that the elements must be separated. That is, multiple elements can be integrated to form a single hardware or software unit, or a single element can be distributed to form multiple hardware or software units. Therefore, even if not mentioned separately, such integrated or distributed examples are also included in the scope of the present disclosure. In the present disclosure, the elements described in the various examples do not necessarily mean required elements, and some can be optional elements. Therefore, examples consisting of a subset of the components described in the embodiments are also included in the scope of the present disclosure. In addition, examples that additionally include another component in the components described in the various examples are also included in the scope of the present disclosure.

[0036] In the present disclosure, for the convenience of description, the expressions of positional relationships used in the present disclosure are described, such as up, down, left, and right. If the drawings shown in the present specification are viewed in reverse, the positional relationships described in the specification can be interpreted in the opposite manner. The present disclosure In the present disclosure, the expressions "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any and all combinations of one or more of the relevant listed items. Hereinafter, reference may be made to Figures 1 to 10 Various examples of the present disclosure are described in detail.

[0037] Figure 1 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown.

[0038] Tracking can be performed by performing object detection and extracting visual features of the object through object re-identification. Object detection may mean creating detection information including the location of the object, bounding box information about the object, the heading angle of the object, etc. For example, object re-identification may mean creating a feature vector for the visual features of the object, such as the size of the object, the color of the object, and / or the shape of the object. If object re-identification is performed after object detection is performed, the network that performs object detection and the network that performs object re-identification may exist separately. The presence of two networks may result in a slower tracking algorithm compared to the case where a single network is used to perform object detection and object re-identification.

[0039] refer to Figure 1 According to an example tracking method of the present disclosure, object detection and object re-identification can be performed simultaneously. For example, the tracking method can perform object detection and object re-identification simultaneously through a network. For example, a first neck for object detection and a second neck for object re-identification can be run as a network through a shared backbone. The tracking method can accelerate the performance of the tracking algorithm by using a model including a neck with a shared backbone. The present disclosure may provide a detailed description of the tracking algorithm.

[0040] According to the Society of Automotive Engineers (SAE), the levels of automation for autonomous vehicles can be categorized as follows. At Level 0, the SAE classification standard may correspond to "no automation," in which the autonomous driving system temporarily engages in emergency situations (e.g., automatic emergency braking) and / or only provides warnings (e.g., blind spot warning, lane departure warning, etc.), and the driver is expected to operate the vehicle. At Level 1, the SAE classification standard may correspond to "driver assistance," in which the system performs some driving functions (e.g., steering, acceleration, braking, lane centering, adaptive cruise control, etc.) while the driver operates the vehicle on normal road conditions, and the driver is expected to determine the operating state and / or timing of the system, perform other driving functions, and respond to (e.g., resolve) the emergency situation. At Level 2, the SAE classification standard may correspond to "partial automation," in which the system performs steering, acceleration, and / or braking under the driver's supervision, and the driver is expected to determine the operating state and / or timing of the system, perform other driving functions, and respond to (e.g., resolve) the emergency situation. At Level 3, the SAE classification standard may correspond to "conditional automation," in which the system drives the vehicle (e.g., performs driving functions such as steering, acceleration, and / or braking) under limited conditions, but transfers driving control to the driver if the required conditions are not met, and the driver is expected to determine the operating state and / or timing of the system and take over control in emergency situations, but does not otherwise operate the vehicle (e.g., steering, acceleration, and / or braking). At Level 4, the SAE classification standard may correspond to "high automation," in which the system performs all driving functions, and the driver is expected to take over control of the vehicle only in emergency situations. At Level 5, the SAE classification standard may correspond to "full automation," in which the system performs all driving functions without any assistance from the driver (including in emergency situations), and the driver is expected to perform no driving functions other than determining the operating state of the system. Although the present disclosure may apply the SAE classification standard to autonomous driving classification, other classification methods and / or algorithms may be used in one or more configurations described in the present disclosure.

[0041] Figure 2 An example of a flow chart illustrating a tracking method according to an example of the present disclosure is shown.

[0042] The detection model (e.g., a neural network) may include a backbone, at least one neck, and / or a head. The backbone may utilize basic features at different resolutions, the neck may fuse features at different resolutions, and at least one head may perform object detection at different resolutions.

[0043] A backbone network can be used in object detection model architectures. The backbone may be responsible for extracting and encoding features from the input data. It can act as a core feature extractor, capturing both low-level and high-level features from the input data.

[0044] The neck may be responsible for further transforming and refining the features extracted by the backbone model. The neck can improve the features extracted by the backbone and give more informative feature representations.

[0045] The backbone may be responsible for extracting initial features from the input data, while the neck enhances and merges these features to improve the performance of the model.

[0046] The head can include task-specific layers designed to produce final predictions or inferences based on the information extracted by the backbone and neck.

[0047] In machine learning, a loss function (also known as a cost function or error function) is a method for evaluating how well a particular algorithm models a given set of data. By comparing the predictions generated by the model with the actual target values, the loss function quantifies the error, or discrepancy. The purpose of the loss function is to guide the training process. When the model makes a prediction, the loss function calculates a value that represents how far the prediction is from the true value. The goal of the learning algorithm is to minimize this loss value by adjusting the model's parameters during the training phase.

[0048] refer to Figure 2 According to an exemplary tracking method of the present disclosure, in S210, the generating device may generate a first feature based on the first frame, for example, by using a backbone. The output of the backbone of the received image may be referred to as a "feature."

[0049] According to the tracking method, in S220, the generating device may generate first detection information based on the first feature using a first neck for object detection, the first detection information indicating a detection result of the first object. The first detection information may include position information of the first object, a heading angle of the first object, and / or bounding box information about the first object.

[0050] According to the tracking method, in S230, the generating device may generate a first feature vector for the visual feature of the first object based on the first feature using the second neck for object re-identification. For example, the first feature vector may be a vector including information about the feature of the first object, such as the shape of the first object, the size of the first object, and / or the color of the first object.

[0051] According to the tracking method, in S240, the tracking device can perform tracking based on the first detection information and / or the first feature vector. For example, the tracking method can perform tracking by performing association based on the object detection information of the previous frame of the first frame, the feature vector of the first frame, the first detection information of the first frame, and / or the first detection information of the first frame. For example, the tracking method can perform tracking by performing association based on the object detection information of the second frame (which may be the next frame of the first frame), the feature vector of the second frame, the first detection information of the first frame, and / or the first detection information of the first frame. A detailed description of tracking will be given later.

[0052] Figure 3 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown.

[0053] refer to Figure 3 In S310, a tracking method according to an example of the present disclosure may perform object detection and / or object re-identification. The tracking method may simultaneously perform object detection and object re-identification using a single network. For example, as described above, a first network for object detection and a second network for object re-identification may share a backbone. For example, object detection and object re-identification may be performed simultaneously.

[0054] In S320, the tracking method may perform association. The association may refer to a process of connecting an object detected in a current frame to an object detected in a previous frame. For example, an association vector of an object of a previous frame may be created by combining object detection information and a feature vector of an object detected in a previous frame. An association vector may be created by combining multiple pieces of object detection information and a feature vector of an object detected in a current frame. For example, the tracking method may perform association by calculating a distance between an association vector of a previous frame and an association vector of a current frame, and / or treating an object whose current frame association vector may be closest to the previous frame association vector as the same object as the previous frame object. Cosine distance, Euclidean distance, Mahalanobis distance and / or intersection over union (IOU) may be used to calculate the distance between association vectors.

[0055] At S330, the tracking method may perform an update. For example, the tracking method may update the state of the object based on the association result. For example, the tracking method may update data about an object that may be located at a first location in a previous frame to change it to data about an object located at a second location in a current frame, but is not limited thereto.

[0056] In S340, the tracking method may perform management. For example, the tracking method may manage the tracking state of the object. For example, the tracking method may remove the object from the tracking target. When the object disappears from the object detection range, tracking of the object may be interrupted.

[0057] In S350, the tracking method may perform registration. For example, when a new object is detected within the object detection range, the tracking method may register the new object as a tracking target.

[0058] At S360, the tracking method may perform a prediction. For example, the tracking method may predict the position of the object in the next frame. For example, the tracking method may predict the position of the object by using a Kalman filter, but is not limited thereto. The position information of the predicted object may be used in the association process. For example, the association may be performed by calculating the distance between the association vector of a detected object within a predetermined distance of the predicted object's position and the association vector of the object performing the prediction. Figure 4 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown.

[0059] refer to Figure 4 , a tracking method according to an example of the present disclosure may perform an inference operation and / or a training operation. During the inference operation, the tracking method may perform object detection and object re-identification simultaneously. For example, by sharing a backbone between a first neck (neck 1) associated with object detection and a second neck (neck 2) associated with object re-identification, object detection and object re-identification may be performed simultaneously. As described in the present disclosure, the tracking method may generate a first feature based on a first frame through the backbone. For example, the tracking method may generate first detection information indicating a detection result of a first object based on the first feature through the first neck. For example, the first detection information may include multiple pieces of information about the category of the object, the 2D bounding box of the object, the 3D bounding box of the object, and / or the heading angle of the object. For example, the tracking method may generate a first feature vector for a visual feature of the first object based on the first feature through the second neck. If association is performed, the generated first detection information and the generated first feature vector may be used.

[0060] The tracking method can perform a training operation. For example, the tracking method can train the backbone and the first neck based on the ground truth (GT) for object detection. In addition, the tracking method can train the second neck by applying knowledge distillation techniques. For example, the second neck can be trained by using the output of the first model pre-trained on object re-identification. For example, the classification vector can be created by inputting the feature vector that can be generated by the second neck into a pooling layer and a fully connected layer (which can be a classification network). The second neck can be trained by comparing the GT for object re-identification with the classification vector. The present disclosure can provide a detailed description of the operations for training the second neck.

[0061] Figure 5 An example of a flow chart illustrating a tracking method according to an example of the present disclosure is shown.

[0062] refer to Figure 5 According to an exemplary tracking method of the present disclosure, in S510, the generating device may generate a first association vector obtained by combining the first detection information and the first feature vector. For example, the first association vector having 130 channels may be created by adding the first detection information about the first object to the first feature vector consisting of 128 channels, but the present invention is not limited thereto.

[0063] According to the tracking method, in S520 , the generating device may generate a second feature based on a second frame through the backbone, and the second frame may be a next frame of the first frame.

[0064] According to the tracking method, in S530 , the generating device may generate a plurality of second detection information about the plurality of second objects based on the second feature through the first neck.

[0065] According to the tracking method, in S540 , the generating device may generate a fourth feature vector of the visual features of the plurality of second objects based on the second feature through the second neck.

[0066] According to the tracking method, in S550 , the generating means may generate second correlation vectors obtained by respectively combining a plurality of pieces of second detection information and the fourth eigenvector.

[0067] According to this tracking method, in S560, the tracking device may perform tracking by associating the first object with the object having the association vector closest to the first association vector in the second association vector. For example, the object corresponding to the vector in the second association vector in the second frame that is closest to the first association vector in the first frame may correspond to the first object. For example, the tracking may be performed by associating the first object in the first frame with the object having the association vector closest to the first association vector.

[0068] Figure 6 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown.

[0069] refer to Figure 6 According to an example of the present disclosure, a tracking method can generate multiple detection information of an object in a first frame and a feature vector of the object. The tracking method can predict the positions of objects M-1 to M-5 in a second frame (frame 2) based on the multiple information generated about the first frame, and the second frame can be the next frame of the first frame (S610). The tracking method can predict the position of the object based on the first frame. For example, if the position of the object in the actual second frame is detected, the tracking method can perform association based on the predicted position and the actual detected position.

[0070] Association can refer to the process of identifying whether an object detected in a new frame corresponds to a tracked object. If a new detection result is received in a frame, association can include finding a match between the tracked object and the detection result. If the object is associated, the newly received detection information (e.g., such as updated location data) can be used to update the state of the object. Over time, this continuous association of object recognition results can allow accurate estimation of information, such as, for example, position and / or velocity.

[0071] For example, if the first object in the first frame is predicted to be located at the first position of the second frame, the distance between the association vector of the object detected in the second frame that may be at a predetermined distance from the first position and the association vector of the first object can be calculated. The tracking method can determine the object having the association vector closest to the association vector of the first object as the same object as the first object, and can perform tracking T-1 to T-5 (S620). For example, the tracking method can predict the position of the object in the third frame (frame 3) based on the multiple pieces of information generated about the second frame, and the third frame can be the next frame of the second frame (S630). Tracking of the second and third frames can be performed by using the tracking method for the first and second frames.

[0072] Figure 7 An example of a flow chart illustrating a tracking method according to an example of the present disclosure is shown.

[0073] According to an example tracking method of the present disclosure, a second model including a backbone, a first neck, and a second neck can be trained based on a first model pre-trained for object re-identification. The first neck can be associated with object detection. The second neck can be associated with object re-identification. The first neck and the second neck can share the backbone.

[0074] The tracking method can train the second model based on the first model by applying knowledge distillation techniques. For example, in a state where the parameters of the backbone and the first neck are fixed, the tracking method can train the second neck by using the first model. For example, the tracking method can train the second neck after freezing the backbone and the first neck. For example, referring to Figure 7 According to the tracking method, in S710, the generating device may generate a second feature vector based on the first input image through the first model.

[0075] According to the tracking method, in S720, the generating device may generate a third feature vector based on the second input image through the backbone and the second neck. The first input image described in the present disclosure may be an image obtained by extracting an object region from the second input image.

[0076] According to the tracking method, in S730, the generating device may generate a first loss based on the second eigenvector and the third eigenvector. For example, the tracking method may train the second neck so that the third eigenvector may be close to the second eigenvector, and the second eigenvector may be the output of the first model pre-trained for object re-identification. The tracking method may generate the first loss by comparing the second eigenvector and the third eigenvector. The second eigenvector may include soft-labeled data. In addition, the third eigenvector may also include soft-labeled data. A soft label may be a label that assigns a probability or confidence score to each of several categories, rather than a single label (hard label) for data probability or confidence score. For example, it can be assumed that dogs, cats, and televisions need to be classified. If the network generates an output of [0.1, 1, 0.8] for a cat input, there may be a method for training the network so that an output of [0, 1, 0] is generated for hard-labeled data, and there may also be a method for training the network so that an output of [0.1, 1, 0.8] is generated for soft-labeled data.

[0077] According to the tracking method, in S740, the generating device may generate a second loss based on the third feature vector and the ground truth. For example, the tracking method may generate a classification vector by inputting the third feature vector into a classification network, and may generate the second loss by comparing the classification vector with the ground truth. For example, the ground truth may include hard-labeled data. For example, the classification network may include a pooling layer and a fully connected layer.

[0078] At least some networks (e.g., neural networks, deep learning networks, etc.) can exploit domain-invariant features of the source and target domains, and some of these label-induced losses require ground truth labels as input. Since the target ground truth labels are unavailable, target predicted labels that can be represented in different forms can be used to formulate label-induced losses.

[0079] Hard labeled data (e.g., in the form of hard labels) can classify samples into one or more clear categories, but it may be overconfident when the quality of the predicted labels is poor. Soft labeled data (e.g., in the form of soft labels) can provide probability values ​​for each category of the sample, which can alleviate the problem of overconfidence.

[0080] According to the tracking method, in S750, the training device can train the second neck based on the first loss and the second loss. For example, the tracking method can train the second neck by using the first loss and the second loss as the total loss for object re-identification.

[0081] Figure 8 An example of a schematic diagram illustrating a tracking method according to an example of the present disclosure is shown.

[0082] refer to Figure 8According to an example of the present disclosure, a tracking method can train a second model 820 using a first model 810. For example, the tracking method can train the second model 820 by applying a knowledge distillation technique based on the first model 810 pre-trained for object re-identification. For example, the first model 810 pre-trained for object re-identification can generate a second feature vector based on the first input image 801. For example, the second model 820 can generate a third feature vector based on the second input image using a second neck for object re-identification. The first input image 801 can be an image obtained by extracting an object region from the second input image 802. The second feature vector and the third feature vector can include soft-labeled data. The tracking method can generate a first loss by comparing the second feature vector and the third feature vector. For example, the tracking method can input the third feature vector into a classification network 830. The third feature vector input into the classification network can be output as a classification vector. The tracking method can generate a second loss based on the classification vector and the ground truth (GT) including hard-labeled data. The tracking method can train the second neck using the first and second losses as a total loss. If the second neck is trained, the backbone and the first neck may be frozen. For example, the tracking method can train the second neck while the parameters of the backbone and the first neck are fixed.For example, the tracking method can train the backbone and the first neck based on data labeled for object detection.

[0083] The tracking method may perform tracking by performing estimation through the trained second model 820. For example, the tracking method may generate a first feature based on the first frame through the backbone. For example, the tracking method may generate first detection information indicating a detection result of the first object based on the first feature through the first neck for object detection. The first detection information may include information about a heat map, the size, depth, heading angle, etc. of the object. The tracking method may generate a first feature vector for a visual feature of the first object based on the first feature through the second neck for object re-identification. The tracking method may perform association by calculating the distance between the association vector of the previous frame of the first frame and the association vector of the first frame. For example, the tracking method may update the state information of the object based on the association result, may perform management by deleting an object whose tracking may be interrupted, and may additionally register a newly detected object in a tracking list.

[0084] Figure 9 An example of a block diagram of a tracking device according to an example of the present disclosure is shown.

[0085] refer to Figure 9 , the tracking device 100 may include a memory 110, a generating device 120, a tracking device 130, and a training device 140. The generating device 120, the tracking device 130, and the training device 140 may correspond to a processor.

[0086] The memory 110 may be configured to store computer-executable instructions.

[0087] The processor may be configured to access the memory 110 to execute instructions. For example, the processor may generate a first feature based on the first frame using the backbone through the generation device 120. Additionally or alternatively, the processor may generate first detection information indicating a detection result of the first object based on the first feature using the first neck for object detection through the generation device 120. Additionally or alternatively, the processor may generate a first feature vector for visual features of the first object based on the first feature using the second neck for object re-identification through the generation device 120.

[0088] For example, the processor may perform tracking based on the first detection information and the first feature vector through the tracking device 130 .

[0089] For example, the processor may train a second model including a backbone, a first neck, and a second neck based on a first model pre-trained for object re-identification through the training device 140. For example, the processor may train the second neck through the training device 140 while the parameters of the backbone and the first neck are fixed.

[0090] Additionally or alternatively, the processor may generate a second feature vector based on the first input image using the first model through the generation device 120. For example, the processor may generate a third feature vector based on the second input image using the backbone and the second neck through the generation device 120. The first input image may be an image obtained by extracting an object region from the second input image. The processor may generate a first loss based on the second feature vector and the third feature vector through the generation device 120. For example, the second feature vector may include soft label data. The processor may generate a second loss based on the third feature vector and GT through the generation device 120. For example, the processor may train the second neck based on the first loss and the second loss through the training device 140.

[0091] The processor may generate a classification vector based on the third feature vector by using the classification network through the generation device 120. Additionally or alternatively, the processor may generate a second loss by comparing the classification vector and the ground truth through the generation device 120. For example, the ground truth may include hard labeled data.

[0092] The processor may generate a first association vector through the generation device 120, the first association vector being obtained by combining the first detection information and the first feature vector. For example, the processor may generate a second feature based on a second frame, which may be a frame next to the first frame, by using the backbone, for example, through the generation device 120. Additionally or alternatively, the processor may generate a plurality of pieces of second detection information about a plurality of second objects based on the second feature, using the first neck, through the generation device 120. For example, the processor may generate a fourth feature vector for visual features of a plurality of second objects based on the second feature, using the second neck, through the generation device 120. For example, the processor may generate a second association vector through the generation device 120, the second association vector being obtained by respectively combining the plurality of pieces of second detection information and the fourth feature vector.

[0093] The processor may perform tracking by associating an object having an association vector closest to the first association vector among the second association vectors with the first object through the tracking device 130. The distance between the vectors may be calculated by using cosine distance, Euclidean distance, or the like.

[0094] According to one aspect of the present disclosure, a tracking method may include: a generating device generates a first feature based on a first frame through a backbone, the generating device generates first detection information indicating a detection result of a first object based on the first feature through a first neck for object detection, the generating device generates a first feature vector for a visual feature of the first object based on the first feature through a second neck for object re-identification, and a tracking device performs tracking based on the first detection information and the first feature vector.

[0095] According to one example, the tracking method may further include training, by a training device, a second model including a backbone, a first neck, and a second neck based on a first model pre-trained for object re-identification.

[0096] According to one example, the training of the second model may include training the second neck by a training device in a state where parameters of the backbone and the first neck are fixed.

[0097] According to one example, the training of the second model may include generating a second eigenvector by a generating device through a first model based on a first input image, generating three eigenvectors by the generating device based on the second input image through a backbone and a second neck, generating a first loss by the generating device based on the second eigenvector and a third eigenvector, and training the second neck based on the first loss by a training device.

[0098] According to one example, the training of the second model may include generating, by a generating device, a second loss based on a third feature vector and a ground truth (GT), and training, by a training device, the second neck based on the first loss and the second loss.

[0099] According to one example, the generating of the second loss may include generating, by the generating device, a classification vector based on the third feature vector through a classification network, and generating, by the generating device, the second loss by comparing the classification vector and the GT.

[0100] According to one example, the GT may include hard labeled data.

[0101] According to one example, the second feature vector may include soft label data.

[0102] According to one example, the first input image may be an image obtained by extracting an object region from the second input image.

[0103] According to one example, performing the tracking may include: generating, by a generating device, a first association vector obtained by combining first detection information and a first feature vector, generating, by a generating device, a second feature based on a second frame which is a next frame of the first frame through a backbone, generating, by a generating device, a plurality of second detection information about a plurality of second objects based on the second feature through a first neck, generating, by a generating device, a fourth feature vector for visual features of a plurality of second objects based on the second feature through a second neck, generating, by a generating device, a second association vector obtained by respectively combining a plurality of second detection information and a fourth feature vector, and performing tracking by a tracking device associating an object having an association vector closest to the first association vector in the second association vector with the first object.

[0104] According to one aspect of the present disclosure, a tracking device may include a memory storing computer-executable instructions and at least one processor that executes the instructions by accessing the memory. The at least one processor may be configured to generate, by a generating device, a first feature based on a first frame using a backbone, generate first detection information indicating a detection result of a first object based on the first feature using a first neck for object detection, generate a first feature vector for a visual feature of the first object based on the first feature using a second neck for object re-identification, and perform tracking based on the first detection information and the first feature vector using a tracking device.

[0105] According to one example, the at least one processor may be configured to train, by a training device, a second model including a backbone, a first neck, and a second neck based on a first model pre-trained for object re-identification.

[0106] For example, the at least one processor may be configured to train the second neck using a training device in a state where parameters of the backbone and the first neck are fixed.

[0107] For example, the at least one processor may be configured to generate a second eigenvector based on a first input image using a first model through a generating device, generate a third eigenvector based on the second input image by using the backbone and the second neck, generate a first loss based on the second eigenvector and the third eigenvector, and train the second neck based on the first loss through the training device.

[0108] For example, the at least one processor may be configured to generate a second loss based on the third feature vector and GT by the generating device, and train the second neck based on the first loss and the second loss by the training device.

[0109] For example, the at least one processor may be configured to generate a classification vector based on the third feature vector using a classification network through the generating device, and generate a second loss by comparing the classification vector and the GT.

[0110] For example, GT may include hard labeled data.

[0111] For example, the second feature vector may include soft label data.

[0112] For example, the first input image may be an image obtained by extracting an object region from the second input image.

[0113] For example, the at least one processor can be configured to: generate, by the generating device, a first association vector obtained by combining the first detection information and the first feature vector, generate a second feature based on a second frame which is the next frame of the first frame by using the backbone, generate a plurality of second detection information about a plurality of second objects based on the second feature by using the first neck, generate a fourth feature vector for visual features of the plurality of second objects based on the second feature by using the second neck, and obtain a second association vector by respectively combining the plurality of second detection information and the fourth feature vector, and perform tracking by the tracking device by associating an object having an association vector closest to the first association vector in the second association vector with the first object.

[0114] Figure 10 An example of a block diagram of a computing system for performing a tracking method according to one example of the present disclosure is shown.

[0115] refer to Figure 10 , the computing system 1000 may include at least one processor 1100 , a memory 1300 , a user interface input device 1400 , a user interface output device 1500 , a storage 1600 , and a network interface 1700 , which may be connected to each other via a bus 1200 .

[0116] The processor 1100 may be a central processing unit (CPU) or a semiconductor device for processing instructions stored in the memory 1300 and / or the storage 1600. The memory 1300 and the storage 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a ROM (read-only memory) 1310 and a RAM (random access memory) 1320.

[0117] Therefore, the process of the method or algorithm described in the examples of the present disclosure can be directly implemented by hardware, software modules or their combination performed by the processor 1100. The software module can reside in a storage medium (i.e., memory 1300 and / or storage 1600), such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, solid-state drive (SSD), removable disk or CD-ROM. An exemplary storage medium is connected to the processor 1100, and the processor 1100 can read information from the storage medium and can write information in the storage medium. In another method, the storage medium can be integrated with the processor 1100. The processor 1100 and the storage medium can reside in an application-specific integrated circuit (ASIC). The ASIC can reside in a user terminal. In another method, the processor 1100 and the storage medium can reside in a user terminal as separate components.

[0118] Although the present disclosure has been described above with reference to exemplary examples and accompanying drawings, the present disclosure is not limited thereto, but can be variously modified and changed by those skilled in the art to which the present disclosure belongs, without departing from the spirit and scope of the present disclosure claimed in the appended claims. The examples of the present disclosure can be used to explain the spirit and scope of the present disclosure, but are not used to limit the spirit and scope of the present disclosure, and therefore the spirit and scope of the present disclosure are not limited by the examples. The scope of the present disclosure can be interpreted according to the appended claims, and all technical ideas within the scope equivalent to the claims can be included within the scope of the present disclosure. The above description is only an example of the technical ideas of the present disclosure, and those skilled in the art can make various modifications and variations without departing from the essential features of the present disclosure. Therefore, the embodiments of the present disclosure are not intended to limit the technical ideas of the present disclosure, but are used to explain the technical ideas of the present disclosure, and the scope and spirit of the present disclosure are not limited by the above-mentioned embodiments. The scope of protection of the present disclosure can be interpreted by the appended claims, and all equivalents thereof can be interpreted as included within the scope of the present disclosure.

[0119] According to a tracking method of an example of the present disclosure, tracking performance can be improved without causing a decrease in object detection performance.

[0120] According to a tracking method of an example of the present disclosure, an object re-identification model may be trained by applying knowledge distillation technology.

[0121] According to a tracking method of an example of the present disclosure, an object re-identification model may be trained by using a pre-trained model.

[0122] According to a tracking method of an example of the present disclosure, object detection and object re-identification may be performed simultaneously.

[0123] A tracking method according to an example of the present disclosure may perform multi-task learning associated with tracking.

[0124] According to an exemplary tracking method of the present disclosure, a decoder for detecting an object and a decoder for re-identifying an object may share a backbone.

[0125] Effects obtained in the present disclosure are not limited to the above-mentioned effects, and other effects that are not mentioned will be clearly understood from the following description by those skilled in the art to which the present disclosure pertains.

[0126] Although the present disclosure has been described above with reference to exemplary examples and accompanying drawings, the present disclosure is not limited thereto but may be variously modified and changed by those skilled in the art without departing from the spirit and scope of the present disclosure as claimed in the following claims.

Claims

1. A method performed by at least one computing device, the method comprising the steps of: Generating a first feature based on the first frame and via a backbone of a detection model; generating first detection information indicating a detection result of a first object based on the first feature and via a first neck portion of the detection model, wherein the first neck portion is configured for object detection; generating a first feature vector for visual features of the first object based on the first features and via a second neck of the detection model, wherein the second neck is configured for object re-identification; and Tracking is performed on at least one object including the first object based on the first detection information and the first feature vector.

2. The method according to claim 1, further comprising the steps of: A second model is trained based on a first model pre-trained for object re-identification, wherein the second model includes the backbone, the first neck, and the second neck.

3. The method according to claim 2, wherein: The steps for training the second model include: The second neck is trained with parameters of the backbone and the first neck fixed.

4. The method according to claim 2, wherein: The steps for training the second model include: generating a second feature vector based on the first input image and via the first model; generating a third feature vector based on the second input image and via the backbone and the second neck; generating a first loss based on the second eigenvector and the third eigenvector; and The second neck is trained based on the first loss.

5. The method according to claim 4, wherein The steps for training the second model include: Based on the third eigenvector and the ground truth GT, a second loss is generated, and The training of the second neck includes: training the second neck based on the first loss and the second loss.

6. The method according to claim 5, wherein: The steps of generating the second loss include: generating a classification vector based on the third feature vector and via a classification network; and The second loss is generated by comparing the classification vector and the ground truth GT.

7. The method according to claim 5, wherein: The ground truth GT consists of hard labeled data.

8. The method according to claim 4, wherein: The second feature vector includes soft label data.

9. The method according to claim 4, wherein: The first input image is an image obtained by extracting an object region from the second input image.

10. The method according to claim 1, wherein The steps to perform a trace include: generating a first correlation vector obtained by combining the first detection information and the first feature vector; generating a second feature based on a second frame and via the backbone, wherein the second frame is a next frame of the first frame; generating a plurality of second detection information about a plurality of second objects based on the second feature and via the first neck; generating, based on the second feature and via the second neck, a fourth feature vector of visual features for the plurality of second objects; generating a second association vector, wherein the second association vector is obtained by respectively combining the plurality of pieces of second detection information and the fourth eigenvector; and Tracking is performed by associating the first object with an object having an association vector closest to the first association vector among the second association vectors.

11. A device comprising: a memory configured to store computer-executable instructions; as well as at least one processor configured to execute the computer-executable instructions by accessing the memory, Wherein, the at least one processor is configured to: Generating a first feature based on the first frame and via a backbone of a detection model; generating first detection information indicating a detection result of a first object based on the first feature and via a first neck portion of the detection model, wherein the first neck portion is configured for object detection; generating a first feature vector for visual features of the first object based on the first features and via a second neck of the detection model, wherein the second neck is configured for object re-identification; and Tracking is performed on at least one object including the first object based on the first detection information and the first feature vector.

12. The device according to claim 11, wherein The at least one processor is configured to: A second model is trained based on a first model pre-trained for the object re-identification, wherein the second model includes the backbone, the first neck, and the second neck.

13. The device according to claim 12, wherein The at least one processor is configured to: The second neck is trained with parameters of the backbone and the first neck fixed.

14. The device according to claim 12, wherein The at least one processor is configured to: generating a second feature vector based on the first input image and via the first model; generating a third feature vector based on the second input image and via the backbone and the second neck; generating a first loss based on the second eigenvector and the third eigenvector; and The second neck is trained based on the first loss.

15. The device according to claim 14, wherein The at least one processor is configured to: generating a second loss based on the third eigenvector and the ground truth GT; and The second neck is trained based on the first loss and the second loss.

16. The device according to claim 15, wherein The at least one processor is configured to: generating a classification vector based on the third feature vector and through a classification network; and The second loss is generated by comparing the classification vector and the ground truth GT.

17. The device according to claim 15, wherein The ground truth GT consists of hard labeled data.

18. The device according to claim 14, wherein The second feature vector includes soft label data.

19. The device according to claim 14, wherein The first input image is an image obtained by extracting an object region from the second input image.

20. The device according to claim 11, wherein The at least one processor is configured to: generating a first correlation vector obtained by combining the first detection information and the first feature vector; generating a second feature based on a second frame and via the backbone, wherein the second frame is a next frame to the first frame; generating a plurality of second detection information about a plurality of second objects based on the second feature and via the first neck; generating, based on the second feature and via the second neck, a fourth feature vector of visual features for the plurality of second objects; generating a second association vector, wherein the second association vector is obtained by respectively combining the plurality of pieces of second detection information and the fourth eigenvector; and Tracking is performed by associating the first object with an object having an association vector closest to the first association vector among the second association vectors.

Images