Method and device for tracking

Abstract

A tracking method according to an example of the present disclosure may include generating a first feature by a generating device based on a first frame through a backbone, generating first detection information indicating a detection result for a first object based on the first feature by a first neck for object detection by the generating device, generating a first feature vector for a visual feature of the first object by the generating device based on the first feature by a second neck for object recognition, and / or performing tracking by a tracking device based on the first detection information and the first feature vector.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to object detection and, more particularly, to a tracking method and apparatus therefor. BACKGROUND

[0002] With the development of deep neural network-based image processing technology in autonomous driving technology, various artificial intelligence models such as object detection, semantic segmentation, depth map estimation, and lane recognition are being investigated. Research is ongoing into a method for performing autonomous driving by tracking objects using artificial intelligence models. For example, an object detection model and an object re-identification model can be used for tracking. A method that can improve the speed of an algorithm without degrading the performance of object detection in terms of tracking may be needed. SUMMARY

[0003] The present disclosure can solve the above-mentioned problems of the prior art while maintaining the advantages achieved by the prior art.

[0004] One aspect of the present disclosure may provide a method and apparatus that improve tracking performance without causing performance degradation in object detection.

[0005] One aspect of the present disclosure may provide a method and apparatus that performs training on an object redetect model by applying a knowledge distillation technique.

[0006] One aspect of the present disclosure may provide a method and apparatus that perform training of a model for object recognition using a pre-trained model.

[0007] One aspect of the present disclosure may provide a method and apparatus that simultaneously performs object detection and object recognition.

[0008] One aspect of the present disclosure may provide a method and apparatus that performs multi-task learning in conjunction with tracking.

[0009] The technical problems to be solved by the present disclosure are not limited to the above-mentioned problems, and any other technical problems not mentioned here will be clearly understood by those skilled in the art to which the present disclosure relates from the following description.

[0010] A device may comprise: a memory configured to store computer-executable instructions; and at least one processor configured to execute the computer-executable instructions by accessing the memory, wherein the at least one processor is configured to: generate a first feature based on a first frame and via a backbone of a detection model; generate first detection information indicative of a detection result for a first object based on the first feature and via a first neck of the detection model, the first neck being configured for object detection; generate a first feature vector for a visual feature of the first object based on the first feature and via a second neck of the detection model, the second neck being configured for object recognition;and, based on the first detection information and the first feature vector, track at least one object comprising the first object;

[0011] The at least one processor may be configured to: train a second model based on a first model pre-trained on object recognition, wherein the second model comprises the backbone, the first neck, and the second neck.

[0012] The at least one processor may be configured to: train the second neck in a state in which the parameters of the backbone and the first neck are fixed.

[0013] The at least one processor may be configured to: generate a second feature vector based on a first input image and via the first model; generate a third feature vector based on a second input image and via the backbone and the second neck; generate a first loss based on the second feature vector and the third feature vector; and train the second neck based on the first loss.

[0014] The at least one processor may be configured to: generate a second loss based on the third feature vector and the ground truth (GT); and train the second neck based on the first loss and the second loss.

[0015] The at least one processor may be configured to: generate a classification vector based on the third feature vector and via a classification network, and generate the second loss by comparing the classification vector and the GT.

[0016] The GT can consist of hard-labeled data.

[0017] The second feature vector can consist of soft-labeled data.

[0018] The first input image may be an image obtained by extracting an object region from the second input image.

[0019] The at least one processor may be configured to: generate a first association vector obtained by combining the first detection information and the first feature vector; generate a second feature based on a second frame and across the backbone, wherein the second frame is a next frame of the first frame; generate pieces of second detection information about a plurality of second objects based on the second feature and across the first neck; generate fourth feature vectors for visual features of the plurality of second objects based on the second feature and across the second neck; generate second association vectors, wherein the second association vectors are obtained by respectively combining the pieces of the second detection information and the fourth feature vectors;and perform the tracking by associating the first object with an object having an association vector among the second association vectors that is closest to the first association vector;

[0020] One or more methods may be performed by at least one computer to perform one or more operations described herein and / or to implement one or more functions described herein.

[0021] The features of the present disclosure briefly summarized above are merely aspects of the detailed description of the present disclosure described below and do not limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings: Fig. 1 shows an example diagram illustrating a tracking method according to an example of the present disclosure; Fig. 2 shows an example flowchart illustrating a tracking method according to an example of the present disclosure; Fig. 3 shows an example diagram illustrating a tracking method according to an example of the present disclosure; Fig. 4 shows an example diagram illustrating a tracking method according to an example of the present disclosure; Fig. 5 shows an example flowchart illustrating a tracking method according to an example of the present disclosure; Fig. 6 shows an example diagram illustrating a tracking method according to an example of the present disclosure; Fig. 7 shows an example of a flowchart illustrating a tracking method according to an example of the present disclosure; Fig. 8 shows an example diagram illustrating a tracking method according to an example of the present disclosure; Fig. 9 shows an example of a block diagram of a tracking device according to an example of the present disclosure; and Fig. 10 shows an example block diagram of a computer system for performing a tracking method according to an example of the present disclosure. DETAILED DESCRIPTION

[0023] Below, various examples of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the examples set forth herein.

[0024] In describing the examples of the present disclosure, the detailed description is omitted when it is believed that a specific description of the related art would obscure the subject matter of the examples of the present disclosure. Furthermore, in the drawings, parts unrelated to the description of the present disclosure are omitted, and similar parts are denoted by similar reference numerals.

[0025] Throughout this disclosure, it is understood that an element described as being "connected" or "coupled" to another element may be connected directly or indirectly to another element. When a part "includes" or "possesses" some elements, unless expressly described to the contrary, it means that other elements may also be included but not excluded.

[0026] In the present disclosure, terms such as "first" or "second," and the like, may express their elements regardless of their priority or importance, and may be used to distinguish one element from another, but are not limited to these components. For example, without departing from the scope of the present disclosure, a first component of one example may be referred to as the second component of another example. Similarly, a second component of one example may be referred to as the first component of another example.

[0027] In the present disclosure, components distinguished from one another serve only to clearly describe features and do not imply that the components are necessarily separate. That is, a plurality of components may be integrated to form a single hardware or software unit, or a single component may be distributed to form a plurality of hardware or software units. Accordingly, such integrated or distributed examples fall within the scope of the present disclosure, even if not specifically mentioned.

[0028] In the present disclosure, the components described in various examples do not necessarily represent essential components, and some may be optional components. Therefore, an example consisting of a subset of the components described in an example also falls within the scope of the present disclosure. Furthermore, an example including another component in addition to the components described in the various examples also falls within the scope of the present disclosure.

[0029] In this disclosure, terms such as "top," "bottom," "left," and "right" are used to describe positions for convenience. If the drawings shown in this specification are viewed in reverse order, the positional relationship described in the specification can be interpreted in reverse.

[0030] In the disclosure, the terms "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any combination of one or more of the listed items.

[0031] Below, various examples of the present disclosure are described with reference to the Fig. 1 to 10 described in detail.

[0032] Fig. 1 shows an example diagram illustrating a tracking method according to an example of the present disclosure.

[0033] Tracking can be performed by performing object detection and extracting an object's visual features through object re-identification. Object detection can involve creating detection information, including the object's location, bounding box information about the object, the object's directional angle, or the like. For example, object re-identification can involve creating a feature vector for an object's visual features, such as the object's size, the object's color, and / or the object's shape. If object re-identification is performed after object detection, a network that performs object detection and a network that performs object re-identification can exist separately.The presence of two networks may result in slower tracking algorithms than in the case where object detection and re-identification of the object are performed with a single network.

[0034] Referring to Fig. 1, the tracking method according to an example of the present disclosure may perform object detection and object re-identification simultaneously. For example, the tracking method may perform object detection and object re-identification simultaneously using a network. For example, a first neck for object detection and a second neck for object re-identification may operate as a network by sharing a backbone. The tracking method may accelerate the performance of the tracking algorithm by using a model that includes necks that share a backbone. Detailed descriptions of the tracking algorithm may be provided here.

[0035] The level of automation of an autonomous vehicle can be classified as follows according to the American Society of Automotive Engineers (SAE). At Level 0 of autonomous driving, the SAE classification standard may correspond to "non-automation," where an autonomous driving system temporarily intervenes in emergency situations (e.g., automatic emergency braking) and / or only issues warnings (e.g., blind spot warning, lane departure warning, etc.), and a driver is expected to operate the vehicle. At Level 1 of autonomous driving, the SAE classification standard may correspond to "driver assistance," where the system performs some driving functions (e.g., steering, acceleration, braking, lane centering, adaptive cruise control, etc.).) while the driver controls the vehicle in a normal operating segment, and the driver is expected to determine the operating state and / or timing of the system, perform other driving functions, and handle (e.g., resolve) emergency situations. At Level 2 of autonomous driving, the SAE classification standard may correspond to "partial automation," where the system performs steering, acceleration, and / or braking under the supervision of the driver, and the driver is expected to determine the operating state and / or timing of the system, perform other driving functions, and handle (e.g., resolve) emergency situations. At Level 3 of autonomous driving, the SAE classification standard may correspond to "conditional automation," where the system controls the vehicle under certain conditions (e.g.,driving functions such as steering, acceleration, and / or braking), but transfers driving control to the driver if the required conditions are not met, and the driver is expected to determine an operating state and / or timing of the system and assume control in emergency situations, but otherwise does not control the vehicle (e.g., steering, acceleration, and / or braking). At Level 4 of autonomous driving, the SAE classification standard may correspond to the "high level of automation," where the system performs all driving functions and the driver is expected to assume control of the vehicle only in emergency situations.At Level 5 autonomous driving, the SAE classification standard may correspond to "full automation," where the system performs all driving functions without any driver assistance, even in emergency situations, and the driver is not expected to perform any driving functions other than determining the system's operating state. Although this disclosure may apply the SAE classification standard for the classification of autonomous driving, other classification methods and / or algorithms may be used in one or more of the configurations described herein.

[0036] Fig. 2 shows an example flowchart illustrating a tracking method according to an example of the present disclosure.

[0037] A detection model (e.g., a neural network) can comprise a backbone, at least one neck, and / or a head. The backbone can exploit the essential features of different resolutions, and the neck can merge the features of different resolutions. At least one head can perform the detection of objects at different resolutions.

[0038] A backbone network can be used in the architectures of object detection models. The backbone network can be responsible for extracting and encoding features from the input data. It can act as a core feature extractor, capturing low-level and high-level features from the input data.

[0039] A neck can be responsible for further transforming and refining the features extracted by the backbone model. The neck can enhance the extracted features of the backbone and provide more informative feature representations.

[0040] The backbone may be responsible for the initial extraction of features from the input data, while the neck enhances and merges these features to improve the performance of the model.

[0041] The head may contain task-specific layers that serve to make the final prediction or conclusion based on the information extracted by the backbone and neck.

[0042] In machine learning, a loss function (also known as a cost function or error function) is a method for evaluating how well a particular algorithm models the given data. By comparing the predicted values ​​produced by the model with the actual target values, the loss function quantifies the error, or difference. The purpose of the loss function is to guide the training process. When the model makes a prediction, the loss function calculates a numerical value that indicates how far the prediction is from the true value. The goal of the learning algorithm is to minimize this loss value by adjusting the model's parameters during the training phase.

[0043] As in Fig. As illustrated in Figure 2, in a tracking method according to the example of the present disclosure, at S210, a generating device may generate a first feature, for example, based on a first image via a backbone. The output of the backbone receiving an image may be referred to as a "feature."

[0044] According to the tracking method, in S220, the generating device may generate first detection information indicating the detection result for a first object based on a first feature by a first neck for object detection. The first detection information may include location information of the first object, a heading angle of the first object, and / or bounding box information about the first object.

[0045] According to the tracking method, in S230, the generating device may generate a first feature vector for the visual feature of the first object based on the first feature by a second neck for object recognition. For example, the first feature vector may be a vector containing information about the properties of the first object, such as the shape of the first object, the size of the first object, and / or the color of the first object.

[0046] According to the tracking method, a tracking device may perform tracking based on the first detection information and / or the first feature vector in S240. For example, the tracking method may perform tracking by performing an association based on object detection information about the previous frame of a first frame, a feature vector of the first frame, first detection information of the first frame, and / or first detection information of the first frame. For example, the tracking method may perform tracking by performing an association based on object detection information about a second image, which may be the next image of the first image, a feature vector of the second image, the first detection information of the first image, and / or the first detection information of the first image. Detailed descriptions of tracking may be given later.

[0047] Fig. 3 shows an example diagram illustrating a tracking method according to an example of the present disclosure.

[0048] As in Fig. As shown in Figure 3, a tracking method according to an example of the present disclosure may perform object detection and / or object re-identification at S310. The tracking method may simultaneously perform object detection and object re-identification using a network. For example, as described above, a first neck for object detection and a second neck for object re-identification may share a backbone. For example, object detection and object re-identification may be performed simultaneously.

[0049] In S320, the tracking method may perform association. Association may refer to a process of connecting an object detected in a current image with an object detected in a previous image. For example, an association vector for the object of the previous frame may be created by combining object detection information and a feature vector for the object detected in the previous frame. Association vectors may be created by combining object detection information and feature vectors for objects detected in the current image.For example, the tracking method may perform an association by calculating a distance between the association vector for the previous frame and the association vectors for the current frame, and / or by considering an object with an association vector of the current frame that is closest to the association vector for the previous frame as an object corresponding to the object of the previous frame. A cosine distance, a Euclidean distance, a Mahalanobis distance, and / or an intersection-over-unit (IOU) distance may be used to calculate the distance between the association vectors.

[0050] In S330, the tracking method may perform an update. For example, the tracking method may update the states of objects based on the matching result. For example, the tracking method may update data about an object located at a first location in the previous image to change it to data about an object located at a second location in the current image, but is not limited thereto.

[0051] In S340, the tracking method may perform management. For example, the tracking method may manage the tracking status of an object. For example, the tracking method may remove an object whose tracking can be interrupted if the object disappears within an object detection area from a tracking target.

[0052] In S350, the tracking process can perform registration. For example, if a new object is detected within the object detection area, the tracking process can register the new object as a tracking target.

[0053] In S360, the tracking method can perform a prediction. For example, the tracking method can predict the positions of objects in the next image. The tracking method can predict the positions of objects, for example, by using a Kalman filter, but is not limited to this. The location information of the predicted objects can be used in the matching process. The matching can be performed, for example, by calculating a distance between the association vectors of objects detected as being within a predetermined distance from the locations of the predicted objects and the association vectors of objects for which a prediction was performed.

[0054] Fig. 4 shows an example diagram illustrating a tracking method according to an example of the present disclosure.

[0055] Referring to Fig. 4, a tracking method according to an example of the present disclosure may perform an inference operation and / or a training operation. The tracking method may simultaneously perform object detection and object re-identification during the inference operation. For example, object detection and object re-identification may be performed simultaneously by sharing a backbone between a first neck (neck1) associated with object detection and a second neck (neck2) associated with object re-identification. As described herein, the tracking method may generate a first feature based on a first frame through the backbone. For example, the tracking method may generate first detection information indicating the detection result for a first object based on a first feature through the first neck.The first detection information may, for example, include information about a class of an object, a 2D bounding box for the object, a 3D bounding box for the object, and / or a heading angle of the object. For example, the tracking method may generate a first feature vector for a visual feature of the first object based on the first feature by the second neck. The generated first detection information and the generated first feature vector may be used when performing a matching.

[0056] The tracking method can perform the training operation. For example, the tracking method can train the backbone and the first neck based on a ground truth (GT) for object detection. Furthermore, the tracking method can train the second neck by applying a knowledge distillation technique. For example, the second neck can be trained using the output of a first model previously trained for object re-identification. For example, a classification vector can be created by inputting the feature vector, which can be generated by the second neck, to a pooling layer and a fully connected layer, which can be a classification network. The second neck can be trained by comparing the GT for object re-identification with the classification vector. Detailed descriptions of training the second neck can be given here.

[0057] Fig. 5 shows an example flowchart illustrating a tracking method according to an example of the present disclosure.

[0058] Referring to Fig. 5, according to a tracking method according to an example of the present disclosure, a generating device may generate a first association vector obtained by combining first detection information and a first feature vector in S510. The first association vector with 130 channels may be created, for example, by adding the first detection information about a first object to the first feature vector consisting of 128 channels, but is not limited thereto.

[0059] According to the tracking method, in S520, the generating device may generate a second feature based on a second image, which may be the next image of a first image, via a backbone.

[0060] According to the tracking method, in S530, the generating device may generate pieces of the second detection information about a plurality of second objects based on the second feature through a first neck.

[0061] According to the tracking method, in S540, the generating device may generate fourth feature vectors for the visual features of a plurality of second objects based on the second feature by a second neck.

[0062] According to the tracking method, in S550, the generating device may generate second association vectors obtained by combining the second detection information and the fourth feature vectors.

[0063] According to the tracking method, a tracking device may perform tracking in S560 by associating an object having the association vector closest to the first association vector among the second association vectors with the first object. For example, an object corresponding to a vector closest to the first association vector in the first image among the second association vectors in the second image may correspond to the first object. For example, tracking may be performed by associating the first object in the first image with an object having the association vector closest to the first association vector.

[0064] Fig. 6 shows an example diagram illustrating a tracking method according to an example of the present disclosure.

[0065] Referring to Fig. 6, a tracking method according to an example of the present disclosure may generate detection information for objects in a first image and feature vectors for the objects. The tracking method may predict the positions of objects M-1 to M-5 in a second frame (frame 2), which may be the next frame of the first frame, based on information generated with respect to the first frame (S610). The tracking method may predict the positions of the objects based on the first image. For example, if the positions of objects are detected in the current second frame, the tracking method may perform a correlation based on the predicted position and the actually detected position.

[0066] Association may refer to the process of identifying whether a detected object in a new image corresponds to a tracked object. If new detection results are received in a frame, association may consist of finding the matching object among the tracked objects and the detection results. If an object is associated, the newly received detection information, e.g., updated location data, can be used to update the object's status. This continuous association of object detection results over time may allow for accurate estimation of information such as location and / or speed.

[0067] For example, if a first object in the first image is predicted to be located at a first location in the second image, a distance between the association vectors of objects that may be a predetermined distance away from the first location may be calculated from the objects detected in the second image and the association vector of the first object. The tracking method may determine that the object with the association vector closest to the association vector of the first object is the same object as the first object and may perform tracking T-1 to T-5 (S620). For example, the tracking method may predict the positions of objects in a third frame (frame 3), which may be the next frame of the second frame, based on information generated regarding the second frame (S630).Tracking for the second image and the third image can be performed using the tracking method for the first image and the second image.

[0068] Fig. 7 shows an example of a flowchart illustrating a tracking method according to an example of the present disclosure.

[0069] A tracking method according to an example of the present disclosure may train a second model comprising a backbone, a first neck, and a second neck based on a first model trained for object re-recognition. The first neck may be associated with object detection. The second neck may be associated with object re-identification. The first neck and the second neck may share the backbone.

[0070] The tracking method can train the second model based on the first model by applying a knowledge distillation technique. For example, the tracking method can train the second neck using the first model in a state where the parameters of the backbone and the first neck are fixed. For example, the tracking method can train the second neck after the backbone and the first neck have been frozen. For example, based on Fig. 7, according to the tracking method in S710, a generating device may generate a second feature vector based on a first input image by a first model.

[0071] According to the tracking method, the generating device may generate a third feature vector based on the second input image through the backbone and the second neck in S720. The first input image described here may be an image obtained by extracting an object region from the second input image.

[0072] According to the tracking method, the generating device may generate a first loss based on the second feature vector and the third feature vector in S730. For example, the tracking method may train the second neck such that the third feature vector is close to the second feature vector, which may be the output of the first model trained to re-identify the object. The tracking method may generate the first loss by comparing the second feature vector and the third feature vector. The second feature vector may include soft-labeled data. Furthermore, the third feature vector may also include soft-labeled data. Soft labeling may be a label that assigns a probability or confidence value to each of multiple classes, rather than a single label (hard labeling) for the data.For example, assume that dogs, cats, and televisions need to be classified. If a network produces an output of [0, 1, 1, 0, 8] for a cat input, there may be a method for training a network to produce an output of [0, 1, 0] for hard-labeled data and a method for training a network to produce an output of [0, 1, 1, 0, 8] for soft-labeled data.

[0073] According to the tracking method, the generating device may generate a second loss based on the third feature vector and the GT in S740. For example, the tracking method may generate a classification vector by inputting the third feature vector to a classification network and generate the second loss by comparing the classification vector and the GT. The GT may, for example, contain hard-labeled data. The classification network may, for example, contain a pooling layer and a fully connected layer.

[0074] At least some networks (e.g., neural networks, deep learning networks, etc.) can exploit domain-invariant features of the source and target domains with some label-induced losses that require ground-truth labels as inputs. Since ground-truth labels are not available, predicted target labels, which can be expressed in various forms, can be used to formulate the label-induced losses.

[0075] Hard-labeled data (e.g., in the form of a hard label) can classify a sample into one or more clear categories, but may be overconfident if the quality of the predicted labels is poor. Soft-labeled data (e.g., in the form of a soft label) can assign probability values ​​to each category in a sample, which can mitigate the problem of overconfidence.

[0076] According to the tracking method, a training device may train the second neck based on the first loss and the second loss at S750. For example, the tracking method may train the second neck by using the first loss and the second loss as the overall loss for re-identifying the object.

[0077] Fig. 8 shows an example diagram illustrating a tracking method according to an example of the present disclosure.

[0078] Referring to Fig. 8, a tracking method according to an example of the present disclosure may train a second model 820 using a first model 810. For example, the tracking method may train the second model 820 by applying a knowledge distillation technique based on the first model 810 trained for object re-identification. For example, the first model 810 trained for object re-identification may generate a second feature vector based on a first input image 801. For example, the second model 820 may generate a third feature vector based on a second input image by a second neck for object re-identification. The first input image 801 may be an image obtained by extracting an object region from a second input image 802. The second feature vector and the third feature vector may include soft-labeled data.The tracking method may generate a first loss by comparing the second feature vector and the third feature vector. For example, the tracking method may input the third feature vector to a classification network 830. The third feature vector input to the classification network may be output as a classification vector. The tracking method may generate a second loss based on the classification vector and the GT including hard label data. The tracking method may train a second neck by using the first loss and the second loss as the overall loss. When training the second neck, the backbone and the first neck may be frozen. For example, the tracking method may train the second neck in a state where the parameters of the backbone and the first neck are fixed.For example, the tracking method can train the backbone and the first neck based on data labeled for object detection.

[0079] The tracking method may perform tracking by performing an estimation by the trained second model 820. For example, the tracking method may generate a first feature based on a first image through the backbone. For example, the tracking method may generate first detection information indicating the detection result for the first object based on the first feature through the first neck for object detection. The first detection information may include information about a heat map, the size of an object, a depth, a heading angle, or the like. The tracking method may generate a first feature vector for the visual feature of the first object based on the first feature through a second neck for object recognition.The tracking method can perform association by calculating the distance between an association vector for the previous frame of the first frame and association vectors for the first frame. For example, the tracking method can update the object's state information based on the association result, perform management by removing an object whose tracking can be interrupted, and additionally register a newly detected object in a tracking list.

[0080] Fig. 9 shows an example of a block diagram of a tracking device according to an example of the present disclosure.

[0081] Referring to Fig. 9, a tracking device 100 may include a memory 110, a generating device 120, a tracking device 130, and a training device 140. The generating device 120, the tracking device 130, and the training device 140 may correspond to a processor.

[0082] The memory 110 may be configured to store computer-executable instructions.

[0083] The processor may be configured to access memory 110 to execute the instructions. For example, the processor may generate a first feature based on a first frame using a backbone by the generating device 120. Additionally or alternatively, the processor may generate first detection information indicating the detection result for a first object based on a first feature by using a first neck for object detection by the generating device 120. Additionally or alternatively, the processor may generate a first feature vector for the visual feature of the first object based on the first feature by using a second neck for object recognition by the generating device 120.

[0084] For example, the processor may perform tracking based on the first detection information and the first feature vector by the tracking device 130.

[0085] For example, the processor may train a second model comprising the backbone, the first neck, and the second neck based on a first model trained using the training device 140 for re-identifying the object. For example, the processor may train the second neck using the training device 140 in a state in which the parameters of the backbone and the first neck are fixed.

[0086] Additionally or alternatively, the processor may generate a second feature vector based on a first input image using the first model by the generating device 120. For example, the processor may generate a third feature vector based on a second input image using the backbone and the second neck by the generating device 120. The first input image may be an image obtained by extracting an object region from the second input image. The processor may generate a first loss based on the second feature vector and the third feature vector by the generating device 120. For example, the second feature vector may include soft-labeled data. The processor may generate a second loss based on the third feature vector and the GT by the generating device 120.For example, the processor may train the second neck based on the first loss and the second loss by the training device 140.

[0087] The processor may generate a classification vector based on the third feature vector using a classification network via the generation device 120. Additionally or alternatively, the processor may generate the second loss by comparing the classification vector and the GT by the generation device 120. The GT may, for example, contain hard-labeled data.

[0088] The processor may generate a first association vector obtained by combining first detection information and the first feature vector by the generating device 120. For example, the processor may generate a second feature based on a second image, which may be the closest image to the first image, for example, by using the backbone by the generating device 120. Additionally or alternatively, the processor may generate pieces of second detection information about a plurality of second objects based on the second feature using the first neck by the generating device 120. For example, the processor may generate fourth feature vectors for visual features of a plurality of second objects based on the second feature using the second neck by the generating device 120.For example, the processor may generate second association vectors through the generating device 120, which can be obtained by appropriately combining the pieces of the second detection information and the fourth feature vectors.

[0089] The processor may perform tracking by associating an object with an association vector that is closest to a first association vector among the second association vectors with the first object via the tracking device 130. The distance between the vectors may be calculated using a cosine distance, a Euclidean distance, or the like.

[0090] According to one aspect of the present disclosure, a tracking method may include the steps of: generating a first feature based on a first image through a backbone by a generating device, generating first detection information indicating a detection result for a first object based on the first feature by the generating device through a first neck for object detection, generating a first feature vector for a visual feature of the first object based on the first feature by a second neck for object recognition by the generating device, and performing tracking based on the first detection information and the first feature vector by a tracking device.

[0091] In one example, the tracking method may further comprise training a second model comprising the backbone, the first neck, and the second neck by a training device based on a first model pre-trained from the object re-identification.

[0092] In one example, training the second model may include the training device training the second neck in a state where the parameters of the backbone and the first neck are fixed.

[0093] According to one example, training the second model may include generating a second feature vector by the generating device based on a first input image by the first model, generating a third feature vector by the generating device based on a second input image by the backbone and the second neck, generating a first loss by the generating device based on the second feature vector and the third feature vector, and training the second neck by the training device based on the first loss.

[0094] According to one example, training the second model may include generating a second loss by the generating device based on the third feature vector and the ground truth (GT), and training the second neck by the training device based on the first loss and the second loss.

[0095] According to one example, generating the second loss may include generating a classification vector by the generating device based on the third feature vector through a classification network and generating the second loss by the generating device by comparing the classification vector and the GT.

[0096] According to one example, the GT can contain hard-labeled data.

[0097] In one example, the second feature vector may contain soft-labeled data.

[0098] In one example, the first input image may be an image obtained by extracting an object region from the second input image.

[0099] According to one example, performing the tracking may include generating, by the generating device, a first association vector obtained by combining the first detection information and the first feature vector; generating, by the generating device, a second feature based on a second frame that is a next frame of the first frame by the backbone; generating, by the generating device, pieces of the second detection information about a plurality of second objects based on the second feature by the first neck; generating, by the generating device, fourth feature vectors for visual features of the plurality of second objects based on the second feature by the second neck; generating, by the generating device, second association vectors obtained by respectively combining the pieces of second detection information and the fourth feature vectors;and performing, by the tracking device, tracking by associating an object with an association vector closest to the first association vector among the second association vectors with the first object.

[0100] According to one aspect of the present disclosure, a tracking device may include a memory storing computer-executable instructions and at least one processor executing the instructions by accessing the memory. The at least one processor may be configured to generate, via a generating device, a first feature based on a first frame using a backbone; to generate first detection information indicative of a detection result for a first object based on the first feature using a first neck for object detection; and to generate a first feature vector for a visual feature of the first object based on the first feature using a second neck for object re-detection; and to perform tracking based on the first detection information and the first feature vector via a tracking device.

[0101] According to one example, the at least one processor may be configured to train, using a training device, a second model comprising the backbone, the first neck, and the second neck based on a first model previously trained using the re-object identification.

[0102] For example, the at least one processor may be configured to train the second neck via the training device in a state in which the parameters of the backbone and the first neck are fixed.

[0103] For example, the at least one processor may be configured to generate, via the generating device, a second feature vector based on a first input image using the first model, to generate a third feature vector based on a second input image using the backbone and the second neck, and to generate a first loss based on the second feature vector and the third feature vector, and to train the second neck based on the first loss by the training device.

[0104] For example, the at least one processor may be configured to generate a second loss based on the third feature vector and the GT by the generating device and to train the second neck based on the first loss and the second loss by the training device.

[0105] For example, the at least one processor may be configured to generate, via the generating device, a classification vector based on the third feature vector using a classification network and to generate the second loss by comparing the classification vector and the GT.

[0106] For example, the GT can contain hard-labeled data.

[0107] The second feature vector can, for example, contain soft-labeled data.

[0108] The first input image may, for example, be an image obtained by extracting an object region from the second input image.

[0109] For example, the at least one processor may be configured to generate, via the generating device, a first association vector obtained by combining the first detection information and the first feature vector; to generate a second feature based on a second frame, which is a next frame of the first frame, using the backbone; to generate pieces of second detection information about a plurality of second objects based on the second feature using the first neck; to generate fourth feature vectors for visual features of the plurality of second objects based on the second feature using the second neck; and to generate second association vectors obtained by respectively combining the pieces of the second detection information and the fourth feature vectors.and that it performs tracking by the tracking device by associating an object with an association vector that is closest to the first association vector among the second association vectors with the first object.,

[0110] Fig. 10 shows an example block diagram of a computer system for performing a tracking method according to an example of the present disclosure.

[0111] With reference to Fig. 10, a computer system 1000 may include at least a processor 1100, a memory 1300, a user interface input device 1400, a user interface output device 1500, a mass storage 1600, and a network interface 1700, which may be interconnected via a bus 1200.

[0112] Processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in memory 1300 and / or mass storage 1600. Memory 1300 and mass storage 1600 may include various types of volatile or non-volatile storage media. For example, memory 1300 may include a ROM (Read Only Memory) 1310 and a RAM (Random Access Memory) 1320.

[0113] Accordingly, the processes of the method or algorithm described with respect to the examples of the present disclosure may be implemented directly by hardware executed by processor 1100, a software module, or a combination thereof. The software module may be located in a storage medium (i.e., memory 1300 and / or mass storage 1600), such as RAM, flash memory, ROM, EPROM, EEPROM, a register, a hard disk, a solid-state drive (SSD), a removable disk, or a CD-ROM. The example storage medium is coupled to processor 1100, and processor 1100 may read information from and write information to the storage medium. In another method, the storage medium may be integrated with processor 1100.The processor 1100 and the storage medium may be located in an application-specific integrated circuit (ASIC). The ASIC may be housed in a user terminal. In another method, the processor 1100 and the storage medium may be housed as separate components in the user terminal.

[0114] Although the present disclosure has been described with reference to exemplary examples and the accompanying drawings, the present disclosure is not limited thereto, but may be variously modified and altered by those skilled in the art to which the present disclosure pertains without departing from the spirit and scope of the present disclosure as claimed in the following claims.

[0115] The examples of the present disclosure may serve to explain the spirit and scope of the present disclosure, but not to limit it, so the spirit and scope of the present disclosure are not limited by the examples. The scope of the present disclosure can be interpreted based on the appended claims, and any technical ideas within the scope corresponding to the claims can be included in the scope of the present disclosure. The above description is merely an example of the technical idea of ​​the present disclosure, and various modifications and alterations can be made by a person skilled in the art without departing from the essential features of the present disclosure.Accordingly, the examples of the present disclosure are not intended to limit the technical idea of ​​the present disclosure, but to illustrate it, and the scope and spirit of the present disclosure are not limited by the above examples. The scope of the present disclosure can be interpreted by the appended claims, and all equivalents thereof can be construed to fall within the scope of the present disclosure.

[0116] According to a tracking method according to the example of the present disclosure, it is possible to improve tracking performance without causing performance degradation in object detection.

[0117] According to a tracking method according to the example of the present disclosure, it is possible to perform training on an object re-identification model by applying a knowledge distillation technique.

[0118] According to a tracking method according to the example of the present disclosure, it is possible to perform training of a model for object re-identification using a pre-trained model.

[0119] According to a tracking method according to the example of the present disclosure, it is possible to perform object detection and object re-identification simultaneously.

[0120] A tracking method according to the example of the present disclosure may perform multi-task learning associated with tracking.

[0121] According to a tracking method according to the example of the present disclosure, a decoder that detects an object and a decoder that re-identifies the object may share a backbone.

[0122] The effects achieved in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art to which the present disclosure belongs from the following description.

[0123] Although the present disclosure has been described with reference to exemplary examples and the accompanying drawings, the present disclosure is not limited thereto, but may be variously modified and altered by those skilled in the art to which the present disclosure pertains without departing from the spirit and scope of the present disclosure as claimed in the following claims.

Claims

[1] A method performed by at least one computing device, the method comprising: Generating a first feature based on a first image and via a backbone of a detection model; Generating, based on the first feature and via a first neck of the detection model, first detection information indicating a detection result for a first object, wherein the first neck is configured for object detection; Generating a first feature vector for a visual feature of the first object based on the first feature and via a second neck of the detection model, the second neck being configured for re-identification of the object; and Performing the tracking of at least one object comprising the first object based on the first detection information and the first feature vector. [2] The method of claim 1, further comprising: Training a second model based on a first model pre-trained on object recognition, the second model comprising the backbone, the first neck, and the second neck. [3] The method of claim 2, wherein training the second model comprises: Training the second neck in a state where the parameters of the backbone and the first neck are fixed. [4] The method of claim 2, wherein training the second model comprises: Generating a second feature vector based on a first input image and using the first model; Generating a third feature vector based on a second input image and across the backbone and the second neck; Generating a first loss based on the second feature vector and the third feature vector; and Training based on the first loss, the second neck. [5] The method of claim 4, wherein training the second model comprises: Generating a second loss based on the third feature vector and ground truth (GT), and wherein training the second neck comprises training the second neck based on the first loss and the second loss. [6] The method of claim 5, wherein generating the second loss comprises: Generating a classification vector based on the third feature vector and via a classification network; and Generate the second loss by comparing the classification vector with the GT. [7] The method of claim 5, wherein the GT comprises hard-tagged data. [8] The method of claim 4, wherein the second feature vector comprises soft-labeled data. [9] The method according to claim 4, wherein the first input image is an image obtained by extracting an object region from the second input image. [10] The method of claim 1, wherein performing the tracking comprises: Generating a first association vector obtained by combining the first detection information and the first feature vector; Generating a second feature based on a second frame and across the backbone, the second frame being a next frame to the first frame; Generating second detection information about a plurality of second objects based on the second feature and the first neck; generating fourth feature vectors for visual features of the plurality of second objects based on the second feature and across the second neck; Generating second association vectors, wherein the second association vectors are obtained by respectively combining the parts of the second detection information and the fourth feature vectors; and Performing the tracking by associating the first object with an object having an association vector among the second association vectors that is closest to the first association vector. [11] Device comprising: a memory configured to store computer-executable instructions; and at least one processor configured to execute the computer-executable instructions by accessing the memory, wherein the at least one processor is configured: to generate a first feature based on a first image and via a backbone of a detection model, to generate first detection information based on the first feature and via a first neck of the detection model, which first neck indicates a detection result for a first object, wherein the first neck is configured for object detection, generate a first feature vector for a visual feature of the first object based on the first feature and via a second neck of the detection model, wherein the second neck is configured for recognizing the object; and to track at least one object comprising the first object based on the first detection information and the first feature vector. [12] The apparatus of claim 11, wherein the at least one processor is configured: to train a second model based on a first model trained on object recognition, the second model comprising the backbone, the first neck and the second neck. [13] The apparatus of claim 12, wherein the at least one processor is configured: to train the second neck in a state where the parameters of the backbone and the first neck are fixed. [14] The apparatus of claim 12, wherein the at least one processor is configured: to generate a second feature vector based on a first input image and the first model; to generate a third feature vector based on a second input image and via the backbone and the second neck; to generate a first loss based on the second feature vector and the third feature vector; and based on the first loss, to train the second neck. [15] The apparatus of claim 14, wherein the at least one processor is configured: generate a second loss based on the third feature vector and the ground truth (GT); and to train the second neck based on the first loss and the second loss. [16] The apparatus of claim 15, wherein the at least one processor is configured: to generate a classification vector based on the third feature vector and via a classification network, and to generate the second loss by comparing the classification vector with the GT. [17] The apparatus of claim 15, wherein the GT comprises hard-tagged data. [18] The apparatus of claim 14, wherein the second feature vector comprises soft-labeled data. [19] The apparatus according to claim 14, wherein the first input image is an image obtained by extracting an object region from the second input image. [20] The apparatus of claim 11, wherein the at least one processor is configured: generate a first association vector obtained by combining the first detection information and the first feature vector; generate a second feature based on a second frame and via the backbone, the second frame being a next frame of the first frame; to generate second detection information about a plurality of second objects based on the second feature and via the first neck; to generate fourth feature vectors for visual features of the plurality of second objects based on the second feature and via the second neck; to generate second association vectors, wherein the second association vectors are obtained by respectively combining the parts of the second detection information and the fourth feature vectors; and to perform the tracking by associating the first object with an object that has an association vector among the second association vectors that is closest to the first association vector.

Images