System and method for image-to-video re-identification

Abstract

A computer-implemented method includes: obtaining a model trained based on at least knowledge distillation between a teacher network and a student network according to a modified triplet loss; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

Description

Technical Field

[0001] The present disclosure relates generally to image-to-video re-identification, and more particularly to image-to-video re-identification via mutual discriminative knowledge transfer. Background Art

[0002] Image-to-video re-identification (I2V Re-ID) involves identifying an object from multiple videos, given an image of the object. For example, a query might include an image of a person or a vehicle. Based on the query, one or more videos capturing the person or vehicle are identified from a collection of surveillance videos. The representation gap between images and videos makes I2V Re-ID challenging to solve. Summary of the Invention

[0003] Various embodiments of the description include, but are not limited to, cloud-based systems, methods, and non-transitory computer-readable media for image-to-video re-identification.

[0004] In some embodiments, one or more non-transitory computer-readable storage media store instructions executable by one or more processors, wherein execution of the instructions causes the one or more processors to perform operations, the operations comprising: obtaining a model trained based on at least knowledge distillation between a teacher network and a student network according to a modified triplet loss; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

[0005] In some embodiments, the teacher network is a video-to-video re-identification network configured to identify matches between a query video and a gallery of videos of the teacher network; and the student network is an image-to-video re-identification network configured to identify matches between a query image and a gallery of videos of the student network.

[0006] In some embodiments, obtaining the model includes: for a first training, training the teacher network based on at least using multiple videos as input training data for the teacher network; and for a second training, training the teacher network and the student network together based on at least: (i) for each of one or more objects, using multiple images of the object in different views as input training data for the teacher network, and (ii) for each of the one or more objects, using a subset of the multiple images as input training data for the student network.

[0007] In some embodiments, for the second training, the teacher network is configured to transfer temporal information to the student network through the knowledge distillation.

[0008] In some embodiments, the knowledge distillation includes self-distillation or cross-distillation.

[0009] In some embodiments, obtaining the model includes: extracting multiple teacher features from the input training data of the teacher network based on at least global average pooling; and extracting multiple student features from the input training data of the student network based on at least global average pooling.

[0010] In some embodiments, obtaining the model further comprises: at least based on the triple loss of the teacher features to train the teacher network.

[0011] In some embodiments, training the teacher network comprises training the teacher network without using a cross-entropy classification loss.

[0012] In some embodiments, obtaining the model further comprises: based on at least a pairwise distance loss (L PD ) to train the student network; and the pairwise distance loss (L PD ) includes the difference between: (i) one or more teacher feature distances among at least a portion of the teacher features, each of the teacher feature distances indicating a distance between two different inputs of the teacher network, and (ii) one or more student feature distances among at least a portion of the student features, each of the student feature distances indicating a distance between two different inputs of the student network.

[0013] In some embodiments, obtaining the model further comprises: based on at least the mutual logarithmic distillation loss (L MKD ) to train the teacher network and the student network; and the mutual logarithmic distillation loss (L MKD) includes (i) a student-to-teacher logits distillation loss based at least on the teacher features and the student features and (ii) a teacher-to-student logits distillation loss based at least on the teacher features and the student features.

[0014] In some embodiments, obtaining the model further comprises: selecting a portion of the teacher features and a portion of the student features by using intra-batch difficult sample mining; and using the selected portion of the teacher features and the selected portion of the student features as inputs to the modified triplet loss.

[0015] In some embodiments, the modified triplet loss includes a mutual third-order distillation loss; and the third-order mutual distillation loss includes a student-to-teacher third-order distillation loss and a teacher-to-student third-order distillation loss.

[0016] In some embodiments, each of the student-to-teacher third-order distillation loss and the teacher-to-student third-order distillation loss is based on at least a first sample distribution and a second sample distribution; the first sample distribution is based on at least a division between: (i) a first function of the distance between the anchor sample of the teacher feature and the positive sample of the teacher feature, and (ii) the sum of the first function and a second function of the distance between the anchor sample of the teacher feature and the negative sample of the teacher feature; and the second sample distribution is based on at least a division between: (i) a third function of the distance between the anchor sample of the student feature and the positive sample of the student feature, and (ii) the sum of the third function and a fourth function of the distance between the anchor sample of the student feature and the negative sample of the student feature.

[0017] In some embodiments, obtaining the model includes: minimizing a total loss (L) including: a triplet loss corresponding to the teacher network, a triplet loss corresponding to the student network, a logarithmic distillation loss, a pairwise distance loss, and the modified triplet loss.

[0018] In some embodiments, providing the image and the multiple videos to the model to obtain one or more first features of the image and one or more second features of each of the multiple videos includes: providing the image and the multiple videos to a trained teacher network or a trained student network to obtain one or more first features of the image and one or more second features of each of the multiple videos.

[0019] In some embodiments, determining the one or more videos from the multiple videos that match the image based on the one or more first features and the one or more second features includes: determining, for each video from the multiple videos, a distance between the one or more first features and the one or more second features; ranking the multiple videos according to the distance starting from the lowest distance; and determining one or more highest-ranked videos or one or more videos below a threshold distance from the multiple videos as one or more matches to the image.

[0020] In some embodiments, the training data of the student network and the training data of the teacher network do not include the image; and the training data of the student network and the training data of the teacher network do not include any of the plurality of videos.

[0021] In some embodiments, a computer-implemented method includes: obtaining a model trained based on at least knowledge distillation between a teacher network and a student network according to a modified triplet loss; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

[0022] In some embodiments, a system includes: one or more processors; and one or more non-transitory computer-readable memories, the one or more non-transitory computer-readable memories being coupled to the one or more processors and configured with instructions, the instructions being executable by the one or more processors to cause the system to perform operations, the operations including: obtaining a model trained based on at least knowledge distillation between a teacher network and a student network according to a modified triplet loss; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

[0023] In some embodiments, a computer system includes: a first acquisition module, which is configured to obtain a model trained based on at least knowledge distillation between a teacher network and a student network according to a modified triplet loss; a second acquisition module, which is configured to obtain an image and multiple videos; a providing module, which is configured to provide the image and the multiple videos to the model to obtain one or more first features of the image and one or more second features of each of the multiple videos; and a determination module, which is configured to determine one or more videos in the multiple videos that match the image based on the one or more first features and the one or more second features.

[0024] In some embodiments, a computer-implemented method includes obtaining a model trained according to a modified triplet loss based on division; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

[0025] In some embodiments, one or more non-transitory computer-readable storage media store instructions executable by one or more processors, wherein execution of the instructions causes the one or more processors to perform operations comprising: obtaining a model trained according to a modified triplet loss based on division; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

[0026] In some embodiments, a system includes: one or more processors; and one or more non-transitory computer-readable memories, wherein the one or more non-transitory computer-readable memories are coupled to the one or more processors and configured with instructions, the instructions being executable by the one or more processors to cause the system to perform operations, the operations including: obtaining a model trained according to a modified triplet loss based on division; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

[0027] In some embodiments, a computer-implemented method includes: obtaining a model trained according to a modified triplet loss; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on a distance between the one or more first features and the one or more second features.

[0028] In some embodiments, one or more non-transitory computer-readable storage media store instructions executable by one or more processors, wherein execution of the instructions causes the one or more processors to perform operations comprising: obtaining a model trained according to a modified triplet loss; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on a distance between the one or more first features and the one or more second features.

[0029] In some embodiments, a system includes: one or more processors; and one or more non-transitory computer-readable memories, wherein the one or more non-transitory computer-readable memories are coupled to the one or more processors and configured with instructions, the instructions being executable by the one or more processors to cause the system to perform operations, the operations including: obtaining a model trained according to a modified triplet loss; obtaining an image and a plurality of videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and determining one or more videos in the plurality of videos that match the image based on a distance between the one or more first features and the one or more second features.

[0030] These and other features of the systems, methods, and non-transitory computer-readable media disclosed herein, as well as the methods of operation and functionality of the related structural elements and the economy of assembly and manufacture of the parts will become more apparent after considering the following description and the appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. However, it should be expressly understood that the drawings are for purposes of illustration and description only and are not intended as a definition of the limits of the specification. It should be understood that the foregoing general description and the following detailed description are exemplary and illustrative only and are not limiting of the specification as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Non-limiting embodiments of the present invention may be more readily understood by reference to the accompanying drawings, in which:

[0032] Figure 1 Illustrated is an exemplary system for image-to-video re-identification in accordance with various embodiments of the present disclosure.

[0033] Figure 2A Illustrated is an exemplary model framework for image-to-video re-identification in accordance with various embodiments of the present disclosure.

[0034] Figure 2B Illustrated is an exemplary method of applying a model for image-to-video re-identification in accordance with various embodiments.

[0035] Figure 3 Illustrated is a visual comparison of models for image-to-video re-identification in accordance with various embodiments.

[0036] Figure 4 Illustrated is an exemplary method for image-to-video re-identification in accordance with various embodiments.

[0037] Figure 5 Illustrated is an exemplary system for image-to-video re-identification in accordance with various embodiments.

[0038] Figure 6 A block diagram illustrating an exemplary computer system in which any of the embodiments described herein may be implemented. DETAILED DESCRIPTION

[0039] Non-limiting embodiments of the present specification will now be described with reference to the accompanying drawings. The specific features and aspects of any embodiment disclosed herein may be used and / or combined with the specific features and aspects of any other embodiment disclosed herein. Such embodiments are provided as examples and illustrate only a few embodiments within the scope of this specification. Various changes and modifications apparent to those skilled in the art to which this specification pertains are deemed to be within the spirit, scope, and intent of this specification as further defined in the appended claims.

[0040] Re-identification (Re-ID) aims to retrieve matching samples with the same identity as a given query in a collection of images. This task of Re-ID can include two main categories: image-based (I2I) Re-ID and video-based (V2V) Re-ID. In I2I Re-ID, both the query and the collection are images, while in V2V Re-ID, they are both videos. However, in many real-world scenarios, the query may include one or more images of an object (e.g., a person or vehicle captured across non-overlapping cameras), while the collection includes videos. The query may include one or more images. This type of Re-ID can be called image-to-video (I2V) Re-ID. An example of an application in the security field is to quickly locate and track a suspect from a large amount of surveillance video based on only one image of the suspect.

[0041] In contrast to I2I Re-ID and V2V Re-ID, I2V Re-ID aims to bridge image and video representations. Current techniques propose projecting images and videos into a shared embedding space or focus on global matching designed for classification tasks or representative learning. However, in the Re-ID task, due to the non-overlapping labels of the training and test sets, the discriminative ability of features is more important. That is, the model ultimately needs to be applied to test objects (e.g., person X1) that may be different from the objects used when training the model (e.g., persons A1, A2, ... A1000). Unlike classifier models that are configured to classify queries into one of multiple trained categories by learning which category the object falls into, Re-ID requires learning feature differences between different objects in order to find matches to the query. In Re-ID applications, the test data may be fundamentally different from the training data used to train the model. Therefore, the model needs to learn the discriminative nature of features across various objects, and the learned features need to be distinguishable across objects.

[0042] In various embodiments, to at least address the representation gap between images and videos in I2V Re-ID, a knowledge distillation (KD) process can be used. KD is a technique for transferring knowledge from a teacher network to a student network. It can achieve both a speedup in inference time and a reduction in memory consumption without a large performance degradation. KD can be roughly divided into two categories: distillation from logits and distillation from feature embeddings. The knowledge distillation used in the disclosed methods and systems can include (i) self-distillation, in which knowledge is transferred from a teacher network to a student network with the same architecture (e.g., both the teacher network and the student network are ResNet-50), or (ii) cross-distillation, in which knowledge is transferred from a teacher network to a student network with a different architecture (e.g., one of the teacher network and the student network is ResNet-50, and the other is ResNet-101).

[0043] KD can significantly improve matching accuracy by transferring richer representations from a video-based teacher network to an image-based student network. In addition, the disclosed systems and methods can achieve feature discriminability for Re-ID during KD, which is useful when applying the model to objects that are not in the training data. For example, the disclosed model can be trained with a training dataset of 5,000 objects (e.g., people, vehicles), and the trained model can be effectively applied to queries for objects that are not in the training dataset (e.g., people or vehicles that are different from the 5,000 objects in the training dataset). To this end, in some embodiments, a mutually discriminative knowledge distillation framework is disclosed to more effectively transfer richer video-based representations to image-based representations. In one embodiment, the framework can introduce a triplet contrast loss (TCL) to transfer local structure, utilize higher-order information, and mitigate misalignment of the heterogeneous outputs of the teacher and student networks. In one embodiment, the discriminative ability of features can be enhanced through soft supervision of the teacher network via hard triplet sampling in the student embedding. In one embodiment, using the disclosed loss L, the classification loss can be eliminated to utilize more discriminative features. In one embodiment, in addition to TCL, mutual learning can also be employed to regularize both the teacher network training and the student network training.

[0044] The technical effects of the disclosed method and system may include at least the following effects. In some embodiments, feature discriminability can be achieved by utilizing TCL instead of trying to preserve the distances between instances from the teacher network. In one embodiment, instead of directly comparing the distances between anchor positive and anchor negative pairs in the triple loss, the probability of these distances can be measured. In one embodiment, unlike other losses that match the global information between the teacher network and the student network, TCL focuses on the local structure including the nearest neighbors for each example. In one embodiment, combined with hard triple sampling in the student embedding, the discriminative ability of features can be enhanced through the soft supervision of the teacher network. In one embodiment, TCL alleviates the misalignment of the heterogeneous outputs of the teacher network and the student network. In one embodiment, TCL also encodes higher-order (e.g., third-order) structured knowledge in the triples and introduces richer similarity comparison information than vanilla triples in the context of knowledge distillation. In one embodiment, TCL is complementary to the conventional KD loss and can be combined with them to improve performance. In one embodiment, TCL greatly improves the performance of I2V Re-ID even without using cross entropy loss. In one embodiment, in addition to the proposed TCL, mutual learning is also employed to regularize both teacher learning and student learning. In one embodiment, by integrating these two techniques, the disclosed method can be referred to as Mutual Discriminative Knowledge Transfer (MDKT) for I2V Re-ID.

[0045] Figure 1 Illustrated is an exemplary system 100 for image-to-video re-identification in accordance with various embodiments. Figure 1 The operations shown and presented below are intended to be illustrative. Figure 1 As shown, the exemplary system 100 may include at least one computing system 102, which includes one or more processors 104 and one or more memories 106. The one or more memories 106 may include one or more non-transitory computer-readable storage media. The one or more memories 106 may store instructions that, when executed by the one or more processors 104, cause the one or more processors 104 to perform the various operations described herein. The system 102 may be implemented on or as various devices such as mobile phones, tablets, servers, computers, wearable devices (smart watches), etc. The above-mentioned system 102 may be installed with appropriate software (e.g., platform programs, etc.) and / or hardware (e.g., wires, wireless connections, etc.) to access other devices of the system 100.

[0046] System 100 may include one or more data stores (e.g., data store 108) and one or more computing devices (e.g., computing device 109) accessible to system 102. In some embodiments, system 102 may be configured to obtain data 122 (e.g., images, videos) from data store 108 (e.g., a database or dataset of images or videos) and / or computing device 109 (e.g., a computer, server, mobile phone, or camera configured to capture images or videos). System 102 may use the obtained data 122 to train a model for image-to-video re-identification.

[0047] One or more data stores, one or more computing devices, and / or system 102 may be configured to store videos. For example, a gallery of videos of objects may be stored for use in identifying matches to a query image of the object.

[0048] System 100 may also include one or more computing devices (e.g., computing devices 110 and 111) coupled to system 102. Computing devices 110 and 111 may include devices such as mobile phones, tablets, in-vehicle computers, wearable devices (smart watches), etc. Computing devices 110 and 111 may transmit signals (e.g., data signals) to system 102 or receive signals (e.g., data signals) from system 102. For example, computing device 110 may transmit a query signal including a query image of an object to system 102. In response, system 102 may identify one or more videos from a gallery of videos that match the object in the query image and send a return signal 128 to computing device 110. A match may mean that one or more videos contain the object in the query image. Return signal 128 may include one or more videos or an identification of one or more videos.

[0049] In some embodiments, the system 102, one or more data stores (e.g., data store 108), and one or more computing devices (e.g., computing device 109) can be integrated into a single device or system. Alternatively, the system 102, one or more data stores, and one or more computing devices can operate as separate devices. The data store can be accessed anywhere by the system 102, for example, in one or more memories 106, in the computing device 109, in another device coupled to the system 102 (e.g., a network storage device), or in another storage location (e.g., a cloud-based storage system, a network file system, etc.). Although the system 102 and the computing device 109 are shown as a single component in this figure, it should be understood that the system 102 and the computing device 109 can be implemented as a single device or as multiple devices coupled together. The system 102 can be implemented as a single system or as multiple systems coupled to each other. Generally speaking, system 102 , computing device 109 , data storage 108 , and computing devices 110 and 111 may be capable of communicating with one another via one or more wired or wireless networks (eg, the Internet) over which data can be transferred.

[0050] Figure 2A An exemplary model 200 for image-to-video re-identification according to various embodiments of the present disclosure is illustrated. The structure and operation of the model 200 presented below are intended to be illustrative. Depending on the implementation, the exemplary operation of the model 200 may include additional, fewer, or alternative steps performed in various orders or in parallel. The following operations of the model 200 may be implemented in various environments, including, for example, by Figure 1 The operations of model 200 may be implemented by one or more components of system 102, computing device 110, computing device 111, etc.

[0051] In some embodiments, the framework of model 200 can be used in the MDKT method for V2V Re-ID and can include two stages: (i) training a teacher network using the V2V Re-ID setting, and (ii) providing frames representing different views of an object (e.g., image frames capturing the object from different perspectives) as input to the teacher network and the student network to perform view knowledge distillation using one or more distillation losses, such as Figure 2A shown.

[0052] In some embodiments, the teacher network 201 and the student network 202 may each include a neural network such as ResNet-34, ResNet-50, ResNet-101, MobileNet-v2, etc. Table 2 below provides different performances of different combinations of the teacher network and the student network. For simplicity, the description herein may follow an example in which the teacher network 201 and the student network 202 each include ResNet-50 as the backbone network.

[0053] In some embodiments, the teacher network 201 can be a video-to-video re-identification network configured to identify matches between a query video and a gallery of videos of the teacher network; and the student network 202 can be an image-to-video re-identification network configured to identify matches between a query image and a gallery of videos of the student network.

[0054] In some embodiments, obtaining the model 200 may include: for a first training (in Figure 2A ), training a teacher network 201 based on at least using multiple videos as input training data for the teacher network (e.g., using a V2V Re-ID setting); and for a second training (in Figure 2A ), a teacher network 201 and a student network 202 are trained together based on at least the following: (i) for each of one or more objects, multiple images of the object in different views (e.g., different images of the object captured under different camera perspectives) are used as input training data for the teacher network, and (ii) for each of the one or more objects, a subset of the multiple images are used as input training data for the student network. During a first training period, the teacher network 201 can be trained without the student network 202. Parameters (e.g., weights) of the teacher network 201 can be updated during the first training period. Then, during a second training period, the parameters of the teacher network 201 can be fixed or not. That is, the parameters of the teacher network 201 may or may not be updated again during the second training period. The parameters of the student network 202 may be updated during the second training period. More details of the second training are provided below.

[0055] In some embodiments, the teacher network 201 can be initialized with pre-trained weights (e.g., weights pre-trained on ImageNet data during the first training). In one embodiment, both the last ReLU activation function and the last classification layer are eliminated in favor of BNNeck (batch normalization followed by a linear layer), and the stride of the last residual block is reduced from 2 to 1.

[0056] In some embodiments, obtaining the model 200 may include: extracting a plurality of teacher features from the input training data of the teacher network based on at least global average pooling; and extracting a plurality of student features from the input training data of the student network based on at least global average pooling. In some embodiments, given P video clips of an object (e.g., a person) Each n Contains T frames. T can be a natural number. For simplicity, unless otherwise specified, T is set to 8. The teacher network 201 (e.g., ResNet-50) can be used to extract features of these frames in each video. In some embodiments, as for feature aggregation, different methods can be used to fuse variable length inputs into a single input. In one example, the video representation f for the teacher network is t (V n ) can be calculated using global average pooling (GAP in step 203), spatial average pooling (SAP), temporal average pooling (TAP), etc. The same applies to the student network in step 204.

[0057] In some embodiments, triplet loss may be used (and without using the cross entropy classification loss during training of the teacher network outside the distillation process) to train the teacher network 201. The cross entropy classification loss (or simply cross entropy loss) can be Formulated as:

[0058]

[0059] where the subscript t stands for “teacher” and refers to the “teacher network”, and denote the one-hot label and the softmax output, respectively. Cross-entropy loss can be eliminated when training the teacher network outside of the distillation process. In other words, the disclosed MDKT can better learn discriminative features without using classification loss. The same applies to the student network.

[0060] In some embodiments, obtaining the model 200 may include: at least a triplet loss based on teacher features To train the teacher network. Through this training, the teacher network can learn the discriminative features among people so that it can determine whether two image frames contain the same person. In some embodiments, in a series of triples {x a ,x p ,x n}Train triplet loss where x a and x p are videos from the same person, denoted as anchor samples and positive samples (subscript a stands for “anchor” and subscript p stands for “positive”), and x n from different people and are denoted as negative samples (subscript n stands for “negative”). Triplet loss is a loss function used in machine learning algorithms where a baseline (anchor) input is compared to positive (true) inputs and negative (false) inputs. The anchor sample can be understood as the sample being processed, and training based on triplet loss can make x a x n Closer to x p That is, the distance from the baseline (anchor) input to the positive (true) input is minimized, and the distance from the baseline (anchor) input to the negative (false) input is maximized. In some embodiments, the triplet loss can be defined as:

[0061]

[0062] where [z] + =max(z,0), and f t (x a ),f t (x p ),f t (x n ) denotes the features of the three videos from the teacher network. α is the margin implemented between positive and negative pairs.

[0063] In some embodiments, obtaining model 200 may include: selecting a portion of the teacher features and a portion of the student features by using in-batch hard sample mining; and using the selected portion of the teacher features and the selected portion of the student features as input to a modified triplet loss. By using in-batch hard samples, the selected portion for training can correspond to samples that are close cases and difficult to distinguish (e.g., images of siblings). This can help improve the accuracy of the student network. In some embodiments, a batch hard strategy is adopted, and there are P identities in each batch, where each identity has K samples. In one embodiment, in equation (1), N is equal to P×K.

[0064] In some embodiments, for the second training, the teacher network is configured to transfer temporal information to the student network through knowledge distillation. For example, through training, the student network can add temporal information to the input query image, enabling the query image to be compared with a gallery of videos (each of which can be considered as multiple temporally related images). In some embodiments, after training the teacher network 201, the knowledge located in multiple views can be transferred to the student network 202 in a teacher-student distillation manner. The distillation of both temporal information and multi-view information can simulate the process of going from multiple inputs to a single or fewer inputs, corresponding to V2V to I2V.

[0065] In some embodiments, for MDKT, it may be mandatory for students to network f s The feature output of (·) and the teacher network f t (·) feature output matching. For example, in order to distill multi-view information from the teacher network to the student network, the teacher network may be allowed to access frames I from different viewpoints. t =(i1,i2,…,i N ), and provide the student network with a subset of the teacher’s input I s =(i1,i2,…,i M ), where the cardinality N>M (e.g., N=8 and M=2). t Randomly select a subset I from s . You can t Evenly sample I s frames in the dataset without replacement. This asymmetry between the teacher and student networks can lead to a self-distillation objective, where the student network is able to achieve better solutions despite inheriting the same architecture. s The training data used as input to the student network can (i) help simulate I2V re-ID (because the query image is a subset of multiple frames of the image matching the video) and (2) utilize information from multiple views of the object to train the model. Therefore, the accuracy of the model 200 can be improved.

[0066] In some embodiments, to transfer knowledge using multiple view frames, MDKT may be formulated as an optimization problem using the following three-level distillation loss.

[0067] (i) Mutual Logarithmic Distillation. In some embodiments, obtaining the model 200 may include at least one step based on the logarithmic distillation loss (L KD ) to train the teacher network and the student network. Logarithmic distillation loss (L KD ) may include a teacher-to-student log distillation loss based at least on teacher features and student features

[0068] In some embodiments, obtaining the model 200 may include generating a model based on at least the reciprocal logarithmic distillation loss (L MKD ) to train the teacher network and the student network. Mutual logarithmic distillation loss (L MKD ) may include (i) a student-to-teacher log distillation loss based at least on teacher features and student features and (ii) a teacher-to-student logarithmic distillation loss based on at least the teacher features and the student features Here the teacher features and student features may have been subjected to the softmax function.

[0069] In some embodiments, a logit-based knowledge distillation loss is employed to distill multi-view knowledge from the teacher to the student:

[0070]

[0071] Where the subscript s stands for “student” and denotes the model of the student network; KL denotes the Kullback-Leibler divergence; and is the output distribution of the teacher network and the student network; τ1 is the non-negative temperature (the larger the value of τ1, the smoother the output). is the first-order distillation loss and forces the student network to keep its predictions consistent with the teacher network.

[0072] In some embodiments, the parameters of the teacher network may not be frozen during the training based on (mutual) logit distillation. That is, the teacher network may be further trained to adjust its parameters based on (mutual) logit distillation.

[0073] In some embodiments, mutual learning techniques are employed to regularize both the teacher network and the student network. For example, another loss from the student-to-teacher logit distillation is added as follows:

[0074]

[0075] Therefore, in some embodiments, the final mutual logarithmic distillation loss L MKD 205 can be the sum of the teacher-to-student log distillation loss and the student-to-teacher log distillation loss:

[0076]

[0077] (ii) Pairwise distances in embedding. In some embodiments, obtaining the model 200 may include at least one pairwise distance loss (L PD ) to train the student network. Pairwise distance loss (L PD) may include the difference between: (i) one or more teacher feature distances among at least a portion (e.g., a small batch) of teacher features, each of the teacher feature distances indicating a distance between two different inputs to the teacher network; and (ii) one or more student feature distances among at least a portion of the student features, each of the student feature distances indicating a distance between two different inputs to the student network.

[0078] In some embodiments, in addition to logit distillation, an embedding-based knowledge distillation loss L can also be used. PD 206:

[0079]

[0080] Among them D t [i,j]=D(f t (I t [i]),f t (I t [j])), indicating the distance induced by the teacher network between the i-th input and the j-th input (similarly, the notation D s [i,j] also applies to the student network); B represents the batch size. L PD The loss is a second-order loss and encourages the student network to mirror the pairwise distances spanned by the teacher network.

[0081] (iii) Triplet contrast loss for discriminative transfer. In some embodiments, the modified triplet loss may include a distillation loss L TCL And the distillation loss L TCL Can include teachers to students

[0082] Distillation loss

[0083] In some embodiments, the modified triplet loss may include a mutual distillation loss L MTCL , and the mutual distillation loss L MTCL Can include student to teacher distillation loss and teacher-to-student distillation losses

[0084] In some embodiments, each of the student-to-teacher distillation loss and the teacher-to-student distillation loss may be based on at least a first sample distribution (e.g., ) and a second sample distribution (e.g., ). The first sample distribution can be based on at least the division between: (i) a first function of the distance between the anchor samples of the teacher feature and the positive samples of the teacher feature (e.g., ), and (ii) the first function and a second function of the distance between the anchor sample of the teacher feature and the negative sample of the teacher feature (e.g., The second sample distribution may be based on at least a division between: (i) a third function of the distance between the anchor sample of the student feature and the positive sample of the student feature (e.g., ), and (ii) the third function and the fourth function of the distance between the anchor sample of the student feature and the negative sample of the student feature (e.g., Further details are described below with reference to equations (7) and (8).

[0085] In some embodiments, the two distillation losses mentioned above may focus on addressing representation learning (global matching) but neglect the transfer of discriminative power (local structure). For Re-ID tasks, discriminative feature learning may be more important because the labels between the training and test sets are different. To address this in the context of I2V Re-ID, a third-order distillation loss, namely triplet contrastive loss (TCL), can be used.

[0086] In some embodiments, for the vanilla triplet loss (Equation 2), the distance between the anchor positive and anchor negative pairs can be used for discriminative feature learning. However, in knowledge distillation, since the outputs of the teacher network and the student network are heterogeneous, the absolute distances between these pairs are not well aligned. In some embodiments, to mitigate the misalignment, the probability of the two distances can be measured. In one example, let And make And the probability is defined as the division:

[0087]

[0088] Among them, P apn Measures how much closer the anchor sample is to the positive sample than the negative sample. If the anchor sample is closer to the positive sample than the negative sample, then P apn large, otherwise it is small. Compared with the vanilla triplet loss of cumulative absolute difference, P apn Using it in the context of knowledge distillation for discriminative transfer in I2V Re-ID leads to higher-order (e.g., third-order) similarity comparisons, thereby encoding structural information in triples. In some embodiments, to transfer local structure, intra-batch hard sample mining can be employed to make the local structure include the nearest neighbors for each example.

[0089] In some embodiments, similar to logit-based distillation, a parameter τ2 can be used to soften the probability:

[0090]

[0091] Leverage resources from the teacher network and from the student network You can define the distribution and Therefore, the TCL corresponding to the distillation of the teacher network to the student network can be formulated as:

[0092]

[0093] Where t2s represents the distance from the teaching network to the student network.

[0094] In some embodiments, similar to mutual logit distillation, matching between the student network and the teacher network can be employed for TCL:

[0095]

[0096] Where s2t represents the student network to the teacher network.

[0097] In some embodiments, the final mutual TCLL for the final optimization can be MTCL 207 is defined as:

[0098]

[0099] In some embodiments, the disclosed MTCL (e.g., a modified triplet loss based on the above-mentioned division) can achieve knowledge distillation between the two networks described herein, because the regular triplet loss is applicable to a single network and cannot be used for knowledge distillation between two networks. In some embodiments, the disclosed MTCL can also provide at least the following technical effects: (i) using triplet loss and batch hard mining for discriminative learning, TCL gives the ability to learn discriminative features in knowledge distillation, so that the trained model can distinguish different objects even if they have a high degree of similarity; (ii) introducing higher-order relationships between samples through division and utilizing structured relationships in triplets; (iii) aligning (e.g., through division) heterogeneous outputs and having the ability to be used in different teacher and student network architectures (that is, TCL allows the teacher network and the student network to be different from each other, for example, one is ResNet-50 and the other is ResNet-101). For example, division can represent relative distances rather than absolute distances, and this enables cross-distillation in addition to self-distillation.

[0100] In some embodiments, in addition to the third-stage distillation loss (L MKD 205, L PD 206, L MTCL 207), triple loss L TR 208 can be used for both the teacher network and the student network and can be formulated as:

[0101]

[0102] in It can be similar to the above description of equation (2) where s replaces t.

[0103] In some embodiments, obtaining the model 200 may include minimizing a total loss L comprising: a triplet loss corresponding to the teacher network, a triplet loss corresponding to the student network, a logarithmic distillation loss, a pairwise distance loss, and a modified triplet loss. In some embodiments, the final objective function may be formulated as a combination of four losses:

[0104] L=L TR +αL MKD +βL PD +γL MTCL (13)

[0105] Where α, β, and γ can be various parameters such as constants. In some embodiments, for triple loss and MTCL, within-batch hard sample mining can be used; for MTCL, hard triples can be generated based on student embeddings. Training model 200 can be based at least on minimizing loss L by adjusting the parameters of model 200.

[0106] Figure 2B An exemplary method 250 for applying a model for image-to-video re-identification in accordance with various embodiments is illustrated. The operations of the method 250 presented below are intended to be illustrative. Depending on the implementation, the exemplary operations of the method 250 may include additional, fewer, or alternative steps performed in various orders or in parallel. The operations of the method 250 below may be implemented in various environments, including, for example, by Figure 1 The operations of method 250 may be implemented by one or more components of system 102, computing device 110, computing device 111, etc.

[0107] In some embodiments, the method for image-to-video re-identification may include: obtaining a triplet loss according to the modified triplet loss (e.g., the above or L MTCL207) A model trained at least based on knowledge distillation between a teacher network and a student network; obtaining an image 251 (e.g., a query image containing an object such as a person or a vehicle) and a plurality of videos 252 (e.g., a gallery of videos, each of which may or may not contain an object); providing the image (which may be provided as an image or as a video comprising a frame) and the plurality of videos to a model (e.g., the student network 202 after being trained or the teacher network 201 after being trained) to obtain one or more first features 253 of the image and one or more second features 254 of each of the plurality of videos (e.g., by spatial average pooling (SAP) and temporal average pooling (TAP)); and determining one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features (e.g., by distance calculation and retrieval). The training of the teacher network 201 and the student network 202 can refer to Figure 2A and other related descriptions herein. For example, by at least L MTCL 207 or the total loss L is minimized to train the teacher network 201 and the student network 202.

[0108] In some embodiments, when applying the trained model 200, each query may be a still image, and one or more matches (if any) to the query are identified from a gallery set comprising multiple video clips. Figure 2B As shown, the network (e.g., the teacher network 201 or the student network 202 after being trained) can be used for both image feature extraction and video feature extraction. In one embodiment, a still image can be considered as a video (e.g., a video of one frame). After feature extraction, the distance between the query feature of the query image and each gallery video feature of the gallery video can be determined, and I2V retrieval can be performed based on these distances. Further details of the dataset, implementation, and results are described below.

[0109] In some embodiments, the disclosed methods and systems can be tested on various datasets including MARS, DukeMTMC-VideoReID (Duke-video), and VeRi-776 (VeRi) benchmarks. MARS is a person Re-ID dataset and includes 19,680 short tracks (tracklets) from 6 different cameras. It captures 1,261 different identities, of which 625 are used for the training set, 635 for the gallery, and 626 for the query. The average length of a person video is 59 frames. MARS is automatically annotated and may include errors and false detections. Duke-video is also a person Re-ID dataset and includes 5,534 video short tracks from 8 different cameras, where each short tracklet has an average of 167 frames. VeRi is a vehicle Re-ID dataset, which is captured from a 1.0 km 2 VeRi is collected from 20 fixed cameras of vehicles moving on a circular road in a region. VeRi contains 18,397 short tracks, each with an average of 6 frames. There are 775 identities, 576 of which are used for training and 200 for testing. The query set shares the same identities as the gallery. However, unlike the other two datasets, VeRi's query set may only include a single image for each (ID, camera) pair.

[0110] In some embodiments, determining one or more videos from a plurality of videos that match an image based on one or more first features and one or more second features may include: determining, for each video from the plurality of videos, a distance between the one or more first features and the one or more second features (e.g., by distance calculation and retrieval); ranking the plurality of videos according to the distance starting from the lowest distance (e.g., ranking a gallery of videos 252 in descending order of degree of matching with the query image 251); and determining, from the plurality of videos, one or more highest-ranked videos (e.g., top 1, top 5) or one or more videos below a threshold distance as one or more matches to the image.

[0111] In some embodiments, the training data for the student network and the training data for the teacher network may or may not include the image 251, and the training data for the student network and the training data for the teacher network may or may not include any of the plurality of videos 252. In some embodiments, the training data for the student network and the training data for the teacher network may not include all of the plurality of videos 252, which makes the discriminative ability of the features more important.

[0112] In some embodiments, to process the dataset, the first frame of each short trajectory can be extracted when testing I2VRe-ID on the MARS and Duke-video datasets. Since the query set only includes a single image in Veri, it can be evaluated in the I2V setting. In some embodiments, cumulative matching characteristics (CMC) and mean average precision (mAP) can be used to evaluate performance. In some embodiments, ResNet-50 can be used for both the teacher network and the student network. Pre-trained models on ImageNet can be used for initialization of the teacher network and / or the student network. The following hyperparameters can be used to test MARS and Duke-video: (i) each batch contains P = 8 identities, where each identity has K = 4 samples; and (ii) each sample includes 8 images equally spaced in the short trajectory. For VeRi, P can be increased to 18, and a single image can be used at a time. All teacher networks can be trained for 300 epochs using the Adam optimizer, where the learning rate is initialized to 10 -4 and decreases 10 times every 100 epochs. During the distillation phase, N=8 images may be provided to the teacher network, and M=2 images (randomly picked) may be provided to the student network. The student network may be trained for 600 epochs, and the learning rate decay may be stepped between 300 and 450. In some embodiments, τ1=10 (Equation 3), τ2=4 (Equation 8), α=10 -1 (Equation 13), β = 10 -4 (Equation 13), and γ = 1000 (Equation 13). In some embodiments, to improve generalization, data expansion can be applied. The teacher network can be set to training mode during distillation, that is, batch normalization statistics can be calculated on a batch basis.

[0113] In some embodiments, MDKT can be applied in self-distillation for I2V Re-ID. The teacher network can be indicated by the name of the backbone (e.g., ResNet-50) and its student (e.g., ResMDKT-50) can be appended with "VKD" (view knowledge distillation) and "MDKT". By mutual training of both the teacher network and the student network, ResNet-50, ResNet-34, and MobileNet-v2 can be used for mutual knowledge transfer. The teacher network (ResNet-101, ResNet-50bam, and DenseNet-121) can be frozen, and only the student network can be trained to verify the effectiveness of discriminative knowledge transfer without mutual training. Table 1 illustrates the comparison of different backbones across different datasets. Table 1 shows that in the vast majority of settings, the student network works better than the teacher network. Compared to VKD, MDKT achieves better results in the I2V setting, especially in the I2V setting. For example, on the MARS dataset, MDKT wins by 1.03% on average for cmc1 across different backbones. The same applies to the I2V setting on VeRi. Therefore, MDKT can achieve better results than VKD regardless of using mutual training or not, which reflects the effectiveness of the disclosed TCL for better feature transfer.

[0114] Table 1. Self-distillation results across datasets, settings, and architectures

[0115]

[0116] In some embodiments, MDKT can be applied in cross-distillation for I2V Re-ID. The disclosed TCL can exploit high-order structured relationships in triples, thereby alleviating the requirement for the same architecture for distillation. Therefore, MDKT allows knowledge to be transferred from a complex architecture to a simpler architecture. Similar to model compression, network complexity can be reduced while enjoying the benefits already obtained through self-distillation. Table 2 shows the cross-distillation results for various combinations of teacher networks and student networks across different datasets. Table 2 shows that with a stronger teacher network, better performance results (e.g., Re-ID accuracy) can be achieved by the student network.

[0117] Table 2. Cross-distillation results across datasets, settings, and architectures.

[0118]

[0119]

[0120] In some embodiments, a thorough ablation study of the last loss term (Equation 13) can be performed on the MARS and Duke-video datasets using ResNet-50, and the results are shown in Table 3. Table 3 shows that when fewer views are provided to the student network and only triplet loss is used for training (TR model), the results are very poor. One possible reason is that with fewer view inputs, the in-batch hard sample mining may not pick up rich triplets for training. The TCL model shows that the performance is greatly improved compared to the TR model using only mutual triplet contrast loss.

[0121] Table 3. Ablation study of the impact of the loss term on MARS and Duke-video datasets using ResNet-50.

[0122]

[0123] In some embodiments, the three-level distillation loss can play an important role in the performance gain (KD+PD+TCL model). As shown, the distillation loss plus the triplet loss can achieve better results for the I2V Re-ID setting. Nevertheless, for the V2V Re-ID setting on the Duke-video dataset, mutual logit distillation can negatively affect the performance, and these results support the importance of higher-order distillation losses.

[0124] In some embodiments, mutual learning can be employed as described to regularize both teacher network training and student network training. To investigate the role of mutual training, three models can be designed, namely, freezing teacher parameters, without mutual training (no student-to-teacher loss term) in Equations 5 and 11, and with mutual learning. The results are illustrated in Table 4. Table 4 shows that mutual learning can improve performance for both I2V and V2V settings. Combining Tables 1 and 4, it is shown that the main improvement may come from the proposed TCL for discriminative feature transfer, and that mutual learning can make the training better.

[0125] Table 4. Ablation study of the impact of mutual learning on MARS and Duke-video datasets using RestNet-50.

[0126]

[0127] In some embodiments, the cross-entropy classification loss may not be used in the final objective function (Equation 13). This may be because, by introducing TCL, together with the other three loss terms, the network is able to learn discriminative features for Re-ID. If the cross-entropy classification loss is added, the final performance may be negatively affected, as shown in Table 5.

[0128] Table 5. Ablation study of the effect of cross-entropy loss on MARS and Duke video datasets using ResNet50.

[0129]

[0130] In some embodiments, after being trained, the teacher network 201 or the student network 202 can be used in Figure 2B In various applications shown in FIG. 6 , in some embodiments, the student network may perform slightly better than the teacher network, as shown in the comparison results in Table 6. Therefore, in some embodiments, the student network 202 is used to identify matches to the query image from the video gallery.

[0131] Table 6. Ablation study of the impact of using ResNet50 on teacher and student on MARS and Duke-video datasets.

[0132]

[0133]

[0134] Figure 3 FIGURE 1 illustrates a visual comparison of models for image to video re-identification according to various embodiments. The operations presented below are intended to be illustrative. In some embodiments, feature maps of VKD and MDKT can be compared. Figure 3 As shown in Figure 3, the image representations learned by MDKT may focus on more foregrounds and accessories compared to VKD, which reflects that MDKT tends to learn more discriminative features.

[0135] In some embodiments, MDKT may outperform other methods such as VKD in terms of correctly identifying a match as the top 1 result returned from the model, correctly identifying a match within the top 5 results returned from the model, match identification accuracy, etc.

[0136] In various embodiments, the Re-ID task requires discriminative feature learning capabilities to overcome non-overlapping labels between the training and test sets. As discussed above, image-to-video Re-ID can be formulated as a teacher-student knowledge distillation approach, and knowledge distillation can be achieved using mutual discriminative knowledge transfer methods. The disclosed systems and methods can utilize triples for local discriminative feature learning and align the heterogeneous outputs of the teacher and student networks. The disclosed systems and methods can provide third-order structural relational knowledge for transfer. Combined with mutual learning, the disclosed systems and methods can achieve I2V Re-ID of objects such as people and cars.

[0137] Figure 4Flowchart illustrating an exemplary method 410 for image to video re-identification according to various embodiments of the present disclosure. The method 410 may be implemented in various environments, including, for example, by Figure 1 4. The exemplary method 410 may be implemented by one or more components of the system 102, computing device 110, computing device 111, etc. For example, one or more non-transitory computer-readable storage media (e.g., one or more memories 106) may store instructions that, when executed by one or more processors (e.g., processor 104), cause the system 102 (e.g., processor 104) to perform the method 410. As another example, the computing device 110 may include one or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the computing device 110 to perform the method 410. The operations of the method 410 presented below are intended to be illustrative. Depending on the implementation, the exemplary method 410 may include additional, fewer, or alternative steps that are performed in various orders or in parallel.

[0138] Block 412 includes obtaining a model trained according to a modified triplet loss based at least on knowledge distillation between a teacher network and a student network. In some embodiments, a model trained according to a modified triplet loss based on division may be obtained. For example, a model may have been trained according to a modified triplet loss (e.g., the above or L MTCL 207) trained a model based at least on knowledge distillation between a teacher network and a student network. In some embodiments, method 410 may be performed by computing device 110. Obtaining the model may include receiving, retrieving, referencing, or invoking a trained model directly from a computing system (e.g., system 102) without performing model training (model training may have been performed by the computing device and / or some other computing device). In some embodiments, method 410 may be performed by computing system 102. Obtaining the model may include training the model. Various training steps are described herein.

[0139] In some embodiments, the teacher network is a video-to-video re-identification network configured to identify matches between a query video and a gallery of videos of the teacher network; and the student network is an image-to-video re-identification network configured to identify matches between a query image and a gallery of videos of the student network.

[0140] In some embodiments, obtaining the model includes: for a first training, training a teacher network based on at least using multiple videos as input training data for the teacher network; and for a second training, training the teacher network and the student network together based on at least the following: (i) for each of the one or more objects, using multiple images of the object in different views as input training data for the teacher network, and (ii) for each of the one or more objects, using a subset of the multiple images as input training data for the student network.

[0141] In some embodiments, for the second training, the teacher network is configured to transfer temporal information to the student network through knowledge distillation.

[0142] In some embodiments, knowledge distillation includes self-distillation or cross-distillation.

[0143] In some embodiments, obtaining the model includes: extracting a plurality of teacher features from input training data of the teacher network based on at least global average pooling; and extracting a plurality of student features from input training data of the student network based on at least global average pooling.

[0144] In some embodiments, obtaining the model further comprises: at least a triplet loss based on the teacher features to train the teacher network.

[0145] In some embodiments, training the teacher network includes training the teacher network without using a cross-entropy classification loss.

[0146] In some embodiments, obtaining the model further comprises: PD ) to train the student network; and the pairwise distance loss (L PD ) includes the difference between: (i) one or more teacher feature distances among at least a portion of the teacher features, each of the teacher feature distances indicating a distance between two different inputs of the teacher network, and (ii) one or more student feature distances among at least a portion of the student features, each of the student feature distances indicating a distance between two different inputs of the student network.

[0147] In some embodiments, obtaining the model further comprises: MKD ) to train the teacher network and the student network; and the mutual logarithmic distillation loss (L MKD ) includes (i) a student-to-teacher log distillation loss based on at least teacher features and student features and (ii) a teacher-to-student log distillation loss based on at least teacher features and student features.

[0148] In some embodiments, obtaining the model further comprises: selecting a portion of the teacher features and a portion of the student features by using intra-batch difficult sample mining; and using the selected portion of the teacher features and the selected portion of the student features as inputs to the modified triplet loss.

[0149] In some embodiments, the modified triplet loss includes a mutual third-order distillation loss; and the third-order mutual distillation loss includes a student-to-teacher third-order distillation loss and a teacher-to-student third-order distillation loss.

[0150] In some embodiments, the modified triplet loss is based on at least a division. In some embodiments, each of the student-to-teacher third-order distillation loss and the teacher-to-student third-order distillation loss is based on at least a first sample distribution and a second sample distribution; the first sample distribution is based on at least a division between: (i) a first function of the distance between an anchor sample of the teacher feature and a positive sample of the teacher feature, and (ii) a sum of the first function and a second function of the distance between the anchor sample of the teacher feature and the negative sample of the teacher feature; and the second sample distribution is based on at least a division between: (i) a third function of the distance between the anchor sample of the student feature and the positive sample of the student feature, and (ii) a sum of the third function and a fourth function of the distance between the anchor sample of the student feature and the negative sample of the student feature.

[0151] In some embodiments, obtaining the model includes minimizing a total loss (L) including: a triplet loss corresponding to the teacher network, a triplet loss corresponding to the student network, a logarithmic distillation loss, a pairwise distance loss, and a modified triplet loss. Further details can be found in Figure 2A and other descriptions above.

[0152] In some embodiments, the training data for the student network and the training data for the teacher network do not include images; and the training data for the student network and the training data for the teacher network do not include any of the plurality of videos.

[0153] Block 414 includes obtaining an image and a plurality of videos. In some embodiments, the trained model may include a trained teacher network and / or a trained student network. Details of the network training may be described above (e.g., by minimizing the total loss (L)).

[0154] Block 416 includes providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos. In some embodiments, providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos includes providing the image and the plurality of videos to a trained teacher network or a trained student network to obtain one or more first features of the image and one or more second features of each of the plurality of videos. For further details, see Figure 2B and other descriptions above.

[0155] Block 418 includes determining one or more videos from the plurality of videos that match the image based on the one or more first features and the one or more second features. In some embodiments, determining one or more videos from the plurality of videos that match the image based on the one or more first features and the one or more second features includes: for each video from the plurality of videos, determining a distance between the one or more first features and the one or more second features; ranking the plurality of videos based on the distance starting from the lowest distance; and determining one or more highest ranked videos from the plurality of videos or one or more videos below a threshold distance as one or more matches to the image. Further details may be found in Figure 2B and other descriptions above.

[0156] Figure 5 The diagram illustrates a block diagram of an exemplary computer system 510 for image-to-video re-identification in accordance with various embodiments. System 510 may be an exemplary implementation of system 102, computing device 110, computing device 111, or one or more similar devices. Method 410 may be implemented by computer system 510. Computer system 510 may include: one or more processors; and one or more non-transitory computer-readable storage media (e.g., one or more memories), the one or more non-transitory computer-readable storage media coupled to the one or more processors and configured with instructions that are executable by the one or more processors to cause the system or device (e.g., processor) to perform method 410. Computer system 510 may include various units / modules corresponding to instructions (e.g., software instructions). In some embodiments, the instructions may correspond to software such as desktop software or an application (APP) installed on a mobile phone, pad, etc.

[0157] In some embodiments, the computer system 510 may include: a first obtaining module 512, which is configured to obtain a model trained based on at least the knowledge distillation between the teacher network and the student network according to the modified triplet loss; a second obtaining module 514, which is configured to obtain an image and a plurality of videos; a providing module 516, which is configured to provide the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and a determining module 518, which is configured to determine one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features.

[0158] Figure 6 6 is a block diagram illustrating a computer system 600 on which any of the embodiments described herein may be implemented. System 600 may correspond to system 102 or computing devices 109, 110, or 111 described above. Computer system 600 includes a bus 602 or another communication mechanism for communicating information, and one or more hardware processors 604 coupled to bus 602 for processing information. Hardware processor 604 may be, for example, one or more general-purpose microprocessors.

[0159] Computer system 600 also includes a main memory 606, such as random access memory (RAM), cache, and / or other dynamic storage device, coupled to bus 602 for storing information and instructions to be executed by processor 604. Main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. Such instructions, when stored in a storage medium accessible to processor 604, render computer system 600 a special-purpose machine customized to perform the operations specified in the instructions. Computer system 600 also includes a read-only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic disk, optical disk, or USB thumb drive (flash drive), is provided and coupled to bus 602 for storing information and instructions.

[0160] The computer system 600 can implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic in conjunction with the computer system to make or program the computer system 600 into a special-purpose machine. According to one embodiment, the techniques herein are performed by the computer system 600 in response to the processor 604 executing one or more sequences of one or more instructions contained in the main memory 606. Such instructions can be read into the main memory 606 from another storage medium, such as the storage device 610. Execution of the sequences of instructions contained in the main memory 606 causes the processor 604 to perform the process steps described herein. In alternative embodiments, hardwired circuitry can be used in place of or in combination with software instructions.

[0161] Main memory 606, ROM 608, and / or storage device 610 may include non-transitory storage media. As used herein, the term "non-transitory media" and similar terms refer to media that store data and / or instructions that cause a machine to operate in a specific manner. Media excludes temporary signals. Such non-transitory media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of non-transitory media may include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, and networked versions thereof.

[0162] Computer system 600 also includes a network interface 618 coupled to bus 602. Network interface 618 provides a bidirectional data communication coupling to one or more network links connected to one or more local networks. For example, network interface 618 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a corresponding type of telephone line. As another example, network interface 618 can be a local area network (LAN) card (or a WAN component communicating with a WAN) for providing a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, network interface 618 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0163] Computer system 600 can send messages and receive data, including program code, through the network, network link, and network interface 618. In the Internet example, a server might transmit the requested code for an application program through the Internet, an ISP, a local network, and network interface 618.

[0164] The received code may be executed by processor 604 as it is received and / or stored in storage device 610 or other non-volatile storage for later execution.

[0165] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in a code module executed by one or more computer systems or computer processors comprising computer hardware and fully or partially automated by them. These processes and algorithms may be implemented partially or entirely in dedicated circuits.

[0166] The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. In addition, certain method or process blocks can be omitted in some implementations. The methods and processes described herein are not limited to any particular order, and the boxes or states associated therewith can be performed in other appropriate orders. For example, the boxes or states described can be performed in an order other than the order specifically disclosed, or multiple boxes or states can be combined in a single box or state. The exemplary boxes or states can be performed serially, in parallel, or in some other manner. Boxes or states can be added to or removed from the disclosed exemplary embodiments. The exemplary systems and components described herein can be configured differently than described. For example, elements can be added to, removed from, or rearranged in comparison to the disclosed exemplary embodiments.

[0167] The various operations of the exemplary methods described herein may be performed, at least in part, by an algorithm. The algorithm may be included in a program code or instruction stored in a memory (e.g., the aforementioned non-transitory computer-readable storage medium). Such an algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly program the computer to perform a function, but is capable of learning from training data to make a predictive model for performing the function.

[0168] The various operations of the exemplary methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Such processors, whether temporarily or permanently configured, may constitute a processor-implemented engine that operates to perform one or more operations or functions described herein.

[0169] Similarly, the methods described herein may be at least partially processor-implemented, where one or more specific processors are examples of hardware. For example, at least some of the operations of the methods may be performed by one or more processors or processor-implemented engines. Furthermore, one or more processors may also be operable to support the execution of the related operations in a "cloud computing" environment or as "software as a service" (SaaS).

[0170] Any process descriptions, elements or blocks in the flowcharts described herein and / or depicted in the accompanying drawings should be understood to potentially represent modules, segments or portions of code, which include one or more executable instructions for implementing specific logical functions or steps in the process. Alternative implementations are included within the scope of the embodiments described herein, where, as will be understood by those skilled in the art, depending on the functionality involved, elements or functions may be deleted from the scope shown or discussed, executed out of order, including substantially simultaneously or in reverse order.

[0171] As used herein, the term "or" may be interpreted in an inclusive or exclusive sense. In addition, multiple instances may be provided for a resource, operation, or structure described herein as a single instance. Furthermore, the boundaries between the various resources, operations, engines, and data registers are somewhat arbitrary, and the particular operations are illustrated in the context of a particular illustrative configuration. Other allocations of functionality are contemplated and may fall within the scope of the various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the exemplary configurations may be implemented as combined structures or resources. Similarly, structures and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements fall within the scope of the embodiments of the present disclosure as represented by the appended claims. Accordingly, the specification and drawings are to be considered in an illustrative rather than a restrictive sense.

[0172] Although the overview of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure. Such embodiments of the subject matter may be referred to herein, individually or collectively, by the term "invention" merely for convenience, without intending to voluntarily limit the scope of this application to any single disclosure or concept if, in fact, more than one disclosure or concept is disclosed.

[0173] The embodiments illustrated herein are described in sufficient detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, the detailed description should not be taken in a limiting sense, and the scope of the various embodiments is limited only by the appended claims and the full scope of equivalents to which such claims are entitled.

Claims

1. One or more non-transitory computer-readable storage media storing instructions executable by one or more processors, wherein execution of the instructions causes the one or more processors to perform operations comprising: Obtaining a model trained based on at least knowledge distillation between a teacher network and a student network according to a modified triplet loss, wherein the modified triplet loss includes a distillation loss from the teacher network to the student network; Acquire images and multiple videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and One or more videos matching the image among the plurality of videos are determined based on the one or more first features and the one or more second features.

2. The one or more non-transitory computer-readable storage media of claim 1 , wherein: The teacher network is a video-to-video re-identification network configured to identify matches between a query video and a gallery of videos of the teacher network; and The student network is an image-to-video re-identification network configured to identify matches between a query image and a gallery of videos of the student network.

3. The one or more non-transitory computer-readable storage media of claim 1 , wherein obtaining the model comprises: For a first training, the teacher network is trained based on at least using a plurality of videos as input training data for the teacher network; as well as For the second training, the teacher network and the student network are trained together based on at least: (i) for each of the one or more objects, multiple images of the object in different views are used as input training data for the teacher network, and (ii) for each of the one or more objects, a subset of the multiple images is used as input training data for the student network.

4. The one or more non-transitory computer-readable storage media of claim 3, wherein: For the second training, the teacher network is configured to transfer temporal information to the student network through the knowledge distillation.

5. The one or more non-transitory computer-readable storage media of claim 4, wherein: The knowledge distillation includes self-distillation or cross-distillation.

6. The one or more non-transitory computer-readable storage media of claim 1 , wherein obtaining the model comprises: extracting a plurality of teacher features from input training data of the teacher network based at least on global average pooling; as well as A plurality of student features are extracted from input training data of the student network based at least on global average pooling.

7. The one or more non-transitory computer-readable storage media of claim 6, wherein obtaining the model further comprises: At least a triplet loss based on the teacher features to train the teacher network.

8. The one or more non-transitory computer-readable storage media of claim 6, wherein obtaining the model further comprises: At least based on the mutual logarithmic distillation loss (L MKD ) to train the teacher network and the student network; and The mutual logarithmic distillation loss (L MKD ) includes (i) a student-to-teacher log distillation loss based at least on the teacher features and the student features and (ii) a teacher-to-student log distillation loss based at least on the teacher features and the student features.

9. The one or more non-transitory computer-readable storage media of claim 6, wherein obtaining the model further comprises: selecting a portion of the teacher features and a portion of the student features by using within-batch difficult sample mining; as well as A selected portion of the teacher features and a selected portion of the student features are used as input to the modified triplet loss.

10. The one or more non-transitory computer-readable storage media of claim 6, wherein: The modified triplet loss The distillation loss of the student network to the teacher network is also included.

11. The one or more non-transitory computer-readable storage media of claim 1 , wherein obtaining the model comprises: Minimize the total loss (L) consisting of the triplet loss corresponding to the teacher network, the triplet loss corresponding to the student network, the logarithmic distillation loss, the pairwise distance loss, and the modified triplet loss.

12. The one or more non-transitory computer-readable storage media of claim 1 , wherein providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos comprises: The image and the plurality of videos are provided to a trained teacher network or a trained student network to obtain one or more first features of the image and one or more second features of each of the plurality of videos.

13. The one or more non-transitory computer-readable storage media of claim 1 , wherein determining the one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features comprises: For each video in the plurality of videos, determining a distance between the one or more first features and the one or more second features; sorting the plurality of videos according to the distance starting from the lowest distance; as well as One or more highest ranked videos or one or more videos below a threshold distance from the plurality of videos are determined as one or more matches to the image.

14. The one or more non-transitory computer-readable storage media of claim 1 , wherein: The training data of the student network and the training data of the teacher network do not include the image; and The training data of the student network and the training data of the teacher network do not include any of the plurality of videos.

15. A computer-implemented method comprising: Obtaining a model trained based on at least knowledge distillation between a teacher network and a student network according to a modified triplet loss, wherein the modified triplet loss includes a distillation loss from the teacher network to the student network; Acquire images and multiple videos; providing the image and the plurality of videos to the model to obtain one or more first features of the image and one or more second features of each of the plurality of videos; and One or more videos matching the image among the plurality of videos are determined based on the one or more first features and the one or more second features.

16. The method according to claim 15, wherein determining the one or more videos in the plurality of videos that match the image based on the one or more first features and the one or more second features comprises: For each video in the plurality of videos, determining a distance between the one or more first features and the one or more second features; ranking the plurality of videos according to the distances starting from a lowest distance; as well as One or more highest ranked videos or one or more videos below a threshold distance from the plurality of videos are determined as one or more matches to the image.

17. The method of claim 15, wherein: The teacher network is a video-to-video re-identification network configured to identify matches between a query video and a gallery of videos of the teacher network; and The student network is an image-to-video re-identification network configured to identify matches between a query image and a gallery of videos of the student network.

18. The method of claim 15, wherein obtaining the model comprises: For a first training, the teacher network is trained based on at least using a plurality of videos as input training data for the teacher network; as well as For the second training, the teacher network and the student network are trained together based on at least: (i) for each of the one or more objects, multiple images of the object in different views are used as input training data for the teacher network, and (ii) for each of the one or more objects, a subset of the multiple images is used as input training data for the student network.

19. The method of claim 15, wherein obtaining the model comprises: extracting a plurality of teacher features from input training data of the teacher network based at least on global average pooling; as well as A plurality of student features are extracted from input training data of the student network based at least on global average pooling.

Images