Target re-recognition device, target re-recognition method thereof and model learning method thereof

Abstract

The invention relates to a target re-recognition device, a target re-recognition method and a model learning method. A target re-recognition technique is used to track an image, wherein a target re-recognition device, a target re-recognition method, and a model learning method can improve target re-recognition performance by using a model trained based on spatial and temporal refinement features. A target re-recognition method includes extracting a first feature from an input video frame, and extracting a second feature by segmenting the first feature based on expert modules within a plurality of expert layers in a sequential relationship, the expert modules being a single expert module activated among the plurality of expert modules of a single expert layer.

Description

Technical Field

[0001] This disclosure relates to in-image target re-identification technology, and more specifically, to a target re-identification device, target re-identification method, and model learning method that utilizes a model trained with spatially and temporally refined features to improve the performance of target re-identification. Background Technology

[0002] Re-identification (re-ID) is a technique for detecting and tracking the same target (e.g., a person, vehicle, etc.) in videos captured by multiple different cameras. It aims to identify whether the target exists in videos captured by different cameras or by the same camera at different times.

[0003] To confirm the presence of the same target in multiple videos, it is crucial to utilize refined features.

[0004] Methods for exploring refined features include attention-based methods and part-level methods.

[0005] Traditional methods train model parameters based on all samples in the dataset. However, the refinement needed to distinguish very similar samples (e.g., a logo on a shirt) may only exist in a subset of the samples.

[0006] When updating model parameters based on all samples in the dataset, refinement components present in subsets of samples may be ignored.

[0007] Therefore, an effective method for learning refined components is needed, as refined components are a key factor in distinguishing samples.

[0008] The aforementioned related technologies are intended only to help understand the background of this disclosure and should not be construed as an endorsement of prior art known to those skilled in the art. Summary of the Invention

[0009] This disclosure proposes a target re-identification device, a corresponding target re-identification method, and a model learning method that can effectively learn detailed components (refined features), which are key factors in distinguishing samples.

[0010] This disclosure provides a target re-identification device, a corresponding target re-identification method, and a model learning method that can effectively learn both spatial refinement differences and temporal refinement differences.

[0011] This disclosure provides an expert-scalable object re-identification device, object re-identification method, and model learning method that can automatically incorporate new experts during training.

[0012] The technical objectives of this disclosure are not limited to those described above, and other objectives not described herein will be clearly understood by those skilled in the art from the following description.

[0013] To achieve the above objectives, the disclosed target re-identification device may include a memory and a processor, the memory being configured to store a target re-identification model and the processor being configured to execute the model.

[0014] According to an embodiment, the processor can extract a first feature from a video frame input, segment the first feature into a second feature based on expert modules within multiple expert layers in a sequential relationship, and classify the second feature in a target re-identification model, wherein the expert module is a single expert module activated among multiple expert modules within a single expert layer.

[0015] According to an embodiment, an expert module can be a single expert module generated during model implementation, or an expert model added during training.

[0016] According to an embodiment, the expert module may be an expert module generated during model implementation or an expert model added during training.

[0017] According to the embodiment, the expert module in the first expert layer can extract segmentation features based on the first feature, and the expert modules in the remaining expert layers can extract segmentation features based on the features output from the expert module of the previous expert layer.

[0018] According to an embodiment, among multiple expert modules within a single expert layer, the expert module can have the highest relevance to the input video.

[0019] According to an embodiment, the expert module can selectively extract spatial segmentation features or temporal segmentation features based on the spatiotemporal importance of the input features.

[0020] According to an embodiment, the expert module may include: an importance assessment module configured to output an importance vector based on input features; a branching module configured to output input features to a spatial feature channel or a temporal feature channel based on the importance vector; a spatial feature extraction module configured to extract spatial features segmented based on features input through the spatial feature channel; and a temporal feature extraction module configured to extract temporal features segmented based on features input through the temporal feature channel.

[0021] According to an embodiment, the importance evaluation module can use max pooling and fully connected layers to output an importance vector that includes the importance values ​​of the input features.

[0022] According to the embodiment, the branch module can generate a binary decision vector based on the importance vector, output the input features to the spatial feature channel when the binary decision vector is 1, and output the input features to the temporal feature channel when the binary decision vector is 0.

[0023] According to an embodiment, the processor may include a selector configured to activate one of the multiple expert modules within a single expert layer during training based on the relevance scores of the multiple expert modules within a single expert layer.

[0024] According to an embodiment, the processor may include a waiting expert module associated with a single expert layer.

[0025] According to an embodiment, the waiting expert module can be selectively included in a single expert layer during training.

[0026] The target re-identification method according to embodiments of the present disclosure may include: extracting a first feature from an input video frame; extracting a second feature by segmenting the first feature based on expert modules within a plurality of expert layers in a sequential relationship; and classifying the second feature in a target re-identification model.

[0027] According to an embodiment, an expert module can be a single expert module generated during model implementation, or an expert model added during training.

[0028] According to an embodiment, the extraction of the second feature may include: extracting segmented features by an expert module within a first expert layer based on the first feature; and extracting segmented features by an expert module within a remaining expert layer based on features output from an expert module in the previous expert layer.

[0029] According to an embodiment, the feature extracted by the expert module within the final expert layer can be a second feature.

[0030] According to an embodiment, the extraction of the second feature may include selectively extracting spatial segmentation features or temporal segmentation features by an expert module based on the temporal and spatial importance of the input features.

[0031] According to an embodiment, the extraction of the second feature may include: extracting an importance vector based on the input feature; outputting the input feature to a spatial feature channel or a temporal feature channel based on the importance vector; and extracting spatial features segmented based on features input through the spatial feature channel or temporal features segmented based on features input through the temporal feature channel.

[0032] According to an embodiment, the extraction of the second feature may include generating a binary decision vector based on the importance vector.

[0033] According to an embodiment, outputting input features to a spatial feature channel or a temporal feature channel may include: outputting input features to a spatial feature channel based on a binary decision vector of 1; and outputting input features to a temporal feature channel based on a binary decision vector of 0.

[0034] According to embodiments of this disclosure, a model learning method for a target re-identification device may include: extracting features based on input samples; activating an expert module for each of a plurality of expert layers in a sequential relationship based on the features; after activating the expert module, using the activated expert module to extract segmentation features based on the features extracted from the input samples; calculating a loss based on the segmentation features; and updating the model based on the loss.

[0035] According to an embodiment, the activation of an expert module may include: a first expert layer among a plurality of expert layers activating one of the plurality of expert modules based on features; and the remaining expert layers among the plurality of expert layers activating one of the plurality of expert modules based on features output from a previous expert layer.

[0036] According to an embodiment, the activation of an expert module may include: evaluating the input features and relevance scores of the expert module by multiple expert modules of each of a plurality of expert layers; and activating the expert module with the highest relevance score within the corresponding expert layer by a selector of each of the plurality of expert modules.

[0037] According to an embodiment, the extraction of segmentation features may include: evaluating the temporal and spatial importance of input features by an activated expert module, and extracting spatial segmentation features or temporal segmentation features based on the evaluation.

[0038] According to an embodiment, the calculation of loss may include: vectorizing the spatial feature parameters or temporal feature parameters output from the activated expert module, and calculating the loss of spatial features and the loss of temporal features by calculating pairwise cosine similarity based on the vectorized spatial feature parameters and temporal feature parameters of each expert layer.

[0039] The target re-identification device, target re-identification method, and model learning method disclosed in the embodiments are advantageous in effectively learning detailed features, which are key factors in distinguishing images.

[0040] The target re-identification device, target re-identification method, and model learning method disclosed in the embodiments are also advantageous in learning spatial refinement features or temporal refinement features based on whether the features are important in the spatial or temporal aspects.

[0041] Therefore, the target re-identification device (or target re-identification method) disclosed in the embodiments has expertise in identifying subtle differences in images and enhancing recognition performance.

[0042] Furthermore, the target re-identification devices, target re-identification methods, and model learning methods are also advantageous in automatically adding and expanding new experts during training.

[0043] Therefore, this eliminates the hassle of staff manually assigning additional experts and the burden of conducting extensive testing to determine the appropriate number of experts.

[0044] The advantages of this disclosure are not limited to those described above, and other advantages not described herein will be readily apparent to those skilled in the art from the following description. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the configuration of a target re-identification device according to an embodiment of the present disclosure;

[0046] Figure 2 This is a block diagram illustrating the functional blocks of a processor according to an embodiment of the present disclosure;

[0047] Figure 3 This is a diagram illustrating a detailed configuration of a feature extraction module according to an embodiment of the present disclosure;

[0048] Figure 4 It is shown Figure 3 A detailed configuration diagram of the expert module in the diagram;

[0049] Figure 5 This is a flowchart illustrating a target re-identification method according to an embodiment of the present disclosure;

[0050] Figure 6 It is shown Figure 5 Detailed operation flowcharts for S510-1 to S510-N; and

[0051] Figure 7 This is a flowchart illustrating a method for training a target re-identification model according to an embodiment of the present disclosure. Detailed Implementation

[0052] It is understood that, as used herein, the terms “vehicle” or “of a vehicle” or other similar terms generally include motor vehicles, such as passenger cars including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, watercraft including various ships and vessels, aircraft, etc., and include hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other vehicles using alternative fuels (e.g., fuels derived from resources other than petroleum). As referred to herein, a hybrid vehicle is a vehicle with two or more power sources, such as a gasoline and electric dual-power vehicle.

[0053] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” as used herein are intended to also include the plural forms. It will be further understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any one and all combinations of one or more of the associated listed items. Throughout this specification, unless explicitly stated otherwise, the word “comprising” and variations such as “including” or “containing” will be understood to imply inclusion of the stated elements without excluding any other elements. Additionally, the terms “unit,” “device,” “component,” and “module” described in the specification refer to a unit for performing at least one function and operation, and can be implemented by hardware components or software components and combinations thereof.

[0054] Furthermore, the control logic of this disclosure can be implemented as a non-transitory computer-readable medium containing executable program instructions that are executed by a processor, controller, etc. Examples of computer-readable media include, but are not limited to, ROM, RAM, optical disc (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable medium can also be distributed across a networked computer system, thereby being stored and executed in a distributed manner, for example, via a telematics server or a controller area network (CAN).

[0055] Furthermore, detailed descriptions of well-known technologies related to the embodiments disclosed in this specification may be omitted to avoid obscuring the subject matter of the embodiments disclosed in this specification. Additionally, the accompanying drawings are provided only for easy understanding of the embodiments disclosed in this specification and do not limit the technical spirit disclosed herein. It should be understood that the embodiments include all modifications, equivalents, and alternatives within the spirit and scope of this disclosure.

[0056] As used herein, terms including ordinal numbers such as “first” and “second” may be used to describe individual components without limiting them. Terms are used only to distinguish one component from another.

[0057] Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well.

[0058] It should be understood that when a component is referred to as "connected to" or "linked to" another component, the component can be directly connected to or linked to the other component, or there may be intermediate components. Conversely, when a component is referred to as "directly connected" or "directly linked to" another component, there are no intermediate components.

[0059] In the following description, embodiments disclosed herein are described with reference to the accompanying drawings, in which the same reference numerals are assigned to refer to the same or similar components, and redundant descriptions of the same or similar components are omitted.

[0060] Figure 1 This is a diagram illustrating the configuration of a target re-identification device 100 according to an embodiment of the present disclosure.

[0061] Reference Figure 1 According to embodiments of the present disclosure, the target re-identification device 100 may be a computing device implemented to perform target re-identification based on an input video sequence.

[0062] For example, video sequences can be input from a single camera or from multiple cameras.

[0063] The target re-identification device 100 can perform target re-identification on video sequences based on a target re-identification model with an artificial intelligence neural network structure.

[0064] The target re-identification model of the target re-identification device 100 can be updated through training.

[0065] According to the embodiments, the target re-identification model can effectively learn the detailed components (or features) of a sample, and can effectively learn spatial refinement differences and temporal refinement differences.

[0066] Furthermore, the object re-identification model can automatically add new experts during training. In this way, by expanding the number of experts during training, the object re-identification model can alleviate the hassle of manually assigning experts and the effort required to conduct numerous tests to determine the appropriate number of experts.

[0067] According to an embodiment, the target re-identification device 100 may include a processor 110, a memory 130, a storage device 150, a user interface 170, and a bus 190.

[0068] The processor 110 can be implemented as a hardware data processing device that utilizes physical structures to perform desired operations.

[0069] Processor 110 controls the overall operation of each component of target re-identification device 100. Processor 110 may be configured to include at least one of a central processing unit (CPU), microprocessor unit (MPU), microcontroller unit (MCU), graphics processing unit (GPU), or any type of processor known in the art.

[0070] Furthermore, the processor 110 can perform calculations for at least one application or program to run methods / operations according to various embodiments of this disclosure.

[0071] Memory 130 stores various data, commands, and / or information. Memory 130 can load one or more computer programs from storage device 150 to run methods / operations according to various embodiments of this disclosure. For example, memory 130 may store a target re-identification model executed by processor 110.

[0072] Examples of memory 130 may include random access memory (RAM) and dynamic RAM (DRAM), but are not limited thereto, and may include at least one form of memory known in the art of this disclosure.

[0073] Storage device 150 may temporarily store one or more computer programs. For example, storage device 150 may include non-volatile memory, such as flash memory, hard disk, removable disk, or any other form of computer-readable recording medium known in the art.

[0074] For example, a computer program may include one or more instructions for implementing methods / operations according to various embodiments of the present disclosure. When the computer program is loaded into memory 130, processor 110 may execute methods / operations according to various embodiments of the present disclosure by running one or more instructions.

[0075] User interface 170 can receive commands, data, and information from external sources to target re-identification device 100. User interface 170 can output the operation results of target re-identification device 100. For example, user interface 170 may include a keyboard, mouse, monitor, and touch screen.

[0076] Bus 190 provides communication functionality between components of target re-identification device 100. Bus 190 can be implemented in various forms, such as address bus, data bus, and control bus.

[0077] Figure 2 This is a block diagram illustrating the functional blocks of a processor 200 according to an embodiment of the present disclosure.

[0078] Figure 2 The processor can be with Figure 1 The processor 110 is the same.

[0079] According to an embodiment, the processor 200 can perform target re-identification based on the input video sequence and can update the target re-identification model by learning from the input samples.

[0080] For this purpose, the processor 200 may be equipped with a target re-identification model. The operation of the processor 200 described below can be performed by the target re-identification model loaded onto the processor 200, and the target re-identification function of the target re-identification device 100 can be implemented by the target re-identification model.

[0081] Reference Figure 2 The processor 200 (or target re-identification model) may include a backbone 210, a feature extraction module 220, and a classification module 230, but the configuration of the processor (or target re-identification model) is not limited thereto.

[0082] The backbone 210 (or backbone network) can be configured based on the ResNet-50 model, but is not limited to it. For example, models such as VGG16 and SpinNet can form the backbone.

[0083] The backbone 210 extracts features from the input video frames and can output the extracted features to the feature extraction module 220. In the following text, the features output from the backbone 210 to the feature extraction module 220 are referred to as the first features.

[0084] The feature extraction module 220 includes multiple expert modules, and can segment the first feature based on the activated expert module among the multiple expert modules to output the second feature. The segmentation within the feature extraction module 220 can be performed multiple times.

[0085] According to an embodiment, the feature extraction module 220 includes multiple expert layers in a sequential relationship, and the multiple expert layers may include multiple expert modules in a parallel relationship.

[0086] Each of the multiple expert layers can extract refined features from the input features based on the activated expert modules among the multiple expert modules included, thereby generating refined features.

[0087] According to an embodiment, during a sample-based learning process, among multiple expert modules, the expert module with the highest relevance to the sample can be activated.

[0088] For example, when there are N expert layers, the output of the first expert module up to the (N-1)th expert module can be input to the next expert module, and the output of the last (Nth) expert module can be the output of the feature extraction module 220.

[0089] In other words, the output of the last layer (Nth layer) expert module can correspond to the second feature output from the feature extraction module 220.

[0090] According to an embodiment, the output of a later expert layer (e.g., the expert layer in the second stage) may include more refined features compared to the features included in the output of an earlier expert layer (e.g., the expert layer in the first stage).

[0091] Therefore, the features used in the training of a later expert module can include more granular information compared to those used in the training of an earlier expert module, and the later expert module gains a more accurate expertise in distinguishing smaller differences compared to the earlier expert module.

[0092] According to an embodiment, during the learning process of the feature extraction module 220, the expert module can selectively learn spatial features and temporal features.

[0093] The expert module evaluates the importance of spatial and temporal aspects based on input features, and can learn spatial or temporal features based on the evaluation results.

[0094] According to an embodiment, the feature extraction module 220 can automatically add new expert modules during the learning process. The addition of new expert modules can occur independently for each expert layer.

[0095] Therefore, in addition to the expert module, the feature extraction module 220 may also include a waiting expert module associated with the expert layer.

[0096] According to an embodiment, when the correlation between the waiting expert module and the sample is greater than the correlation between other expert modules and the sample, the waiting expert module can be added to the corresponding expert layer. Furthermore, after adding the waiting expert module to the expert layer, the feature extraction module 220 can generate a new waiting expert module.

[0097] To clearly represent the ability of the feature extraction module 220 to extend expert modules and distinguish them from the existing expert modules included in the expert layer, the waiting expert modules included in the expert layer can be referred to as "extended expert modules".

[0098] Preferably, the expert modules in the feature extraction module 220 learn to refine features to identify different identities. In this regard, preferably, the similarity between each expert module within the expert layer is low.

[0099] According to an embodiment, the feature extraction module 220 may apply diversity loss to limit the pairwise similarity of experts. Here, diversity loss may include spatial feature loss and temporal feature loss.

[0100] The feature extraction module 220 can vectorize the spatial feature-related parameters and temporal feature-related parameters of each expert module.

[0101] Then, the feature extraction module 220 can calculate the spatial feature loss (spatial feature loss) and the temporal feature loss (temporal feature loss) based on the pairwise cosine similarity of each expert layer.

[0102] However, the similarity calculation method used by the feature extraction module 220 for loss calculation is not limited to this.

[0103] The feature extraction module 220 can calculate the diversity loss by aggregating the spatial feature loss and temporal feature loss calculated for each expert layer.

[0104] The feature extraction module 220 can update the model parameters by applying diversity loss. According to an embodiment, the feature extraction module 220 can further apply conventional re-identification (Re-ID) losses, such as cross-entropy loss and batch hard triplet loss, to update the model parameters.

[0105] The method of calculating re-identification loss based on cross-entropy and batch hard triples is a well-known technique and will not be described in detail here.

[0106] The classification module 230 can classify categories based on the output of the feature extraction module 220. For example, the classification module 230 can use various classification algorithms to perform classification, such as Naive Bayes classifier, Support Vector Machine (SVM), Random Forest, Decision Tree, Gradient Boosting Tree (GBT), SGD classifier, and AdaBoost.

[0107] Figure 3 This is a diagram illustrating a detailed configuration of the feature extraction module 300 according to an embodiment of the present disclosure.

[0108] Figure 3 The feature extraction module 300 can be used with Figure 2 The feature extraction module 220 is the same.

[0109] Reference Figure 3 The feature extraction module 300 may include multiple expert layers 300-1 to 300-N in a sequential relationship.

[0110] According to an embodiment, each of the plurality of expert layers 300-1 to 300-N may include a plurality of expert modules 310-1 to 310-N connected in parallel and selectors 320-1 to 320-N.

[0111] Although Figure 3 Examples are given for the first-stage expert layer 300-1, which includes five expert modules; the second-stage expert layer 300-2, which includes three expert modules; and the Nth-stage expert layer 300-N, which includes four expert modules. However, this configuration is not exhaustive.

[0112] Expert modules can be modules generated during model implementation or modules added during training.

[0113] In the embodiment, the fifth expert module of the first stage expert layer 300-1 The third expert module of the second stage expert level 300-2 And the fourth expert module of expert layer 300-N in stage N. It could be an "Extended Expert Module".

[0114] Here, wl is a subscript indicating that the corresponding expert module is an "extended expert module". As mentioned above, an "extended expert module" refers to an expert module added to the expert layer during training.

[0115] Expert layers 300-1 to 300-N can segment input features and output segmented features.

[0116] To achieve this, each expert layer 300-1 to 300-N may include activated expert modules from among multiple expert modules 310-1 to 310-N. and

[0117] Although Figure 3 An example illustrates the first expert module in the first stage expert layer 300-1. The third expert module in the second stage expert layer 300-2 and the third expert module in the Nth stage expert layer 300-N The activation is possible, but the configuration is not exhaustive.

[0118] As described above, the activation of the expert module is determined by selector 320 (320-1 to 320-N) during training.

[0119] During training, selectors 320-1 to 320-N can receive correlation scores r from multiple expert modules 310-1 to 310-N within each expert layer 300-1 to 300-N. l And activate the corresponding expert layer with the highest relevance score r within the range of 300-1 to 300-N. l The expert module.

[0120] According to an embodiment, selectors 320-1 to 320-N can be based on the input relevance score r. l This generates a one-hot vector in which only one index is set to 1 and the rest are set to 0.

[0121] For example, selectors 320-1 to 320-N can be evaluated using the Cumbel-Softmax algorithm to assess the relevance score r. l And assign the value 1 to the index of the highest relevance score, and assign 0 to the index of the remaining expert modules, thereby generating a one-hot vector.

[0122] Based on the one-hot vector, selectors 320-1 to 320-N can activate the corresponding expert layer with the highest relevance score r. l The expert module.

[0123] Based on the example, the first expert module of the first expert layer 300-1 Input features can be segmented And generate segmentation features The third expert module of the second expert level 300-2 Input features can be segmented And generate segmentation features The third expert module of the Nth expert layer (300-N) Input features can be segmented And generate segmentation features f out Here, the third expert module of the Nth expert layer (300-N) is used. Output feature f out It can become the final output of the feature extraction module 300.

[0124] The output f of the Nth expert layer 300-N out The output can be connected to the trunk via synthesizer 330. Combine them, and then input them into the classification module.

[0125] Figure 4 It is shown Figure 3 A diagram showing the detailed configuration of the expert module in the software.

[0126] Reference Figure 3 and Figure 4 Expert Module 310 (310-1 to 310-N) The characteristics of the entry The inputs are fed into the mapping module 311 to map them to the feature space and obtain the mapped features f. i l .

[0127] Here, "l" indicates that the expert module belongs to the l-th expert layer, and "i" indicates that the expert module is the i-th expert module within that expert layer.

[0128] therefore, The i-th expert module represents the l-th expert layer.

[0129] Based on the activation state of expert module 310, the mapping feature f i l It can be selectively input into importance assessment module 313 and branch module 314.

[0130] Therefore, the mapping feature f i l The input is fed into the filtering module 312, which, based on the state vector values ​​input from the selector 320, can map the feature f. i l It can be passed downstream, or its transmission downstream can be prevented.

[0131] Here, the state vector value is output from selector 320 to determine the activation state of the corresponding expert module 310, and can have a value of 0 or 1.

[0132] When the state vector value is 1 (i.e., the corresponding expert module 310 is activated), the filtering module 312 can map the feature f. i l The output is sent downstream, specifically to the importance assessment module 313 and the branch module 314.

[0133] When the state vector value is 0 (i.e., the corresponding expert module 310 is disabled), the filtering module 312 prevents the mapping of feature f. i l This could potentially terminate the operation of the corresponding expert module 310.

[0134] Importance assessment module 313 can receive mapping feature f i l As input, and generate a mapping feature f i l Importance vector

[0135] According to an embodiment, the importance evaluation module 313 can utilize max pooling and fully connected layers to output an importance vector. This importance vector includes the mapping feature f. i l A single importance value.

[0136] Here, importance values ​​(or importance vectors) can indicate the mapping features f. i l Is space or time more important?

[0137] The maximum pool and the fully connected layer are well-known technologies in the technical field of this invention, and detailed descriptions thereof are omitted here.

[0138] Importance vector It can be entered into branch module 314.

[0139] Branch module 314 can receive the mapping feature f i l and importance vector Both are used as input.

[0140] Branch module 314 can be based on importance vector Generate a binary decision vector, and based on this binary decision vector, map the feature f. i l Output to spatial feature extraction module 315 or temporal feature extraction module 316.

[0141] The path between branch module 314 and spatial feature extraction module 315 can be called the "spatial feature channel", while the path between branch module 314 and temporal feature extraction module 316 can be called the "temporal feature channel".

[0142] For example, branch module 314 may use discretization methods such as semantic hashing to generate binary decision vectors, but the algorithms used to generate binary decision vectors are not limited to this.

[0143] When the value of the binary decision vector is 1, branch module 314 can map the feature f. i l The output is sent to the spatial feature extraction module 315. Here, This can refer to the "mapped feature f" that is branched onto the spatial feature extraction module 315 (or the spatial feature channel). i l (Hereinafter referred to as spatial branching feature).

[0144] When the value of the binary decision vector is 0, branch module 314 can map the feature f. i l Output to the time feature extraction module 316. This can refer to the "mapping feature f" that is branched onto the time feature extraction module 316 (or the time feature channel). i l (Hereinafter referred to as time branch feature).

[0145] The spatial feature extraction module 315 can extract features from spatial branch features. Extract refined features.

[0146] For example, the spatial feature extraction module 315 can be constructed using a 1×3×3 convolutional layer to extract spatial information; however, the implementation of the spatial feature extraction module 315 is not limited to this.

[0147] The time feature extraction module 316 can extract features from time branch features. Extract refined features.

[0148] For example, the time feature extraction module 316 can be constructed using a 3×1×1 convolutional layer to extract time information; however, this implementation of the time feature extraction module 316 is not limited to this.

[0149] Therefore, for a single video frame, the expert module can extract either the spatial or temporal features of the segmentation. In other words, for a single video frame, the expert module outputs either the spatial or temporal features of the segmentation, but not both together.

[0150] Synthesizer 317 can combine the outputs of spatial feature extraction module 315 and temporal feature extraction module 316 to generate an output.

[0151] The output of synthesizer 317 can be the output of the corresponding expert module 310.

[0152] On the other hand, during the training process, the mapping features f i l It can be input into the correlation assessment module 318.

[0153] The correlation assessment module 318 can assess the correlation between the corresponding expert module 310 and the sample.

[0154] According to an embodiment, the correlation evaluation module 318 can use max pooling and fully connected layers to generate correlation values, and use the tanh function to obtain a correlation score.

[0155] The correlation assessment module 318 can assign correlation scores. Output to the selector 320 of the corresponding expert layer.

[0156] Therefore, selector 320 can obtain the relevance scores of all expert modules within the corresponding expert layer. Based on this, the expert layer with the highest relevance score r can be activated. l The expert module.

[0157] Figure 5 This is a flowchart illustrating a target re-identification method according to an embodiment of the present disclosure;

[0158] Figure 5 The step-by-step operations shown can be referenced Figure 1 , Figure 2 , Figure 3 and Figure 4 The target re-identification device 100 or target re-identification model described herein shall be used to perform this action.

[0159] Reference Figure 1 , Figure 2 , Figure 3 , Figure 4 and Figure 5 In operation of S500, the backbone 210 can extract the first feature from the input video frame.

[0160] Subsequently, in operation S510, the feature extraction module 220 can refine the first feature based on expert modules 310-1 to 310-N within multiple expert layers 300-1 to 300-N to generate the second feature.

[0161] According to an embodiment, expert modules 310-1 to 310-N may be expert modules activated during training by selectors 320 within expert layers 300-1 to 300-N based on their highest relevance to the samples.

[0162] According to the embodiments, expert modules 310-1 to 310-N may be expert modules generated during model implementation or expert modules added during training.

[0163] In operation S510, the segmentation performed by multiple expert modules 310-1 to 310-N can be executed multiple times across operations S510-1 to S510-N.

[0164] In each of operations S510-1 to S510-N, expert modules 310-1 to 310-N can map input features to a feature space and evaluate whether the mapped features are important in space or time.

[0165] Based on this assessment, expert modules 310-1 to 310-N can extract spatial segmentation features or temporal segmentation features from the mapping features.

[0166] Subsequently, in operation S520, the classification module 230 can classify the category based on the second feature output by operation S510.

[0167] Figure 6 It is shown Figure 5 A detailed flowchart of the operation of S510-1 to S510-N.

[0168] Reference Figure 3 , Figure 4 and Figure 6 In operation S511, the mapping module 311 can convert the input features Map to the feature space and obtain the mapped features f i l .

[0169] Mapping feature f i lInput can be entered into importance assessment module 313 and branch module 314.

[0170] Importance assessment module 313 can evaluate mapping feature f in operation S512. i l Is it more important in terms of space or time?

[0171] During operation S512, the importance assessment module 313 can output an importance vector. This importance vector includes the mapping feature f. i l A single importance value.

[0172] Importance vector It can be entered into branch module 314.

[0173] In operation S513, branch module 314 receives the mapping feature f. i l and importance vector As input, and based on the importance vector The output can be selectively sent to the spatial feature extraction module 315 or the temporal feature extraction module 316.

[0174] In operation S513, branch module 314 can be based on importance vector. To generate a binary decision vector, and based on this binary decision vector, to map the feature f i l Output to spatial feature extraction module 315 or temporal feature extraction module 316.

[0175] When the value of the binary decision vector in operation S513 is 1, branch module 314 can map the feature f. i l Output to spatial feature extraction module 315.

[0176] When the value of the binary decision vector in operation S513 is 0, branch module 314 can map the feature f. i l Output to the time feature extraction module 316.

[0177] Subsequently, in operation S514, the spatial feature extraction module 315 can extract features from the input features. Spatial segmentation features can be extracted from the input features, or in operation S515, the temporal feature extraction module 316 can extract features from the input features. Extract time segmentation features.

[0178] Subsequently, in operation S516, the synthesizer 317 can combine the outputs of the spatial feature extraction module 315 and the temporal feature extraction module 316 to generate an output.

[0179] Figure 7 This is a flowchart illustrating a method for training a target re-identification model according to an embodiment of the present disclosure.

[0180] The target re-identification device 100 can train the target re-identification model based on preset model parameters.

[0181] According to an embodiment, the target re-identification device 100 can receive samples in operation S700 and extract features based on the samples using the backbone 210 in operation S710.

[0182] Next, in operation of S720, the target re-identification device 100 can activate the expert module for each of the multiple expert layers based on features.

[0183] When operating the S720, the first expert layer among multiple expert layers can activate one of the multiple expert modules based on features input from the backbone 210.

[0184] When operating the S720, expert layers other than the first expert layer can activate one of multiple expert modules based on features output from the previous expert layer.

[0185] In operation S720, each expert module within multiple expert layers uses the relevance evaluation module 318 to evaluate the relevance of the input features and the relevance score of the expert module, allowing the selector 320 of each expert layer to activate the expert module with the highest relevance score within that layer.

[0186] After activating an expert module in each of the multiple expert layers, in operation S730, the target re-identification device 100 can use the expert modules activated in the multiple expert layers to extract segmentation features based on features extracted from the input sample.

[0187] When operating the S730, the activated expert module can extract spatial segmentation features (segmented spatial features) or temporal segmentation features (segmented temporal features) from the input features.

[0188] When operating the S730, the activated expert module can evaluate the temporal and spatial importance of the input features and extract the spatial or temporal features of the segmentation based on the evaluation results.

[0189] Subsequently, the target re-identification device 100 can calculate the loss based on the features extracted by the activated expert module in operation S740, and update the model based on the loss in operation S750.

[0190] Here, the loss is calculated based on the features extracted by the activated expert module, and model updates can be performed on the activated expert module.

[0191] According to an embodiment of this disclosure, during operation S740, the target re-identification device 100 can calculate the diversity loss.

[0192] According to an embodiment, the target re-identification device 100 can vectorize the spatial feature-related parameters and temporal feature-related parameters of each expert module.

[0193] Then, the target re-identification device 100 can calculate the spatial feature loss (spatial feature loss) and the temporal feature loss (temporal feature loss) by calculating the pairwise cosine similarity of the vectorized spatial feature parameters and temporal feature parameters of each expert layer.

[0194] The target re-identification device 100 can calculate the diversity loss by aggregating the spatial feature loss and temporal feature loss calculated for each expert layer.

[0195] When operating the S740, the target re-identification device 100 can additionally calculate existing re-identification (Re-ID) losses, such as cross-entropy loss and batch hard triplet loss.

[0196] Table 1 shows the results of testing the target re-identification model according to embodiments of the present disclosure and the conventional target re-identification model.

[0197] Table 1

[0198]

[0199] The tests were conducted on the public large-scale datasets MARS and LS-VID, with evaluations including mAP, rank-1, rank-5, and rank-20 for the MARS dataset, and mAP and rank-1 for the LS-VID dataset.

[0200] As can be seen from Table 1, the target re-identification model according to the embodiments of this disclosure outperforms the performance of traditional target re-identification models.

[0201] Although this disclosure has been described and illustrated with reference to specific embodiments, it will be apparent to those skilled in the art that various modifications and changes may be made to the embodiments without departing from the spirit of this disclosure or the scope of the appended claims.

Claims

1. A target re-identification device, the device comprising: The memory stores the target re-identification model; as well as The processor runs the model. Wherein, the processor: Extract the first feature from the input video frame. Based on expert modules within multiple expert layers in a sequential relationship, the first feature is segmented into a second feature, and The second feature is classified in the target re-identification model. The expert module is a single expert module that is activated among multiple expert modules within a single expert layer.

2. The device according to claim 1, wherein, The expert module is an expert module generated during the implementation of the model, or an expert model added during training.

3. The device according to claim 1, wherein, The expert modules in the first expert layer extract segmentation features based on the first feature, and the expert modules in the remaining expert layers extract segmentation features based on the features output from the expert modules of the previous expert layer.

4. The device according to claim 1, wherein, The expert module has the highest correlation with the input video frame among the multiple expert modules within the single expert layer.

5. The device according to claim 1, wherein, The expert module selectively extracts spatial segmentation features or temporal segmentation features based on the spatial-temporal importance of the input features.

6. The device according to claim 1, wherein, The expert module includes: The importance assessment module outputs an importance vector based on the input features; The branching module, based on the importance vector, outputs the input features to the spatial feature channel or the temporal feature channel; The spatial feature extraction module extracts spatial features segmented based on features input through the spatial feature channel; and The time feature extraction module extracts time features segmented based on features input through the time feature channel.

7. The device according to claim 6, wherein, The importance evaluation module uses max pooling and fully connected layers to output the importance vector, which includes the importance values ​​of the input features.

8. The device according to claim 6, wherein, The branch module generates a binary decision vector based on the importance vector, outputs the input feature to the spatial feature channel based on the binary decision vector being 1, and outputs the input feature to the temporal feature channel based on the binary decision vector being 0.

9. The device according to claim 1, wherein, The processor includes a selector that, during training, activates one of the multiple expert modules within the single expert layer based on the relevance scores of the multiple expert modules within the single expert layer.

10. The device according to claim 1, wherein, The processor further includes a waiting expert module associated with the single expert layer, the waiting expert module being selectively included in the single expert layer during training.

11. A target re-identification method, implemented by a processor running a target re-identification model stored in runtime memory, the method comprising: The processor extracts a first feature from the input video frame; The processor extracts the second feature by segmenting the first feature based on expert modules within multiple expert layers that are in a sequential relationship; as well as The processor classifies the second feature in the target re-identification model. The expert module is a single expert module that is activated among multiple expert modules within a single expert layer.

12. The method according to claim 11, wherein, Extracting the second feature includes: The expert module within the first expert layer extracts segmentation features based on the first feature; and The expert modules in the remaining expert layers extract segmentation features based on the features output by the expert modules in the previous expert layers, and the features extracted by the expert modules in the last expert layer are the second features.

13. The method according to claim 11, wherein, Extracting the second feature includes: The expert module selectively extracts spatial segmentation features or temporal segmentation features based on the spatial-temporal importance of the input features.

14. The method according to claim 11, wherein, Extracting the second feature includes: Importance vectors are extracted based on input features; Based on the importance vector, the input features are output to either the spatial feature channel or the temporal feature channel; and Extract spatial features segmented based on features input through the spatial feature channel or temporal features segmented based on features input through the temporal feature channel.

15. The method according to claim 14, wherein, Extracting the second feature includes generating a binary decision vector based on the importance vector, and outputting the input feature to a spatial feature channel or a temporal feature channel includes outputting the input feature to the spatial feature channel based on the binary decision vector being 1, and outputting the input feature to the temporal feature channel based on the binary decision vector being 0.

16. A model learning method for a target re-identification device, the method comprising: The processor extracts features based on the input samples; The processor activates the expert module of each of the multiple expert layers in a sequential relationship based on the features. After the expert module is activated, the processor uses the activated expert module to extract segmentation features based on the features extracted from the input sample; The processor calculates the loss based on the segmentation features; as well as The processor updates the model based on the loss.

17. The method according to claim 16, wherein, Activating the expert module includes: One of the multiple expert layers is activated by the first expert layer based on the aforementioned feature; and One of the multiple expert modules is activated by the remaining expert layer among the multiple expert layers based on the features output from the previous expert layer.

18. The method according to claim 16, wherein, Activating the expert module includes: The input features and relevance scores of the expert module are evaluated by multiple expert modules of each of the plurality of expert layers; and The expert module with the highest relevance score within the corresponding expert layer is activated by the selector of each of the plurality of expert modules.

19. The method of claim 16, wherein, Extracting the segmentation features includes: The spatial-temporal importance of the input features is evaluated by the activated expert module; and Based on the evaluation, spatial segmentation features or temporal segmentation features are extracted.

20. The method according to claim 19, wherein, The calculation of the loss includes: Vectorize the spatial or temporal feature parameters output from the activated expert module; and The spatial feature loss and temporal feature loss are calculated by computed pairwise cosine similarity of vectorized spatial feature parameters and vectorized temporal feature parameters based on each expert layer.

Images