Cross-mirror tracking method based on association reasoning

Abstract

The invention relates to a cross-mirror tracking method based on association reasoning, and the method comprises the steps: obtaining all pictures shot by a plurality of cameras, adding identity tags to the pictures, and forming a tag sample set; constructing a cross-mirror tracking model, wherein the cross-mirror tracking model comprises a three-branch network, an auxiliary classifier, a clothing classifier, a multi-modal coordination module, a fusion classifier and an identity classifier; training a cross-mirror tracking model by using the labeled sample set to obtain a loss function of each module; the loss functions of all the modules are integrated, total loss is obtained, parameters of the cross-mirror tracking model are updated through total loss back propagation, and loop iteration is carried out until convergence; multi-lens pedestrian pictures needing to be detected are input into the trained cross-lens tracking model, and pedestrian identities are predicted; compared with the prior art, the method can use the cross-mirror tracking model to stably identify the pedestrian identity in the clothing change scene.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a cross-camera tracking method based on association reasoning. Background Art

[0002] Cross-camera tracking is a computer vision technology designed to match pedestrians of the same identity across different camera perspectives based on their images. This technology has broad applications in intelligent surveillance, security systems, smart transportation, criminal investigation, and other fields. With the widespread use of urban surveillance networks, the massive amount of video data generated has made manual pedestrian retrieval extremely difficult. Consequently, deep learning-based cross-camera tracking technology has rapidly developed, enabling efficient retrieval of target pedestrians from massive amounts of data and significantly enhancing the intelligence of surveillance systems. Currently, cross-camera tracking technology primarily relies on deep learning models to automatically extract image features and perform cross-camera matching.

[0003] Existing cross-camera tracking methods still face numerous challenges in addressing clothing-changing scenarios. Most current cross-camera tracking methods rely heavily on clothing features for identity matching. When a pedestrian changes clothing, the model can easily misidentify them as different identities, leading to recognition failure. Although some methods attempt to alleviate this problem by removing background information or extracting local features, they still cannot effectively distinguish between identity features and clothing features, resulting in reduced recognition accuracy. Furthermore, in recent years, some studies have introduced multimodal information, such as text descriptions or pedestrian attributes, to reduce reliance on clothing features. However, since text and images belong to different modalities, efficiently fusing this information in the feature space remains a challenge. Existing methods typically employ simple concatenation or weighted fusion strategies, which fail to fully consider the interaction of cross-modal information and result in limited fusion effectiveness. Furthermore, some attribute-based cross-camera tracking methods attempt to utilize pedestrian attributes (such as gender, backpack, hairstyle, etc.) to assist in identity recognition. However, these attributes still contain a large amount of clothing-related information, such as "wearing a red top" or "black pants." The presence of these clothing attributes still affects the final identity prediction, making it difficult for the model to truly overcome clothing interference. Furthermore, most current cross-camera tracking methods are based on data-driven deep learning frameworks, relying primarily on data distribution to learn identity features without analyzing the correlations between identity features and their formation. Because clothing features are strongly correlated with identity features in the training data, the model can easily learn spurious correlations, causing the recognition process to still be affected by clothing variations and failing to truly decouple identity and clothing features.

[0004] Chinese patent CN119541002A discloses a clothing-changing pedestrian re-identification method based on clothing-aware blending and foreground enhancement. This method simulates pedestrian clothing-changing behavior while preserving clothing style and texture to confuse the model's learning of clothing, thus breaking away from traditional methods that erase or randomly sample clothing pixels. However, single-network processing cannot achieve the goal of refined feature extraction. Therefore, this invention extracts foreground features from clothing-erased images by fusing a Vision Transformer with a ResNet network, compensating for the network's shortcomings. It also simulates clothing-changing behavior by extracting pure clothing fabric and performing swapping. A multi-network fusion approach is used to enhance foreground features, enriching their semantic expression and improving their discriminability for pedestrian identification. However, this invention only simulates clothing changes through clothing-aware blending, failing to clearly separate identity features from clothing features and insufficiently suppressing clothing interference. Clothing changes are simulated through fabric clipping and padding, but actual clothing changes involve multi-dimensional variations such as texture, style, and lighting, limiting the generalization capability of a single padding strategy. The feature fusion of this invention also relies on the splicing of ResNet and Vision Transformer, mainly focusing on local feature enhancement, but lacks fine-grained cross-modal interaction and insufficient cross-modal interaction.

[0005] While existing technologies have improved cross-camera tracking capabilities in clothing-changing scenarios to a certain extent, they are still hampered by issues such as interference from clothing features, insufficient cross-modal fusion capabilities, and difficulty removing spurious correlations, resulting in insufficient robustness in recognition. Therefore, more effective methods are still needed to improve the recognition accuracy and generalization capabilities of cross-camera tracking tasks, ensuring that the model can stably match the same pedestrian identity despite clothing changes. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a cross-camera tracking method based on associative reasoning.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A cross-camera tracking method based on association reasoning, the method comprising:

[0009] Images captured by multiple cameras are input into a pre-trained cross-camera tracking model to predict pedestrian identities. The cross-camera tracking model includes a three-branch network, an auxiliary classifier, a clothing classifier, a multimodal coordination module, a fusion classifier, and an identity classifier.

[0010] The process of predicting pedestrian identity by the cross-camera tracking model includes: using a three-branch network to extract identity features, clothing features and text features from multi-camera pedestrian images; using an auxiliary classifier to further separate identity features and clothing features, and using a clothing classifier to optimize clothing features; fusing the identity features and clothing features through bilinear pooling to obtain visual features, and using multimodal coordination module features to fuse the visual features and text features to obtain fused features; using an identity classifier and a fusion classifier to process the identity features and fused features respectively to obtain pedestrian identity prediction results.

[0011] Furthermore, the three-branch network includes an identity branch, a clothing branch, and a parallel text branch; wherein the identity features are extracted by a pre-trained ResNet network in the identity branch, the clothing features are extracted by a pyramid matching strategy in the clothing branch; and the text features are extracted by an attribute extraction and mask module in the parallel text branch;

[0012] The process of extracting text features by the attribute extraction and masking module includes extracting a fixed attribute list based on a pre-trained SOLIDER model and converting it into a 0-1 binary vector, using a predefined attribute template to mask clothing-related attributes to obtain text features without clothing interference.

[0013] Furthermore, the process of training the cross-camera tracking model includes:

[0014] Obtain all images taken by multiple cameras and add identity tags to the images to form a labeled sample set;

[0015] Build a cross-camera tracking model, using a three-branch network to extract identity features, clothing features, and text features from multi-camera pedestrian images, and calculate the minimum recognition loss of the identity branch and the clothing classification loss of the clothing branch;

[0016] Input the identity features and clothing features into the auxiliary classifier and the clothing classifier, and calculate the KL divergence loss;

[0017] fusing the identity features and clothing features through bilinear pooling to obtain visual features, and fusing the visual features and text features using multimodal coordination module features to obtain fused features;

[0018] The fusion features and identity features are input into the fusion classifier and identity classifier respectively, and the association loss is obtained through the association reasoning mechanism;

[0019] The recognition loss, clothing classification loss, KL divergence loss and association loss are comprehensively minimized, and the parameters of the cross-camera tracking model are updated through loss backpropagation, and the cycle is iterated until convergence.

[0020] Furthermore, the expression for minimizing the recognition loss is:

[0021]

[0022] Among them, L ID To minimize the recognition loss, y i represents the label of the i-th sample, P ID (y i |x i ) represents the identity classifier δ ID For the i-th image x i The predicted probability of , N represents the number of samples in the training batch;

[0023] The expression of the clothing classification loss is:

[0024]

[0025] Among them, L C represents the clothing classification loss, represents the label of the i-th sample, represents the clothing classifier δ C For the i-th image x i The predicted probability, N C Indicates the number of samples for clothing classification.

[0026] Furthermore, the expression of the KL divergence loss is:

[0027]

[0028] in, F C to F ID The KL distance, F ID to F C KL distance;

[0029]

[0030] Among them, F C For clothing features, F ID For identity characteristics, Indicates that the clothing classifier δ C From clothing features F C The probability distribution obtained is, Indicates that the auxiliary classifier δ A From the identity feature F ID The probability distribution obtained, M represents the number of samples in the mini-batch.

[0031] Furthermore, the expression of the association loss is:

[0032]

[0033] Among them, y i is the label of the i-th sample, δ ID is the identity classifier, δ' ID is a fusion classifier that shares weights with the identity classifier, represents the bilinear pooling fusion operation, F ID is the identity feature, F V is the visual feature, F A is the text feature, and N is the number of samples in the training batch.

[0034] Furthermore, the multimodal coordination module includes a token mixer and a channel mixer.

[0035] Furthermore, the token mixer fuses the information of different time series of the visual features and text features, and the fusion expression is:

[0036] Z *,i =X′ *,i +W2Φ(W1Norm(X′ *,i ))

[0037] Where i represents the number of rows, Φ is the GELU activation function, Norm represents layer normalization, W1 and W2 are the weights of the linear layer, X is the input feature, and X′ *,i To represent all rows of the i-th column of the input feature matrix, Z *,i is the intermediate feature representation of the i-th token after passing through the token mixer.

[0038] Furthermore, the channel mixer fuses the information of different feature dimensions of the visual features and text features, and the fusion expression is:

[0039] Y j,* =Z j,* +W4Φ(W3Norm(Z j,* ))

[0040] Among them, j represents the number of columns, Φ is the GELU activation function, Norm represents layer normalization, W3 and W4 are the weights of the linear layer, Z j,* Represents all columns of the jth row of the intermediate feature matrix Z, Y j,* It is the final feature representation of the jth channel after the channel mixer.

[0041] Furthermore, the SOLIDER model used to extract text features in the parallel text branch is a pre-trained model and is in a frozen state during the training of the cross-camera tracking model.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention achieves accurate cross-camera tracking by constructing and training a cross-camera tracking model. In the cross-camera tracking model of the present invention, the three-branch network includes an identity branch, a clothing branch, and a parallel text branch. Text features are extracted through an attribute extraction and masking module. In the attribute extraction and masking module, the pre-trained SOLIDER model extracts basic text features and uses a predefined attribute template to mask the relevant attributes of clothing in the text features, so that the text features remove clothing interference and effectively remove the interference of clothing features on identity features, so that the model can focus more on stable identity representation and improve the accuracy of cross-clothing matching. The present invention supports rapid target retrieval of massive surveillance videos and can be expanded to multiple fields such as security, criminal investigation, and commerce, promoting the practicality and intelligence of cross-camera tracking technology.

[0044] 2. In the present invention, the multimodal coordination network module effectively integrates identity features, clothing features and masked text features, which can compensate for the limitations of a single modality (such as visual deviation caused by clothing changes) and enhance the model's comprehensive understanding of pedestrian identity; through the token mixer and channel mixer of the multimodal coordination network module, dual interactions are performed at the modality level and feature level respectively to fully integrate visual and textual information, which can improve the model's ability to represent pedestrian identity and make it more robust when handling cross-camera tracking tasks.

[0045] 3. The present invention uses the association reasoning mechanism and adopts the association loss L CAL Constrained fusion features enable the model to automatically separate clothing interference during training, significantly reduce the interference of clothing features on identity representation, improve the cross-camera tracking accuracy in clothing-changing scenarios, and enable the model to focus on stable identity features, obtaining more stable and unbiased identity features.

[0046] 4. This invention leverages the complementary nature of textual and visual features to compensate for the shortcomings of visual features in situations where clothing changes significantly. Textual features provide identity cues that are unaffected by clothing. When integrated with visual features, they further enhance the model's global understanding of pedestrian characteristics, overcoming the limitations of a single modality. This enhances the model's comprehensive understanding of pedestrian identity and improves recognition robustness in clothing-changing scenarios.

[0047] 5. This paper uses the KL divergence loss to constrain the distribution consistency of the clothing classifier and the auxiliary classifier, suppressing the potential interference of clothing information on identity features. This enables the model to stably extract identity features unrelated to clothing even in complex clothing-changing scenarios, significantly improving the generalization ability of cross-camera matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1Flow chart of the method of the present invention;

[0049] Figure 2 This is a schematic diagram of the model training process of the present invention;

[0050] Figure 3 Schematic diagram of text feature extraction and masking in the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0052] Example 1

[0053] This embodiment discloses a cross-camera tracking method based on association reasoning, the method is as follows Figure 1 As shown, the specific steps include:

[0054] Step S1: Obtain all images taken by multiple cameras and add identity tags to the images to form a labeled sample set;

[0055] Step S2, constructing a cross-camera tracking model;

[0056] Step S3: Use the labeled sample set to train the cross-camera tracking model, obtain the loss function of each module, integrate the loss functions of each module, and update the parameters of the cross-camera tracking model through backpropagation of the loss function. Repeat the cycle until convergence;

[0057] In step S4, the multi-lens pedestrian images to be detected are input into the trained cross-lens tracking model to predict the pedestrian's identity.

[0058] In step S1, all images taken by multiple cameras are collected from a camera monitoring network.

[0059] The labeled sample set is divided into a training set, a query set, and a gallery set. Each batch contains pedestrian images, their identity labels, camera labels, and clothing labels. These batches of data are used as input during training. During the model training phase, the training set is used for training. During the testing phase, the query set and gallery set are used for testing. The goal is to find pedestrian photos in the gallery set that share the same identity as those in the query set.

[0060] In step S2, the cross-camera tracking model includes a three-branch network, an auxiliary classifier, a clothing classifier, a multimodal coordination module, a fusion classifier, and an identity classifier.

[0061] The process of predicting pedestrian identity by the cross-camera tracking model includes: using a three-branch network to extract identity features, clothing features and text features from multi-camera pedestrian images; using an auxiliary classifier to further separate identity features and clothing features, and using a clothing classifier to optimize clothing features; fusing identity features and clothing features through bilinear pooling to obtain visual features, and using multimodal coordination module features to fuse visual features and text features to obtain fused features; using an identity classifier and a fusion classifier to process identity features and fused features respectively to obtain pedestrian identity prediction results.

[0062] The process of training the cross-camera tracking model in step S3 is as follows: Figure 2 Shown, including:

[0063] Obtain all images taken by multiple cameras and add identity tags to the images to form a labeled sample set;

[0064] Build a cross-camera tracking model, using a three-branch network to extract identity features, clothing features, and text features from multi-camera pedestrian images, and calculate the minimum recognition loss of the identity branch and the clothing classification loss of the clothing branch;

[0065] Input identity features and clothing features into the auxiliary classifier and clothing classifier, and calculate the KL divergence loss;

[0066] The identity features and clothing features are fused through bilinear pooling to obtain visual features, and the visual features and text features are fused using the multimodal coordination module features to obtain fused features.

[0067] The fusion features and identity features are input into the fusion classifier and identity classifier respectively, and the association loss is obtained through the association reasoning mechanism;

[0068] The recognition loss, clothing classification loss, KL divergence loss and association loss are comprehensively minimized, and the parameters of the cross-camera tracking model are updated through loss backpropagation. The cycle is iterated until convergence.

[0069] After the model is trained, the query set and gallery set are fed into the model as input to predict the identity of the pedestrians in the query set and find pictures of the same identity in the gallery set.

[0070] The three-branch network consists of an identity branch (clothing feature extraction network), a clothing branch (identity feature extraction network), and a parallel text branch (text feature extraction network), which respectively extract identity features, clothing features, and text features from multi-shot pedestrian images. The parallel text branch includes an Attribute Extraction and Masking Module (AEM).

[0071] The identity branch is based on ResNet-50, modifies the step size of layer4, adds multiple pooling methods, introduces batch normalization layers, and deletes the fully connected layers to extract the identity feature F.ID , the identity feature is passed through the identity encoder F ID =E(x i ) and is obtained by minimizing the recognition loss L ID To optimize, the formula is expressed as:

[0072]

[0073] Among them, F ID =E(x i ) represents the identity encoder E from the input image x i The identity feature F extracted from ID , L ID To minimize the recognition loss, the model is optimized by minimizing the loss, y i represents the label of the i-th sample, P ID (y i |x i ) represents the identity classifier δ ID For the i-th image x i The predicted probability of , N represents the number of samples in the training batch.

[0074] The clothing branch first extracts features from ResNet-50, and then uses the pyramid matching strategy to extract clothing features F C , a strategy for horizontally slicing feature maps at different pyramid scales for subsequent pooling operations. At each scale, the pyramid matching strategy independently learns each spatial block feature and enhances the classification ability by combining the features of average pooling and maximum pooling, thereby enhancing the local feature representation of clothing. The final clothing feature can be obtained through the clothing encoder and the pyramid matching strategy, and the formula is expressed as:

[0075] F C =PM(E C (x i ))

[0076] Among them, E C (x i ) represents the clothing encoder E C From the input image x i The features extracted from ,PM represents the pyramid matching strategy.

[0077] The clothing branch uses clothing classifier δ C And through clothing classification loss L C For training, the formula is expressed as:

[0078]

[0079] Among them, L CRepresents clothing classification loss, and the model is optimized by minimizing this loss so that it can classify clothing features. represents the label of the i-th sample, represents the clothing classifier δ C For the i-th image x i The predicted probability, N C Indicates the number of samples for clothing classification.

[0080] The parallel text branch obtains text features through attribute extraction and masking modules. The attribute extraction and masking modules are as follows: Figure 3 As shown, the text features are extracted from the attribute extraction and masking module, which extracts the attributes and masks the attributes related to clothing to obtain the text features F A First, we use the SOLIDER model trained on the PETA dataset to identify pedestrian attributes. Given a sample image x i , input SOLIDER model to obtain the attribute list of pedestrians And convert it into a 0-1 binary vector; set the attributes related to clothing to 0, implement the mask operation of clothing information, and obtain the attribute description data after masking

[0081] This processing method effectively removes the interference of clothing features on identity features, allowing the model to focus more on stable identity representation and improve the accuracy of cross-clothing matching.

[0082] The SOLIDER model used to extract text features in the parallel text branch is a pre-trained model and is frozen during the training of the cross-camera tracking model.

[0083] Identity feature F ID With clothing features F C The input to the auxiliary classifier and clothing classifier is constrained by KL divergence loss, as follows:

[0084] In order to make the clothing feature F C Perceived identity characteristics F ID The clothing bias in , KL divergence is used to fit the distribution of clothing bias and identity representation. The auxiliary classifier and clothing classifier share the same weight, and the formula is expressed as follows:

[0085]

[0086] From F C to F ID The KL distance calculation formula is expressed as:

[0087]

[0088] in, Indicates that the clothing classifier δ C From clothing features F C The probability distribution obtained is Indicates that the auxiliary classifier δ A From the identity feature F ID The probability distribution obtained, M represents the number of samples in the mini-batch;

[0089] From F ID to F C The KL distance calculation formula is expressed as:

[0090]

[0091] The total KL divergence loss formula is expressed as:

[0092]

[0093] Identity feature F ID With clothing features F C Fusion obtains visual features F V The multimodal coordination network module is used to fuse visual features with text features. The multimodal coordination network module includes a token mixer and a channel mixer, which can efficiently fuse visual features with text features to enhance the diversity and recognition of feature expression. The details are as follows:

[0094] Visual feature F V By identity feature F ID With clothing features F C Obtained through bilinear pooling fusion, the visual feature F is obtained through the multimodal coordination network module V With text feature F A fusion of

[0095] The multimodal coordination network module includes a token mixer and a channel mixer. The token mixer acts on the time dimension of the input features, allowing information interaction between different time series. The output expression of the token mixer is:

[0096] Z *,i =X′ *,i +W2Φ(W1Norm(X′ *,i ))

[0097] Where i represents the number of rows, Φ is the GELU activation function, Norm represents layer normalization, W1 and W2 are the weights of the linear layer, X∈R t×d are input features, where t is the length of the time series, d is the number of modalities, and X′ *,i Represents all rows of the i-th column of the input feature matrix, Z *,i is the intermediate feature representation of the i-th token after passing through the token mixer;

[0098] The output expression of the channel mixer is:

[0099] Y j,* =Z j,* +W4Φ(W3Norm(Z j,* ))

[0100] Among them, j represents the number of columns, Φ is the GELU activation function, Norm represents layer normalization, W3 and W4 are the weights of the linear layer, Z j,* Represents all columns of the jth row of the intermediate feature matrix Z, Y j,* It is the final feature representation of the jth channel after the channel mixer.

[0101] In another embodiment, the multimodal coordination network module (MCN) and the attribute extraction and masking module (AEM) belong to the text branch and are part of a three-branch network.

[0102] The fusion features and identity features are input into the fusion classifier and identity classifier respectively, and the association loss is obtained through the association reasoning mechanism. The specific idea is to establish the association relationship between identity features, clothing features and their influence paths based on the association graph, and calculate the total direct effect (TDE). The total direct effect value is used to adjust the loss function (association loss), as follows:

[0103] First, the association relationship between identity features, clothing features and their influence paths is established based on the association graph, and the total direct influence is calculated. The total direct influence is obtained by comparing the identity prediction D under the influence of clothing. A,C (B) Identity prediction D only affected by clothing C (B), that is, by calculating D A,C (B)-D C (B) to remove the interference of clothing features on identity recognition.

[0104] D A,C (B)-D C (B) Applied to logarithmic loss, the expression of output association loss is:

[0105]

[0106] Among them, y i is the label of the i-th sample, δ ID is the identity classifier, δ' ID is a fusion classifier that shares weights with the identity classifier, represents the bilinear pooling fusion operation, F ID is the identity feature, F V is the visual feature, FA is the text feature, and N is the number of samples in the training batch.

[0107] This reflects D A,C (B)-D C The idea of ​​(B) can eliminate the interference of clothing on identity features, and the association loss L CAL Constrained fusion features enable the model to automatically separate clothing interference during training and obtain more stable and unbiased identity features.

[0108] The present invention introduces a multimodal feature fusion method, combining text features extracted by the parallel text branch with visual features extracted by the identity branch and clothing branch to achieve identity recognition of pedestrians in different states. Through the attribute extraction and masking module, the pedestrian's text attributes are masked to reduce the interference of clothing information on identity recognition. The multimodal coordination network module is then used to perform feature fusion to enhance the discriminability of the model. The cross-camera tracking model obtained through the above training can significantly improve the efficiency and accuracy of pedestrian target re-identification in different fields of view, and can be specifically applied in:

[0109] A. Video surveillance in large public places: Utilizing the trained model of this invention, efficient correlation discrimination and behavioral analysis can be performed on pedestrian target identities within the surveillance network. In large public places such as airports, train stations, and shopping malls, the surveillance system can analyze pedestrian behavior in real time, identify potential security threats, and track targets, thereby improving the efficiency and accuracy of public safety management. Specific applications include detecting abnormal behavior, such as prolonged stops, wandering, or rapid running; lost and found identification, which helps locate owners by identifying pedestrians; and crowd monitoring, which monitors crowd density, identifies crowded areas, and guides evacuation to prevent stampedes.

[0110] B. Criminal Investigation: The trained model can replace human effort in identifying and locating target suspects, reducing case-solving time and improving efficiency. For example, during an investigation, surveillance video can identify and track a suspect's movements, helping police lock onto their tracks and quickly locate them. During evidence collection, the system can filter out relevant suspect activity from a large amount of surveillance video, providing strong evidentiary support. In cross-regional collaboration, the system can identify and track suspects across cities or regions, helping police across regions collaborate on solving cases.

[0111] C. Target Retrieval: The cross-camera tracking algorithm of this invention can also efficiently search for specified targets within massive video streams, and is widely used in scenarios such as intelligent security systems, business analytics, and finding missing persons. In places like residential communities and schools, the system can quickly identify specific individuals entering the venue, such as key surveillance personnel or unauthorized entrants. In commercial venues such as shopping malls, cross-camera tracking technology can be used to analyze customer behavior, understand their spending habits and preferences, and provide decision-making support for businesses. When searching for missing persons, by inputting their characteristic information, the system can quickly retrieve and locate the missing person, helping family members and police to find the missing person promptly.

[0112] Example 2

[0113] Based on Example 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the cross-mirror tracking method based on association reasoning as described above.

[0114] At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above-mentioned cross-mirror tracking method based on associative reasoning. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0115] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0116] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0117] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A cross-camera tracking method based on associative reasoning, characterized in that: The method comprises: Images captured by multiple cameras are input into a pre-trained cross-camera tracking model to predict pedestrian identities. The cross-camera tracking model includes a three-branch network, an auxiliary classifier, a clothing classifier, a multimodal coordination module, a fusion classifier, and an identity classifier. The process of predicting pedestrian identity by the cross-camera tracking model includes: using a three-branch network to extract identity features, clothing features and text features from multi-camera pedestrian images; using an auxiliary classifier to further separate identity features and clothing features, and using a clothing classifier to optimize clothing features; fusing the identity features and clothing features through bilinear pooling to obtain visual features, and using multimodal coordination module features to fuse the visual features and text features to obtain fused features; using an identity classifier and a fusion classifier to process the identity features and fused features respectively to obtain pedestrian identity prediction results.

2. The cross-camera tracking method based on associative reasoning according to claim 1, characterized in that: The three-branch network includes an identity branch, a clothing branch, and a parallel text branch; wherein the identity features are extracted by a pre-trained ResNet network in the identity branch, the clothing features are extracted by a pyramid matching strategy in the clothing branch; and the text features are extracted by an attribute extraction and mask module in the parallel text branch; The process of extracting text features by the attribute extraction and masking module includes extracting a fixed attribute list based on a pre-trained SOLIDER model and converting it into a 0-1 binary vector, using a predefined attribute template to mask clothing-related attributes to obtain text features without clothing interference.

3. The cross-camera tracking method based on associative reasoning according to claim 1, characterized in that: The process of training the cross-camera tracking model includes: Obtain all images taken by multiple cameras and add identity tags to the images to form a labeled sample set; Build a cross-camera tracking model, using a three-branch network to extract identity features, clothing features, and text features from multi-camera pedestrian images, and calculate the minimum recognition loss of the identity branch and the clothing classification loss of the clothing branch; Input the identity features and clothing features into the auxiliary classifier and the clothing classifier, and calculate the KL divergence loss; fusing the identity features and clothing features through bilinear pooling to obtain visual features, and fusing the visual features and text features using multimodal coordination module features to obtain fused features; The fusion features and identity features are input into the fusion classifier and identity classifier respectively, and the association loss is obtained through the association reasoning mechanism; The recognition loss, clothing classification loss, KL divergence loss and association loss are comprehensively minimized, and the parameters of the cross-camera tracking model are updated through loss backpropagation, and the cycle is iterated until convergence.

4. The cross-camera tracking method based on associative reasoning according to claim 3 is characterized in that: The expression for minimizing the recognition loss is: Among them, L ID To minimize the recognition loss, y i represents the label of the i-th sample, P ID (y i |x i ) represents the identity classifier δ ID For the i-th image x i The predicted probability of , N represents the number of samples in the training batch; The expression of the clothing classification loss is: Among them, L C represents the clothing classification loss, represents the label of the i-th sample, represents the clothing classifier δ C For the i-th image x i The predicted probability, N C Indicates the number of samples for clothing classification.

5. The cross-camera tracking method based on associative reasoning according to claim 3 is characterized in that: The expression of the KL divergence loss is: in, F C to F ID The KL distance, F ID to F C KL distance; Among them, F C is the clothing feature, F ID For identity characteristics, Indicates that the clothing classifier δ C From clothing features F C The probability distribution obtained is, Indicates that the auxiliary classifier δ A From the identity feature F ID The probability distribution obtained, M represents the number of samples in the mini-batch.

6. The cross-camera tracking method based on associative reasoning according to claim 3 is characterized in that: The expression of the association loss is: Among them, y i is the label of the i-th sample, δ ID is the identity classifier, δ ' ID is a fusion classifier that shares weights with the identity classifier, represents the bilinear pooling fusion operation, F ID is the identity feature, F V is the visual feature, F A is the text feature, and N is the number of samples in the training batch.

7. The cross-camera tracking method based on associative reasoning according to claim 3 is characterized in that: The multimodal coordination module includes a token mixer and a channel mixer.

8. The cross-camera tracking method based on associative reasoning according to claim 7 is characterized in that: The token mixer fuses the information of different time series of the visual features and text features. The fusion expression is: Z *,i =X′ *,i +W2Φ(W1Norm(X′ *,i )) Where i represents the number of rows, Φ is the GELU activation function, Norm represents layer normalization, W1 and W2 are the weights of the linear layer, X is the input feature, and X′ *,i To represent all rows of the i-th column of the input feature matrix, Z *,i is the intermediate feature representation of the i-th token after passing through the token mixer.

9. The cross-camera tracking method based on associative reasoning according to claim 7, characterized in that: The channel mixer fuses the information of different feature dimensions of the visual features and text features. The fusion expression is: Y j,* =Z j,* +W4Φ(W3Norm(Z j,* )) Among them, j represents the number of columns, Φ is the GELU activation function, Norm represents layer normalization, W3 and W4 are the weights of the linear layer, Z j,* Represents all columns of the jth row of the intermediate feature matrix Z, Y j,* It is the final feature representation of the jth channel after the channel mixer.

10. The cross-camera tracking method based on associative reasoning according to claim 1, characterized in that: The SOLIDER model used to extract text features in the parallel text branch is a pre-trained model and is in a frozen state during the training of the cross-camera tracking model.

Images