Cross-modal pedestrian re-identification method based on multi-scale cross attention Transform

Abstract

The invention discloses a cross-modal pedestrian re-identification method based on a multi-scale cross attention Transform, and the method comprises the following steps: step 1, image acquisition, step 2, feature set acquisition, step 3, feature fusion, and step 4, feature determination: in an initial stage of a network, two standard data sets are used to acquire RGB and infrared images, and then, the RGB and infrared images are subjected to feature fusion; the method comprises the following steps: extracting key features from RGB and an infrared image by using MSCAViT to obtain two feature sets, then converting the RGB image into a grayscale image, inputting the grayscale image into an MSCAViT model to obtain a third feature set, determining an optimal weight by using an OSCSO method through weighted fusion of the three feature sets, finally obtaining a fused feature, inputting a result feature into a decoding end of the MSCAViT, and finally obtaining a final image through the decoding end of the MSCAViT. According to the method, features finally used for image matching are generated, the model is evaluated through multiple indexes, compared with a traditional method, and the result shows that the proposed method shows a more excellent effect on solving the modal difference problem in a VI Re-ID task.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal pedestrian recognition, and specifically to a cross-modal pedestrian re-identification method based on a multi-scale cross-attention Transformer. Background Art

[0002] Cross-modal pedestrian recognition technology mainly focuses on the ability to identify the same pedestrian between visible light and infrared images. This technology is particularly important in nighttime surveillance, because criminals who commit crimes at night are often difficult to be effectively captured by visible light cameras. By using infrared cameras to capture the appearance of pedestrians, the efficiency of solving cases can be greatly improved, which has very important practical significance for public safety and criminal investigation. In addition, with the advancement of science and technology, a large number of infrared cameras have been used in video surveillance systems, which not only strengthens social management and prevents the occurrence of criminal acts, but also shows broad application prospects in maintaining national security. The challenges of pedestrian recognition mainly include the large differences between images captured in the two modalities. For example, RGB images have three channels, including visible light color information of red, green and blue, while infrared images have only one channel, including intensity information of near-infrared light. In addition, different clarity and lighting conditions may produce significant differences in the two types of images. Despite these challenges, cross-modal pedestrian recognition technology uses deep learning and other methods, such as attention mechanism, collaborative learning, intermediate domain modality and other technical means to solve the huge color differences, different styles and insufficient network constraints of cross-modal pedestrian images, thereby improving the accuracy of existing cross-modal pedestrian re-identification models.

[0003] The main purpose of VI Re-ID is to solve the problem of cross-modal pedestrian recognition, that is, pedestrian recognition and matching between visible light images and infrared images, which is very important in practical applications because monitoring systems usually capture different types of image data at different time periods, such as day and night. Through VI Re-ID, all-weather monitoring can be achieved, improving the reliability of pedestrian tracking and safety monitoring.

[0004] However, traditional pedestrian tracking has the following disadvantages:

[0005] There are significant differences in visual features between visible light images and infrared images, which makes cross-modal feature matching very difficult. Changes in lighting, weather, and environment will affect image quality and consistency, and also increase the difficulty of recognition. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-modal pedestrian re-identification method based on a multi-scale cross-attention Transformer to solve the problem that there are significant differences in visual features between visible light images and infrared images proposed in the above background technology, which makes cross-modal feature matching very difficult. Changes in lighting, weather and environment will affect image quality and consistency, and also increase the difficulty of recognition.

[0007] To achieve the above object, the present invention provides the following technical solution: a cross-modal person re-identification method based on a multi-scale cross-attention Transformer, comprising the following steps:

[0008] Step 1: Image acquisition: In the initial stage of the network, the image is classified through Vision Transformer, and two standard datasets are used to obtain RGB and infrared images;

[0009] Step 2: Feature set acquisition: Use MSCAViT to extract key features from RGB and infrared images to obtain two feature sets, convert the RGB image into a grayscale image, and input it into the MSCAViT model to obtain the third feature set;

[0010] Step 3: Feature fusion: Through the weighted fusion of the above three feature sets, the OSCSO method is used to form the mathematical model of OSCSO to determine the optimal weight, and finally the fusion feature is obtained;

[0011] Step 4: Feature determination: Input the resulting features into the decoding end of MSCAViT to generate the final features used for image matching.

[0012] As a preferred technical solution of the present invention, the specific process of image classification by Vision Transformer in step 1 is as follows: S1, image segmentation: first, the input image is segmented into image blocks of fixed size, each image block is of size PXP pixels, and each image block is flattened into a vector. For an image of size HXWXC, it is segmented into image blocks, the size of the flattened vector of each image block is, and the flattened vector of each image block is mapped to a fixed dimension D using a linear layer to obtain the sequence representation of NXD; S2. Fixed image position: ViT uses a position encoding similar to Transformer to add fixed position information to the vector representation of each image block; S3. Image classification processing: An additional classification tag vector is introduced at the beginning of the sequence, and after passing through multiple layers of encoders, the final representation of the classification tag is used for classification.

[0013] As a preferred technical solution of the present invention, in the step 1, a multi-head self-attention mechanism and a feedforward neural network are provided in the Vision Transformer. The multi-head self-attention performs multi-head self-attention calculation on the input sequence, and each head calculates different attention weights, thereby capturing different contextual information. When calculating the self-attention, the input sequence is mapped to query, key and value vectors, and the weighted sum is calculated. The feedforward neural network applies two linear transformations after each attention head, uses a nonlinear activation function ReLU in the middle, and uses residual connections and layer normalization to stabilize the training. The input of ViT is represented by t0, and the expression of the pth block is as follows: t0=[t cls ||t patch ]+t pos , k p =t p-1 +MHSA(LN(t p-1 )), t p =k p +FFN(LN(k p )), where t cls is the classification mark, t patch is the image patch vector, t pos is the position code, t p-1 is the output of the previous layer, LN(·) represents layer normalization, MHSA(·) represents the multi-head self-attention mechanism, and FFN(·) represents the feedforward neural network. Through these steps, the ViT model extracts the global information of the image layer by layer, and finally, the classification task is performed through the representation of the classification label.

[0014] As a preferred technical solution of the present invention, in the step 2, MSCAViT is composed of T multi-scale transformer encoders, wherein each encoder is composed of two branches, the two branches include a B branch and an S branch, the B branch is a large-scale branch, using a coarse-grained patch size, which contains more encoders, the S branch is a small-scale branch, using a fine-grained patch size, which contains fewer encoders, the two branches are fused K times through cross-attention, and finally the CLS tokens of the two branches are obtained as the output features of the encoder, and the cls tokens of each branch are added with position encoding for learning the position information in ViT.

[0015] As a preferred technical solution of the present invention, the input of the MSCAViT model in step 2 is specifically to use the CLS token of each branch as an agent to exchange information between image block vectors from other branches, and then return it to its own branch. Since the CLS token has learned abstract information in all image block vectors in its own branch, the interaction with image block vectors in other branches helps to contain information at different scales. After fusing with other branch tokens, the CLS token interacts with its own image block vector again on the next encoder, passing the information learned from other branches to its own image block vector to enrich the representation of each image block vector. Specifically, for the B branch, first collect the image block vectors from the S branch, and connect its own CLS token to the image block vector. The formula is as follows: Where F(·) is the dimension-aligned projection function, and then the module is and t' b Cross attention is performed between them. Since the image block feature information has been fused into the cls token, the CLS token is used as the query. Mathematically, the CA expression formula is: In the formula k=t' b W k , W q , W k , W v , are learnable parameters, D and e are the embedding dimensions and number of heads. In addition, as with self-attention, multiple heads are used in CA and denoted as MHCA. Finally, for a given t' b Perform layer normalization and residual connection, the formula is as follows: Among them, G b (·) are dimensionally aligned back-projection functions, respectively. Therefore, small-scale image patches capture more detail information, while large-scale image patches focus more on capturing global features. The multi-scale cross-attention mechanism combines features of different scales to enable the model to focus on both details and global information, thereby improving the ability to understand image content.

[0016] As a preferred technical solution of the present invention, the OSCSO method in step three is specifically an OSCSO algorithm based on the existing SCSO, in which sand cats are regarded as input populations, sand cats enter the search area to find prey, and each search agent searches for these prey. The advantage of SCSO is that it provides better local search solutions and faster convergence speed.

[0017] As a preferred technical solution of the present invention, the mathematical model preparation process of OSCSO in step three is specifically as follows: S1, initialization: set the number of sand cats to N, the maximum number of iterations to M, and these sand cats are randomly initialized in the search area. In addition, the fitness cost of each sand cat is obtained by evaluating the defined fitness function. At the end of each iteration, the sand cat with the best cost is selected as the current best solution; S2, search for prey: the prey search mechanism of the sand cat depends on the emission of low-frequency noise. The OSCSO algorithm utilizes the auditory ability of the sand cat in low-frequency detection. The sensitivity range G decreases linearly from 2 to 0, simulating the sensitivity of the sand cat to low frequencies below 2KHz. For example, if the maximum number of iterations is 100, the value of G will be greater than 1 in the first half of the iterations, and will be less than 1 in the last 50 iterations. Among them, H F The value is inspired by the auditory characteristics of sand cats and is set to 2. t is the current number of iterations and T is the maximum number of iterations to control the balance between the exploration and development phases. The L parameter is added. Through this adaptive strategy, the transition between the two phases will be more balanced. L = 2xGxrand (0, 1) - G. In order to avoid falling into the local optimal trap, the sensitivity range of each sand cat is different. r = Gxrand (0, 1). Each search agent sand cat is based on the best candidate position x' best 、Current position(x t ) and sensitivity range (r) to update its own position so that the sand cat can find other best prey locations. S3. Attacking the prey: Modeling the attack phase of the OSCSO algorithm to represent the optimal position (x best ) and the current position (x t ), the position update formula at this stage is as follows: Among them, rand(0,1) represents any random number between [0,1]. Assuming that the sensitivity range of the sand cat is a circle, the direction of movement is determined by the random angle (θ) on the circle. Since the selected random angle is between 0 and 360, its value will be between -1 and 1. In this way, each member of the population moves in different circular directions in the search space; S4. Exploration and development: Exploration and development are switched through adaptive G and L parameters. Since the L parameter depends on G, its fluctuation range will also decrease. When the value of the G parameter remains uniformly distributed, the value of L will also remain balanced. In other words, the value of L is a random value within [-2G, 2G], where G decreases linearly from 2 to 0 during the iteration process. When the value of L is in [-1,1], the sand cat is guided to attack the prey, otherwise, the cat is tasked to find a new solution in the global area, that is: A traditional position update is generated, which is enhanced by the optimal value as shown below: FP = x t+1*BtSn, where BtSn represents the optimal weight factor used to adjust the position update, the balancing behavior of the proposed algorithm and the effort to find other local regions in the global space have a fast and accurate convergence rate, thus showing good performance in high-dimensional and multi-objective problems.

[0018] As a preferred technical solution of the present invention, the three feature sets in step 2 are specifically: and The image matching features in step 4 are specifically S1, weighted fusion features: the network analyzes three sets of features during training and testing, which brings about the problem of too high dimension, which often leads to performance degradation, increased computational burden and increased network time consumption. Therefore, the OSCSO algorithm is used to determine the weight value, and the three sets of features are spliced ​​using the optimal weighting factor. The mathematical formula for weighted feature fusion is as follows: In the formula, w1 and w2 are the best weights obtained from the OSCSO algorithm. In order to find the best weight value, the objective function formula is set: In this formula, by adjusting the values ​​of w1 and w2, the CrCt correlation coefficient is maximized to achieve the best feature fusion effect. CrCt is used to measure the similarity between fused features. It represents the correlation between features. The value range is between -1 and 1. The closer the value is to 1, the stronger the linear correlation between the two features. The formula is as follows: Among them, h and g represent two sets of features, and Represent the mean of features h and g respectively. By maximizing CrCt, it is ensured that the fused features retain the most useful information and reduce redundancy, thereby improving the effect and performance of feature fusion. S2. Person re-identification using MSCAViT: In the MSCAViT architecture model, the encoder extracts features, then fuses them using the optimal weights, and finally passes them to the decoding layer of MSCAViT. The features obtained are represented by two modes, namely and The model mainly uses infrared images to identify the identity in visible light images. Therefore, the recognition formula of the decoding layer is as follows: I =p I w q , Among them, the prototype and query of infrared image are represented by p I and q I In addition, the visible light related features are marked as F R ', k R and v R Represents the key and value respectively, Finally, the complete visible light and infrared image features for pedestrian recognition are obtained, and the formula is as follows: S3. Loss function: In cross-modal person re-identification, there are significant differences between different modalities such as RGB images and infrared images. These differences will lead to inconsistency in feature space and affect matching performance. Heterogeneous center loss reduces the feature differences of the same identity under different modalities and enhances the consistency of cross-modal features: in, and represent the feature vectors of the i-th sample in visible light and infrared modes, respectively, c v and c I Represent the centers of the visible light and infrared modal feature vectors respectively, N is the number of samples, λ is a hyperparameter used to balance the weights of different loss terms, ||·||2 represents the L2 norm, and the heterogeneous center loss makes the intra-modal features more concentrated by minimizing the distance from each sample feature to the corresponding modality center. By minimizing the distance between the feature centers of different modalities, the distribution differences between different modalities are reduced, making cross-modal matching more effective. The ID loss ensures that the model accurately distinguishes different identities and improves the recognition accuracy by providing a strong supervision signal: In the formula, p i,yi Indicates that the i-th sample is predicted to be its true identity y i The probability of y i is the true identity label of the i-th sample. The ID loss improves the model’s ability to distinguish pedestrian identities by minimizing the difference between the predicted distribution and the actual identity distribution. Therefore, the combination of heterogeneous center loss and ID loss enables MSCAViT to effectively solve the two key problems of cross-modal feature distribution differences and pedestrian identity distinction. The combination of the two enables the comprehensive optimization of feature alignment and identity distinction, significantly improving the overall performance and robustness of the re-identification task, as shown in the following formula: L = L hc +L id .

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: in the initial stage of the network, two standard data sets are used to obtain RGB and infrared images, then, key features are extracted from the RGB and infrared images using MSCAViT to obtain two feature sets, then, the RGB images are converted into grayscale images and input into the MSCAViT model to obtain the third feature set, through the weighted fusion of the three feature sets, the OSCSO method is used to determine the optimal weights, and finally the fused features are obtained, the resulting features are input into the decoding end of MSCAViT to generate the final features for image matching, the model is evaluated by a variety of indicators and compared with traditional methods, and the results show that the proposed method shows a superior effect in solving the modal difference problem in the VI Re-ID task. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1is a flow chart of the present invention;

[0021] Figure 2 This is a flowchart of image classification by Vision Transformer of the present invention;

[0022] Figure 3 A flow chart prepared for the mathematical model of OSCSO of the present invention;

[0023] Figure 4 A flow chart showing the features of image matching of the present invention;

[0024] Figure 5 A network diagram for deep learning of the present invention;

[0025] Figure 6 Schematic diagram of the architecture of the Transformer encoder of the present invention;

[0026] Figure 7 Schematic diagram of the architecture of MSCAViT for person re-identification of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] See also Figure 1-7 The present invention provides a cross-modal person re-identification method based on a multi-scale cross-attention Transformer, comprising the following steps:

[0029] Step 1: Image acquisition: In the initial stage of the network, the image is classified through Vision Transformer, and two standard datasets are used to obtain RGB and infrared images;

[0030] Step 2: Feature set acquisition: Use MSCAViT to extract key features from RGB and infrared images to obtain two feature sets, convert the RGB image into a grayscale image, and input it into the MSCAViT model to obtain the third feature set;

[0031] Step 3: Feature fusion: Through the weighted fusion of the above three feature sets, the OSCSO method is used to form the mathematical model of OSCSO to determine the optimal weight, and finally the fusion feature is obtained;

[0032] Step 4: Feature determination: Input the resulting features into the decoding end of MSCAViT to generate the final features used for image matching.

[0033] In step 1, the specific process of image classification by Vision Transformer is as follows: S1. Image segmentation: First, the input image is segmented into image blocks of fixed size, each image block is of size PXP pixels, and each image block is flattened into a vector. For an image of size HXWXC, it is segmented into image blocks, the size of the flattened vector of each image block is, and the flattened vector of each image block is mapped to a fixed dimension D using a linear layer to obtain the sequence representation of NXD; S2. Fixed image position: ViT uses a position encoding similar to Transformer to add fixed position information to the vector representation of each image block; S3. Image classification processing: An additional classification tag vector is introduced at the beginning of the sequence, and after passing through multiple layers of encoders, the final representation of the classification tag is used for classification.

[0034] In step 1, the Vision Transformer is equipped with a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention performs multi-head self-attention calculations on the input sequence. Each head calculates different attention weights to capture different contextual information. When calculating self-attention, the input sequence is mapped to query, key, and value vectors, and the weighted sum is calculated. The feedforward neural network applies two linear transformations after each attention head, and uses the nonlinear activation function ReLU in the middle. Residual connections and layer normalization are used to stabilize training. The input of ViT is represented by t0, and the expression of the pth block is as follows: t0 = [t cls ||t patch ]+t pos , k p =t p-1 +MHSA(LN(t p-1 )), t p =k p +FFN(LN(k p )), where t cls is the classification mark, t patch is the image patch vector, t pos is the position code, t p-1 is the output of the previous layer, LN(·) represents layer normalization, MHSA(·) represents the multi-head self-attention mechanism, and FFN(·) represents the feedforward neural network. Through these steps, the ViT model extracts the global information of the image layer by layer, and finally, the classification task is performed through the representation of the classification label.

[0035] In step 2, MSCAViT consists of T multi-scale transformer encoders, each of which consists of two branches, including B branch and S branch. B branch is a large-scale branch, using coarse-grained patch size, which contains more encoders, and S branch is a small-scale branch, using fine-grained patch size, which contains fewer encoders. The two branches are fused K times through cross attention, and finally the CLS tokens of the two branches are obtained as the output features of the encoder, and the cls tokens of each branch are added with position encoding to learn the position information in ViT.

[0036] The input MSCAViT model in step 2 specifically uses the CLS token of each branch as an agent to exchange information between image block vectors from other branches, and then returns it to its own branch. Since the CLS token has learned abstract information in all image block vectors in its own branch, the interaction with image block vectors in other branches helps to contain information at different scales. After fusing with other branch tokens, the CLS token interacts with its own image block vector again on the next encoder, passing the information learned from other branches to its own image block vector to enrich the representation of each image block vector. Specifically, for the B branch, first collect the image block vectors from the S branch and connect its own CLS token to the image block vector. The formula is as follows: Where F(·) is the dimension-aligned projection function, and then the module is and t' b Cross attention is performed between them. Since the image block feature information has been fused into the cls token, the CLS token is used as the query. Mathematically, the CA expression formula is: In the formula k=t' b W k , W q , W k , W v , are learning parameters, D and e are the embedding dimensions and number of heads. In addition, as with self-attention, multiple heads are used in CA and denoted as MHCA. Finally, for a given t' b Perform layer normalization and residual connection, the formula is as follows:

[0037] Among them, G b (·) are dimensionally aligned back-projection functions, respectively. Therefore, small-scale image patches capture more detail information, while large-scale image patches focus more on capturing global features. The multi-scale cross-attention mechanism combines features of different scales to enable the model to focus on both details and global information, thereby improving the ability to understand image content.

[0038] The OSCSO method in step 3 is specifically that the OSCSO algorithm is based on the existing SCSO, in which sand cats are regarded as input populations, sand cats enter the search area to find prey, and each search agent searches for these prey. The advantage of SCSO is that it provides better local search solutions and faster convergence speed.

[0039] The mathematical model preparation process of OSCSO in step 3 is as follows: S1. Initialization: Set the number of sand cats to N and the maximum number of iterations to M. These sand cats are randomly initialized in the search area. In addition, the fitness cost of each sand cat is obtained by evaluating the defined fitness function. At the end of each iteration, the sand cat with the best cost is selected as the current best solution; S2. Search for prey: The prey search mechanism of the sand cat relies on the emission of low-frequency noise. The OSCSO algorithm uses the auditory ability of the sand cat in low-frequency detection. The sensitivity range G decreases linearly from 2 to 0, simulating the sensitivity of the sand cat to low frequencies below 2KHz. For example, if the maximum number of iterations is 100, the value of G will be greater than 1 in the first half of the iterations, and will be less than 1 in the last 50 iterations. Among them, H F The value is inspired by the auditory characteristics of sand cats and is set to 2. t is the current number of iterations and T is the maximum number of iterations to control the balance between the exploration and development phases. The L parameter is added. Through this adaptive strategy, the transition between the two phases will be more balanced. L = 2xGxrand (0, 1) - G. In order to avoid falling into the local optimal trap, the sensitivity range of each sand cat is different. r = Gxrand (0, 1). Each search agent sand cat is based on the best candidate position x' best 、Current position(x t ) and sensitivity range (r) to update its own position so that the sand cat can find other best prey locations. S3. Attacking the prey: Modeling the attack phase of the OSCSO algorithm to represent the optimal position (x best ) and the current position (x t ), the position update formula at this stage is as follows: Among them, rand(0,1) represents any random number between [0,1]. Assuming that the sensitivity range of the sand cat is a circle, the direction of movement is determined by the random angle (θ) on the circle. Since the selected random angle is between 0 and 360, its value will be between -1 and 1. In this way, each member of the population moves in different circular directions in the search space; S4. Exploration and development: Exploration and development are switched through adaptive G and L parameters. Since the L parameter depends on G, its fluctuation range will also decrease. When the value of the G parameter remains uniformly distributed, the value of L will also remain balanced. In other words, the value of L is a random value within [-2G, 2G], where G decreases linearly from 2 to 0 during the iteration process. When the value of L is in [-1,1], the sand cat is guided to attack the prey, otherwise, the cat is tasked to find a new solution in the global area, that is: A traditional position update is generated, which is enhanced by the optimal value as shown below: FP = x t+1 *BtSn, where BtSn represents the optimal weight factor used to adjust the position update, the balancing behavior of the proposed algorithm and the effort to find other local regions in the global space have a fast and accurate convergence rate, thus showing good performance in high-dimensional and multi-objective problems.

[0040] The three feature sets in step 2 are: and The features of image matching in step 4 are specifically S1, weighted fusion features: the network analyzes three sets of features during training and testing, which brings about the problem of too high dimension, which often leads to performance degradation, increased computational burden and increased network time consumption. Therefore, the OSCSO algorithm is used to determine the weight value, and the three sets of features are spliced ​​using the optimal weighting factor. The mathematical formula for weighted feature fusion is as follows: In the formula, w1 and w2 are the best weights obtained from the OSCSO algorithm. In order to find the best weight value, the objective function formula is set: In this formula, by adjusting the values ​​of w1 and w2, the CrCt correlation coefficient is maximized to achieve the best feature fusion effect. CrCt is used to measure the similarity between fused features. It represents the correlation between features. The value range is between -1 and 1. The closer the value is to 1, the stronger the linear correlation between the two features. The formula is as follows: Among them, h and g represent two sets of features, and Represent the mean of features h and g respectively. By maximizing CrCt, it is ensured that the fused features retain the most useful information and reduce redundancy, thereby improving the effect and performance of feature fusion. S2. Person re-identification using MSCAViT: In the MSCAViT architecture model, the encoder extracts features, then fuses them using the optimal weights, and finally passes them to the decoding layer of MSCAViT. The features obtained are represented by two modes, namely and The model mainly uses infrared images to identify the identity in visible light images. Therefore, the recognition formula of the decoding layer is as follows: I =p I w q , Among them, the prototype and query of infrared image are represented by p I and q I In addition, the visible light related features are marked as F R ', k R and v R Represents the key and value respectively, Finally, the complete visible light and infrared image features for pedestrian recognition are obtained, and the formula is as follows: S3. Loss function: In cross-modal person re-identification, there are significant differences between different modalities such as RGB images and infrared images. These differences will lead to inconsistency in feature space and affect matching performance. Heterogeneous center loss reduces the feature differences of the same identity under different modalities and enhances the consistency of cross-modal features: in, and represent the feature vectors of the i-th sample in visible light and infrared modes, respectively, c v and c I Represent the centers of the visible light and infrared modal feature vectors respectively, N is the number of samples, λ is a hyperparameter used to balance the weights of different loss terms, ||·||2 represents the L2 norm, and the heterogeneous center loss makes the intra-modal features more concentrated by minimizing the distance from each sample feature to the corresponding modality center. By minimizing the distance between the feature centers of different modalities, the distribution differences between different modalities are reduced, making cross-modal matching more effective. The ID loss ensures that the model accurately distinguishes different identities and improves the recognition accuracy by providing a strong supervision signal: In the formula, p i,yi Indicates that the i-th sample is predicted to be its true identity y i The probability of y iis the true identity label of the i-th sample. The ID loss improves the model’s ability to distinguish pedestrian identities by minimizing the difference between the predicted distribution and the actual identity distribution. Therefore, the combination of heterogeneous center loss and ID loss enables MSCAViT to effectively solve the two key problems of cross-modal feature distribution differences and pedestrian identity distinction. The combination of the two enables the comprehensive optimization of feature alignment and identity distinction, significantly improving the overall performance and robustness of the re-identification task, as shown in the following formula: L = L hc +L id .

[0041] In the present invention, at the initial stage of the network, the image is classified by Vision Transformer, and two standard data sets are used to obtain RGB and infrared images; the key features are extracted from the RGB and infrared images by MSCAViT to obtain two feature sets, the RGB images are converted into grayscale images, and input into the MSCAViT model to obtain the third feature set; through the weighted fusion of the above three feature sets, the OSCSO method is used to form the mathematical model of OSCSO to determine the optimal weight, and finally the fusion feature is obtained; the result feature is input into the decoding end of MSCAViT to generate the feature finally used for image matching, and the fusion feature is finally obtained by combining the appendix of the specification. Figure 5 ,Compared with the traditional method, the results show that the proposed method has a better effect in solving the modality difference problem in VI Re-ID tasks. Figure 6 , the proposed OSCSO pseudo code is as follows:

[0042]

[0043]

[0044] In conjunction with the instruction manual Figure 7 ,Schematic diagram of weighted fusion.

[0045] Detailed description of the invention Results

[0046] This implementation uses the publicly available visible light-infrared cross-modal pedestrian re-identification dataset SYSU-MM01. The details of the dataset are as follows:

[0047] The SYSU-MM01 dataset was released by a research team from Sun Yat-sen University to solve the problem of pedestrian re-identification between visible light and infrared images. The dataset contains images of pedestrians captured by six different cameras, involving a variety of challenging conditions such as perspective conversion, light changes, occlusion, and background interference. The images cover a variety of indoor and outdoor scenes, such as porches, corridors, and indoor halls. The database contains images of each pedestrian in both visible light and infrared modes.

[0048] In order to verify the superiority of this implementation (Ours), this implementation is compared with several existing pedestrian re-identification methods between visible light and infrared images, including AGW, DDAG, NFS and other methods. The mean average accuracy (mAP) and hit rate (Rank-n) of these methods on the above public datasets will be compared. The specific data comparison is shown in Table 1.

[0049] Table 1 Accuracy of visible light and infrared image datasets (%)

[0050]

[0051] The comparison results on the SYSU-MM01 dataset are shown in Table 1. The proposed method outperforms recent CNN-based and Transformer-based methods. Compared with CM-PReID in global search mode, the results of our method in R1, R10, and mAP are 69.44%, 97.32%, and 65.23%, respectively, which are 2.19%, 1.94%, and 0.94% higher than CM-PReID. In global search mode, our method is 12.53%, 12.93%, and 9.78% higher than NFS, which automatically selects the best features using neural feature search, in R1, R10, and mAP, respectively. In addition, compared with Transformer-based methods such as SPOT and CPA, our method leads by 4.1%, 4.59%, and 2.98% and 10.68%, 4.15%, and 7.56% in R1, R10, and mAP, respectively, in full search mode. Extensive experiments show that our method outperforms existing methods.

[0052] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-modal person re-identification method based on multi-scale cross-attention Transformer, characterized by: The following steps are involved: Step 1: Image acquisition: In the initial stage of the network, the image is classified through Vision Transformer, and two standard datasets are used to obtain RGB and infrared images; Step 2: Feature set acquisition: Use MSCAViT to extract key features from RGB and infrared images to obtain two feature sets, convert the RGB image into a grayscale image, and input it into the MSCAViT model to obtain the third feature set; Step 3: Feature fusion: Through the weighted fusion of the above three feature sets, the OSCSO method is used to form the mathematical model of OSCSO to determine the optimal weight, and finally the fusion feature is obtained; Step 4: Feature determination: Input the resulting features into the decoding end of MSCAViT to generate the final features used for image matching.

2. The cross-modal person re-identification method based on multi-scale cross-attention Transformer according to claim 1, characterized in that: The specific process of image classification by Vision Transformer in step 1 is as follows: S1. Image segmentation: First, the input image is segmented into image blocks of fixed size, each image block is of size PXP pixels, and each image block is flattened into a vector. For an image of size H×W×C, it is segmented into image blocks, the size of the flattened vector of each image block is, and the flattened vector of each image block is mapped to a fixed dimension D using a linear layer to obtain the sequence representation of NXD; S2. Fixed image position: ViT uses a position encoding similar to Transformer to add fixed position information to the vector representation of each image block; S3. Image classification processing: An additional classification tag vector is introduced at the beginning of the sequence, and after passing through multiple layers of encoders, the final representation of the classification tag is used for classification. In the step 1, a multi-head self-attention mechanism and a feedforward neural network are provided in the Vision Transformer. The multi-head self-attention performs multi-head self-attention calculation on the input sequence. Each head calculates different attention weights, thereby capturing different contextual information. When calculating self-attention, the input sequence is mapped to query, key and value vectors, and the weighted sum is calculated. The feedforward neural network applies two linear transformations after each attention head, uses a nonlinear activation function ReLU in the middle, and uses residual connections and layer normalization to stabilize training. The input of ViT is represented by t0, and the expression of the pth block is as follows: t0=[t cls ||t patch ]+t pos , k p =t p-1 +MHSA(LN(t p-1 )), t p =k p +FFN(LN(k p )), where t cls is the classification mark, t patch is the image patch vector, t pos is the position code, t p-1 is the output of the previous layer, LN(·) represents layer normalization, MHSA(·) represents the multi-head self-attention mechanism, and FFN(·) represents the feedforward neural network. Through these steps, the ViT model extracts the global information of the image layer by layer, and finally, the classification task is performed through the representation of the classification label.

3. The cross-modal person re-identification method based on multi-scale cross-attention Transformer according to claim 1, characterized in that: In the step 2, MSCAViT is composed of T multi-scale transformer encoders, each of which is composed of two branches, including a B branch and an S branch. The B branch is a large-scale branch using a coarse-grained patch size and contains more encoders. The S branch is a small-scale branch using a fine-grained patch size and contains fewer encoders. The two branches are fused K times through cross-attention, and finally the CLS tokens of the two branches are obtained as the output features of the encoder, and the cls tokens of each branch are added with position encoding for learning the position information in ViT. The input of the MSCAViT model in step 2 is specifically to use the CLS token of each branch as an agent to exchange information between the image block vectors from other branches, and then return it to its own branch. Since the CLS token has learned abstract information in all the image block vectors in its own branch, the interaction with the image block vectors in other branches helps to contain information at different scales. After fusing with the other branch tokens, the CLS token interacts with its own image block vector again on the next encoder, passing the information learned from other branches to its own image block vector to enrich the representation of each image block vector. Specifically, for the B branch, first collect the image block vectors from the S branch and connect its own CLS token to the image block vector. The formula is as follows: Where F(·) is the dimension-aligned projection function, and then the module is and t' b Cross attention is performed between them. Since the image block feature information has been fused into the cls token, the CLS token is used as the query. Mathematically, the CA expression formula is: In the formula k=t' b W k , W q , W k , W v , are learnable parameters, D and e are the embedding dimensions and number of heads. In addition, as with self-attention, multiple heads are used in CA and denoted as MHCA. Finally, for a given t' b Perform layer normalization and residual connection, the formula is as follows: Among them, G b (·) are dimensionally aligned back-projection functions, respectively. Therefore, small-scale image patches capture more detail information, while large-scale image patches focus more on capturing global features. The multi-scale cross-attention mechanism combines features of different scales to enable the model to focus on both details and global information, thereby improving the ability to understand image content.

4. The cross-modal person re-identification method based on multi-scale cross-attention Transformer according to claim 1, characterized in that: The OSCSO method in step 3 is specifically that the OSCSO algorithm is based on the existing SCSO, in which sand cats are regarded as input populations, sand cats enter the search area to find prey, and each search agent searches for these prey. The advantage of OSCSO is that it provides better local search solutions and faster convergence speed. The mathematical model preparation process of OSCSO in step 3 is specifically as follows: S1, initialization: set the number of sand cats to N, the maximum number of iterations to M, and these sand cats are randomly initialized in the search area. In addition, the fitness cost of each sand cat is obtained by evaluating the defined fitness function. At the end of each iteration, the sand cat with the best cost is selected as the current best solution; S2, search for prey: the prey search mechanism of the sand cat depends on the emission of low-frequency noise. The OSCSO algorithm uses the auditory ability of the sand cat in low-frequency detection. The sensitivity range G decreases linearly from 2 to 0, simulating the sensitivity of the sand cat to low frequencies below 2KHz. For example, if the maximum number of iterations is 100, the value of G will be greater than 1 in the first half of the iterations, and will be less than 1 in the last 50 iterations. Among them, H F The value is inspired by the auditory characteristics of sand cats and is set to 2. t is the current number of iterations and T is the maximum number of iterations to control the balance between the exploration and development phases. The L parameter is added. Through this adaptive strategy, the transition between the two phases will be more balanced. L = 2xGxrand (0, 1) - G. In order to avoid falling into the local optimal trap, the sensitivity range of each sand cat is different. r = Gxrand (0, 1). Each search agent sand cat is based on the best candidate position x' best 、Current position(x t ) and sensitivity range (r) to update its own position so that the sand cat can find other best prey locations. S3. Attacking the prey: Modeling the attack phase of the OSCSO algorithm to represent the optimal position (x best ) and the current position (x t ), the position update formula at this stage is as follows: Among them, rand(0,1) represents any random number between [0,1]. Assuming that the sensitivity range of the sand cat is a circle, the direction of movement is determined by the random angle (θ) on the circle. Since the selected random angle is between 0 and 360, its value will be between -1 and 1. In this way, each member of the population moves in different circular directions in the search space; S4. Exploration and development: Exploration and development are switched through adaptive G and L parameters. Since the L parameter depends on G, its fluctuation range will also decrease. When the value of the G parameter remains uniformly distributed, the value of L will also remain balanced. In other words, the value of L is a random value within [-2G, 2G], where G decreases linearly from 2 to 0 during the iteration process. When the value of L is in [-1,1], the sand cat is guided to attack the prey, otherwise, the cat is tasked to find a new solution in the global area, that is: A traditional position update is generated, which is enhanced by the optimal value as shown below: FP = x t+1 *BtSn, where BtSn represents the optimal weight factor used to adjust the position update, the balancing behavior of the proposed algorithm and the effort to find other local regions in the global space have a fast and accurate convergence rate, thus showing good performance in high-dimensional and multi-objective problems.

5. The cross-modal person re-identification method based on multi-scale cross-attention Transformer according to claim 1, characterized in that: The three feature sets in step 2 are specifically: and The image matching features in step 4 are specifically S1, weighted fusion features: the network analyzes three sets of features during training and testing, which brings about the problem of too high dimension, which often leads to performance degradation, increased computational burden and increased network time consumption. Therefore, the OSCSO algorithm is used to determine the weight value, and the three sets of features are spliced ​​using the optimal weighting factor. The mathematical formula for weighted feature fusion is as follows: In the formula, w1 and w2 are the best weights obtained from the OSCSO algorithm. In order to find the best weight value, the objective function formula is set: In this formula, by adjusting the values ​​of w1 and w2, the CrCt correlation coefficient is maximized to achieve the best feature fusion effect. CrCt is used to measure the similarity between fused features. It represents the correlation between features. The value range is between -1 and 1. The closer the value is to 1, the stronger the linear correlation between the two features. The formula is as follows: Among them, h and g represent two sets of features, and Represent the mean of features h and g respectively. By maximizing CrCt, it is ensured that the fused features retain the most useful information and reduce redundancy, thereby improving the effect and performance of feature fusion. S2. Person re-identification using MSCAViT: In the MSCAViT architecture model, the encoder extracts features, then fuses them using the optimal weights, and finally passes them to the decoding layer of MSCAViT. The features obtained are represented by two modes, namely and The model mainly uses infrared images to identify the identity in visible light images. Therefore, the recognition formula of the decoding layer is as follows: I =p I w q , Among them, the prototype and query of infrared image are represented by p I and q I In addition, the visible light related features are marked as F R ', k R and v R Represents the key and value respectively, Finally, the complete visible light and infrared image features for pedestrian recognition are obtained, and the formula is as follows: S3. Loss function: In cross-modal person re-identification, there are significant differences between different modalities such as RGB images and infrared images. These differences will lead to inconsistency in feature space and affect matching performance. Heterogeneous center loss reduces the feature differences of the same identity under different modalities and enhances the consistency of cross-modal features: in, and represent the feature vectors of the i-th sample in visible light and infrared modes, respectively, c v and c I Represent the centers of the visible light and infrared modal feature vectors respectively, N is the number of samples, λ is a hyperparameter used to balance the weights of different loss terms, ||·||2 represents the L2 norm, and the heterogeneous center loss minimizes the distance from each sample feature to the corresponding modality center, making the intra-modality features more concentrated. By minimizing the distance between the feature centers of different modalities, the distribution differences between different modalities are reduced, making cross-modal matching more effective. The ID loss provides a strong supervision signal to ensure that the model accurately distinguishes different identities and improves the recognition accuracy: In the formula, p i,yi Indicates that the i-th sample is predicted to be its true identity y i The probability of y i is the true identity label of the i-th sample. The ID loss improves the model’s ability to distinguish pedestrian identities by minimizing the difference between the predicted distribution and the actual identity distribution. Therefore, the combination of heterogeneous center loss and ID loss enables MSCAViT to effectively solve the two key problems of cross-modal feature distribution differences and pedestrian identity distinction. The combination of the two enables the comprehensive optimization of feature alignment and identity distinction, significantly improving the overall performance and robustness of the re-identification task, as shown in the following formula: L = L hc +L id .

Images