Cross-modal pedestrian re-identification method based on coal mine scene

Abstract

The invention belongs to the field of cross-modal pedestrian re-identification, and discloses a cross-modal pedestrian re-identification method based on a coal mine scene, and the method comprises the steps: 1, obtaining an identity representing a pedestrian image and an identity tag corresponding to the pedestrian image, and carrying out the preprocessing; step 2, constructing a cross-modal pedestrian re-identification backbone network; step 3, constructing a multi-scale feature enhancement module and adding the multi-scale feature enhancement module into the cross-modal pedestrian re-identification backbone network; 4, determining a loss function, and completing the final training of the cross-modal pedestrian re-identification backbone network; and 5, taking one modal pedestrian image as a query set, taking the other modal pedestrian image as a to-be-retrieved set, and matching the pedestrian image in the query set with the pedestrian image in the to-be-retrieved set to obtain a cross-modal pedestrian re-identification result. According to the method, cross-modal image features can be effectively fused, feature information of pedestrian images can be effectively mined, and the prediction ability and generalization ability of the model in a cross-modal pedestrian image retrieval scene are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal pedestrian re-identification, and specifically relates to a cross-modal pedestrian re-identification method based on a coal mine scene. Background Art

[0002] With the rapid development of economy and technology, it is increasingly important to protect personal and property safety in public places. The government and relevant departments have increased their investment in intelligent security, and more and more cameras have been put into use in public places. Intelligent monitoring has penetrated into people's lives and has become an important tool for dealing with safety issues and emergency events. In this context, pedestrian re-identification (Person Re-Identification, Re-ID) technology has attracted widespread attention from scholars and has achieved rapid development. Most existing pedestrian re-identification methods are dedicated to retrieving visible light images captured by cameras during the day. However, these methods cannot effectively retrieve pedestrian information in low-light environments, especially dark environments, in coal mine scenes, which limits their effective practical application. For this reason, cross-modal pedestrian re-identification has emerged. In recent years, researchers have proposed methods for solving the problem of cross-modal pedestrian re-identification, which can be divided into two categories: first, methods based on feature sharing, focusing on the learning of shared features of pedestrians in cross-modal images, so as to narrow the differences between modalities; second, methods based on feature complementation, learning features of different parts to achieve the purpose of mutual complementation.

[0003] However, the cross-modal pedestrian re-identification in the existing technology has the following defects: First, the existing cross-modal datasets usually have obvious deficiencies in sample diversity and richness. The number and coverage of pedestrian samples are far less than those of single-modal datasets. Especially in coal mine environments, environmental changes caused by factors such as lighting changes, personnel clothing, dust and smoke further limit the representativeness and diversity of the dataset, and the generalization ability is greatly limited; due to the common environmental factors such as low light, high temperature, and smoke in coal mine workplaces, infrared imaging is greatly affected, and the resolution is often poor, and the image details are relatively blurred. Low-quality images not only increase the difficulty of feature extraction, but also easily introduce noise and artifacts, further affecting the training effect of the model.

[0004] Second, the existing pedestrian re-identification methods based on feature sharing face the problem of serious loss of image feature information and a large amount of effective frequency information loss when applied in complex environments such as coal mines. Specifically, the acquired feature information is easily affected by factors such as changes in the posture of workers in coal mine scenes and noise interference in coal mine workplaces, which makes it difficult for the model to access enough diverse information such as identity, posture, perspective and environmental changes during training, which directly leads to the loss of important image feature information, especially the loss of a large amount of effective frequency information.

[0005] Third, the existing pedestrian re-identification methods based on feature supplementation face many challenges in their application in special environments such as coal mines. There is a lack of sufficient interaction between pedestrian samples and it is difficult to extract more effective pedestrian features. Specifically, due to the low contrast of infrared images in coal mine environments and insufficient texture information, the supplementary features are often difficult to fully reflect the real identity information of pedestrians. The model faces great difficulties in extracting effective pedestrian features and is difficult to handle the differences between heterogeneous information. In addition, due to the complex environment of coal mine operating areas, smoke, insufficient light and other factors, and the feature supplementation method mainly focuses on the feature enhancement of a single sample, lacks in-depth mining of the relationship between samples, resulting in a lack of sufficient interaction and information sharing between pedestrian samples, which limits the model's accurate judgment of pedestrian identities and affects its generalization and robustness in complex scenarios. Summary of the invention

[0006] In order to solve the problems in the prior art such as insufficient samples of pedestrian image datasets, easy loss of pedestrian feature information and generally low quality of pedestrian images, the present invention provides a cross-modal pedestrian re-identification method based on coal mine scenes, which effectively integrates image features across modalities and effectively mines the feature information of pedestrian images.

[0007] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0008] The present invention is a cross-modal pedestrian re-identification method based on a coal mine scene, which specifically includes the following steps:

[0009] Step 1: Obtain the identity of the pedestrian image and the identity label corresponding to the pedestrian image from the coal mine infrared image and the coal mine visible light image. Each pedestrian image of different modalities contains a visible light image and an infrared light image. The visible light image and the infrared light image are preprocessed. The visible light image and the infrared light image are preprocessed by a text pre-training model to obtain the visible light image features and the infrared light image features after the text pre-training model supplements the missing color information;

[0010] Step 2: construct a cross-modal pedestrian re-identification backbone network based on relational modeling and spectrum transformation, wherein the cross-modal pedestrian re-identification backbone network mainly includes: a color encoder, a text encoder, a fragment relational modeling module, a channel spectrum transformation module, a visible light texture encoder, a joint relational encoder, and an infrared texture encoder; wherein the fragment relational modeling module includes a visual area image modeling module and a text area semantic modeling module, and a global-local cross attention mechanism is added to each part of the visual area image modeling and the text area semantic modeling, and by capturing the contextual information in the fragment relational modeling module, discriminative cross-modal embedding knowledge is learned, and then the overall embedding is learned;

[0011] Step 3: construct a multi-scale feature enhancement module and add it to the cross-modal pedestrian re-identification backbone network in step 2 to enhance the perception of the cross-modal pedestrian re-identification backbone network to the target object and obtain richer pedestrian image features;

[0012] Step 4: Determine the loss function and complete the final training of the cross-modal person re-identification backbone network;

[0013] Step 5: Use one of the modal pedestrian images as the query set and the other modal pedestrian images as the retrieval set, match the pedestrian images in the query set with the pedestrian images in the retrieval set, and obtain the cross-modal pedestrian re-identification results.

[0014] A further improvement of the present invention is that in step 2, the cross-modal pedestrian re-identification backbone network further includes a visible light texture encoder, a joint relationship encoder, and an infrared texture encoder. The visible light image and the infrared light image are constructed through a cross-modal pedestrian re-identification backbone network, and the paths of the visible light image and the infrared light image are respectively: a visible light path, an infrared path, and a visible light-infrared shared path:

[0015] For the visible light path, the input is a visible light image. The process of the visible light image passing through the cross-modal pedestrian re-identification backbone network is as follows: the visible light image enters the color encoder And through the visual area image modeling module Obtain color features and use the visible light texture encoder The acquired texture features are jointly fed into the visible light joint relation encoder Fully integrate;

[0016] For the infrared path, the input is the image features after the missing color information is supplemented by the text pre-trained model in step 1. The process of passing through the cross-modal pedestrian re-identification backbone network is as follows: Enter the text encoder And through the text region semantic modeling module Get text features and use infrared texture encoder The acquired texture features are jointly fed into the infrared joint relation encoder Fully integrate;

[0017] For the visible light-infrared shared path, the process is as follows: the texture features obtained by the visible light path and the infrared path are fed into the parameter-sharing feature enhancement network Improve the regional perception ability of the cross-modal pedestrian re-identification backbone network for visible light images and infrared images, and use the channel spectrum change module Obtain more useful spectrum information and finally constrain it through the loss function.

[0018] A further improvement of the present invention is that the global-local cross attention mechanism in step 2 learns discriminative cross-modal embedding knowledge, and then learns the overall embedding, which specifically includes the following steps:

[0019] Step 2.1.1, by calculating the product of the attention matrix from the low layer of the fragment relationship modeling module to the high layer of the fragment relationship modeling module, the cumulative attention score of the specified block in the area from the low layer of the fragment relationship modeling module to the high layer of the fragment relationship modeling module is calculated to obtain the attention map. The calculation process is described as:

[0020]

[0021] in, represents the renormalization of the attention weights, S represents the weighted attention matrix of each layer, Represents matrix multiplication operation, 1-i represents from the 1st layer to the i-th layer, thus, the information propagated from the input layer to the higher layer;

[0022] Step 2.1.2: Mining high response areas by aggregating attention maps. It is represented as the cumulative weighted class embedding and is obtained from the query matrix Q i Select the query vectors ranked in the top R to reconstruct the new query matrix Q l , used to represent the most concerned local embedded information;

[0023] Step 2.1.3: Construct a new query matrix Q l And the global set of the entire key-value pair, calculate the cross attention to get the attention weight matrix. The specific calculation process is:

[0024]

[0025] Among them, d is the dimension of the input feature, softmax is the activation function operation, K gT is the new query vector that matches the query vector, V is the value vector, and Y att Represented as the final cross-attention output weight.

[0026] A further improvement of the present invention is that in step 2, the visual area image modeling module includes three parts: area mapping, area feature semantics and sample segment interaction, which specifically includes the following steps:

[0027] Step 2.2.1: The input coal mine infrared image and coal mine visible light image are embedded in the region mapping part to dynamically adjust the dimension size of the vector, and add a fully connected layer to map the mapping region feature to the d-dimensional local feature as a visual fragment, which is recorded as the visible light initial region feature. where n r is the number of regional features;

[0028] Step 2.2.2: Obtained initial region features A semantic fully connected relational graph network feature representing the visual area is constructed to obtain the local spatial information of the target in the visible light image of the coal mine, and the contextual semantics of the initial regional features are learned through relational interaction. The graph nodes represent the visible light initial regional features R, and the edges represent their corresponding semantic relations. The global-local cross attention mechanism is used to capture the semantic relations of the visible light initial regional features R and learn the local features enhanced by mutual relations. The semantic relations are implied in the attention weights to obtain the relation-enhanced regional features.

[0029] Step 2.2.3: Obtain the global visual embedding feature Y by aggregating the initial visible light region feature R and the relationship enhanced region feature v , and its calculation process can be described as:

[0030] Y v =λ·MaxPool(R)+(1-λ)·AvgPool(R v )

[0031] Among them, the initial region feature R of visible light uses the maximum pooling MaxPool, the enhanced region feature uses the average pooling AvgPool operation, and λ is the proportional coefficient.

[0032] A further improvement of the present invention is that in step 2, the text region semantic modeling includes three parts: region mapping, vocabulary feature semantics and sample segment interaction, specifically including the following steps:

[0033] Step 2.3.1. Input the infrared image and visible light image of the coal mine, dynamically adjust the embedding vector dimension through the region mapping part, and add a fully connected layer to map the regional features to d-dimensional local features as text fragments, and record them as infrared initial regional features. where n t is the number of regional features;

[0034] Step 2.3.2: Obtained infrared light initial area features A word connection relationship graph network representing the text area is constructed to obtain the local position information of the target in the word, and the word dependency information and position information are utilized through relationship interaction. The graph nodes represent the initial regional features T of infrared light, and the edges represent the corresponding semantic relations. The global-local cross attention mechanism is used to capture the semantic relations and learn the local features enhanced by mutual relations. The semantic relations are implied in the attention weights to obtain the relationship-enhanced vocabulary features.

[0035] Step 2.3.3: Obtain the global visual embedding feature U by aggregating the infrared initial region feature T and the relation-enhanced vocabulary feature v , and its calculation process can be described as:

[0036] U v =λ·MaxPool(T)+(1-λ)·AvgPool(T v )

[0037] Among them, the infrared light initial area feature T uses the maximum pooling MaxPool, the relationship enhancement vocabulary feature uses the average pooling AvgPool operation, and λ is the proportional coefficient.

[0038] A further improvement of the present invention is that in step 2, a channel spectrum transformation module is added to the global-local cross attention mechanism, and a discrete cosine transform is performed through the channel spectrum transformation module. The specific transformation process includes the following steps:

[0039] Step 2.4.1. Input X i It is divided into multiple parts along the channel dimension and expressed as [X 0 ,X 0 ,…,X n-1 ], where X i ∈R C′×H×W , i∈{0, 1, ..., k-1}, And assign the corresponding two-dimensional discrete cosine transform frequency component to each part to obtain the discrete cosine transform result Y;

[0040] Step 2.4.2: The obtained discrete cosine transform result is used as the compression result and expressed as:

[0041]

[0042] where i∈{0, 1, ..., k-1}, [u i ,v i ] the corresponding 2D index of the corresponding frequency component, is the calculated discrete cosine value, For input X i When the input value of height h and width w is specified, Y freq That is the compressed C'-dimensional vector, and the global compressed vector is obtained by cascading, which is expressed as:

[0043]

[0044] Among them, cat means concatenating multiple tensors together;

[0045] Step 2.4.3, after obtaining the weight of each channel through the fully connected layer and passing through the spectral channel, the output is specifically expressed as:

[0046] Y att =sigmoid(fc(Y freq ))

[0047] Among them, sigmoid represents a smooth and monotonic activation function that maps any real value to the interval (0,1).

[0048] A further improvement of the present invention is that the multi-scale feature enhancement module in step 3 adopts a convolution generation structure with three different branches, each branch structure has a different convolution configuration, extracts multiple distinguishable semantic information, enhances the feature representation of the target, and performs feature enhancement through the multi-scale feature enhancement module. The specific process is:

[0049] Step 3.1: Input feature f i Enter three different branches respectively and get three different outputs F1, F2, F3. The calculation process can be described as:

[0050]

[0051] in, They represent convolution operations with kernel sizes of 3×3, 1×1, 1×3, and 3×1, respectively. Represents a dilated convolution operation with a kernel size of 3×3 and a dilation rate of 5;

[0052] Step 3.2: After the input features pass through three different branches, they are merged and spliced ​​with the residual connection after 1×1 convolution to obtain the final output f out , and its calculation process can be described as:

[0053]

[0054] Among them, F cat Indicates the concatenation operation of the feature map. Represents a standard convolution operation with a kernel size of 1×1.

[0055] A further improvement of the present invention is that in step 4, the orthogonal projection loss function, the cross entropy loss function, the triple loss function, the center-guided pair mining loss function, and the KL divergence loss function are used to jointly constrain the model training process. Encoder and The features after the encoder are optimized by the KL divergence loss function. Encoder and The features after the encoder are optimized through the triplet loss function, and the features after the visible light-infrared shared path are jointly optimized through the center-guided mining loss function, the cross entropy loss function and the orthogonal projection loss function.

[0056] A further improvement of the present invention is that the calculation process of the orthogonal projection loss function is:

[0057] First, define the calculation formula of the cosine similarity operator of two vectors, as shown in the following formula:

[0058]

[0059] Among them, ‖g‖2 represents the norm calculation process of the norm l2 operator;

[0060] Secondly, the clustering calculation process of samples of the same category and the orthogonality calculation process of samples of different categories are defined as follows:

[0061]

[0062] Among them, <x i ,y i > is an input-output pair and calculates the cosine similarity operator of the two vectors, represents the output of the middle layer of the network, and B represents the size of the mini-batch;

[0063] Finally, the total orthogonal projection loss function is expressed as:

[0064] L ort =(1-d)+|h|

[0065] Among them, |·| is the absolute value operator.

[0066] The cross entropy loss function is used for identity recognition in the pedestrian retrieval process. The cross-modal pedestrian re-identification task is regarded as an image classification task, and its calculation process can be expressed as:

[0067]

[0068] Among them, n represents the number of training samples, x i Represents the input of pedestrian image, y i represents its corresponding identity label, p(y i |x i ) represents the predicted probability of the input image and its corresponding identity label being identified after classification by the Softmax function;

[0069] The triplet loss function is used to optimize the triplet relationship between the unified identity and different identities, ensuring that the feature distance of the same pedestrian image is smaller than the feature distance between different pedestrians. The cross-modal pedestrian re-identification task is regarded as an image retrieval task, and its calculation process can be expressed as:

[0070]

[0071] in, and Represent anchor samples, positive samples and negative samples respectively. and The IDs of people in the same group, and Then use the IDs of different pedestrians respectively, F(·) represents the feature extraction function, It represents the process of solving the Euclidean distance between positive and negative samples. The most difficult positive sample is the sample that belongs to the same category as the anchor sample and is the farthest away. The most difficult negative sample is the sample that belongs to a different category from the anchor sample and is closest to it. P represents the number of pedestrians in a single batch, K represents the number of pictures corresponding to each pedestrian, and α is the edge parameter.

[0072] The center-guided pair mining loss function is used to constrain the generated embedding information, and the generated embedding is constrained by the following three attribute constraints, which increases the distance between the generated embedding and the original embedding, shortens the distance between the generated embedding of the visible light image and the original embedding of the infrared image, and shortens the distance between the generated embedding of the original image and the original embedding of the visible light image, and ensures that the intra-class distance is smaller than the inter-class distance, where α is a balance parameter, and its calculation process is expressed as:

[0073]

[0074] Similarly, for the control of the embedded class center generated by the infrared image, the calculation process can be expressed as:

[0075]

[0076] The final center-guided pair mining loss function can be expressed as:

[0077]

[0078] The calculation process of the KL divergence loss function is:

[0079]

[0080] Among them, p n,m For {f n ,f m} pairing probability, q n,m The constant ε is set to 1e for its true matching probability.-8 .

[0081] The beneficial effects of the present invention are:

[0082] Aiming at the scarcity of pedestrian samples in existing coal mine scenarios, a fragment relationship modeling architecture is proposed, focusing on the fragment-level interaction between samples, effectively capturing the contextual information of pedestrian fragments, and enhancing the feature representation of the network;

[0083] The present invention proposes a channel spectrum transformation process, which compresses the channel and utilizes multiple frequency components of discrete cosine transform to mine potential useful frequency information that is not noticed by general channel attention;

[0084] The present invention proposes a multi-scale feature enhancement module, which makes full use of local information and global context information, fully combines high-resolution low-level features and low-resolution high-level features, effectively enhances the network's perception of the target object, and obtains richer pedestrian image features.

[0085] In summary, the present invention effectively enhances the interaction between pedestrian cross-modal features and mines richer pedestrian image features through relational modeling and spectral transformation, effectively improving the prediction and generalization capabilities of the model in cross-modal pedestrian image retrieval scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 It is a flow chart of the cross-modal pedestrian re-identification method of the present invention.

[0087] Figure 2 Schematic diagram of the cross-modal person re-identification backbone network of the present invention.

[0088] Figure 3 It is a structural diagram of the cross-modal pedestrian re-identification backbone network of the present invention.

[0089] Figure 4 It is a schematic diagram of the visual area image modeling module of the present invention.

[0090] Figure 5 It is a schematic diagram of the channel spectrum conversion process of the present invention.

[0091] Figure 6 Schematic diagram of the multi-scale feature enhancement module of the present invention. DETAILED DESCRIPTION

[0092] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0093] In view of the complex environment of coal mine operation areas and the susceptibility to factors such as smoke, insufficient light, etc., this paper effectively extracts pedestrian image features and reduces inter-modal differences by adding fragment relationship modeling structure and channel spectrum transformation process. Specifically, a cross-modal pedestrian re-identification method based on coal mine scenes is proposed. Figure 1-2 As shown, the method specifically comprises the following steps:

[0094] Step 1: Obtain the identity of the pedestrian image and the identity label corresponding to the pedestrian image from the coal mine infrared image and the coal mine visible light image. Each pedestrian image of different modalities contains a visible light image and an infrared light image. The visible light image and the infrared light image are preprocessed. The visible light image and the infrared light image are preprocessed by a text pre-training model to obtain the visible light image features and the infrared light image features after the text pre-training model supplements the missing color information;

[0095] Step 2. Construct a cross-modal pedestrian re-identification backbone network based on relation modeling and spectral transformation. The cross-modal pedestrian re-identification backbone network mainly includes: a color encoder, a text encoder, a fragment relation modeling module, a channel spectrum transformation module, a visible light texture encoder, a joint relation encoder, and an infrared texture encoder; wherein the fragment relation modeling module includes a visual area image modeling module and a text area semantic modeling module, and a global-local cross-attention mechanism is added to each part of the visual area image modeling and the text area semantic modeling. By capturing the contextual information in the fragment relation modeling module, discriminative cross-modal embedding knowledge is learned, and then the overall embedding is learned.

[0096] The overall network framework is as follows Figure 3 As shown, the cross-modal pedestrian re-identification backbone network also includes a visible light texture encoder, a joint relationship encoder, and an infrared texture encoder. The visible light image and the infrared light image are constructed through a cross-modal pedestrian re-identification backbone network. The paths of the visible light image and the infrared light image are respectively: a visible light path, an infrared path, and a visible light-infrared shared path:

[0097] For the visible light path, the input is a visible light image. The process of the visible light image passing through the cross-modal pedestrian re-identification backbone network is as follows: the visible light image enters the color encoder And through the visual area image modeling module Obtain color features and use the visible light texture encoder The acquired texture features are jointly fed into the visible light joint relation encoder Fully integrate;

[0098] For the infrared path, the input is the image features after the missing color information is supplemented by the text pre-trained model in step 1. The process of passing through the cross-modal pedestrian re-identification backbone network is as follows: Enter the text encoder And through the text region semantic modeling module Get text features and use infrared texture encoder The acquired texture features are jointly fed into the infrared joint relation encoder Fully integrate;

[0099] For the visible light-infrared shared path, the process is as follows: the texture features obtained by the visible light path and the infrared path are fed into the parameter-sharing feature enhancement network Improve the regional perception ability of the cross-modal pedestrian re-identification backbone network for visible light images and infrared images, and use the channel spectrum change module Obtain more useful spectrum information and finally constrain it through the loss function.

[0100] The global-local cross attention mechanism acquires richer image relationships and semantic relationships by strengthening the interaction between the global image and the local salient regions, learns discriminative cross-modal embedding knowledge, and then learns the overall embedding. Specifically, it includes the following steps:

[0101] Step 2.1.1, by calculating the product of the attention matrix from the low layer of the fragment relationship modeling module to the high layer of the fragment relationship modeling module, the cumulative attention score of the specified block in the area from the low layer of the fragment relationship modeling module to the high layer of the fragment relationship modeling module is calculated to obtain the attention map. The calculation process is described as:

[0102]

[0103] in, represents the renormalization of the attention weights, S represents the weighted attention matrix of each layer, Represents matrix multiplication operation, 1-i represents from the 1st layer to the i-th layer, thus, the information propagated from the input layer to the higher layer;

[0104] Step 2.1.2: Mining high response areas by aggregating attention maps. It is represented as the cumulative weighted class embedding and is obtained from the query matrix Q i Select the query vectors ranked in the top R to reconstruct the new query matrix Q l , used to represent the most concerned local embedded information;

[0105] Step 2.1.3: Construct a new query matrix Q lAnd the global set of the entire key-value pair, calculate the cross attention to get the attention weight matrix. The specific calculation process is:

[0106]

[0107] Among them, d is the dimension of the input feature, softmax is the activation function operation, K gT is the new query vector that matches the query vector, V is the value vector, and Y att Represented as the final cross-attention output weight.

[0108] like Figure 4 As shown, the visual region image modeling module includes three parts: region mapping, region feature semantics, and sample segment interaction. The region feature semantic part learns the attention weights of different image relationships in the region features through the image relationship graph between visual regions, and learns the context information of the region features and extracts image relationships through the global-local cross attention mechanism. The sample segment interaction part captures the relationship features between samples through the maximum pooling operation and the average pooling operation to learn the pedestrian image features with enhanced relationships, which specifically includes the following steps:

[0109] Step 2.2.1: The input coal mine infrared image and coal mine visible light image are embedded in the region mapping part to dynamically adjust the dimension size of the vector, and add a fully connected layer to map the mapping region feature to the d-dimensional local feature as a visual fragment, which is recorded as the visible light initial region feature. where n r is the number of regional features;

[0110] Step 2.2.2: Obtained initial region features A semantic fully connected relational graph network feature representing the visual area is constructed to obtain the local spatial information of the target in the visible light image of the coal mine, and the contextual semantics of the initial regional features are learned through relational interaction. The graph nodes represent the visible light initial regional features R, and the edges represent their corresponding semantic relations. The global-local cross attention mechanism is used to capture the semantic relations of the visible light initial regional features R and learn the local features enhanced by mutual relations. The semantic relations are implied in the attention weights to obtain the relation-enhanced regional features.

[0111] Step 2.2.3: Obtain the global visual embedding feature Y by aggregating the initial visible light region feature R and the relationship enhanced region feature v , and its calculation process can be described as:

[0112] Y v =λ·MaxPool(R)+(1-λ)·AvgPool(R v )

[0113] Among them, the initial region feature R of visible light uses the maximum pooling MaxPool, the enhanced region feature uses the average pooling AvgPool operation, and λ is the proportional coefficient.

[0114] The text region semantic modeling includes three parts: region mapping, lexical feature semantics and sample segment interaction. The lexical feature semantic part generates a bidirectional language representation result through the text relationship graph in the text region, aggregates important semantic information through the global-local cross attention mechanism, filters irrelevant information, and extracts text relationships. The sample segment interaction part captures the relationship features between samples through the maximum pooling operation and the average pooling operation to learn the pedestrian text features with enhanced relationships, which specifically includes the following steps:

[0115] Step 2.3.1. Input the infrared image and visible light image of the coal mine, dynamically adjust the embedding vector dimension through the region mapping part, and add a fully connected layer to map the regional features to d-dimensional local features as text fragments, and record them as infrared initial regional features. Where n t is the number of regional features;

[0116] Step 2.3.2: Obtained infrared light initial area features A word connection relationship graph network representing the text area is constructed to obtain the local position information of the target in the word, and the word dependency information and position information are utilized through relationship interaction. The graph nodes represent the initial regional features T of infrared light, and the edges represent the corresponding semantic relations. The global-local cross attention mechanism is used to capture the semantic relations and learn the local features enhanced by mutual relations. The semantic relations are implied in the attention weights to obtain the relationship-enhanced vocabulary features.

[0117] Step 2.3.3: Obtain the global visual embedding feature U by aggregating the infrared initial region feature T and the relation-enhanced vocabulary feature v , and its calculation process can be described as:

[0118] U v =λ·MaxPool(T)+(1-λ)·AvgPool(T v )

[0119] Among them, the infrared light initial area feature T uses the maximum pooling MaxPool, the relationship enhancement vocabulary feature uses the average pooling AvgPool operation, and λ is the proportional coefficient.

[0120] The present invention adds a channel spectrum transformation module, namely the discrete cosine transform process, to the global-local cross attention mechanism, and designs a channel spectrum transformation process that can be applied to the field of cross-modal person re-identification. While compressing the channel, it utilizes multiple frequency components of the discrete cosine transform to mine potential useful frequency information that is not noticed by general channel attention. Figure 5 As shown, the input feature vector uses multiple frequency components based on discrete cosine transform, and provides frequency component selection criteria for selection, and finally obtains more information including multiple frequency components.

[0121] The specific transformation process includes the following steps:

[0122] Step 2.4.1. Input X i It is divided into multiple parts along the channel dimension and expressed as [X 0 ,X 0 ,…,X n-1 ], where X i ∈R C′×H×W , i∈{0, 1, …, k-1}, And assign the corresponding two-dimensional discrete cosine change frequency component to each part, and get the discrete cosine transform result Y. Figure 2 As shown, enter X i is the output of the texture encoder;

[0123] Step 2.4.2: The obtained discrete cosine transform result is used as the compression result and expressed as:

[0124]

[0125] where i∈{0, 1, …, k-1}, [u i ,v i ] the corresponding 2D index of the corresponding frequency component, is the calculated discrete cosine value, For input X i When the input value of height h and width w is specified, Y freq That is the compressed C'-dimensional vector, and the global compressed vector is obtained by cascading, which is expressed as:

[0126]

[0127] Among them, cat means concatenating multiple tensors together;

[0128] Step 2.4.3, after obtaining the weight of each channel through the fully connected layer and passing through the spectral channel, the output is specifically expressed as:

[0129] Y att =sigmoid(fc(Y freq))

[0130] Among them, sigmoid represents a smooth and monotonic activation function that maps any real value to the interval (0,1).

[0131] Step 3: construct a multi-scale feature enhancement module and add it to the cross-modal pedestrian re-identification backbone network in step 2 to enhance the perception of the cross-modal pedestrian re-identification backbone network on the target object and obtain richer pedestrian image features. Figure 6 As shown in the figure, the multi-scale feature enhancement module adopts a convolutional generation structure with three different branches. Each branch structure has a different convolution configuration, extracts multiple distinguishable semantic information, enhances the feature representation of the target, and performs feature enhancement through the multi-scale feature enhancement module. The specific process is as follows:

[0132] Step 3.1: Input feature f i Enter three different branches respectively and get three different outputs F1, F2, F3. The calculation process can be described as:

[0133]

[0134] in, They represent convolution operations with kernel sizes of 3×3, 1×1, 1×3, and 3×1, respectively. represents a dilated convolution operation with a kernel size of 3×3 and a dilation rate of 5; Figure 2 As shown, feature f i is the output obtained by the texture encoder, which is used as the input of the multi-scale feature enhancement module.

[0135] Step 3.2: After the input features pass through three different branches, they are merged and concatenated with the residual connection after 1×1 convolution to obtain the final output f out , and its calculation process can be described as:

[0136]

[0137] Among them, F cat Indicates the concatenation operation of the feature map. Represents a standard convolution operation with a kernel size of 1×1.

[0138] Step 4: Determine the loss function, training environment, and training details, complete the final training of the cross-modal pedestrian re-identification backbone network, use one modality of pedestrian images as the query set and the other modality of pedestrian images as the retrieval set, match the pedestrian images in the query set with the pedestrian images in the retrieval set, and obtain the cross-modal pedestrian re-identification results.

[0139] The present invention uses orthogonal projection loss function, cross entropy loss function, triple loss function, center-guided pair mining loss function, and KL divergence loss function to jointly constrain the model training process. Encoder and The features after the encoder are optimized by the KL divergence loss function. Encoder and The features after the encoder are optimized through the triplet loss function, and the features after the visible light-infrared shared path are jointly optimized through the center-guided mining loss function, the cross entropy loss function and the orthogonal projection loss function.

[0140] The present invention uses an orthogonal projection loss function, implements orthogonalization constraints in the intermediate feature space and uses a projection method to explore the possibility of implementing in small batches to enhance the intrinsic distinguishing features between samples, maximize the separation of different features generated by different categories, supplement the angular distinguishability of the output space, and further enhance the distinguishability and generalization ability of the model's learned feature representation during the training process.

[0141] First, define the calculation formula of the cosine similarity operator of two vectors, as shown in the following formula:

[0142]

[0143] Among them, ‖g‖2 represents the norm calculation process of the norm l2 operator;

[0144] Secondly, the clustering calculation process of samples of the same category and the orthogonality calculation process of samples of different categories are defined as follows:

[0145]

[0146] Among them, <x i ,y i > is an input-output pair and calculates the cosine similarity operator of the two vectors, represents the output of the middle layer of the network, and B represents the size of the mini-batch;

[0147] Finally, the total orthogonal projection loss function is expressed as:

[0148] L ort =(1-d)+|h|

[0149] Among them, |·| is the absolute value operator.

[0150] The orthogonal projection loss function utilizes efficient vectorization implementation and direct gradient calculation and propagation, and orthogonalization in the feature space to optimize the different goals of inter-class separation and intra-class clustering. It forces all categories to be orthogonal to each other by clustering different intra-class samples at the same time. Maximizing the difference may lead to negative correlation between classes, ensuring that the embeddings generated from different branches can capture different information features.

[0151] The cross entropy loss function prevents the model from overfitting by improving the ability to identify identity information. This function is used for identity recognition in the pedestrian retrieval process during the training process. The cross-modal pedestrian re-identification task is regarded as an image classification task. Its calculation process can be expressed as:

[0152]

[0153] Among them, n represents the number of training samples, x i Represents the input of pedestrian image, y i represents its corresponding identity label, p(y i |x i ) represents the predicted probability of the input image and its corresponding identity label being identified after classification by the Softmax function;

[0154] The triplet loss function is used to optimize the triplet relationship between the unified identity and different identities, ensuring that the feature distance of the same pedestrian image is smaller than the feature distance between different pedestrians. The cross-modal pedestrian re-identification task is regarded as an image retrieval task, and its calculation process can be expressed as:

[0155]

[0156] in, and Represent anchor samples, positive samples and negative samples respectively. and The IDs of people in the same group, and Then use the IDs of different pedestrians respectively, F(·) represents the feature extraction function, It represents the process of solving the Euclidean distance between positive and negative samples. The most difficult positive sample is the sample that belongs to the same category as the anchor sample and is the farthest away. The most difficult negative sample is the sample that belongs to a different category from the anchor sample and is closest to it. P represents the number of pedestrians in a single batch, K represents the number of pictures corresponding to each pedestrian, and α is the margin parameter, which is set to 0.3.

[0157] The center-guided pair mining loss function is used to constrain the generated embedding information, and the generated embedding is constrained by the following three attribute constraints, which increases the distance between the generated embedding and the original embedding, decreases the distance between the generated embedding of the visible light image and the original embedding of the infrared image, and decreases the distance between the generated embedding of the original image and the original embedding of the visible light image, and ensures that the intra-class distance is smaller than the inter-class distance, where α is a balance parameter, which is set to 0.2. The calculation process is expressed as:

[0158]

[0159] Similarly, for the control of the embedded class center generated by the infrared image, the calculation process can be expressed as:

[0160]

[0161] The final center-guided pair mining loss function can be expressed as:

[0162]

[0163] The calculation process of the KL divergence loss function is:

[0164]

[0165] Among them, p n,m For {f n ,f m} pairing probability, q n,m The constant ε is set to 1e for its true matching probability. -8 .

[0166] The present invention jointly uses an orthogonal projection loss function, a cross entropy loss function, a triplet loss function, a center-guided pair mining loss function and a KL divergence loss, wherein: Encoder and The features after the encoder are optimized by the KL divergence loss function. Encoder and The features after the encoder are optimized through the triplet loss function, and the features after the visible light-infrared shared path are jointly optimized through the center-guided mining loss function, the cross entropy loss function and the orthogonal projection loss function.

[0167] In the training phase, the size of the input images was uniformly adjusted to 384×144. Each training batch consisted of 4 identities, each of which contained 4 visible light images and 4 infrared images, and random cropping, random horizontal flipping, and random erasing were performed for data enhancement. The number of training iterations was 80 epochs, and the sgd optimizer was used for optimization. The learning rate was set to 0.1 in the first round, and the learning rate was decayed to 0.01 and 0.001 at the 20th and 50th epochs, respectively.

[0168] When training on the RegDB dataset, due to the small number of pedestrian images, the first three stages of training in the ResNet-50 structure are used, and the fragment relationship modeling structure and channel spectrum transformation process are added accordingly, and a feature enhancement module is added in the second stage.

[0169] The technical effects of the present invention are described in detail below in conjunction with performance tests. The present invention aims to improve the prediction and generalization capabilities of the model in cross-modal pedestrian image retrieval scenarios. To prove the performance of the model in coal mine scenarios, the present invention uses a cross-modal pedestrian re-identification dataset for full testing and compares it with existing algorithms in recent years. The results are shown in the following table:

[0170] Table 1 Comparison of results with existing methods on the SYSU-MM01 dataset (%)

[0171]

[0172]

[0173] In the All-search of the dataset, the R-1, R-10, R-20 and mAP of the proposed method reached 84.8%, 99.1%, 99.9% and 81.5%, 1.5% respectively, and in the Indoor-search, the R-1, R-10, R-20 and mAP reached 92.5%, 99.5%, 100.0% and 92.6% respectively, which fully confirmed the effectiveness of the method.

[0174] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A cross-modal pedestrian re-identification method based on coal mine scenes, characterized by: The coal mine scene cross-modal pedestrian re-identification method specifically includes the following steps: Step 1: Obtain the identity of the pedestrian image and the identity label corresponding to the pedestrian image from the coal mine infrared image and the coal mine visible light image. Each pedestrian image of different modalities contains a visible light image and an infrared light image. The visible light image and the infrared light image are preprocessed. The visible light image and the infrared light image are preprocessed by a text pre-training model to obtain the visible light image features and the infrared light image features after the text pre-training model supplements the missing color information; Step 2: construct a cross-modal pedestrian re-identification backbone network based on relational modeling and spectrum transformation, wherein the cross-modal pedestrian re-identification backbone network mainly includes: a color encoder, a text encoder, a fragment relational modeling module, a channel spectrum transformation module, a visible light texture encoder, a joint relational encoder, and an infrared texture encoder; wherein the fragment relational modeling module includes a visual area image modeling module and a text area semantic modeling module, and a global-local cross attention mechanism is added to each part of the visual area image modeling and the text area semantic modeling, and by capturing the contextual information in the fragment relational modeling module, discriminative cross-modal embedding knowledge is learned, and then the overall embedding is learned; Step 3: construct a multi-scale feature enhancement module and add it to the cross-modal pedestrian re-identification backbone network in step 2 to enhance the perception of the cross-modal pedestrian re-identification backbone network to the target object and obtain richer pedestrian image features; Step 4: Determine the loss function and complete the final training of the cross-modal person re-identification backbone network; Step 5: Use one of the modal pedestrian images as the query set and the other modal pedestrian images as the retrieval set, match the pedestrian images in the query set with the pedestrian images in the retrieval set, and obtain the cross-modal pedestrian re-identification results.

2. According to the coal mine scene-based cross-modal pedestrian re-identification method of claim 1, it is characterized by: In step 2, the cross-modal pedestrian re-identification backbone network also includes a visible light texture encoder, a joint relationship encoder, and an infrared texture encoder. The visible light image and the infrared light image are constructed through a cross-modal pedestrian re-identification backbone network, and the paths of the visible light image and the infrared light image are respectively: a visible light path, an infrared path, and a visible light-infrared shared path: For the visible light path, the input is a visible light image. The process of the visible light image passing through the cross-modal pedestrian re-identification backbone network is as follows: the visible light image enters the color encoder And through the visual area image modeling module Obtain color features and use the visible light texture encoder The acquired texture features are jointly fed into the visible light joint relation encoder to integrate; For the infrared path, the input is the image features after the missing color information is supplemented by the text pre-trained model in step 1. The process of passing through the cross-modal pedestrian re-identification backbone network is as follows: Enter the text encoder And through the text region semantic modeling module Get text features and use infrared texture encoder The acquired texture features are jointly fed into the infrared joint relation encoder to integrate; For the visible light-infrared shared path, the process is as follows: the texture features obtained by the visible light path and the infrared path are fed into the parameter-sharing feature enhancement network Improve the regional perception ability of the cross-modal pedestrian re-identification backbone network for visible light images and infrared images, and use the channel spectrum change module Obtain more useful spectrum information and finally constrain it through the loss function.

3. According to the coal mine scene-based cross-modal pedestrian re-identification method of claim 1, it is characterized by: The global-local cross attention mechanism in step 2 learns discriminative cross-modal embedding knowledge, and then learns the overall embedding, which specifically includes the following steps: Step 2.1.1, by calculating the product of the attention matrix from the low layer of the fragment relationship modeling module to the high layer of the fragment relationship modeling module, the cumulative attention score of the specified block in the area from the low layer of the fragment relationship modeling module to the high layer of the fragment relationship modeling module is calculated to obtain the attention map. The calculation process is described as: in, represents the renormalization of the attention weights, S represents the weighted attention matrix of each layer, Represents matrix multiplication operation, 1-i represents from the 1st layer to the i-th layer, thus, the information propagated from the input layer to the higher layer; Step 2.1.2: Mining high response areas by aggregating attention maps. It is represented as the cumulative weighted class embedding and is obtained from the query matrix Q i Select the query vectors ranked in the top R to reconstruct the new query matrix Q l , used to represent the most concerned local embedded information; Step 2.1.3: The new query matrix Q constructed by l And the global set of the entire key-value pair, calculate the cross attention to get the attention weight matrix. The specific calculation process is: Among them, d is the dimension of the input feature, softmax is the activation function operation, K gT is the new query vector that matches the query vector, V is the value vector, and Y att Represented as the final cross-attention output weight.

4. The cross-modal person re-identification method based on coal mine scene according to claim 1 is characterized by: In step 2, the visual region image modeling module includes three parts: region mapping, region feature semantics, and sample segment interaction, and specifically includes the following steps: Step 2.2.1: The input coal mine infrared image and coal mine visible light image are embedded in the region mapping part to dynamically adjust the dimension size of the vector, and add a fully connected layer to map the mapping region feature to the d-dimensional local feature as a visual fragment, which is recorded as the visible light initial region feature. Where n r is the number of regional features; Step 2.2.2: Obtained initial region features A semantic fully connected relational graph network feature representing the visual area is constructed to obtain the local spatial information of the target in the visible light image of the coal mine, and the contextual semantics of the initial regional features are learned through relational interaction. The graph nodes represent the visible light initial regional features R, and the edges represent their corresponding semantic relations. The global-local cross attention mechanism is used to capture the semantic relations of the visible light initial regional features R and learn the local features enhanced by mutual relations. The semantic relations are implied in the attention weights to obtain the relation-enhanced regional features. Step 2.2.3: Obtain the global visual embedding feature Y by aggregating the initial visible light region feature R and the relationship enhanced region feature v , and its calculation process can be described as: Y v =λ·MaxPool(R)+(1-λ)·AvgPool(R v ) Among them, the initial region feature R of visible light uses the maximum pooling MaxPool, the enhanced region feature uses the average pooling AvgPool operation, and λ is the proportional coefficient.

5. The cross-modal person re-identification method based on coal mine scene according to claim 1 is characterized by: In step 2, the text region semantic modeling includes three parts: region mapping, vocabulary feature semantics, and sample segment interaction, and specifically includes the following steps: Step 2.3.

1. Input the infrared image and visible light image of the coal mine, dynamically adjust the embedding vector dimension through the region mapping part, and add a fully connected layer to map the regional features to d-dimensional local features as text fragments, and record them as infrared initial regional features. Where n t is the number of regional features; Step 2.3.2: Obtained infrared light initial area features A word connection relationship graph network representing the text area is constructed to obtain the local position information of the target in the word, and the word dependency information and position information are utilized through relationship interaction. The graph nodes represent the initial regional features T of infrared light, and the edges represent the corresponding semantic relations. The global-local cross attention mechanism is used to capture the semantic relations and learn the local features enhanced by mutual relations. The semantic relations are implied in the attention weights to obtain the relationship-enhanced vocabulary features. Step 2.3.3: Obtain the global visual embedding feature U by aggregating the infrared initial region feature T and the relation-enhanced vocabulary feature v , and its calculation process can be described as: U v =λ·MaxPool(T)+(1-λ)·AvgPool(T v ) Among them, the infrared light initial area feature T uses the maximum pooling MaxPool, the relationship enhancement vocabulary feature uses the average pooling AvgPool operation, and λ is the proportional coefficient.

6. The cross-modal person re-identification method based on coal mine scene according to claim 1 is characterized by: In step 2, a channel spectrum transformation module is added to the global-local cross attention mechanism, and discrete cosine transform is performed through the channel spectrum transformation module. The specific transformation process includes the following steps: Step 2.4.

1. Input X i It is divided into multiple parts along the channel dimension and expressed as [X 0 ,X 0 ,…,X n-1 ], where X i ∈R C '×H×W , i∈{0, 1, ..., k-1}, And assign the corresponding two-dimensional discrete cosine transform frequency component to each part to obtain the discrete cosine transform result Y; Step 2.4.2: The obtained discrete cosine transform result is used as the compression result and expressed as: where i∈{0, 1, ..., k-1}, [u i ,v i ] the corresponding 2D index of the corresponding frequency component, is the calculated discrete cosine value, For input X i When the input value of height h and width w is specified, Y freq That is the compressed C'-dimensional vector, and the global compressed vector is obtained by cascading, which is expressed as: Among them, cat means concatenating multiple tensors together; Step 2.4.3, after obtaining the weight of each channel through the fully connected layer and passing through the spectral channel, the output is specifically expressed as: Y att =sigmoid(fc(Y freq )) Among them, sigmoid represents a smooth and monotonic activation function that maps any real value to the interval (0,1).

7. The cross-modal person re-identification method based on coal mine scene according to claim 1 is characterized by: The multi-scale feature enhancement module in step 3 adopts a convolutional generation structure with three different branches. Each branch structure has a different convolution configuration, extracts multiple distinguishable semantic information, enhances the feature representation of the target, and performs feature enhancement through the multi-scale feature enhancement module. The specific process is as follows: Step 3.1: Input feature f i Enter three different branches respectively and get three different outputs F1, F2, F3. The calculation process can be described as: in, They represent convolution operations with kernel sizes of 3×3, 1×1, 1×3, and 3×1, respectively. Represents a dilated convolution operation with a kernel size of 3×3 and a dilation rate of 5; Step 3.2: After the input features pass through three different branches, they are merged and concatenated with the residual connection after 1×1 convolution to obtain the final output f out , and its calculation process can be described as: Among them, F cat Indicates the concatenation operation of the feature map. Represents a standard convolution operation with a kernel size of 1×1.

8. The cross-modal person re-identification method based on coal mine scene according to claim 2 is characterized by: In step 4, the orthogonal projection loss function, the cross entropy loss function, the triple loss function, the center-guided pair mining loss function, and the KL divergence loss function are used to jointly constrain the model training process. Encoder and The features after the encoder are optimized by the KL divergence loss function. Encoder and The features after the encoder are optimized through the triplet loss function, and the features after the visible light-infrared shared path are jointly optimized through the center-guided mining loss function, the cross entropy loss function and the orthogonal projection loss function.

9. The cross-modal person re-identification method based on coal mine scene according to claim 8 is characterized in that: The calculation process of the orthogonal projection loss function is: First, define the calculation formula of the cosine similarity operator of two vectors, as shown in the following formula: Among them, ‖g‖2 represents the norm calculation process of the norm l2 operator; Secondly, the clustering calculation process of samples of the same category and the orthogonality calculation process of samples of different categories are defined as follows: Among them, <x i ,y i > is an input-output pair and calculates the cosine similarity operator of the two vectors, represents the output of the middle layer of the network, and B represents the size of the mini-batch; Finally, the total orthogonal projection loss function is expressed as: L ort =(1-d)+h· Among them, |·| is the absolute value operator.

10. The cross-modal person re-identification method based on coal mine scene according to claim 8, characterized in that: The cross entropy loss function is used for identity recognition in the pedestrian retrieval process. The cross-modal pedestrian re-identification task is regarded as an image classification task, and its calculation process can be expressed as: Among them, n represents the number of training samples, x i Represents the input of pedestrian image, y i represents its corresponding identity label, p(y i |x i ) represents the predicted probability of the input image and its corresponding identity label being identified after classification by the Softmax function; The triplet loss function is used to optimize the triplet relationship between the unified identity and different identities, ensuring that the feature distance of the same pedestrian image is smaller than the feature distance between different pedestrians. The cross-modal pedestrian re-identification task is regarded as an image retrieval task, and its calculation process can be expressed as: in, and Represent anchor samples, positive samples and negative samples respectively. and The IDs of people in the same group, and Then use the IDs of different pedestrians respectively, F(·) represents the feature extraction function, It represents the process of solving the Euclidean distance between positive and negative samples. The most difficult positive sample is the sample that belongs to the same category as the anchor sample and is the farthest away. The most difficult negative sample is the sample that belongs to a different category from the anchor sample and is closest to it. P represents the number of pedestrians in a single batch, K represents the number of pictures corresponding to each pedestrian, and α is the edge parameter. The center-guided pair mining loss function is used to constrain the generated embedding information, and the generated embedding is constrained by the following three attribute constraints, which increases the distance between the generated embedding and the original embedding, shortens the distance between the generated embedding of the visible light image and the original embedding of the infrared image, and shortens the distance between the generated embedding of the original image and the original embedding of the visible light image, and ensures that the intra-class distance is smaller than the inter-class distance, where α is a balance parameter, and its calculation process is expressed as: Similarly, for the control of the embedded class center generated by the infrared image, the calculation process can be expressed as: The final center-guided pair mining loss function can be expressed as: The calculation process of the KL divergence loss function is: Among them, p n,m For {f n ,f m } pairing probability, q n,m The constant ε is set to 1e for its true matching probability. -8 .

Images