A method and device for occluded pedestrian re-identification

Abstract

The application discloses a kind of occluded pedestrian re-identification methods, comprising: the original image of target pedestrian is input into visual encoder and obtains high-dimensional visual embedding;High-dimensional visual embedding is input into prompt learner, and obtain target text prompt, occlusion text prompt and background text prompt;High-dimensional occlusion text embedding and high-dimensional background text embedding are fused by fusion layer and obtain fusion background text embedding;High-dimensional visual embedding and high-dimensional target text embedding and fusion background text embedding are cosine similarity operation, and obtain target attention graph and background attention graph;Enhanced visual feature is obtained;And target pedestrian is stored according to enhanced visual feature using memory bank.The occluded pedestrian re-identification method disclosed in the application improves the performance of pedestrian re-identification by designing independent target prompt learner, occlusion prompt learner and background prompt learner.

Description

Technical Field

[0001] This invention belongs to the field of computer vision image processing, and more specifically, relates to a method and apparatus for re-identifying occluded pedestrians. Background Technology

[0002] Person re-identification (ReID) is an important and challenging problem in computer vision, aiming to accurately identify a specific individual by comparing images from different camera perspectives. This task is widely used in multi-object tracking, motion analysis, crowd monitoring, and video surveillance. The core objective of ReID is to achieve accurate matching and identification of the same pedestrian captured from different viewpoints and under different conditions, thereby providing reliable support for intelligent surveillance and behavior analysis.

[0003] Although ReID technology has advanced to a certain extent, in complex urban environments, especially in densely populated areas with diverse objects, traditional pedestrian re-identification methods often struggle to perform adequately due to occlusion issues. This is because in complex urban settings, occlusion can occur at different locations and in different parts between pedestrians and between pedestrians and various objects. This results in key features of the target pedestrian not being fully captured, leading to inconsistencies or missing features. Traditional ReID methods struggle to effectively match and identify pedestrians based on incomplete pedestrian feature information caused by partial or complete occlusion.

[0004] In recent years, person re-identification methods based on Contrastive Language-Image Pretraining (CLIP) have been proposed and have achieved significant progress. When dealing with the problem of occluded person re-identification, the main advantage of CLIP-based ReID methods lies in their ability to understand the target object at the textual level, thus providing supplementary descriptions and information for different parts of the pedestrian. CLIP-ReID, through multimodal learning, overcomes the limitations of traditional methods to some extent, enabling the model to perform better feature matching when faced with partially occluded pedestrians.

[0005] However, despite its advantages, the CLIP-ReID method still faces several challenges in its application to occluded person re-identification. First, existing CLIP-ReID methods typically use a single target cue to align with the image, or multiple cues to capture features from different parts of the target and align them with the image. When multiple pedestrians are occluded, these methods often struggle to effectively handle the occluded targets because they lack sufficient text understanding capabilities to cope with complex occlusion scenarios. Especially when multiple pedestrians are simultaneously occluded, existing CLIP image-text alignment methods are prone to mismatches, making it difficult to correctly identify occluded pedestrians. Furthermore, existing CLIP-ReID methods often use a contrast loss based on the entire image for training, which fails to fully utilize the contextual information of the occluded region, resulting in limited recognition capabilities for occluded areas. Therefore, although CLIP-ReID can improve the performance of person re-identification under certain conditions, its performance remains significantly limited under severe occlusion. Summary of the Invention

[0006] To address the shortcomings of existing technologies and reduce false positives and false negatives in pedestrian re-identification in occluded scenarios, this invention discloses an occluded pedestrian re-identification method that improves pedestrian re-identification performance by designing independent target cue learners, occlusion cue learners, and background cue learners. This invention provides an occluded pedestrian re-identification method, comprising:

[0007] S1, Input the original image of the target pedestrian into the visual encoder to obtain a high-dimensional visual embedding;

[0008] S2, the high-dimensional visual embedding is input into the prompt learner to obtain the target text prompt, the occlusion text prompt and the background text prompt respectively;

[0009] S3, after inputting the target text prompt, the occlusion text prompt and the background text prompt shown into the text encoder, high-dimensional target text embedding, high-dimensional occlusion text embedding and high-dimensional background text embedding are obtained, and the high-dimensional occlusion text embedding and the high-dimensional background text embedding are fused through the fusion layer to obtain the fused background text embedding;

[0010] S4, after projecting the high-dimensional visual embedding to the same dimension as the text embedding through the projection layer, perform cosine similarity operation with the high-dimensional target text embedding and the fused background text embedding to obtain the target attention map and the background attention map;

[0011] S5, the target attention map is filtered to obtain target pixel regions, and feature enhancement based on an attention mechanism is performed using the high-dimensional visual embedding and the high-dimensional target text embedding of the target pixel regions to obtain enhanced visual features; and

[0012] S6, the target pedestrian is stored in a memory bank according to the enhanced visual features, and updated in the form of momentum during training.

[0013] The specific method of step S1 is as follows: input the original image. Obtain multi-layer high-dimensional visual embeddings, where each layer has a dimension of . Where h and w are the number of blocks in the height and width dimensions after the image is segmented. The channel dimension is embedded in the high-dimensional vision.

[0014] The specific method for step S2 is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Layer-height visual embedding Global embedding With local embedding After performing cross-attention, the following formula (1) is used to obtain the first... Layer text prompts :

[0015] ...(1)

[0016] in For cross attention, For projection layer, To generate the prompt layer, For the number of visual embedding groups, For the first The target text tooltip, the occlusion text tooltip, or the background text tooltip.

[0017] Step S2 further includes: calculating and obtaining the learnable prompts of the target text prompt, the occlusion text prompt and the background text prompt according to the formula (1) to obtain the target text prompt, the occlusion text prompt and the background text prompt of the final layer.

[0018] Furthermore, in step S3: the high-dimensional target text embedding is... The fused background text is embedded as The two dimensions are ,in Set the text to a fixed length. For text embedding dimensions.

[0019] Furthermore, step S4 specifically involves: the target attention map and the background attention map being calculated using the following formula (2):

[0020] ...(2)

[0021] in This represents a target attention map or a background attention map. This represents the cosine similarity operation. It is a resampling function that upsamples the calculated attention map to the original image size. , This represents the projection matrix, which projects the visual embedding of each layer onto the same dimension as the text embedding.

[0022] Specifically, step S5 involves enhancing visual features. It is calculated using the following formula (3):

[0023] ... (3)

[0024] in It is an eigenvector concatenation operation. It is an attention-based feature enhancement module. This indicates the global embedding at the last level. Indicates target text embedding, This represents the filtered local visual embedding.

[0025] Preferably, step S6 further includes centroid aggregation of the feature vector of the target pedestrian, specifically including the following steps:

[0026] S61, Create a memory bank K , used to store the centroid vector of the target pedestrian, assuming the momentum update coefficient is . m ;

[0027] S62, Extract target pedestrian p eigenvectors ;

[0028] S63, inspect the target pedestrian p Does the centroid vector exist in the memory bank? K In the middle, when the target pedestrian p When the centroid vector does not exist, the eigenvector Record as the target pedestrian p The initial centroid vector, i.e. When the target pedestrian p When the centroid vector of the target pedestrian exists, the target pedestrian is updated using the following formula (4). p The centroid vector :

[0029] ... (4)

[0030] S64, Calculate the feature vector With the centroid vector The distance loss is used to bring the model to convergence. Specifically, the memory loss function is calculated using the following formula (5). :

[0031] ...(5)

[0032] Where c represents the total number of pedestrians in the training target.

[0033] Preferably, step S6 further includes supervision during training using the following loss function:

[0034] The re-identification function is supervised specifically by a weighted average of the cross-entropy loss function and the triplet loss function, and the re-identification function is calculated using formula (6). :

[0035] ... (6)

[0036] in, Represents the cross-entropy loss function. Represents the triplet loss function. This represents the enhanced re-identification features;

[0037] The mask supervision function is calculated using the following formula (7) based on the external mask and the target attention map and the fused background attention map obtained in the model. :

[0038] ... (7)

[0039] in, For focal loss, For dice loss, For external mask, For the target attention map, This is a background attention map.

[0040] The present invention also provides a pedestrian re-identification device for occlusion, characterized in that the device comprises:

[0041] A visual encoder is used to process the raw image of the target pedestrian to obtain a high-dimensional visual embedding.

[0042] A target cue learner, connected to a visual encoder, is used to obtain target text cues through the high-dimensional visual embedding;

[0043] An occlusion cue learner, connected to a visual encoder, is used to obtain occlusion text cues through high-dimensional visual embedding;

[0044] A background cue learner, connected to a visual encoder, is used to obtain background text cues through high-dimensional visual embedding.

[0045] A text encoder, connected to a target cue learner, an occlusion cue learner, and a background cue learner, is used to obtain high-dimensional target text embeddings, high-dimensional occlusion text embeddings, and high-dimensional background text embeddings based on the target text cue, the occlusion text cue, and the indicated background text cue.

[0046] The cross-modal interaction module, connected to the text encoder, is used to fuse the high-dimensional occlusion text embedding and the high-dimensional background text embedding through a fusion layer to obtain a fused background text embedding; to project the high-dimensional visual embedding to the same dimension as the text embedding through a projection layer; and to perform cosine similarity calculation on the projected visual embedding, the high-dimensional target text embedding and the fused background text embedding to obtain a target attention map and a background attention map.

[0047] The computation and enhancement module, connected to the visual encoder and the cross-modal interaction module, is used to filter the target attention map to obtain target pixel regions, and to perform attention-based feature enhancement using the high-dimensional visual embedding and the high-dimensional target text embedding of the target pixel regions to obtain enhanced visual features; and

[0048] The memory bank, connected to the computational enhancement module, is used to store the enhanced visual features of the target pedestrian and is updated in momentum form during training.

[0049] This invention utilizes independent target cue learners, occlusion cue learners, and background cue learners. This enables the model to update and learn text cue layer by layer based on the features of different regions and levels of the image, using shallower-level text cue combined with deeper-level visual features. This strengthens the correlation between visual features and text cue layer by layer and enhances the model's ability to distinguish between targets, occlusions, and background.

[0050] This invention combines the alignment capabilities of the CLIP visual encoder and text encoder to extract visual and textual features, obtaining accurate target and background attention maps. Through a cross-modal interaction module and an attention mechanism, it effectively fuses textual and visual features. This design allows the model to selectively focus on specific regions in occluded scenarios, improving its ability to accurately locate targets in occluded environments.

[0051] This invention employs supervision from a re-identification function, a mask supervision function, and a memory loss function during training. This guides the model to gradually acquire feature extraction capabilities and simultaneously directs the target attention map and background attention map to focus on accurate locations, enabling the model to distinguish between different targets. By combining supervision from these three loss functions, this invention effectively reduces confusion caused by occlusion or similar appearances, thereby significantly improving the performance of the pedestrian re-identification model in complex occlusion scenarios.

[0052] Therefore, the occluded pedestrian re-identification method disclosed in this invention reduces false detections and false negatives in pedestrian re-identification in occluded scenarios, thereby improving the performance of pedestrian re-identification. Attached Figure Description

[0053] Figure 1 This is a flowchart of a pedestrian re-identification method for occlusion provided in an embodiment of the present invention;

[0054] Figure 2 This is a block diagram of an occluded pedestrian re-identification device provided in one embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram of the structure of the prompting learner provided in an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the cross-modal interaction module provided in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of the calculation and enhancement module provided in an embodiment of the present invention;

[0058] Figure 6 The diagram illustrates the average recognition accuracy and first selection accuracy of the occluded pedestrian re-identification method provided in this embodiment of the invention on three occluded scene pedestrian re-identification datasets (Occluded-Duke, Occluded-ReID, P-DukeMTMC) and two general scene pedestrian re-identification datasets (Market-1501, DukeMTMC). Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0060] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0061] Figure 1 This is a flowchart of an occluded pedestrian re-identification method according to an embodiment of the present invention, as shown below. Figure 1 As shown, the occluded pedestrian re-identification method disclosed in this invention includes the following steps:

[0062] Step S1: Input the original image of the target pedestrian into the visual encoder to obtain a high-dimensional visual embedding; specifically, input the original image containing the target pedestrian. Obtain multi-layer high-dimensional visual embeddings, where each layer has a dimension of . h and w are the number of blocks in the height and width dimensions of the image after it has been segmented. This refers to the high-dimensional visual embedding channel dimension. In one embodiment, the 12 layers in the encoder are divided into four groups: layers 1-3 form the first group, layers 4-6 form the second group, layers 7-9 form the third group, and layers 10-12 form the fourth group. The high-dimensional visual embeddings extracted from the last layer of each group, i.e., layers 3, 6, 9, and 12, are denoted as […]. , , , ,in Recorded as the number i The high-dimensional visual embedding of the layer, the dimension of the extracted high-dimensional visual embedding of the layer is 1. The number of embeddings is This is because the visual encoder adds an extra global feature to each image during encoding. Correspondingly, the remaining local visual embeddings represent the corresponding blocks of the image, denoted as... The method of selecting a specific layer in the visual encoder to extract high-dimensional visual embeddings and the number of groups into which the visual encoder is divided are merely one example, not a limitation.

[0063] Step S2: Input the high-dimensional visual embedding into the cue learner to obtain the target text cue, the occluded object text cue, and the background text cue. In one embodiment, the 12 layers in the encoder are divided into four groups, and the high-dimensional visual embeddings obtained from layers 3, 6, 9, and 12 are extracted and denoted as follows: , , , ,in Recorded as the number i High-dimensional visual embedding of layers, high-dimensional visual embedding After being input into the visual cue learner, learnable text cues describing the visual image are obtained. The target, occlusions, and background in the visual image are learned using target cue learners, occlusion cue learners, and background cue learners, respectively. Within the cue learner, multiple cross-attention mechanisms and a cue generation layer consisting of projection layers form a chained operation, with each layer corresponding to the processing of its own high-dimensional visual embedding. Specifically, the high-dimensional visual embedding... Global embedding With local embedding After cross-attention, it is connected to the cues generated in the previous layer. The cues learner obtains the text cues for this layer using the following formula (1):

[0064] ...(1)

[0065] in, For cross attention, For projection layer, To generate the prompt layer, For the number of visual embedding groups, For the first The target text tooltip, the occlusion text tooltip, or the background text tooltip.

[0066] Step S3: Input the target text hint, occlusion text hint, and background text hint into the text encoder to obtain high-dimensional target text embedding, high-dimensional occlusion text embedding, and high-dimensional background text embedding. Then, fuse the high-dimensional occlusion text embedding and high-dimensional background text embedding through a fusion layer to obtain the fused background text embedding. Specifically, input the multiple text hints learned in the hint learner into the text encoder to obtain the high-dimensional target text embedding. Blend background text embedded as The two dimensions are ,in Set the text to a fixed length. For text embedding dimensions.

[0067] Step S4: Project the high-dimensional visual embedding to the same dimension as the text embedding through the projection layer, and then perform cosine similarity calculation with the high-dimensional target text embedding and the fused background text embedding to obtain the target attention map and the background attention map; specifically, the cosine similarity is calculated according to the following formula (2):

[0068] ...(2)

[0069] in This represents a target attention map or a background attention map; This represents the cosine similarity operation; It is a resampling function that upsamples the calculated attention map to the original image size; ,when hour, For the target attention diagram, hour, Background image for attention; among which This is a projection matrix that projects the visual embeddings of each layer onto the same dimension as the text embeddings. Specifically, the visual embeddings are aligned with the text embeddings, allowing each visual embedding to be judged for its similarity to the target and background embeddings, and the degree of membership of the visual embedding to the target is obtained through a softmax operation. Through a multi-layer mask supervision mechanism, the text prompts corresponding to the target and background can be continuously aligned with their respective visual embeddings.

[0070] Step S5: Filter the target attention map to obtain target pixel regions, and use the high-dimensional visual embeddings and high-dimensional target text embeddings of the target pixel regions for attention-based feature enhancement to obtain enhanced visual features. Specifically, based on the membership degree of the high-dimensional visual embeddings to the high-dimensional target text embeddings, all visual embeddings can be sorted. In one embodiment of the present invention, the last layer of visual embeddings is selected and interactively calculated with the target text embedding and the fused background text embedding to obtain the target attention map and the background attention map. The result of softmax normalization between the two maps is used as the membership degree of the visual embedding to the target text embedding. The visual embeddings of the last layer of the visual encoder 21 are sorted, and visual embeddings are selected proportionally for calculating enhanced features. After performing attention-based feature enhancement using the selected visual embeddings and target text embeddings, the enhanced visual features are obtained by concatenating them with the visual global embedding. It is calculated using the following formula (3):

[0071] ... (3)

[0072] in It is an eigenvector concatenation operation; It is an attention-based feature enhancement module, consisting of a cross attention module and two self-attention modules connected in series with the feedforward network; This indicates the global embedding at the last level; Indicates target text embedding; This represents the filtered local visual embedding.

[0073] Step S6: Store the enhanced visual features of the target pedestrian using a memory bank and update them in momentum form during training; specifically, step S6 also includes aggregating the centroid of the feature vector of the target pedestrian. p Taking the update process as an example, it is implemented through the following steps:

[0074] Step S61, Create a bank memory K , used to store the centroid vector of the target pedestrian, let the momentum update coefficient be. m ;

[0075] Step S62, extract the target pedestrian p eigenvectors The target pedestrian here p This represents one of the target pedestrians;

[0076] Step S63, inspect the target pedestrian p Does the centroid vector exist in the memory? K In the middle, when the target pedestrian p When the centroid vector does not exist, the eigenvector Record as target pedestrian p The initial centroid vector, i.e. When the target pedestrian p When the centroid vector exists, the target pedestrian is updated using the following formula (4). p centroid vector :

[0077] ... (4)

[0078] Step S64, calculate the feature vector With the centroid vector The distance loss is used to bring the model to convergence. Specifically, the memory loss function is calculated using the following formula (5). :

[0079] ...(5)

[0080] Where c represents the total number of pedestrians in the training target.

[0081] Furthermore, according to an embodiment of the present invention, when using the occluded pedestrian re-identification method disclosed in the present invention, in addition to the memory bank loss function used in step S64... In addition to supervision, it also includes supervision during training through the following loss function:

[0082] The re-identification function is supervised by a weighted average of the cross-entropy loss function and the triplet loss function, and the re-identification function is calculated using formula (6). :

[0083] ... (6)

[0084] in, Represents the cross-entropy loss function. Represents the triplet loss function. This indicates the enhanced re-identification features and

[0085] The mask supervision function is calculated using the following formula (7) based on the external mask and the target attention map and fused background attention map obtained in the model. :

[0086] ... (7)

[0087] in, For focal loss, For dice loss, For external mask, For the target attention map, This is a background attention map.

[0088] Figure 2 This is a block diagram of an occluded pedestrian re-identification device provided in one embodiment of the present invention. Figure 2 As shown, the occluded pedestrian re-identification device 200 includes: a visual encoder 21, a cue learner 22, a text encoder 23, a cross-modal interaction module 24, a calculation and enhancement module 25, and a memory bank 26. The cue learner 22 includes a target cue learner 221, an occlusion cue learner 222, and a background cue learner 223. The cue learner 22 is connected to the visual encoder 21 and the text encoder 23. Specifically, the target cue learner 221, the occlusion cue learner 222, and the background cue learner 223 in the cue learner 22 are all connected to the visual encoder 21 and the text encoder 23. The cross-modal interaction module 24 is connected to the text encoder 23. The calculation and enhancement module 25 is connected to the visual encoder 21 and the cross-modal interaction module 24. The memory bank 26 is connected to the calculation and enhancement module 25.

[0089] Specifically, the visual encoder 21 processes the original image of the target pedestrian to obtain a high-dimensional visual embedding; the target cue learner 221 obtains target text cue through the high-dimensional visual embedding; the occlusion cue learner 222 obtains occlusion text cue through the high-dimensional visual embedding; the background cue learner 223 obtains background text cue through the high-dimensional visual embedding; the text encoder 23 obtains high-dimensional target text embedding, high-dimensional occlusion text embedding, and high-dimensional background text embedding based on the target text cue, occlusion text cue, and background text cue; and the cross-modal interaction module 24 integrates the high-dimensional occlusion text embedding and the high-dimensional background text embedding. This embedding is achieved by fusing the background text embedding through a fusion layer, and by projecting the high-dimensional visual embedding onto the same dimension as the text embedding. Cosine similarity is then performed on the high-dimensional visual embedding, the high-dimensional target text embedding, and the fused background text embedding to obtain the target attention map and the background attention map. The calculation and enhancement module 25 is used to filter and obtain the target pixel region based on the target attention map, and to perform feature enhancement based on the attention mechanism using the high-dimensional visual embedding and the high-dimensional target text embedding of the corresponding target pixel region to obtain enhanced visual features. The memory bank 26 is used to store the enhanced visual features of the target pedestrian and is updated in the form of momentum during training.

[0090] Specifically, the occluded pedestrian re-identification device disclosed in this invention is mainly constructed based on the CLIP model method. The CLIP model is a multimodal pre-trained neural network that is pre-trained using a large amount of paired image and text data to learn the alignment relationship between images and text. The visual encoder 21 includes two forms: one using a deep residual model as its basic architecture, and the other employing a visual Transformer model. The visual Transformer model separates the image into multiple small blocks and unfolds them pixel by pixel. It then uses multiple Transformer layers to extract features from each small block, thereby obtaining the visual embedding of the image. Because the visual encoder 21 adopts a multi-layer architecture, it can extract the processing results of each layer while obtaining the final high-dimensional embedding.

[0091] The text encoder 23 employs a Transformer architecture to extract high-dimensional embeddings of the text, which are then used to align the high-dimensional visual embeddings with the high-dimensional target text. The process of text encoder 23 in handling visual embeddings and text alignment is as follows: first, the text is segmented into multiple markers, and these markers are mapped into discrete integer targets according to inherent rules; then, the Transformer further transforms these integer targets into dense embedding representations. Both the visual encoder 21 and the text encoder 23 mentioned above use Transformer encoders pre-trained on a large number of paired image-text sequences, and the information they extract only provides preliminary alignment performance.

[0092] Figure 3This is a schematic diagram of the structure of the prompt learner provided in an embodiment of the present invention. The prompt learner 22 includes a target prompt learner 221, an occlusion prompt learner 222, and a background prompt learner 223. These three independent learners are respectively used to learn the text descriptions of the target, occlusion, and background in the image to form corresponding text prompts. Specifically, as shown... Figure 3 As shown, each independent cue learner is composed of a chain of multiple cross-attention and cue-generating layers. This cue learner learns the textual description of the image using multi-layered high-dimensional visual embeddings extracted by the visual encoder 21. The cue learner consists of two parts: a fixed textual description and a learnable description. When the model is supervised using prior information from external masks, the feature learning part adaptively focuses on and describes different locations within the same image as the supervision is passed on. The textual cue obtained by the cue learner is finally encoded by the text encoder 23 to obtain the corresponding target text embedding, occlusion text embedding, and background text embedding.

[0093] In one embodiment, the specific process of selecting an image to extract a high-dimensional visual embedding, learning target, occlusion, and background cues, and obtaining the final corresponding text embedding is as follows: Input image The visual encoder 21 encodes multi-layer high-dimensional visual embeddings. The 12-layer visual encoder is divided into four groups: layers 1-3 form the first group, layers 4-6 the second group, layers 7-9 the third group, and layers 10-12 the fourth group. The high-dimensional visual embeddings extracted from the last layer of each group, i.e., layers 3, 6, 9, and 12, are denoted as […]. , , , ,in Recorded as the number i High-dimensional visual embedding of layers , , , The dimension of each visual embedding layer is 1. h and w are the number of blocks in the height and width dimensions after the image is segmented. For visual embedding channels, the number of embeddings is [number missing]. This is because the visual encoder 21 adds an extra global feature to each image during encoding. Correspondingly, the remaining local visual embeddings represent the corresponding blocks of the image, denoted as... .like Figure 3 As shown, each section corresponds to processing the visual embedding of each layer. Specifically, the first... i High-dimensional visual embedding of layers Global embedding With local embedding After cross-attention, it is connected to the text prompt generated in the previous section to enter the prompt generation layer to obtain the text prompt for this section. The specific method for obtaining the text prompt is referred to the above formula (1), and will not be described in detail here to avoid redundancy. Those skilled in the art should understand that dividing the visual encoder into four groups here is only for better description of the present invention and should not be regarded as a limitation of the present invention. Other forms of grouping may also be used.

[0094] In three independent cue learners, learnable cues for the learning target, occlusion, and background are obtained through formula (1). Finally, the learned cues are combined with text cues: the target text cue is "A photo of an occluded person"; the occlusion text cue is "A photo of someone blocking others"; and the background text cue is "A photo of background". The target text cue, occlusion text cue, and background text cue in the final layer are extracted by the text encoder 23 to obtain their corresponding text embeddings.

[0095] Figure 4 This is a schematic diagram of the structure of the cross-modal interaction module 24 provided in an embodiment of the present invention. Figure 4 As shown, the cross-modal interaction module 24 includes a projection layer and a fusion layer. The projection layer is used to map the high-dimensional visual embedding to a unified dimension of the text embedding; the fusion layer is used to fuse the occluded text embedding and the background text embedding to obtain a fused background text embedding. Figure 4 It can be seen that the projected visual embedding and text embedding have the same dimension and have alignment properties. Therefore, the visual embedding, target text embedding, and fused background text embedding can be used to perform cosine similarity operations to obtain the target attention map and background attention map.

[0096] Figure 5 This is a schematic diagram of the structure of the calculation and enhancement module 25 provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the computation and enhancement module 25 includes: a cross-attention layer, two self-attention layers, and two feedforward network layers, for... Figure 4 The target attention map is filtered to obtain the target pixel region, and the target text is embedded and the filtered visual embedding is performed. That is, the high-dimensional visual embedding containing the target pixel region is enhanced by the attention mechanism to obtain enhanced visual features.

[0097] Figure 6This diagram illustrates the average recognition accuracy and first-selection accuracy of the occluded pedestrian re-identification method provided in this embodiment of the invention on three occluded scene pedestrian re-identification datasets (Occluded-Duke, Occluded-ReID, P-DukeMTMC) and two general scene pedestrian re-identification datasets (Market-1501, DukeMTMC). Figure 6 It can be seen that on multiple public datasets for occluded and general scenarios, the system performs well in terms of first-choice accuracy (Rank-1) and average accuracy (mAP), which are important in the field of re-identification.

[0098] This invention utilizes independent target cue learners, occlusion cue learners, and background cue learners. This enables the model to update and learn text cue layer by layer based on the features of different regions and levels of the image, using shallower-level text cue combined with deeper-level visual features. This strengthens the correlation between visual features and text cue layer by layer and enhances the model's ability to distinguish between targets, occlusions, and background.

[0099] This invention combines the alignment capabilities of the CLIP visual encoder and text encoder to extract visual and textual features, obtaining accurate target and background attention maps. Through a cross-modal interaction module and an attention mechanism, it effectively fuses textual and visual features. This design allows the model to selectively focus on specific regions in occluded scenarios, improving its ability to accurately locate targets in occluded environments.

[0100] This invention employs supervision from a re-identification function, a mask supervision function, and a memory loss function during training. This guides the model to gradually acquire feature extraction capabilities and simultaneously directs the target attention map and background attention map to focus on accurate locations, enabling the model to distinguish between different targets. By combining supervision from these three loss functions, this invention effectively reduces confusion caused by occlusion or similar appearances, thereby significantly improving the performance of the pedestrian re-identification model in complex occlusion scenarios.

[0101] Therefore, the occluded pedestrian re-identification method disclosed in this invention reduces false detections and false negatives in pedestrian re-identification in occluded scenarios, thereby improving the performance of pedestrian re-identification.

[0102] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for re-identifying occluded pedestrians, characterized in that, Includes the following steps: S1, Input the original image of the target pedestrian into the visual encoder to obtain a high-dimensional visual embedding; S2, the high-dimensional visual embedding is input into the prompt learner to obtain the target text prompt, the occlusion text prompt and the background text prompt respectively; S3, the target text prompt, the occlusion text prompt, and the background text prompt are input into the text encoder to obtain high-dimensional target text embedding, high-dimensional occlusion text embedding, and high-dimensional background text embedding. The high-dimensional occlusion text embedding and the high-dimensional background text embedding are then fused through a fusion layer to obtain a fused background text embedding. The high-dimensional target text embedding is... The fused background text is embedded as The two dimensions are ,in Set the text to a fixed length. For text embedding dimensions; S4, after projecting the high-dimensional visual embedding to the same dimension as the text embedding through a projection layer, a cosine similarity operation is performed with the high-dimensional target text embedding and the fused background text embedding to obtain a target attention map and a background attention map, wherein the target attention map and the background attention map are calculated using the following formula: in This represents a target attention map or a background attention map. This represents the cosine similarity operation. It is a resampling function that upsamples the calculated attention map to the original image size. , This represents the projection matrix, which projects the visual embedding of each layer onto the same dimension as the text embedding. Indicates local embedding; S5, the target attention map is filtered to obtain target pixel regions, and attention-based feature enhancement is performed using the high-dimensional visual embedding and the high-dimensional target text embedding of the target pixel regions to obtain enhanced visual features, wherein the enhanced visual features It is calculated using the following formula: in It is an eigenvector concatenation operation. It is an attention-based feature enhancement module. This indicates the global embedding at the last level. Indicates target text embedding, This represents the filtered local visual embedding; and S6, the target pedestrian is stored in a memory bank according to the enhanced visual features, and updated in the form of momentum during training.

2. The method for re-identifying occluded pedestrians according to claim 1, characterized in that, The specific method for step S1 is as follows: Input the original image Obtain multi-layer high-dimensional visual embeddings, where each layer has a dimension of . Where h and w are the number of blocks in the height and width dimensions after the image is segmented. The channel dimension is embedded in the high-dimensional vision.

3. The method for re-identifying occluded pedestrians according to claim 1, characterized in that, The specific method of step S2 is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a Layer-height visual embedding Global embedding With local embedding After performing cross-attention, the following formula is used to obtain the first... Layer text hints : in For cross attention, For projection layer, To generate the prompt layer, For the number of visual embedding groups, For the first The target text tooltip, the occlusion text tooltip, or the background text tooltip.

4. The method for re-identifying occluded pedestrians according to claim 3, characterized in that, Step S2 further includes: according to the first... Layer text hints The formulas are used to calculate and obtain the learnable prompts for the target text prompt, the occlusion text prompt, and the background text prompt, respectively, to obtain the final layer of target text prompt, the final layer of occlusion text prompt, and the final layer of background text prompt.

5. The method for re-identifying occluded pedestrians according to claim 1, characterized in that, Step S6 further includes centroid aggregation of the feature vector of the target pedestrian, specifically including the following steps: S61, Create a memory bank K , used to store the centroid vector of the target pedestrian, assuming the momentum update coefficient is . m ; S62, Extract target pedestrian p eigenvectors ; S63, inspect the target pedestrian p Does the centroid vector exist in the memory bank? K In the middle, when the target pedestrian p When the centroid vector does not exist, the eigenvector Record as the target pedestrian p The initial centroid vector, i.e. When the target pedestrian p When the centroid vector of the target pedestrian exists, the target pedestrian is updated using the following formula. p The centroid vector : S64, Calculate the feature vector With the centroid vector The distance loss is used to bring the model to convergence. Specifically, the memory loss function is calculated using the following formula. : Where c represents the total number of pedestrians in the training target.

6. The method for re-identifying occluded pedestrians according to claim 5, characterized in that, Step S6 further includes supervision during training using the following loss function: The re-identification function is supervised, specifically by a weighted average of the cross-entropy loss function and the triplet loss function. The re-identification function is calculated using the following formula. : in, Represents the cross-entropy loss function. Represents the triplet loss function. This represents the enhanced re-identification features; The mask supervision function is calculated using the following formula based on the external mask and the target attention map and fused background attention map obtained from the model. : in, For focal loss, For dice loss, For external mask, For the target attention map, This is a background attention map.

7. A pedestrian re-identification device for obscured pedestrians, characterized in that, include: A visual encoder is used to process the raw image of the target pedestrian to obtain a high-dimensional visual embedding. A target cue learner, connected to a visual encoder, is used to obtain target text cues through the high-dimensional visual embedding; An occlusion cue learner, connected to a visual encoder, is used to obtain occlusion text cues through high-dimensional visual embedding; A background cue learner, connected to a visual encoder, is used to obtain background text cues through high-dimensional visual embedding. A text encoder, connected to a target cue learner, an occlusion cue learner, and a background cue learner, is used to obtain high-dimensional target text embeddings, high-dimensional occlusion text embeddings, and high-dimensional background text embeddings based on the target text cue, the occlusion text cue, and the indicated background text cue. A cross-modal interaction module, connected to a text encoder, is used to fuse the high-dimensional occlusion text embedding and the high-dimensional background text embedding through a fusion layer to obtain a fused background text embedding, and to project the high-dimensional visual embedding to the same dimension as the text embedding through a projection layer. Then, cosine similarity calculation is performed on the projected visual embedding, high-dimensional target text embedding, and fused background text embedding to obtain the target attention map and the background attention map; The computation and enhancement module, connected to the visual encoder and the cross-modal interaction module, is used to filter the target attention map to obtain target pixel regions, and to perform attention-based feature enhancement using the high-dimensional visual embedding and the high-dimensional target text embedding of the target pixel regions to obtain enhanced visual features. as well as A memory bank, connected to the computational enhancement module, is used to store the enhanced visual features of the target pedestrian and is updated in momentum form during training. The high-dimensional target text embedding is... The fused background text is embedded as The two dimensions are ,in Set the text to a fixed length. For the text embedding dimension, the target attention map and the background attention map are calculated using the following formula: in This represents a target attention map or a background attention map. This represents the cosine similarity operation. It is a resampling function that upsamples the calculated attention map to the original image size. , This represents the projection matrix, which projects the visual embedding of each layer onto the same dimension as the text embedding. This indicates local embedding, the enhanced visual features It is calculated using the following formula: in It is an eigenvector concatenation operation. It is an attention-based feature enhancement module. This indicates the global embedding at the last level. Indicates target text embedding, This represents the filtered local visual embedding.

Images