Shielded pedestrian re-identification method and device

Abstract

The invention discloses an occluded pedestrian re-identification method. The method comprises the following steps: inputting an original image of a target pedestrian into a visual encoder to obtain high-dimensional visual embedding; inputting the high-dimensional visual embedding into a prompt learning device, and obtaining a target text prompt, a shelter text prompt and a background text prompt; fusing the high-dimensional shelter text embedding and the high-dimensional background text embedding through a fusion layer to obtain a fused background text embedding; performing cosine similarity operation on high-dimensional visual embedding, high-dimensional target text embedding and fused background text embedding to obtain a target attention map and a background attention map; enhanced visual features are obtained; and storing the target pedestrian by using a memory bank according to the enhanced visual features. According to the sheltered pedestrian re-identification method disclosed by the invention, the pedestrian re-identification performance is improved by designing the independent target prompt learner, the sheltered object prompt learner and the background prompt learner.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision image processing, and more specifically, relates to a method and device for re-identifying an occluded pedestrian. Background Art

[0002] Person Re-Identification (ReID) is an important and challenging key problem in the field of computer vision. It aims to accurately identify a specific individual by comparing images from different camera perspectives. This task is widely used in fields such as multi-target tracking, motion analysis, group monitoring, and video surveillance. The core goal of ReID is to achieve accurate matching and recognition of the same pedestrian captured across different perspectives and under different conditions, thereby providing reliable support for intelligent monitoring and behavior analysis.

[0003] Although the relevant technologies of ReID have been developed to a certain extent, in complex urban environments, especially in the case of dense crowds and diverse objects, general pedestrian re-identification methods will have difficulty in performing as expected due to occlusion problems. This is because in complex urban backgrounds, pedestrians and pedestrians and various objects may be occluded at different positions and parts, which will cause the key parts of the target pedestrians to be unable to be fully captured, resulting in inconsistent or missing features, and traditional ReID methods are difficult to effectively match and identify based on incomplete pedestrian feature information due to partial or complete occlusion.

[0004] In recent years, pedestrian re-identification methods based on contrastive language-image pretraining (CLIP) have been proposed and have made significant progress. When dealing with the problem of occluded pedestrian re-identification, the main advantage of the CLIP-based ReID method is that it can understand the target object at the text level, thereby providing supplementary descriptions and information for different parts of the pedestrian. CLIP-ReID overcomes the limitations of traditional methods to a certain extent through multimodal learning, enabling the model to better perform feature matching when facing partially occluded pedestrians.

[0005] However, despite the advantages of the CLIP-ReID method, its application in occluded pedestrian re-identification still faces some problems. First, existing CLIP-ReID methods usually use a single target cue to align with the image, or use multiple cues to capture the features of different parts of the target and align with the image. When multiple pedestrians are occluded, these methods often have difficulty in effectively handling occluded targets because they lack sufficient text understanding capabilities to cope with complex occlusion scenarios. In particular, when multiple pedestrians are occluded at the same time, existing CLIP image-text alignment methods are prone to mismatching and have difficulty in correctly identifying occluded pedestrians. In addition, existing CLIP-ReID methods often use contrast loss based on the entire image for training. This method fails to fully utilize the contextual information of the occluded area, resulting in limited recognition capabilities for the occluded area. Therefore, although CLIP-ReID can improve the performance of pedestrian re-identification under certain conditions, its performance is still greatly limited in the case of severe occlusion. Summary of the invention

[0006] In view of the defects of the prior art, the occluded pedestrian re-identification method disclosed in the present invention reduces the false detection and missed detection of pedestrian re-identification in occluded scenes, and improves the performance of pedestrian re-identification by designing independent target prompt learners, occluded object prompt learners, and background prompt learners. The present invention provides an occluded pedestrian re-identification method, comprising:

[0007] S1, input the original image of the target pedestrian into the visual encoder to obtain a high-dimensional visual embedding;

[0008] S2, inputting the high-dimensional visual embedding into a prompt learner to obtain target text prompts, occluder text prompts, and background text prompts respectively;

[0009] S3, inputting the target text prompt, the occluder text prompt and the background text prompt into a text encoder to obtain a high-dimensional target text embedding, a high-dimensional occluder text embedding and a high-dimensional background text embedding, and fusing the high-dimensional occluder text embedding and the high-dimensional background text embedding through a fusion layer to obtain a fused background text embedding;

[0010] S4, after projecting the high-dimensional visual embedding to the same dimension as the text embedding through the projection layer, performing a cosine similarity operation with the high-dimensional target text embedding and the fused background text embedding to obtain a target attention map and a background attention map;

[0011] S5, screening the target attention map to obtain a target pixel area, and using the high-dimensional visual embedding of the target pixel area and the high-dimensional target text embedding to perform feature enhancement based on the attention mechanism to obtain enhanced visual features; and

[0012] S6, storing the target pedestrian in a memory bank according to the enhanced visual features, and updating it in a momentum form during training.

[0013] The specific method of step S1 is as follows: input the original image I∈R H×W×3 , obtain multi-layer high-dimensional visual embedding, where the dimension of each layer is Where h and w are the number of small blocks in the height and width dimensions after the image is divided into small blocks, C v Embedding channel dimensions for high-dimensional vision.

[0014] The specific method of step S2 is: embed the i-th layer of high-dimensional vision into V i Global embedding of With local embedding After cross attention, the text prompt P of the i-th layer is obtained by the following formula (1): i :

[0015]

[0016] Among them, CA is cross attention, Proj is the projection layer, PG is i is to generate the hint layer, S is the number of visual embedding groups, P i It is one of the target text prompt, occluder text prompt or background text prompt of the i-th layer.

[0017] The step S2 also includes: calculating and obtaining the learnable prompts of the target text prompt, the occluder text prompt and the background text prompt respectively according to the formula (1), and obtaining the target text prompt of the final layer, the occluder text prompt of the final layer and the background text prompt of the final layer.

[0018] Furthermore, in step S3: the high-dimensional target text is embedded as T Tar , the fused background text is embedded as The dimensions of both are Where L is the fixed length of the text, C t Embedding dimension for text.

[0019] Furthermore, the step S4 is specifically as follows: the target attention map and the background attention map are calculated by the following formula (2):

[0020]

[0021] Among them, M j represents the target attention map or the background attention map, cos(·,·) represents the cosine similarity operation, and φ is a resampling function that upsamples the calculated attention map to the original image size. Wi represents the projection matrix, which projects the visual embedding of each layer to the same dimension as the text embedding.

[0022] Wherein, the step S5 specifically comprises: the enhanced visual feature F e It is calculated by the following formula (3):

[0023]

[0024] Where concat(·,·) is a feature vector concatenation operation, FSE(·) is an attention-based feature enhancement module, represents the global embedding of the last layer, T Tar represents the target text embedding, Represents the filtered local visual embedding.

[0025] Preferably, the step S6 further includes aggregating the centroid of the feature vector of the target pedestrian, specifically including the following steps:

[0026] S61, creating a memory bank K for storing the centroid vector of the target pedestrian, and setting the momentum update coefficient to m;

[0027] S62, extract the feature vector of the target pedestrian p

[0028] S63, checking whether the centroid vector of the target pedestrian p exists in the memory bank K. When the centroid vector of the target pedestrian p does not exist, the feature vector Recorded as the initial centroid vector of the target pedestrian p, that is, When the centroid vector of the target pedestrian p exists, the centroid vector K[p] of the target pedestrian p is updated by the following formula (4):

[0029]

[0030] S64, calculating the feature vector The distance loss from the centroid vector K[p] makes the model converge. Specifically, the memory library loss function is calculated by the following formula (5):

[0031]

[0032] Where c is the total number of training target pedestrians.

[0033] Preferably, step S6 also includes supervising the training through the following loss function:

[0034] The re-identification function supervision is specifically performed through the weighted cross entropy loss function and the triplet loss function. The re-identification function is calculated using formula (6):

[0035]

[0036] in,

[0037] represents the cross entropy loss function, represents the triple loss function, F e represents the enhanced re-identification features;

[0038] Mask supervision function supervision, the mask supervision function is calculated by the following formula (7) between the target attention map and the fused background attention map obtained in the external mask and the model:

[0039]

[0040] in, is the focal loss, is dice loss, G is the external mask, M Tar is the target attention map, is the background attention map.

[0041] The present invention also provides a device for re-identifying an obstructed pedestrian, characterized in that the device comprises:

[0042] The visual encoder is used to process the original image of the target pedestrian to obtain a high-dimensional visual embedding;

[0043] a target cue learner, connected to the visual encoder, for obtaining a target text cue through the high-dimensional visual embedding;

[0044] The occluder hint learner is connected to the visual encoder and is used to obtain occluder text hints through high-dimensional visual embeddings;

[0045] A background cue learner, connected to the visual encoder, is used to obtain background text cues through high-dimensional visual embeddings;

[0046] A text encoder connected to the target prompt learner, the occluder prompt learner and the background prompt learner, and configured to obtain a high-dimensional target text embedding, a high-dimensional occluder text embedding and a high-dimensional background text embedding according to the target text prompt, the occluder text prompt and the background text prompt;

[0047] A cross-modal interaction module is connected to the text encoder and is used to fuse the high-dimensional occluder text embedding and the high-dimensional background text embedding through a fusion layer to obtain a fused background text embedding, project the high-dimensional visual embedding to the same dimension as the text embedding through a projection layer; and perform cosine similarity operations on the projected visual embedding, the high-dimensional target text embedding, and the fused background text embedding to obtain a target attention map and a background attention map;

[0048] a calculation and enhancement module, connected to the visual encoder and the cross-modal interaction module, for screening the target attention map to obtain a target pixel region, and performing feature enhancement based on an attention mechanism using the high-dimensional visual embedding of the target pixel region and the high-dimensional target text embedding to obtain enhanced visual features; and

[0049] The memory bank is connected to the computing enhancement module, and is used to store the enhanced visual features of the target pedestrian and update them in the form of momentum during the training process.

[0050] The present invention uses independent target cue learners, occluder cue learners, and background cue learners to enable the model to update and learn text cue layer by layer according to the features of different areas and levels of the image, using shallower-level text cueing combined with deeper-level visual features, thereby strengthening the association between visual features and text cueing layer by layer and enhancing the model's ability to distinguish targets, occluders, and backgrounds.

[0051] The present invention combines the CLIP visual encoder and the text encoder to extract the alignment capabilities of visual features and text features, obtains accurate target attention maps and background attention maps, and effectively fuses text and visual features through a cross-modal interaction module combined with an attention mechanism. This design enables the model to selectively focus on specific areas in occluded scenes, and can improve the ability to accurately locate targets in occluded environments.

[0052] The present invention uses the supervision of the re-identification function, mask supervision function, and memory loss function during the training process, which can guide the model to gradually acquire the ability to extract features, while guiding the target attention map and background attention map to focus on the accurate position and distinguish different targets. By combining the supervision of the three loss functions, the present invention effectively reduces the confusion caused by occlusion or similar appearance, thereby significantly improving the performance of the pedestrian re-identification model in complex occlusion scenarios.

[0053] Therefore, the occluded pedestrian re-identification method disclosed in the present invention can reduce false detection and missed detection of pedestrian re-identification in occluded scenes, and improve the performance of pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1is a flow chart of a method for re-identifying an occluded pedestrian provided by an embodiment of the present invention;

[0055] Figure 2 is a module diagram of a device for re-identifying an occluded pedestrian provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the structure of a prompt learner provided by an embodiment of the present invention;

[0057] Figure 4 A schematic diagram of the structure of a cross-modal interaction module provided by an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the structure of a calculation and enhancement module provided in an embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the average recognition accuracy and first choice accuracy index results of the occluded pedestrian re-identification method provided in an embodiment of the present invention for three occluded scene pedestrian re-identification datasets (Occluded-Duke, Occluded-ReID, P-DukeMTMC) and two general scene pedestrian re-identification datasets (Market-1501, DukeMTMC) in occluded scenes. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0061] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0062] Figure 1 FIG. 1 is a flow chart of a method for re-identifying an occluded pedestrian provided by an embodiment of the present invention. Figure 1 As shown, the method for re-identifying an occluded pedestrian disclosed in the present invention comprises the following steps:

[0063] Step S1: Input the original image of the target pedestrian into the visual encoder to obtain a high-dimensional visual embedding; specifically, input the original image I∈R containing the target pedestrian H×W×3 , obtain multi-layer high-dimensional visual embedding, where the dimension of each layer is R (h ×w+1)×Cv , h and w are the number of small blocks in the height and width dimensions after the image is divided into small blocks, Cv In one embodiment, the 12 layers in the encoder are evenly divided into four groups, i.e., layers 1-3 are the first group, layers 4-6 are the second group, layers 7-9 are the third group, and layers 10-12 are the fourth group. The high-dimensional visual embeddings extracted from the last layers of the four groups, i.e., layers 3, 6, 9, and 12, are recorded as V 1 、V 2 、V 3 、V 4 , where V i Denoted as the high-dimensional visual embedding of the i-th layer, the dimension of the high-dimensional visual embedding of the extraction layer is R (h×w+1)×Cv The number of embeddings is h×w+1 because the visual encoder adds an additional global feature V to each image when encoding cls , accordingly, the remaining local visual embeddings represent the corresponding blocks of the image, denoted as V pat , here, the method of selecting a specific layer in the visual encoder to extract high-dimensional visual embedding and the number of groups into which the visual encoder is divided are merely an embodiment and not a limitation.

[0064] Step S2: Input the high-dimensional visual embedding into the prompt learner to obtain the target text prompt, the occluder text prompt and the background text prompt. In one embodiment, the 12 layers in the encoder are evenly divided into four groups, and the high-dimensional visual embedding obtained by extracting the 3rd, 6th, 9th and 12th layers is recorded as V 1 、V 2 、V 3 、V 4 , where V i Denoted as the high-dimensional visual embedding of the i-th layer, the high-dimensional visual embedding V i After being input into the visual cue learner, learnable textual cues describing the visual image are obtained. The target, occluder, and background in the visual image are learned using the target cue learner, occluder cue learner, and background cue learner, respectively. In the cue learner, multiple cross-attention and generative cue layers composed of projection layers form a chain operation, in which each layer corresponds to processing the high-dimensional visual embedding of each layer. Specifically, the high-dimensional visual embedding V i Global embedding of With local embedding After cross attention, it is connected with the prompt generated by the previous layer, and the prompt learner obtains the text prompt of this layer through the following formula (1):

[0065]

[0066] Among them, CA is cross attention, Proj is the projection layer, PG is i is to generate the hint layer, S is the number of visual embedding groups, P iIt is one of the target text prompt, occluder text prompt or background text prompt of the i-th layer.

[0067] Step S3: After inputting the target text prompt, the occluder text prompt and the background text prompt into the text encoder, a high-dimensional target text embedding, a high-dimensional occluder text embedding and a high-dimensional background text embedding are obtained, and the high-dimensional occluder text embedding and the high-dimensional background text embedding are fused through a fusion layer to obtain a fused background text embedding; specifically, the multiple text prompts learned in the prompt learner are input into the text encoder to obtain a high-dimensional target text embedding as T Tar , the fused background text is embedded as The two dimensions are Where L is the fixed length of the text, C t Embedding dimension for text.

[0068] Step S4: After the high-dimensional visual embedding is projected to the same dimension as the text embedding through the projection layer, a cosine similarity operation is performed with the high-dimensional target text embedding and the fused background text embedding to obtain the target attention map and the background attention map; specifically, the cosine similarity calculation is performed according to the following formula (2):

[0069]

[0070] Among them, M j represents the target attention map or the background attention map; cos(·,·) represents the cosine similarity operation; φ is a resampling function that upsamples the calculated attention map to the original image size; When T j =T Tar When M j is the target attention map, When M j is the background attention map; where W i is a projection matrix that projects the visual embedding of each layer to the same dimension as the text embedding. Specifically, the visual embedding is aligned with the text embedding, and each visual embedding can be judged to be similar to the target and background embeddings, and the degree of membership of the visual embedding to the target is obtained through the softmax operation. Through the multi-layer mask supervision mechanism, the textual prompt information corresponding to the target and background can be continuously aligned with their respective visual embeddings.

[0071] Step S5: Filter the target attention map to obtain the target pixel area, and use the high-dimensional visual embedding of the target pixel area and the high-dimensional target text embedding to perform feature enhancement based on the attention mechanism to obtain enhanced visual features. Specifically, all visual embeddings can be sorted according to the degree of membership of the high-dimensional visual embedding to the high-dimensional target text embedding. In one embodiment of the present invention, the last layer of visual embedding is selected to interact with the target text embedding and the fused background text embedding to obtain the target attention map and the background attention map. The result of softmax normalization between the two maps is used as the degree of membership of the visual embedding to the target text embedding. The visual embedding of the last layer of the visual encoder 21 is sorted, and the visual embedding is proportionally filtered for calculating the enhanced features. After the filtered visual embedding and the target text embedding are used for feature enhancement based on the attention mechanism, they are connected with the visual global embedding to obtain the enhanced visual feature F e , calculated by the following formula (3):

[0072]

[0073] Where concat(·,·) is a feature vector concatenation operation; FSE(·) is an attention-based feature enhancement module, which consists of a cross attention and two self-attention modules connected in series with a feedforward network; represents the global embedding of the last layer; T Tar represents the target text embedding; Represents the filtered local visual embedding.

[0074] Step S6: The target pedestrian is stored in a memory bank according to the enhanced visual features, and is updated in the form of momentum during the training process; specifically, step S6 also includes aggregating the centroid of the feature vector of the target pedestrian. The update process of the target pedestrian p is taken as an example to illustrate, which is specifically implemented by the following steps:

[0075] Step S61, create a memory bank K for storing the centroid vector of the target pedestrian, and set the momentum update coefficient to m;

[0076] Step S62: extract the feature vector of the target pedestrian p Here, the target pedestrian p represents one of the target pedestrians;

[0077] Step S63, check whether the centroid vector of the target pedestrian p exists in the memory bank K. When the centroid vector of the target pedestrian p does not exist, the feature vector Recorded as the initial centroid vector of the target pedestrian p, that is When the centroid vector of the target pedestrian p exists, the centroid vector K[p] of the target pedestrian p is updated by the following formula (4):

[0078]

[0079] Step S64, calculate the feature vector The distance loss from the centroid vector K[p] makes the model converge. Specifically, the memory library loss function is calculated by the following formula (5):

[0080]

[0081] Where c is the total number of training target pedestrians.

[0082] Further, according to an embodiment of the present invention, when the occluded pedestrian re-identification method disclosed in the present invention is used, in addition to the memory library loss function used in step S64 In addition to supervision, it also includes supervision during training through the following loss function:

[0083] The re-identification function is supervised by weighting the cross entropy loss function and the triplet loss function. The re-identification function is calculated using formula (6):

[0084]

[0085] in, represents the cross entropy loss function, represents the triple loss function, F e represents the enhanced re-identification features and

[0086] Mask supervision function supervision, based on the external mask and the target attention map obtained in the model and the fused background attention map, the mask supervision function is calculated using the following formula (7):

[0087]

[0088] in, is the focal loss, is dice loss, G is the external mask, M Tar is the target attention map, is the background attention map.

[0089] Figure 2 FIG. 1 is a block diagram of a device for re-identifying a pedestrian under occlusion provided by an embodiment of the present invention. Figure 2As shown, the occluded pedestrian re-identification device 200 includes: a visual encoder 21, a prompt learner 22, a text encoder 23, a cross-modal interaction module 24, a calculation and enhancement module 25 and a memory bank 26, wherein the prompt learner 22 includes a target prompt learner 221, an occluder prompt learner 222 and a background prompt learner 223. The prompt learner 22 is connected to the visual encoder 21, and the prompt learner 22 is connected to the text encoder 23. Specifically, the target prompt learner 221, the occluder prompt learner 222 and the background prompt learner 223 in the prompt learner 22 are all connected to the visual encoder 21 and the text encoder 23, the cross-modal interaction module 24 is connected to the text encoder 23, the calculation and enhancement module 25 is connected to the visual encoder 21 and the cross-modal interaction module 24, and the memory bank 26 is connected to the calculation and enhancement module 25.

[0090] Specifically, the visual encoder 21 is used to process the original image of the target pedestrian to obtain a high-dimensional visual embedding; the target prompt learner 221 is used to obtain the target text prompt through the high-dimensional visual embedding; the occluder prompt learner 222 is used to obtain the occluder text prompt through the high-dimensional visual embedding; the background prompt learner 223 is used to obtain the background text prompt through the high-dimensional visual embedding; the text encoder 23 is used to obtain the high-dimensional target text embedding, the high-dimensional occluder text embedding and the high-dimensional background text embedding according to the target text prompt, the occluder text prompt and the background text prompt; the cross-modal interaction module 24 is used to embed the high-dimensional occluder text embedding and the high-dimensional background text This embedding is fused through a fusion layer to obtain a fused background text embedding, and the high-dimensional visual embedding is projected to the same dimension as the text embedding, and the high-dimensional visual embedding, the high-dimensional target text embedding and the fused background text embedding are subjected to cosine similarity operations to obtain a target attention map and a background attention map; the calculation and enhancement module 25 is used to filter and obtain a target pixel area according to the target attention map, and to perform feature enhancement based on an attention mechanism using the high-dimensional visual embedding and the high-dimensional target text embedding of the corresponding target pixel area to obtain enhanced visual features; and the memory bank 26 is used to store the enhanced visual features of the target pedestrian and update them in the form of momentum during the training process.

[0091] Specifically, the occluded pedestrian re-identification device disclosed in the present invention is mainly constructed based on the method of the CLIP model. The CLIP model is a multimodal pre-trained neural network. The model is pre-trained using a large amount of paired data of images and texts to learn the alignment relationship between images and texts. The visual encoder 21 includes two forms, one using a deep residual model as the basic architecture, and the other using a visual Transformer model. The visual Transformer model separates the image into multiple small blocks and expands them pixel by pixel, and uses a multi-layer Transformer to extract each small block as a feature, thereby obtaining a visual embedding of the image. Since the visual encoder 21 adopts a multi-layer architecture, it is possible to extract the processing results of each layer while obtaining the final high-dimensional embedding.

[0092] The text encoder 23 uses the Transformer architecture to extract high-dimensional embeddings of text and align high-dimensional visual embeddings with high-dimensional target text. The process of the text encoder 23 processing visual embeddings and text alignment is as follows: first, the text is divided into multiple tokens, and the tokens are mapped into discrete integer targets according to inherent rules; the Transformer further converts the integer targets into dense embedding representations. The above-mentioned visual encoder 21 and text encoder 23 both use Transformer encoders pre-trained based on a large number of paired image texts, and the information extracted only has a preliminary alignment performance.

[0093] Figure 3 The schematic diagram of the structure of the prompt learner provided in the embodiment of the present invention, wherein the prompt learner 22 includes a target prompt learner 221, an occluder prompt learner 222 and a background prompt learner 223, and the three independent learners are used to learn the text descriptions of the target, occluder and background in the image respectively to form corresponding text prompts. Specifically, Figure 3 As shown, each independent prompt learner is composed of a chain combination of multiple cross-attention and generative prompt layers. The prompt learner uses the multi-layer high-dimensional visual embedding extracted by the image through the visual encoder 21 to learn the text description of the image. The prompt learner consists of two parts: a fixed text description and a learnable description. When the model is supervised as a whole using external mask prior information, the feature learning part also adaptively pays attention to and describes different positions of the same image as the supervision is transmitted. The text prompt obtained by the prompt learner is finally encoded by the text encoder 23 to obtain the corresponding target text embedding, occluder text embedding and background text embedding.

[0094] In one embodiment, the specific process of selecting an image to extract high-dimensional visual embedding, learning the target, occluder and background prompts, and obtaining the final corresponding text embedding is as follows: Input image I∈R H×W×3, the visual encoder 21 encodes to obtain multi-layer high-dimensional visual embedding. The 12-layer visual encoder is divided into four groups, where layers 1-3 are the first group, layers 4-6 are the second group, layers 7-9 are the third group, and layers 10-12 are the fourth group. The last layer in each group, i.e., the high-dimensional visual embedding extracted from layers 3, 6, 9, and 12 is recorded as V 1 、V 2 、V 3 、V 4 , where V i Denoted as the high-dimensional visual embedding of the i-th layer, V 1 、V 2 、V 3 、V 4 The dimension of each layer of visual embedding is h and w are the number of small blocks in the height and width dimensions after the image is divided into small blocks, C v is the visual embedding channel dimension. The number of embeddings is h×w+1 because the visual encoder 21 adds an additional global feature V to each image when encoding. cls , correspondingly, the remaining local visual embeddings represent the corresponding blocks of the image, denoted as V pat .like Figure 3 As shown in the figure, each section deals with the visual embedding of each layer respectively. Specifically, the high-dimensional visual embedding V of the i-th layer i Global embedding of With local embedding After cross attention, it is connected with the text prompt generated in the previous section, and enters the prompt generation layer to obtain the text prompt of this section. The specific method of obtaining the text prompt refers to the above formula (1). To avoid redundancy, it will not be described in detail here. Those skilled in the art should understand that the visual encoder is divided into four groups here only for better description of the present invention, and cannot be regarded as a limitation of the present invention. Specifically, other forms of grouping may also be used.

[0095] In three independent prompt learners, the learnable prompts of the target, occluder, and background are learned by formula (1). Finally, the learned prompts are combined with the text prompts, and the target text prompt is: A photo of occluded person; the occluder text prompt is: A photo of someone blocking others; and the background text prompt is: A photo of background. The target text prompt, occluder text prompt, and background text prompt of the final layer are extracted by the text encoder 23 to obtain the corresponding text embedding.

[0096] Figure 4 A schematic diagram of the structure of the cross-modal interaction module 24 provided in an embodiment of the present invention. Figure 4As shown, the cross-modal interaction module 24 includes a projection layer and a fusion layer. The projection layer is used to map the high-dimensional visual embedding to the uniform dimension of the text embedding; the fusion layer is used to fuse the occluder text embedding and the background text embedding through the fusion layer to obtain a fused background text embedding. Figure 4 It can be seen that the projected visual embedding has the same dimension as the text embedding and has the alignment property, so the visual embedding, target text embedding, and fused background text embedding can perform cosine similarity operations to obtain the target attention map and background attention map.

[0097] Figure 5 A schematic diagram of the structure of the calculation and enhancement module 25 provided in an embodiment of the present invention is shown in FIG. Figure 5 As shown, the calculation and enhancement module 24 includes: a cross attention layer, two self-attention layers and two feedforward network layers. Figure 4 The target attention map is filtered to obtain the target pixel area, and the target text is embedded and the filtered visual embedding, that is, the high-dimensional visual embedding containing the target pixel area is enhanced based on the attention mechanism to obtain enhanced visual features.

[0098] Figure 6 Schematic diagram of the average recognition accuracy and first choice accuracy of the pedestrian re-identification method provided by the embodiment of the present invention in three pedestrian re-identification datasets (Occluded-Duke, Occluded-ReID, P-DukeMTMC) and two general scene pedestrian re-identification datasets (Market-1501, DukeMTMC). Figure 6 It can be seen that on multiple public datasets of occluded scenes and general scenes, it can be seen that the first choice accuracy (Rank-1) and average recognition accuracy (mAP) that are important in the field of re-identification have good performance.

[0099] The present invention uses independent target cue learners, occluder cue learners, and background cue learners to enable the model to update and learn text cue layer by layer according to the features of different areas and levels of the image, using shallower-level text cueing combined with deeper-level visual features, thereby strengthening the association between visual features and text cueing layer by layer and enhancing the model's ability to distinguish targets, occluders, and backgrounds.

[0100] The present invention combines the CLIP visual encoder and the text encoder to extract the alignment capabilities of visual features and text features, obtains accurate target attention maps and background attention maps, and effectively fuses text and visual features through a cross-modal interaction module combined with an attention mechanism. This design enables the model to selectively focus on specific areas in occluded scenes, and can improve the ability to accurately locate targets in occluded environments.

[0101] The present invention uses the supervision of the re-identification function, mask supervision function, and memory loss function during the training process, which can guide the model to gradually acquire the ability to extract features, while guiding the target attention map and background attention map to focus on the accurate position and distinguish different targets. By combining the supervision of the three loss functions, the present invention effectively reduces the confusion caused by occlusion or similar appearance, thereby significantly improving the performance of the pedestrian re-identification model in complex occlusion scenarios.

[0102] Therefore, the occluded pedestrian re-identification method disclosed in the present invention can reduce false detection and missed detection of pedestrian re-identification in occluded scenes, and improve the performance of pedestrian re-identification.

[0103] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for re-identifying an occluded pedestrian, characterized in that: The following steps are involved: S1, input the original image of the target pedestrian into the visual encoder to obtain a high-dimensional visual embedding; S2, inputting the high-dimensional visual embedding into a prompt learner to obtain target text prompts, occluder text prompts, and background text prompts respectively; S3, inputting the target text prompt, the occluder text prompt and the background text prompt into a text encoder to obtain a high-dimensional target text embedding, a high-dimensional occluder text embedding and a high-dimensional background text embedding, and fusing the high-dimensional occluder text embedding and the high-dimensional background text embedding through a fusion layer to obtain a fused background text embedding; S4, after projecting the high-dimensional visual embedding to the same dimension as the text embedding through the projection layer, performing a cosine similarity operation with the high-dimensional target text embedding and the fused background text embedding to obtain a target attention map and a background attention map; S5, screening the target attention map to obtain a target pixel area, and using the high-dimensional visual embedding of the target pixel area and the high-dimensional target text embedding to perform feature enhancement based on the attention mechanism to obtain enhanced visual features; as well as S6, storing the target pedestrian in a memory bank according to the enhanced visual features, and updating it in a momentum form during training.

2. The method for re-identifying an occluded pedestrian according to claim 1, characterized in that: The specific method of step S1 is: Input the original image I∈R H×W×3 , obtain multi-layer high-dimensional visual embedding, where the dimension of each layer is R (h×w+1)×Cv , where h and w are the number of small blocks in the height and width dimensions after the image is divided into small blocks, C v Embedding channel dimensions for high-dimensional vision.

3. The method for re-identifying an occluded pedestrian according to claim 1, characterized in that: The specific method of step S2 is: embedding the i-th layer of high-dimensional vision into V i Global embedding of With local embedding After cross attention, the text prompt P of the i-th layer is obtained by the following formula (1): i : Among them, CA is cross attention, Proj is the projection layer, PG is i is to generate the hint layer, S is the number of visual embedding groups, P i It is one of the target text prompt, occluder text prompt or background text prompt of the i-th layer.

4. The method for re-identifying an occluded pedestrian according to claim 3, characterized in that: The step S2 also includes: calculating and obtaining the learnable prompts of the target text prompt, the occluder text prompt and the background text prompt respectively according to the formula (1), and obtaining the target text prompt of the final layer, the occluder text prompt of the final layer and the background text prompt of the final layer.

5. The method for re-identifying an occluded pedestrian according to claim 1, characterized in that: In step S3: the high-dimensional target text is embedded as T Tar , the fused background text is embedded as The dimensions of both are Where L is the fixed length of the text, C t Embedding dimension for text.

6. The method for re-identifying an occluded pedestrian according to claim 1, characterized in that: The step S4 is specifically as follows: the target attention map and the background attention map are calculated by the following formula (2): Among them, M j represents the target attention map or the background attention map, cos(·,·) represents the cosine similarity operation, and φ is a resampling function that upsamples the calculated attention map to the original image size. W i represents the projection matrix, which projects the visual embedding of each layer to the same dimension as the text embedding.

7. The method for re-identifying an occluded pedestrian according to claim 1, characterized in that: The step S5 specifically includes: enhancing the visual feature F e It is calculated by the following formula (3): Where concat(·,·) is a feature vector concatenation operation, FSE(·) is an attention-based feature enhancement module, represents the global embedding of the last layer, T Tar represents the target text embedding, Represents the filtered local visual embedding.

8. The method for re-identifying an occluded pedestrian according to claim 1, characterized in that: The step S6 further includes aggregating the centroid of the feature vector of the target pedestrian, specifically including the following steps: S61, creating a memory bank K for storing the centroid vector of the target pedestrian, and setting the momentum update coefficient to m; S62, extract the feature vector of the target pedestrian p S63, checking whether the centroid vector of the target pedestrian p exists in the memory bank K. When the centroid vector of the target pedestrian p does not exist, the feature vector Recorded as the initial centroid vector of the target pedestrian p, that is, When the centroid vector of the target pedestrian p exists, the centroid vector K[p] of the target pedestrian p is updated by the following formula (4): S64, calculating the feature vector The distance loss from the centroid vector K[p] makes the model converge. Specifically, the memory library loss function is calculated by the following formula (5): Where c is the total number of training target pedestrians.

9. The method for re-identifying an occluded pedestrian according to claim 8, characterized in that: The step S6 also includes supervision during training by the following loss function: The re-identification function supervision is specifically performed through the weighted cross entropy loss function and the triplet loss function. The re-identification function is calculated using formula (6) in, represents the cross entropy loss function, represents the triple loss function, F e represents the enhanced re-identification features; Mask supervision function supervision, the mask supervision function is calculated by the following formula (7) between the target attention map and the fused background attention map obtained in the external mask and the model: in, is the focal loss, is dice loss, G is the external mask, M Tar is the target attention map, M BG* is the background attention map.

10. A device for re-identifying an obstructed pedestrian, characterized in that: include: The visual encoder is used to process the original image of the target pedestrian to obtain a high-dimensional visual embedding; a target cue learner, connected to the visual encoder, for obtaining a target text cue through the high-dimensional visual embedding; The occluder hint learner is connected to the visual encoder and is used to obtain occluder text hints through high-dimensional visual embeddings; A background cue learner, connected to the visual encoder, is used to obtain background text cues through high-dimensional visual embeddings; A text encoder connected to the target prompt learner, the occluder prompt learner and the background prompt learner, and configured to obtain a high-dimensional target text embedding, a high-dimensional occluder text embedding and a high-dimensional background text embedding according to the target text prompt, the occluder text prompt and the background text prompt; A cross-modal interaction module, connected to the text encoder, for fusing the high-dimensional occluder text embedding and the high-dimensional background text embedding through a fusion layer to obtain a fused background text embedding, and projecting the high-dimensional visual embedding to the same dimension as the text embedding through a projection layer; The projected visual embedding, high-dimensional target text embedding and fused background text embedding are subjected to cosine similarity calculation to obtain the target attention map and background attention map; A calculation and enhancement module, connected to the visual encoder and the cross-modal interaction module, for screening the target attention map to obtain a target pixel area, and performing feature enhancement based on an attention mechanism using the high-dimensional visual embedding of the target pixel area and the high-dimensional target text embedding to obtain enhanced visual features; as well as The memory bank is connected to the computing enhancement module and is used to store the enhanced visual features of the target pedestrian and update them in the form of momentum during the training process.

Images