Cross-modal grouping alignment method for integrating multi-modal pedestrian re-identification

Abstract

The invention discloses a cross-modal grouping alignment method for integrating multi-modal pedestrian re-identification, which comprises the following steps of: S1, performing feature extraction on an RGB image and a sketch by using a shared same image feature extractor, and performing feature extraction on a text by using a text feature extractor; s2, performing feature fusion on the features of the sketch and the text by using a fusion feature extractor; s3, redundant features of the local features are filtered out through filtering processing, inter-modal fine-grained feature deep fusion is carried out through a contrast learning module in a cross-modal domain, and fine-grained feature alignment is realized; s4, the global features constrain the distribution of the global features of the three modals of the same pedestrian id in the shared space to a hyperplane through a hyperplane constraint module; and S5, the global features in the same hyperplane align the three modals through comparative learning, and finally, a text retrieval RGB image, a sketch retrieval RGB image and a text fusion sketch retrieval RGB image are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to a cross-modal grouping alignment method for integrated multi-modal pedestrian re-identification. Background Art

[0002] Person re-identification (ReID) aims to identify the same individual across different times, locations, and camera views, and match a given pedestrian image in a large-scale image or video database. It has a wide range of applications, including intelligent surveillance, public safety, traffic management, and smart retail. ReID can be divided into single-modality and cross-modal retrieval.

[0003] Single-modality ReID only matches RGB images, while cross-modal ReID allows RGB images to be retrieved through multiple modalities such as infrared, text, and sketches. Text-based retrieval allows users to find pedestrian images through natural language descriptions, while sketches provide more intuitive visual details. However, traditional models are usually trained on only a single modality, limiting the flexible use of multimodal data. In a typical urban surveillance scenario, the surveillance system searches for missing women based on eyewitness descriptions. Traditional feature grouping strategies may analyze the video image features from the camera and the text or sketch features extracted from the eyewitness description separately, failing to achieve fine-grained cross-modal feature fusion. When the details of the eyewitness description are not obvious in the image, such as the woman's scarf is red in the text description, but due to the change in ambient light in the actual surveillance scene, the scarf appears to be darker and no longer red, which may cause traditional algorithms to fail to correctly match pedestrians.

[0004] UNIReID integrates different retrieval methods (such as text, sketches, and RGB images) to meet diverse practical needs. However, UNIReID only relies on global features for cross-modal alignment, which may ignore key local details. Existing local learning paradigms usually use hard segmentation strategies to extract local information, ignoring the global relationship between features, resulting in incomplete information understanding, failing to fully utilize the potential complementary information between images and text, and failing to fully utilize the advantages of the fusion of different modal features. Summary of the invention

[0005] To solve the above problems, the present invention proposes a cross-modal grouping alignment method for integrated multi-modal person re-identification.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A cross-modal grouping alignment method for integrated multi-modal person re-identification is characterized by comprising the following steps: S1, using the same shared image feature extractor to extract features from the RGB image and the sketch, and using the text feature extractor to extract features from the text; S2, using a fusion feature extractor to fuse the features of the sketch and the text, so as to support the retrieval method of fusion of sketch and text; S3: The local features are filtered to remove redundant features, and then the fine-grained features between modalities are deeply fused through the cross-modal domain contrast learning module to achieve fine-grained feature alignment; S4, global features, through the hyperplane constraint module, constrain the distribution of the three modal global features of the same pedestrian ID in the shared space to a hyperplane; The specific process of step S4 is: S41, the distance between features belonging to different categories but the same modality is less than a threshold, that is, the distance between the RGB modality features of different pedestrian categories and the distance difference between the sketch modality, text modality and fusion modality of the same modality are less than the threshold, formally represented as the difference between the four edges of the three-dimensional figure is less than the threshold; S42. For any two different pedestrian IDs, the distance between the RGB modal features of pedestrian ID1 and the sketch, text and fusion modal features of pedestrian ID2 is equal to the distance between the sketch, text and fusion modal features of pedestrian ID1 and the RGB modal features of pedestrian ID2, which is formally represented as the lengths of the diagonals on two planes and one diagonal surface of the three-dimensional figure are equal; S43, through the hyperplane constraint, the distribution of the features of the four modalities of RGB modality, sketch modality, text modality and fusion modality of the same pedestrian ID in the shared space will present a hyperplane; S5. The global features in the same hyperplane align the three modalities through contrast learning, and finally realize text retrieval RGB image, sketch retrieval RGB image, and text fusion sketch retrieval RGB image.

[0007] Preferably, the specific process of step S1 is: S11. During the training process, the training data is defined as , where N represents the number of samples within a batch, I represents the RGB modality, S represents the sketch modality, T represents the text modality, and k represents the sample number; S12, inputting the data of the RGB modality and the sketch modality into the same image feature extractor respectively, and inputting the data of the text modality into the text feature extractor, to obtain global features of the RGB modality, the sketch modality, and the text modality; S13. Perform feature extraction on patch tokens of RGB modality and sketch modality to obtain RGB local features and sketch local features, respectively; perform feature extraction on vocabulary tokens of text modality to obtain text local features.

[0008] Preferably, the specific process of step S2 is: splicing the global features of the sketch and the text, adding a CLS mark, and then passing through a feature extractor, taking the CLS mark through the output corresponding to the feature extractor As a fusion feature of sketch and text.

[0009] Preferably, the specific process of the filtering processing in step S3 is: calculating the similarity scores of the global features and local features within the modality, then arranging them in descending order, selecting and retaining the top-ranked local features, and eliminating the remaining local features.

[0010] Preferably, the specific process of deep fusion of fine-grained features between modalities in step S3 is: S31. For the image domain, the filtered local features of the text are mapped to the image domain through a linear layer. The calculation formula is: , where Represents the text features after mapping, Indicates linear mapping of the local features of the filtered text. Represents the feature dimension after mapping; S32. For the text domain, the filtered RGB local features and sketch local features are mapped to the text domain through a linear layer. The calculation formula is: , , where represents the mapped RGB features, Indicates linear mapping of the local features of the filtered RGB image. Represents the feature dimension after mapping; Represents the mapped sketch feature, Indicates linear mapping of the filtered local features of the sketch. Represents the feature dimension after mapping; S33, the characteristics of the image domain are , the characteristics of the text domain are , the features of the image domain and the text domain are fused through a cross-modal grouping strategy; for the image domain, the local features of each modality in the image domain are first randomly shuffled and divided into n equal parts, and then the local features of the three modalities are combined to generate n groups, each group contains the local features of three different modalities; for the text domain, the local features of each modality in the text domain are first randomly shuffled and divided into n equal parts, and then the local features of the three modalities are combined to generate n groups, each group contains the local features of three different modalities; S34, extracting the global features within each group of the image domain and the text domain through an image feature extractor and a text feature extractor, respectively. The global features within the group of the pth group in the image domain are: , the global feature within the group p of the text domain is, , and then stack the global features of the same group number of different pedestrian IDs to form a feature matrix. The p-th feature matrices of the image domain and the text domain are: , , where represents the feature matrix of the pth stack in the image domain, represents the global feature set within the pth group of all pedestrians in the image domain, represents the feature matrix of the pth stack of the text domain, represents the global feature set within the pth group of all pedestrians in the text domain, Represents the dimensions of the stacked feature matrix; All feature matrices in the image domain and text domain are expressed as: , , where Represents the feature matrix of all stacks in the image domain, Represents the collection of n stacked feature matrices in the image domain, Represents the collection of all stacked feature matrices of the text domain, Represents the collection of n stacked feature matrices of the text domain, Represents the dimension of the collection of all stacked feature matrices in the image domain; All feature matrices in the matrix are compared to calculate the loss, and there will be loss components, and finally the loss in the image domain is obtained; All feature matrices in the matrix are compared to calculate the loss, and there will be loss components, and finally the loss of the text domain is obtained.

[0011] Preferably, the specific process of step S5 is: after passing the hyperplane constraint, contrastive learning is used for the three retrieval methods of text retrieval RGB image, sketch retrieval RGB image, and text fusion sketch retrieval RGB image, respectively, to maximize the cosine similarity between true sample pairs and minimize the cosine similarity between error sample pairs.

[0012] After adopting the above technical scheme, the present invention has the following beneficial effects: the present invention randomly mixes and groups local features from pedestrian images, sketches (i.e., sketches) and texts, breaks the modal barriers, and realizes deep and fine-grained cross-modal feature integration. The cross-modal grouping alignment method (CMGA) for integrated multimodal pedestrian re-identification of the present invention aligns the fine-grained features of different modalities, thereby enhancing the accuracy of matching. The motivation is that by grouping the red scarf features in the image and sketch with the corresponding text description "red scarf", cross-modal information interaction can accurately identify the color of the scarf under illumination changes or occlusion. To this end, the present invention designs cross-modal grouping domain contrast learning (CGIC), which maps RGB, sketch and text features to the image and text domains for intra-domain cross-modal grouping to achieve fine-grained feature fusion. In addition, the present invention also uses contrastive learning to optimize the cross-modal embedding space. In addition, traditional methods often lead to chaotic feature distribution during the initial optimization process, resulting in unstable optimization. The present invention further introduces hyperplane constraint (HC) contrastive learning to ensure that different modal features of the same pedestrian ID are distributed on a common hyperplane in the shared space, which significantly improves feature optimization and model performance, and improves the performance on multimodal pedestrian re-identification tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of the present invention; Figure 2 It is a framework structure diagram of the present invention; Figure 3 It is a structural schematic diagram of the filtering process of the present invention; Figure 4 It is a schematic diagram of the structure of the hyperplane constraint of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0015] like Figures 1 to 4 As shown, a cross-modal grouping alignment method for integrated multi-modal person re-identification includes the following steps: S1, using the same shared image feature extractor to extract features from the RGB image and the sketch, and using the text feature extractor to extract features from the text; The specific process of step S1 is: S11. During the training process, the training data is defined as , where N represents the number of samples within a batch, I represents the RGB modality, S represents the sketch modality, T represents the text modality, and k represents the sample number; S12, inputting the data of the RGB modality and the sketch modality into the same image feature extractor respectively, and inputting the data of the text modality into the text feature extractor, to obtain global features of the RGB modality, the sketch modality, and the text modality; S13, performing feature extraction on the patch tokens of the RGB modality and the sketch modality to obtain RGB local features and sketch local features respectively; performing feature extraction on the vocabulary tokens of the text modality to obtain text local features; S2, using a fusion feature extractor to fuse the features of the sketch and the text, so as to support the retrieval method of fusion of sketch and text; The specific process of step S2 is: splicing the global features of the sketch and text, adding a CLS mark, and then passing it through a feature extractor, taking the CLS mark through the output corresponding to the feature extractor as a fusion of sketch and text; S3: The local features are filtered to remove redundant features, and then the fine-grained features between modalities are deeply fused through the cross-modal domain contrast learning module to achieve fine-grained feature alignment; The specific process of the filtering process in step S3 is: calculating the similarity scores of the global features and local features within the modality, and then sorting them in descending order, selecting and retaining the top-ranked local features, and eliminating the remaining local features; The specific process of deep fusion of fine-grained features between modalities in step S3 is as follows: S31. For the image domain, the filtered local features of the text are mapped to the image domain through a linear layer. The calculation formula is: , where Represents the text features after mapping, Indicates linear mapping of the local features of the filtered text. Represents the feature dimension after mapping; S32. For the text domain, the filtered RGB local features and sketch local features are mapped to the text domain through a linear layer. The calculation formula is: , , where represents the mapped RGB features, Indicates linear mapping of the local features of the filtered RGB image. Represents the feature dimension after mapping; Represents the mapped sketch feature, Indicates linear mapping of the filtered local features of the sketch. Represents the feature dimension after mapping; S33, the characteristics of the image domain are , the characteristics of the text domain are , the features of the image domain and the text domain are fused through a cross-modal grouping strategy; for the image domain, the local features of each modality in the image domain are first randomly shuffled and divided into n equal parts, and then the local features of the three modalities are combined to generate n groups, each group contains the local features of three different modalities; for the text domain, the local features of each modality in the text domain are first randomly shuffled and divided into n equal parts, and then the local features of the three modalities are combined to generate n groups, each group contains the local features of three different modalities; S34, extracting the global features within each group of the image domain and the text domain through an image feature extractor and a text feature extractor, respectively. The global features within the group of the pth group in the image domain are: , the global feature within the group p of the text domain is, , and then stack the global features of the same group number of different pedestrian IDs to form a feature matrix. The p-th feature matrices of the image domain and the text domain are: , , where represents the feature matrix of the pth stack in the image domain, represents the global feature set within the pth group of all pedestrians in the image domain, represents the feature matrix of the pth stack of the text domain, represents the global feature set within the pth group of all pedestrians in the text domain, The dimensions of the feature matrix representing the stack; All feature matrices in the image domain and text domain are expressed as: , , where Represents the feature matrix of all stacks in the image domain, Represents the collection of n stacked feature matrices in the image domain, Represents the collection of all stacked feature matrices of the text domain, Represents the collection of n stacked feature matrices of the text domain, Represents the dimension of the collection of all stacked feature matrices in the image domain; All feature matrices in the matrix are compared to calculate the loss, and there will be loss components, and finally the loss in the image domain is obtained; All feature matrices in the matrix are compared to calculate the loss, and there will be loss components, and finally get the loss of the text domain; S4, global features, through the hyperplane constraint module, constrain the distribution of the three modal global features of the same pedestrian ID in the shared space to a hyperplane; The specific process of step S4 is: S41, the distance between features belonging to different categories but the same modality is less than a threshold, that is, the distance between the RGB modality features of different pedestrian categories and the distance difference between the sketch modality, text modality and fusion modality of the same modality are less than the threshold, formally represented as the difference between the four edges of the three-dimensional figure is less than the threshold; S42. For any two different pedestrian IDs, the distance between the RGB modal features of pedestrian ID1 and the sketch, text and fusion modal features of pedestrian ID2 is equal to the distance between the sketch, text and fusion modal features of pedestrian ID1 and the RGB modal features of pedestrian ID2, which is formally represented as the lengths of the diagonals on two planes and one diagonal surface of the three-dimensional figure are equal; S43, through the hyperplane constraint, the distribution of the features of the four modalities of RGB modality, sketch modality, text modality and fusion modality of the same pedestrian ID in the shared space will present a hyperplane; S5. The global features in the same hyperplane align the three modalities through contrast learning, and finally realize text retrieval RGB image, sketch retrieval RGB image, and text fusion sketch retrieval RGB image; The specific process of step S5 is as follows: after the hyperplane constraint, contrastive learning is used for the three retrieval methods of text retrieval RGB image, sketch retrieval RGB image, and text fusion sketch retrieval RGB image to maximize the cosine similarity between true sample pairs and minimize the cosine similarity between error sample pairs.

[0016] Performance Test:

[0017] In this test, three multimodal datasets constructed from the paper Chen et al. (2023) (i.e. Cuiqun Chen published in CVPR2023 Towards Modality-Agnostic Person Re-identification with Descriptive Query) were used: Tri-CUHK-PEDES, Tri-ICFG-PEDES, and Tri-RSTPReid. These datasets cover three modalities: RGB images (i.e. visible light images), text descriptions, and hand-drawn sketches (i.e. sketch images). The sketch images are synthesized using the Meitu API suitable for low-resolution pedestrian images. This method generates hand-drawn style sketch images of pedestrians based on RGB images after background removal. Detailed statistical information of the dataset is shown in Table 1. Multimodal data not only enriches the information source for pedestrian re-identification, but also provides an experimental basis for verifying the effectiveness and robustness of the method proposed in this invention.

[0018] Table 1: Statistics of the dataset

[0019] For the Tri-CUHK-PEDES dataset, the training set contains 34,054 RGB and sketch images with 11,003 pedestrian identities; the validation set contains 3,078 RGB and sketch images with 1,000 pedestrian identities; and the test set contains 3,074 RGB and sketch images with 1,000 pedestrian identities. For the Tri-ICFG-PEDES dataset, the training set contains 34,674 RGB and sketch images with 3,102 identities; the test set contains 19,848 RGB and sketch images with 1,000 pedestrian identities. For the Tri-RSTPReid dataset, the training set contains 18,505 RGB and sketch images with 3,701 pedestrian identities; the validation set contains 1,000 RGB and sketch images with 200 pedestrian identities; and the test set contains 1,000 RGB and sketch images with 200 pedestrian identities.

[0020] The cross-modal group alignment method (CMGA) for integrated multi-modal person re-identification of the present invention is compared with other existing methods. In order to ensure a fair comparison with existing methods, the cross-modal group alignment method (CMGA) for integrated multi-modal person re-identification of the present invention is tested on three retrieval tasks using the original gallery sets of three datasets. The test results are shown in Table 2.

[0021] Table 2: Test results of CMGA of the present invention and other methods

[0022] The results show that the proposed dual encoder for learning visual and textual modal features is more effective than most CNN-based methods. CMGA adopts cross-modal grouping and hyperplane constraint strategies to achieve fine-grained cross-modal feature integration and unified initial feature optimization direction, thereby achieving superior retrieval performance. As shown in Table 2, the Rank1 accuracy of the text fusion sketch retrieval RGB image task on the three datasets reached 88.42%, 84.13% and 76.20%, respectively, which is 2.13%, 1.96% and 3% higher than UNIReid, respectively. This significant performance improvement demonstrates the effectiveness and potential of the cross-modal grouping alignment method (CMGA) for integrated multimodal person re-identification in multimodal retrieval tasks.

[0023] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A cross-modal grouping alignment method for integrated multi-modal person re-identification, characterized in that: The following steps are involved: S1, using the same shared image feature extractor to extract features from the RGB image and the sketch, and using the text feature extractor to extract features from the text; S2, using a fusion feature extractor to fuse the features of the sketch and the text, so as to support the retrieval method of fusion of sketch and text; S3: The local features are filtered to remove redundant features, and then the fine-grained features between modalities are deeply fused through the cross-modal domain contrast learning module to achieve fine-grained feature alignment; S4, global features, through the hyperplane constraint module, constrain the distribution of the three modal global features of the same pedestrian ID in the shared space to a hyperplane; The specific process of step S4 is: S41, the distance between features belonging to different categories but the same modality is less than a threshold, that is, the distance between the RGB modality features of different pedestrian categories and the distance difference between the sketch modality, text modality and fusion modality of the same modality are less than the threshold, formally represented as the difference between the four edges of the three-dimensional figure is less than the threshold; S42. For any two different pedestrian IDs, the distance between the RGB modal features of pedestrian ID1 and the sketch, text and fusion modal features of pedestrian ID2 is equal to the distance between the sketch, text and fusion modal features of pedestrian ID1 and the RGB modal features of pedestrian ID2, which is formally represented as the lengths of the diagonals on two planes and one diagonal surface of the three-dimensional figure are equal; S43, through the hyperplane constraint, the distribution of the features of the four modalities of RGB modality, sketch modality, text modality and fusion modality of the same pedestrian ID in the shared space will present a hyperplane; S5. The global features in the same hyperplane align the three modalities through contrast learning, and finally realize text retrieval RGB image, sketch retrieval RGB image, and text fusion sketch retrieval RGB image.

2. The cross-modal grouping alignment method for integrated multi-modal person re-identification according to claim 1, characterized in that: The specific process of step S1 is: S11. During the training process, the training data is defined as , where N represents the number of samples within a batch, I represents the RGB modality, S represents the sketch modality, T represents the text modality, and k represents the sample number; S12, inputting the data of the RGB modality and the sketch modality into the same image feature extractor respectively, and inputting the data of the text modality into the text feature extractor, to obtain global features of the RGB modality, the sketch modality, and the text modality; S13. Perform feature extraction on patch tokens of RGB modality and sketch modality to obtain RGB local features and sketch local features, respectively; perform feature extraction on vocabulary tokens of text modality to obtain text local features.

3. The cross-modal grouping alignment method for integrated multi-modal person re-identification according to claim 1, characterized in that: The specific process of step S2 is: splicing the global features of the sketch and text, adding a CLS mark, and then passing it through a feature extractor, taking the CLS mark through the output corresponding to the feature extractor As a fusion feature of sketch and text.

4. The cross-modal grouping alignment method for integrated multi-modal person re-identification according to claim 1, characterized in that: The specific process of the filtering process in step S3 is: calculating the similarity scores of the global features and local features within the modality, then arranging them in descending order, selecting and retaining the top-ranked local features, and eliminating the remaining local features.

5. The cross-modal grouping alignment method for integrated multi-modal person re-identification according to claim 1, characterized in that: The specific process of deep fusion of fine-grained features between modalities in step S3 is as follows: S31. For the image domain, the filtered local features of the text are mapped to the image domain through a linear layer. The calculation formula is: , where Represents the text features after mapping, Indicates linear mapping of the local features of the filtered text. Represents the feature dimension after mapping; S32. For the text domain, the filtered RGB local features and sketch local features are mapped to the text domain through a linear layer. The calculation formula is: , , where Represents the mapped RGB features, Indicates linear mapping of the local features of the filtered RGB image. Represents the feature dimension after mapping; Represents the mapped sketch feature, Indicates linear mapping of the filtered local features of the sketch. Represents the feature dimension after mapping; S33, the characteristics of the image domain are , the characteristics of the text domain are , the features of the image domain and the text domain are fused through a cross-modal grouping strategy; for the image domain, the local features of each modality in the image domain are first randomly shuffled and divided into n equal parts, and then the local features of the three modalities are combined to generate n groups, each group contains the local features of three different modalities; for the text domain, the local features of each modality in the text domain are first randomly shuffled and divided into n equal parts, and then the local features of the three modalities are combined to generate n groups, each group contains the local features of three different modalities; S34, extracting the global features within each group of the image domain and the text domain through an image feature extractor and a text feature extractor, respectively. The global features within the group of the pth group in the image domain are: , the global feature within the group p of the text domain is, , and then stack the global features of the same group number of different pedestrian IDs to form a feature matrix. The p-th feature matrices of the image domain and the text domain are: , , where represents the feature matrix of the pth stack in the image domain, represents the global feature set within the pth group of all pedestrians in the image domain, represents the feature matrix of the pth stack of the text domain, represents the global feature set within the pth group of all pedestrians in the text domain, The dimensions of the feature matrix representing the stack; All feature matrices in the image domain and text domain are expressed as: , , where Represents the feature matrix of all stacks in the image domain, Represents the collection of n stacked feature matrices in the image domain, Represents the collection of all stacked feature matrices of the text domain, Represents the collection of n stacked feature matrices of the text domain, Represents the dimension of the collection of all stacked feature matrices in the image domain; All feature matrices in the matrix are compared to calculate the loss, and there will be loss components, and finally the loss in the image domain is obtained; All feature matrices in the matrix are compared to calculate the loss, and there will be loss components, and finally the loss of the text domain is obtained.

6. The cross-modal grouping alignment method for integrated multi-modal person re-identification according to claim 1, characterized in that: The specific process of step S5 is as follows: after the hyperplane constraint, contrastive learning is used for the three retrieval methods of text retrieval RGB image, sketch retrieval RGB image, and text fusion sketch retrieval RGB image to maximize the cosine similarity between true sample pairs and minimize the cosine similarity between error sample pairs.

Images