A Key Semantic Completeness Alignment Method for Cross-modal Person Search Based on Large Model Knowledge

Abstract

The present application belongs to the technical field of computer vision and natural language processing. The present application provides a method for complete alignment of key semantics in cross-modal pedestrian search based on large model knowledge. The disclosed embodiment utilizes the external general knowledge contained in the pre-trained visual language large model to extract more discriminative visual features and text features; designs a noise filtering module, and filters out noise in images and texts that is not related to pedestrians by referring to the relationship between global and local semantics within the modality, and only retains fine-grained key semantic information related to pedestrians; proposes a complete learning module for joint optimization of key semantic saliency and integrity, and considers the synergistic relationship between saliency and integrity at the same time, to achieve complete alignment of cross-modal pedestrian key semantic information. It can filter out noise information that is not related to pedestrians, enhance the model's attention to key information related to pedestrians, and explore the correspondence between cross-modal fine-grained information, thereby improving the accuracy of cross-modal pedestrian retrieval.

Description

Technical Field

[0001] The disclosed embodiments relate to the technical fields of computer vision and natural language processing, and in particular to a cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge. Background Art

[0002] Natural language cross-modal pedestrian search aims to find images corresponding to target persons from an image library using text descriptions. It has broad applications in security surveillance, intelligent video analysis, and other fields. As a fine-grained cross-modal retrieval task, related research has explored resolving the modal heterogeneity between images and text. However, previous methods have ignored the inherent noise in both images and text (for example, environmental background in images and pedestrian-unrelated words in text), which inevitably interferes with cross-modal alignment. This results in poor discrimination of pedestrian details and difficulty distinguishing similar pedestrian images.

[0003] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0004] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the Invention

[0005] The purpose of the embodiments of the present disclosure is to provide a cross-modal pedestrian search key semantics complete alignment method based on large model knowledge, thereby overcoming one or more problems caused by the limitations and defects of related technologies to at least a certain extent.

[0006] According to an embodiment of the present disclosure, a cross-modal pedestrian search key semantics completion alignment method based on large model knowledge is provided, the method comprising:

[0007] Constructing a pre-trained multimodal large model; wherein the pre-trained multimodal large model includes a pre-trained visual language large model, a noise filtering module, and a key semantic saliency and integrity joint optimization complete learning module, and the key semantic saliency and integrity joint optimization complete learning module includes a salient semantic alignment submodule and a complete semantic alignment submodule;

[0008] The pre-trained visual language model is used to extract the visual global features and visual local features of the training set images, and the text global features and text local features of the word encodings;

[0009] Obtaining cross-modal global similarity and pedestrian classification based on the visual global features and the text global features;

[0010] Using the noise filtering module to filter out noise information irrelevant to pedestrians in the visual local features and the text local features to obtain visual key semantics and text key semantics;

[0011] Processing the visual key semantics and the text key semantics using the salient semantics alignment submodule to obtain visual salient semantics and text salient semantics;

[0012] Obtaining cross-modal saliency similarity based on the visual salient semantics and the textual salient semantics;

[0013] Processing the visual key semantics and the text key semantics using the complete semantic alignment submodule to obtain visual completeness key semantics and text completeness key semantics;

[0014] Obtaining a cross-modal integrity similarity of the image-text pair according to the visual integrity key semantics and the text integrity key semantics;

[0015] Based on the pedestrian classification, the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity, fine-tuning the pre-trained multimodal large model using the Adam optimization algorithm to obtain the fine-tuned pre-trained multimodal large model;

[0016] The query statement is input into the fine-tuned pre-trained multimodal large model to obtain a retrieval result sequence.

[0017] Furthermore, the steps of respectively extracting the visual global features and visual local features of the image, and the text global features and text local features of the text using the pre-trained visual language model include:

[0018] Normalizing the training set images to a uniform size, and performing data augmentation on the training set images by image flipping and random cropping;

[0019] The data-enhanced training set images are input into the pre-trained visual language model, and the visual encoder in the pre-trained visual language model is used to extract visual features. Among them, Recognized as the visual global feature ,Will Recognized as the visual local features , is the number of local regions in an image;

[0020] Encoding the word into a word vector using a word vector embedding method;

[0021] Input the word vector into the pre-trained visual language model, and use the text encoder in the pre-trained visual language model to extract text features. Among them, Recognized as the global feature of the text ,Will Recognized as the local features of the text , is the number of words in each text, is the feature vector corresponding to the text start mark.

[0022] Furthermore, the step of obtaining cross-modal global similarity and pedestrian classification based on the visual global features and the text global features includes:

[0023] The visual global feature and the global features of the text By calculating the first cosine similarity and mapping it to a unified encoding space for coarse-grained semantic alignment, the cross-modal global similarity is obtained. ;

[0024] The visual global feature and the global features of the text Mapping from the feature space to the identity space to obtain the pedestrian classification.

[0025] Furthermore, the step of using the noise filtering module to filter out noise information irrelevant to pedestrians in the visual local features and the text local features to obtain the visual key semantics and the text key semantics includes:

[0026] Get the visual self-attention map from the last layer of Transformer module in the visual encoder , and according to the visual self-attention map Get the visual global-local correlation ; Wherein, the visual self-attention map The relationship between the visual special label [CLS] and the local area of ​​the image is described in [C].

[0027] Introducing selection ratio , to obtain the number of retained local image regions ;in, Indicates rounding down;

[0028] According to the visual global-local correlation , get the maximum correlation The local image area corresponding to the value is regarded as the area related to the pedestrian, so as to obtain the local visual features. Extract The features of the image are used as the key visual semantics related to pedestrians ;

[0029] Get the text self-attention map from the last layer of Transformer module in the text encoder , and according to the text self-attention map Get global-local correlation ; Wherein, the text self-attention map What is depicted is the connection between the text special tag [EOS] and the text word;

[0030] Introducing the selection ratio , to get the number of retained text words ;in, , The maximum word length predefined in the text encoder;

[0031] According to the global-local correlation of the text From the local features of the text Extract The features of the words are used as the key semantics of the text related to pedestrians .

[0032] Furthermore, the step of processing the visual key semantics and the text key semantics using the salient semantics alignment submodule to obtain visual salient semantics and text salient semantics includes:

[0033] Enhance the visual key semantics using the first multi-layer perceptron module Semantic representation ability to obtain visually significant key semantics :

[0034]

[0035] in, is the first multi-layer perceptron module, express Regularization;

[0036] By aggregating the visually salient key semantics , to obtain the visually significant semantics :

[0037]

[0038] in, It is the maximum pooling operation;

[0039] Using the second multi-layer perceptron module to enhance the key semantics of the text Semantic representation ability to obtain the key semantics of text saliency :

[0040]

[0041] By aggregating the key semantics of the text , to obtain the significant semantics of the text :

[0042]

[0043] in, It is the second multi-layer perceptron module.

[0044] Furthermore, the step of obtaining cross-modal saliency similarity based on the visual salient semantics and the textual salient semantics includes:

[0045] Calculate the visual salient semantics and the text is semantically significant The second cosine similarity between them is used to obtain the cross-modal saliency similarity .

[0046] Furthermore, the step of processing the visual key semantics and the text key semantics using the complete semantic alignment submodule to obtain the visual integrity key semantics and the text integrity key semantics includes:

[0047] The third multi-layer perceptron module is used to obtain the visual key semantics Key semantics of pedestrian-related visual completeness :

[0048]

[0049] in, is the third multi-layer perceptron module;

[0050] Use the fourth multi-layer perceptron module to obtain the key semantics of the text Key semantics of pedestrian-related text integrity :

[0051]

[0052] in, It is the fourth multi-layer perceptron module.

[0053] Furthermore, the step of obtaining the cross-modal integrity similarity of the image-text pair based on the visual integrity key semantics and the text integrity key semantics includes:

[0054] According to the visual integrity key semantics and the text integrity key semantics , get the third cosine similarity;

[0055] According to the third cosine similarity, the relationship matrix from the local area of ​​the image to the direction of the text word is obtained ;in, Each row represents the connection between a local area of ​​an image selected from an image and all words selected from a text;

[0056] For each local area of ​​the image, the similarity of the word with the highest similarity is selected to represent the correlation between each local area of ​​the image and the entire text, so as to obtain the correlation between each local area of ​​the image and the text in an image. :

[0057]

[0058] The correlation between all local areas of the image and the text Calculate the average value to get the first image-text similarity from the local area of ​​the image to the text word direction :

[0059]

[0060] in, represents the mean operation;

[0061] For each text word, the similarity of the local image area with the highest similarity is selected to represent the correlation between each text word and the local image area, so as to obtain the correlation between each text word in a text and the local image area. :

[0062]

[0063] The association between all text words and local image regions Calculate the average value to obtain the second image text similarity from the text word to the local area of ​​the image for:

[0064] .

[0065] Furthermore, based on the pedestrian classification, the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity, the pre-trained multimodal large model is fine-tuned using the Adam optimization algorithm to obtain the fine-tuned pre-trained multimodal large model, including:

[0066] Mapping the visual global features and the text global features into a unified identity space and performing pedestrian classification using a cross entropy loss function;

[0067] The cross-modal global similarity, the cross-modal saliency similarity and the cross-modal completeness similarity are cross-modally matched using a contrast loss function, and then the pre-trained multimodal large model is fine-tuned using an Adam optimizer to obtain the fine-tuned pre-trained multimodal large model.

[0068] Furthermore, the step of inputting the query statement into the fine-tuned pre-trained multimodal large model to obtain a retrieval result sequence includes:

[0069] Inputting the query statement into the fine-tuned pre-trained multimodal large model to extract the global text features and the local text features of the query statement;

[0070] Calculating the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity between the query statement and a plurality of pre-stored images based on the global text features and the local text features of the query statement;

[0071] The cross-modal global similarity, the cross-modal saliency similarity and the cross-modal integrity similarity are summed to obtain multiple image-text similarities. ;

[0072] According to the similarity of all the images and texts to obtain the retrieval result sequence.

[0073] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0074] In the embodiments disclosed herein, the aforementioned large-model-based cross-modal pedestrian search key semantics complete alignment method utilizes the external general knowledge contained in the pre-trained visual language large model to extract more discriminative visual and text features. A noise filtering module is then designed to filter out noise irrelevant to pedestrians in images and text by referencing the relationship between global and local semantics within the modality, retaining only the fine-grained key semantic information related to pedestrians. Furthermore, a complete learning module is proposed for joint optimization of key semantic saliency and completeness, simultaneously considering the synergistic relationship between saliency and completeness to achieve complete alignment of cross-modal pedestrian key semantic information. This approach is the first to collaboratively consider the alignment of cross-modal pedestrian key semantic information from both saliency and completeness. By referencing the relationship between global and local semantics within the modality, key semantics related to pedestrians in images and text are obtained. A strategy for collaborative tuning of saliency and completeness is then used to fully explore the correspondence between fine-grained cross-modal key semantic information, thereby more accurately distinguishing different pedestrians with similar appearances, achieving significant performance improvements in the field of natural language cross-modal pedestrian search. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0076] Figure 1 A diagram showing the steps of a cross-modal pedestrian search key semantics completion alignment method based on large model knowledge in an exemplary embodiment of the present disclosure;

[0077] Figure 2 A schematic diagram showing a framework of a pre-trained multimodal large model in an exemplary embodiment of the present disclosure is shown;

[0078] Figure 3 A schematic diagram showing the results of a query obtained by using a cross-modal pedestrian search key semantics completion alignment method based on large model knowledge according to a query statement in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0080] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0081] This example implementation provides a cross-modal pedestrian search key semantics complete alignment method based on large model knowledge. Figure 1 As shown in , the cross-modal pedestrian search key semantics completion alignment method based on large model knowledge may include: steps S101 to S110.

[0082] Step S101: constructing a pre-trained multimodal large model; wherein the pre-trained multimodal large model includes a pre-trained visual language large model, a noise filtering module, and a key semantic saliency and integrity joint optimization complete learning module, and the key semantic saliency and integrity joint optimization complete learning module includes a salient semantic alignment submodule and a complete semantic alignment submodule;

[0083] Step S102: using the pre-trained visual language model to extract the visual global features and visual local features of the training set images, and the text global features and text local features of the word encodings;

[0084] Step S103: obtaining cross-modal global similarity and pedestrian classification based on the visual global features and the text global features;

[0085] Step S104: using the noise filtering module to filter out noise information irrelevant to pedestrians in the visual local features and the text local features to obtain visual key semantics and text key semantics;

[0086] Step S105: using the salient semantics alignment submodule to process the visual key semantics and the text key semantics to obtain visual salient semantics and text salient semantics;

[0087] Step S106: Obtaining cross-modal saliency similarity based on the visual salient semantics and the textual salient semantics;

[0088] Step S107: Processing the visual key semantics and the text key semantics using the complete semantic alignment submodule to obtain visual completeness key semantics and text completeness key semantics;

[0089] Step S108: Obtaining a cross-modal integrity similarity of the image-text pair based on the visual integrity key semantics and the text integrity key semantics;

[0090] Step S109: Based on the pedestrian classification, the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity, the pre-trained multimodal large model is fine-tuned using the Adam optimization algorithm to obtain the fine-tuned pre-trained multimodal large model;

[0091] Step S110: Input the query statement into the fine-tuned pre-trained multimodal large model to obtain a retrieval result sequence.

[0092] Through the aforementioned large-model knowledge-based method for complete alignment of key semantics for cross-modal pedestrian search, the external general knowledge contained in the pre-trained visual language large model is utilized to extract more discriminative visual and textual features. A noise filtering module is then designed to filter out noise irrelevant to pedestrians in images and text by referencing the relationship between global and local semantics within the modality, retaining only the fine-grained key semantic information relevant to pedestrians. Furthermore, a complete learning module is proposed to jointly optimize the saliency and completeness of key semantics, simultaneously considering the synergistic relationship between saliency and completeness to achieve complete alignment of cross-modal pedestrian key semantic information. This approach is the first to jointly consider the saliency and completeness aspects of cross-modal pedestrian key semantics. By referencing the relationship between global and local semantics within the modality, key semantics related to pedestrians in images and text are obtained. A strategy of co-tuning saliency and completeness is then used to fully exploit the correspondence between fine-grained cross-modal key semantic information, thereby more accurately distinguishing different pedestrians with similar appearances, achieving significant performance improvements in the field of natural language cross-modal pedestrian search.

[0093] Below, we will refer to Figures 1 to 3 Each step of the above-mentioned cross-modal pedestrian search key semantics completion alignment method based on large model knowledge in this example implementation is described in more detail.

[0094] In step S101, a pre-trained multimodal large model is constructed; wherein, the pre-trained multimodal large model includes a pre-trained visual language large model, a noise filtering module, and a key semantic saliency and integrity joint optimization complete learning module, and the key semantic saliency and integrity joint optimization complete learning module includes a salient semantic alignment submodule and a complete semantic alignment submodule.

[0095] In step S102, for the image side, the training set images are first normalized to a uniform size, and data enhancement is performed by image flipping and random cropping, and the visual features are extracted using the visual encoder of the pre-trained visual language model that carries external general knowledge. .Will Recognized as global visual features , Local visual features ,in, is the number of local regions in an image.

[0096] For the text side, the word is encoded into a word vector through the word vector embedding method, and then the text features are obtained through the text encoder of the pre-trained large model .Will Considered as a global feature of the text , Local features of text ,in, is the number of words in each text.

[0097] In step S103, in order to extract the key semantic information related to pedestrians and mine the corresponding relationship between images and texts with high quality, the visual global features are used to extract the key semantic information related to pedestrians and mine the corresponding relationship between images and texts. and global features of text By calculating the cosine similarity and mapping it to a unified encoding space for coarse-grained semantic alignment, we can obtain the cross-modal global similarity. At the same time, in order to utilize the identity constraints of pedestrians, the global features of images and texts are mapped from the feature space to the identity space for pedestrian classification.

[0098] In step S104, to reduce the impact of noise in the data (for example, environmental background in images and pedestrian-unrelated words in text) on cross-modal alignment, a noise filtering module (NNF) is proposed. By referring to the relationship between global and local semantics within the modality, it filters out noise unrelated to pedestrians from the visual and text local features, retaining the fine-grained key semantic information related to pedestrians.

[0099] For an image, we first obtain the self-attention map from the last layer of the Transformer module in the visual encoder. ,in It describes the relationship between the visual special label [CLS] and the local area of ​​the image, that is, the relationship between global semantics and local semantics. Global-local correlation This is used as the basis for judging the relevance between local image areas and pedestrians. To determine the number of image regions to retain :

[0100]

[0101] in, Indicates rounding down.

[0102] Then, according to the global-local correlation , find the one with the highest correlation The local image area corresponding to the value is regarded as the area related to the pedestrian, and the rest is regarded as environmental noise unrelated to the pedestrian.

[0103] Finally, according to the global-local correlation From the visual local features Obtain pedestrian-related visual key semantics .

[0104] In the noise filtering on the text side, similar to the image side, the text self-attention map is obtained from the last layer of the Transformer module in the text encoder. , and based on the text self-attention map Get global-local correlation ; Among them, the text self-attention map What is depicted is the connection between the text special tag [EOS] and the text word.

[0105] According to the global-local correlation of the text From the local features of the text Extract The features of words are used as pedestrian-related text key semantics , The calculation method is:

[0106]

[0107] in, , The maximum word length predefined in the text encoder.

[0108] In step S105, a complete learning module is proposed to jointly optimize the key semantic saliency and integrity, taking into account the synergistic relationship between saliency and integrity, and to achieve complete alignment of cross-modal pedestrian key semantic information. This module contains two sub-modules: Prominent Semantics Alignments (PSA) and Integral Semantics Alignments (ISA). The proposed salient semantic alignment module uses the salient semantics of images and text to achieve cross-modal alignment. For pedestrian-related visual key semantics, First, we use a multi-layer perceptron to enhance its semantic representation ability and obtain the key semantics of visual significance :

[0109]

[0110] in, is a multi-layer perceptron module, express Regularization. Subsequently, visual salient semantics are obtained by aggregating salient key semantics :

[0111]

[0112] in, It is the maximum pooling operation.

[0113] Textual salient semantics The acquisition of is similar to the acquisition of visual salient semantics. First, the pedestrian-related text key semantics Using multi-layer perceptron to obtain key semantics of text saliency These key semantics are then aggregated to obtain the salient semantics of the text , the calculation process is as follows:

[0114]

[0115]

[0116] In step S106, in the salient semantic alignment module, the visual salient semantics are calculated. and textual salient semantics The cosine similarity between them is used to obtain the cross-modal saliency similarity .

[0117] In step S107, a complete semantic alignment module is proposed to prevent the model from focusing only on significant semantics and causing local optimal solutions. Applying multi-layer perceptron to obtain pedestrian-related integrity key semantics

[0118]

[0119] By analyzing the key semantics of pedestrian-related text Applying multi-layer perceptron to obtain pedestrian-related integrity key semantics

[0120]

[0121] In step S108, in order to explore the relationship between image regions and text words, the key semantics of visual integrity are calculated. Key semantics with text integrity The cosine similarity between them is used to obtain the relationship matrix from the image region to the text word direction ,in Each row represents the connection between an image region selected from an image and all the words selected from a sentence. For each image region, the similarity of the word with the highest similarity is used to represent the correlation between each image region and the entire text, and finally the correlation between each image region in an image and the text is obtained. , the calculation process is as follows:

[0122]

[0123] To obtain completeness information, the correlation between all image regions and text Calculate the average value to obtain the image-text similarity from the image area to the text word direction , the calculation process is as follows:

[0124]

[0125] in, Represents the mean operation.

[0126] Similarly, for each text word, the similarity of the local image area with the highest similarity is selected to represent the correlation between each text word and the local image area, so as to obtain the correlation between each text word in a text and the local image area. :

[0127]

[0128] The association between all text words and local image regions Calculate the average value to obtain the second image text similarity from the text word to the local area of ​​the image for:

[0129]

[0130] Finally, the cross-modal completeness similarity of each image-text pair in the complete semantic alignment module is obtained :

[0131] =

[0132] In step S109, the cross-modal global similarity , cross-modal saliency similarity and cross-modal completeness similarity The training objective is optimized, and identity constraints are added to the global visual and text features. The pre-trained multimodal large model is fine-tuned using the Adam optimization algorithm until the model converges, resulting in a fine-tuned pre-trained multimodal large model.

[0133] Specifically, the visual global features and text global features are mapped into a unified identity space and the cross entropy loss function is used to perform pedestrian classification;

[0134] Cross-modal matching is performed on the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity using a contrast loss function, and then the pre-trained multimodal large model is fine-tuned using the Adam optimizer until the model converges to obtain a fine-tuned pre-trained multimodal large model.

[0135] In step S110, during the test phase, text is input and the fine-tuned pre-trained multimodal large model encodes the text to obtain text features; based on the text features, the cross-modal global similarity between the text and the pre-stored multiple images is calculated according to steps S103, S106 and S108. , cross-modal saliency similarity and cross-modal completeness similarity ; Then the cross-modal global similarity , cross-modal saliency similarity and cross-modal completeness similarity Sum up to get the final image-text similarity ; Finally, based on the similarity between different samples Sort by size to get the retrieval result sequence.

[0136] In one embodiment, the proposed large-model knowledge-based cross-modal pedestrian search key semantics complete alignment method introduces: a multimodal large model, a noise filtering module, and a complete learning module that jointly optimizes key semantic saliency and completeness. We use a multimodal pre-trained large model and leverage its external general knowledge to extract more discriminative visual and textual features; the noise filtering module filters out pedestrian-irrelevant noise in images and text, retaining key semantic information related to pedestrians; and the complete learning module that jointly optimizes key semantic saliency and completeness considers both the saliency and completeness of key semantics, deeply explores the correspondence between cross-modal key semantics, and achieves more comprehensive cross-modal fine-grained alignment. In response to the noise interference and fine-grained alignment problems encountered by existing technologies in natural language cross-modal pedestrian search tasks, the proposed improved method can filter out pedestrian-irrelevant noise information, enhance the model's attention to pedestrian-related key information, explore the correspondence between cross-modal fine-grained information, and thus improve the accuracy of cross-modal pedestrian retrieval, achieving a significant improvement in accuracy compared to existing methods.

[0137] In one embodiment, a key semantic saliency and completeness alignment network based on large model knowledge is proposed. A multimodal large model is innovatively used as the backbone network to obtain visual and text features, and noise information irrelevant to pedestrians in images and texts is filtered out to obtain key semantic information. Subsequently, accurate alignment of key semantics is achieved from the perspective of joint optimization of saliency and completeness, enhancing the model's attention to fine-grained key semantic information related to pedestrians and solving the problems of noise interference and fine-grained alignment.

[0138] In one embodiment, a pre-trained cross-modal large model is used as the backbone network, and the external general knowledge it carries is used to obtain more discriminative visual and textual features.

[0139] In one embodiment, a noise filtering module is proposed to adaptively filter out pedestrian-irrelevant noise in images and texts, retain key semantic information, and alleviate the interference of noise on cross-modal alignment.

[0140] In one embodiment, a complete learning module is proposed for joint optimization of key semantic saliency and completeness, while considering the synergistic relationship between saliency and completeness to achieve complete alignment of cross-modal pedestrian key semantic information and improve the model's ability to discriminate different pedestrians with similar appearances.

[0141] In one embodiment, for the natural language cross-modal pedestrian search task, the proposed cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge achieved significant improvement in retrieval accuracy compared with existing solutions.

[0142] In a specific embodiment, Figure 3 As shown in the figure, according to the query statement, the cross-modal pedestrian search key semantic complete alignment method based on large model knowledge is used to retrieve the target person image in the image library; the figure gives the top ten retrieval results corresponding to each query statement in descending order according to the similarity between the text and the image, among which the green frame image is the correct target image corresponding to the query statement.

[0143] Through the aforementioned large-model knowledge-based method for complete alignment of key semantics for cross-modal pedestrian search, the external general knowledge contained in the pre-trained visual language large model is utilized to extract more discriminative visual and textual features. A noise filtering module is then designed to filter out noise irrelevant to pedestrians in images and text by referencing the relationship between global and local semantics within the modality, retaining only the fine-grained key semantic information relevant to pedestrians. Furthermore, a complete learning module is proposed to jointly optimize the saliency and completeness of key semantics, simultaneously considering the synergistic relationship between saliency and completeness to achieve complete alignment of cross-modal pedestrian key semantic information. This approach is the first to jointly consider the saliency and completeness aspects of cross-modal pedestrian key semantics. By referencing the relationship between global and local semantics within the modality, key semantics related to pedestrians in images and text are obtained. A strategy of co-tuning saliency and completeness is then used to fully exploit the correspondence between fine-grained cross-modal key semantic information, thereby more accurately distinguishing different pedestrians with similar appearances, achieving significant performance improvements in the field of natural language cross-modal pedestrian search.

[0144] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly indicate the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0145] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0146] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge, characterized by: The method includes: Constructing a pre-trained multimodal large model; wherein the pre-trained multimodal large model includes a pre-trained visual language large model, a noise filtering module, and a key semantic saliency and integrity joint optimization complete learning module, and the key semantic saliency and integrity joint optimization complete learning module includes a salient semantic alignment submodule and a complete semantic alignment submodule; The pre-trained visual language model is used to extract the visual global features and visual local features of the training set images, and the text global features and text local features of the word encodings; Obtaining cross-modal global similarity and pedestrian classification based on the visual global features and the text global features; Using the noise filtering module to filter out noise information irrelevant to pedestrians in the visual local features and the text local features to obtain visual key semantics and text key semantics; Processing the visual key semantics and the text key semantics using the salient semantics alignment submodule to obtain visual salient semantics and text salient semantics; Obtaining cross-modal saliency similarity based on the visual salient semantics and the textual salient semantics; Processing the visual key semantics and the text key semantics using the complete semantic alignment submodule to obtain visual completeness key semantics and text completeness key semantics; Obtaining a cross-modal integrity similarity of the image-text pair according to the visual integrity key semantics and the text integrity key semantics; Based on the pedestrian classification, the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity, fine-tuning the pre-trained multimodal large model using the Adam optimization algorithm to obtain the fine-tuned pre-trained multimodal large model; The query statement is input into the fine-tuned pre-trained multimodal large model to obtain a retrieval result sequence.

2. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 1 is characterized in that: The steps of respectively extracting the visual global features and the visual local features of the image, and the text global features and the text local features of the text using the pre-trained visual language model include: Normalizing the training set images to a uniform size, and performing data augmentation on the training set images by image flipping and random cropping; The data-enhanced training set images are input into the pre-trained visual language model, and the visual encoder in the pre-trained visual language model is used to extract visual features. Among them, Recognized as the visual global feature ,Will Recognized as the visual local features , is the number of local regions in an image; Encoding the word into a word vector using a word vector embedding method; Input the word vector into the pre-trained visual language model, and use the text encoder in the pre-trained visual language model to extract text features. Among them, Recognized as the global feature of the text ,Will Recognized as the local features of the text , is the number of words in each text, is the feature vector corresponding to the text start mark.

3. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 2 is characterized by: The step of obtaining cross-modal global similarity and pedestrian classification based on the visual global features and the text global features includes: The visual global feature and the global features of the text By calculating the first cosine similarity and mapping it to a unified encoding space for coarse-grained semantic alignment, the cross-modal global similarity is obtained. ; The visual global feature and the global features of the text Mapping from the feature space to the identity space to obtain the pedestrian classification.

4. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 3 is characterized by: The step of using the noise filtering module to filter out noise information irrelevant to pedestrians in the visual local features and the text local features to obtain visual key semantics and text key semantics includes: Get the visual self-attention map from the last layer of Transformer module in the visual encoder , and according to the visual self-attention map Get the visual global-local correlation ; Wherein, the visual self-attention map The relationship between the visual special label [CLS] and the local area of ​​the image is described in [C]. Introducing selection ratio , to obtain the number of retained local image regions ;in, Indicates rounding down; According to the visual global-local correlation , get the maximum correlation The local image area corresponding to the value is regarded as the area related to the pedestrian, so as to obtain the local visual features. Extract The features of the image are used as the key visual semantics related to pedestrians ; Get the text self-attention map from the last layer of Transformer module in the text encoder , and according to the text self-attention map Get global-local correlation ; Wherein, the text self-attention map What is depicted is the connection between the text special tag [EOS] and the text word; Introducing the selection ratio , to get the number of retained text words ;in, , The maximum word length predefined in the text encoder; According to the global-local correlation of the text From the local features of the text Extract The features of the words are used as the key semantics of the text related to pedestrians .

5. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 4 is characterized in that: The step of processing the visual key semantics and the text key semantics using the salient semantics alignment submodule to obtain visual salient semantics and text salient semantics includes: Enhance the visual key semantics using the first multi-layer perceptron module Semantic representation ability to obtain visually significant key semantics : in, is the first multi-layer perceptron module, express Regularization; By aggregating the visually salient key semantics , to obtain the visually significant semantics : in, It is the maximum pooling operation; Using the second multi-layer perceptron module to enhance the key semantics of the text Semantic representation ability to obtain the key semantics of text saliency : By aggregating the key semantics of the text , to obtain the significant semantics of the text : in, It is the second multi-layer perceptron module.

6. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 5 is characterized in that: The step of obtaining cross-modal salient similarity based on the visual salient semantics and the textual salient semantics includes: Calculate the visual salient semantics and the text is semantically significant The second cosine similarity between them is used to obtain the cross-modal saliency similarity .

7. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 6 is characterized in that: The step of processing the visual key semantics and the text key semantics by using the complete semantic alignment submodule to obtain the visual integrity key semantics and the text integrity key semantics includes: The third multi-layer perceptron module is used to obtain the visual key semantics Key semantics of pedestrian-related visual completeness : in, is the third multi-layer perceptron module; Use the fourth multi-layer perceptron module to obtain the key semantics of the text Key semantics of pedestrian-related text integrity : in, It is the fourth multi-layer perceptron module.

8. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 7 is characterized in that: The step of obtaining the cross-modal integrity similarity of the image-text pair according to the visual integrity key semantics and the text integrity key semantics includes: According to the visual integrity key semantics and the text integrity key semantics , get the third cosine similarity; According to the third cosine similarity, the relationship matrix from the local area of ​​the image to the direction of the text word is obtained ;in, Each row represents the connection between a local area of ​​an image selected from an image and all words selected from a text; For each local area of ​​the image, the similarity of the word with the highest similarity is selected to represent the correlation between each local area of ​​the image and the entire text, so as to obtain the correlation between each local area of ​​the image and the text in an image. : The correlation between all local areas of the image and the text Calculate the average value to get the first image-text similarity from the local area of ​​the image to the text word direction : in, represents the mean operation; For each text word, the similarity of the local image area with the highest similarity is selected to represent the correlation between each text word and the local image area, so as to obtain the correlation between each text word in a text and the local image area. : The association between all text words and local image regions Calculate the average value to obtain the second image text similarity from the text word to the local area of ​​the image for: 。 9. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 8 is characterized in that: The step of fine-tuning the pre-trained multimodal large model using the Adam optimization algorithm based on the pedestrian classification, the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity to obtain the fine-tuned pre-trained multimodal large model includes: Mapping the visual global features and the text global features into a unified identity space and performing pedestrian classification using a cross entropy loss function; The cross-modal global similarity, the cross-modal saliency similarity and the cross-modal completeness similarity are cross-modally matched using a contrast loss function, and then the pre-trained multimodal large model is fine-tuned using an Adam optimizer to obtain the fine-tuned pre-trained multimodal large model.

10. The cross-modal pedestrian search key semantics completeness alignment method based on large model knowledge according to claim 9 is characterized in that: Inputting the query statement into the fine-tuned pre-trained multimodal large model to obtain a retrieval result sequence includes: Inputting the query statement into the fine-tuned pre-trained multimodal large model to extract the global text features and the local text features of the query statement; Calculating the cross-modal global similarity, the cross-modal saliency similarity, and the cross-modal completeness similarity between the query statement and a plurality of pre-stored images based on the global text features and the local text features of the query statement; The cross-modal global similarity, the cross-modal saliency similarity and the cross-modal integrity similarity are summed to obtain multiple image-text similarities. ; According to the similarity of all the images and texts to obtain the retrieval result sequence.

Images