An integrated cross-modal pedestrian retrieval method for detecting and retrieving tasks in parallel

Abstract

The application discloses a kind of detection and retrieval task parallel integrated cross-modal pedestrian retrieval method, it is related to computer vision technical field, solve the technical problem that cross-modal pedestrian retrieval method memory occupation is high, inference time delay is big.The method includes: obtaining the whole image to be retrieved and text query, after processing, input cross-modal weight adapter, generate dynamic weight increment;Convert fixed visual coding into text-guided visual coding;Share the regression query vector of the output of the first layer of coding layer, detect branch predicts target pedestrian bounding box;Based on regression query vector, retrieve branch gets image retrieval feature;Locate and mark target pedestrian in the whole image to be retrieved, output target frame, retrieval score and sorting result;Detection branch and retrieval branch are jointly trained to obtain retrieval model algorithm.The application reduces memory occupation and time delay, meets the demand of pedestrian retrieval.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks. Background Technology

[0002] Currently, most image-based cross-modal pedestrian retrieval methods follow a two-stage paradigm of "detection first, retrieval later." In the detection stage, the image-side first uses an object detection network to regress and output all potential pedestrian candidate boxes in the global image. Then, through region feature extraction, local visual representations of each candidate pedestrian are cropped from the global visual feature map. In the retrieval stage, the text-side uses a language encoder to encode the natural language query into a high-dimensional semantic feature vector. Finally, in a unified metric space, the similarity between the text semantics and each visual representation is calculated to achieve the final matching and ranking. Throughout the entire process, the target localization and cross-modal matching subtasks operate independently and sequentially.

[0003] While this type of method can adapt to raw surveillance images, the detection and retrieval tasks are usually completed by different networks or different processing stages. This serial processing mode has problems such as high memory consumption and large inference latency. In actual operation, there are obvious performance bottlenecks and logical defects, making it difficult to meet the requirements of real-time retrieval and end-to-end optimization.

[0004] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art: Existing two-stage text-image cross-modal pedestrian retrieval methods have high memory consumption and large inference latency, making it difficult to meet the needs of real-time pedestrian retrieval. Summary of the Invention

[0005] The purpose of this invention is to provide an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks in parallel, thereby addressing the technical problems of existing two-stage text-image cross-modal pedestrian retrieval methods, which suffer from high memory consumption and large inference latency, making it difficult to meet the requirements of real-time pedestrian retrieval. The various technical effects of the preferred solutions among the many technical solutions provided by this invention are detailed below.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks, comprising the following steps: S100: acquiring the entire image to be retrieved. and text query The visual feature map is obtained through processing. and text retrieval features Visual feature map Text retrieval features Input a cross-modal weight adaptor to generate dynamic weight increments for the attention projection layer in the visual encoder. S200: Dynamic weight increment At least one of the query projection weight, key projection weight, and value projection weight applied to the multi-head self-attention module converts the fixed visual encoding into text-guided visual encoding; S300: To The language adaptive visual coding layer, composed of a layer transformer encoder, takes the query vector as input. This layer serves as a shared coding layer for the detection and retrieval branches. The shared coding layer outputs the first... Layer regression query vector The detection branch is based on the last layer of regression query vector. Predict target pedestrian bounding box S400: Based on regression query vector Search branch prediction Two-dimensional offset of each sampling point and sampling weights In visual feature map Local feature sampling is performed on the image, and the resulting image retrieval features are obtained by fusing them using a lightweight transform encoder. S500: During the inference phase, the image retrieval features output from the last layer of the retrieval branch are used. Text retrieval features Normalization and similarity calculation are performed to obtain the text query. The matching score between the candidate target and the target pedestrian bounding box is used to determine the matching score. Association, in the corresponding entire image to be retrieved The system locates and marks the target pedestrian, outputs the target bounding box, retrieval score, and ranking result; S600: During the training phase, based on the target bounding box, retrieval score, and ranking result, the detection branch and retrieval branch are simultaneously optimized and jointly trained with loss weighting to obtain an integrated cross-modal pedestrian retrieval model algorithm, and pedestrian retrieval is performed based on the integrated cross-modal pedestrian retrieval model algorithm.

[0007] Preferably, in step S100, the text query... Used to describe the appearance attributes of the target pedestrian, text query Input any one of the following: general text encoder, BERT model, RoBERTa model, or CLIP text encoder, to obtain text retrieval features. .

[0008] Preferably, in step S100, the text query... When inputting a general text encoder, a pre-trained language model is used, and the global semantic vector corresponding to the first classification label of the sequence is taken and passed through a fully connected layer to obtain text retrieval features. .

[0009] Preferably, in step S100, the entire image is... The image is divided into blocks and positional encoding is added. These blocks are then input into a ViT visual encoder, a Swing Transformer visual encoder, a hybrid backbone of a convolutional network and a Transformer, or a lightweight visual backbone to obtain visual feature maps. .

[0010] Preferably, in step S100, the dynamic weight increment of the attention projection layer is generated using the following formula. and output hidden features : , , in, To utilize the hidden features obtained after superimposing dynamic weight increments, These are the pre-trained static weights in the visual encoder. For dynamic weight increments, The input feature vector of the attention projection layer, and As a low-rank adapting substrate, These are the dynamic coefficients generated from the current text and image input. For the number of low-rank adapting bases, is the ordinal number of the low-rank basis.

[0011] Preferably, in step S100, the visual feature map is... Text retrieval features The specific steps for inputting the cross-modal weighted adaptor are as follows: Mapping textual and visual features to a shared latent space: ; Calculate the spatial correlation weight graph : ; Based on this correlation weight map, text-related visual embeddings are aggregated. E : E ; Dynamic coefficients are generated jointly by text vectors and visual feedback. : ; in, For text queries The global text semantic vector obtained after linear mapping and feature pooling For feature pooling operations, For text queries, Visual feature map The original visual feature matrix after flattening This is the shared latent space visual feature matrix obtained after linear mapping. This is a spatial correlation weighting diagram. It is the Sigmoid activation function. For text-related visual embedding, For bias terms, , , It is a learnable mapping matrix.

[0012] Preferably, in step S200, the formula for the dynamic weight increment ΔW simultaneously acting on the query projection weight, key projection weight, and value projection weight of the multi-head self-attention module is as follows: ; in, This represents the visual feature matrix input to the multi-head self-attention module; , and These represent the query projection weight matrix, the key projection weight matrix, and the value projection weight matrix, respectively. , and These represent the dynamic weight increments of the query, key, and value projections generated by the cross-modal weighted adaptor based on the current text and image input; , and These represent the query matrix, key matrix, and value matrix formed after dynamic compensation, respectively.

[0013] Preferably, in step S400, image retrieval features The expression is: , in, As a reference point, For the first Each sampling point offset, For sampling weights, The number of sampling points. Represents visual feature maps. Represents the sampling function. This indicates that the sampled local features are fed into the lightweight Transformer encoder for fusion.

[0014] Preferably, in step S600, the loss function expression for the loss-weighted joint training of the detection branch and the retrieval branch is: , , , in, Weighting coefficients for detecting loss; This represents the loss function for the detection branch. Indicates Smooth L1 loss, Indicates GIOU loss; This represents the loss function for the retrieval branch. This represents the similarity distribution matching loss. This indicates the loss in identity classification.

[0015] Preferably, in step S600, during the joint training phase of the integrated cross-modal pedestrian retrieval model algorithm, the outputs of encoders from layer 1 to layer 6 are all incorporated into the retrieval branch for supervision; during the inference phase of the integrated cross-modal pedestrian retrieval model, only layer 6 of the encoder is used for output.

[0016] Implementing one of the above-described technical solutions of the present invention has the following advantages or beneficial effects: In this application, the detection branch and the retrieval branch share image-text fusion features, avoiding the repeated extraction of visual features in the two-stage method, reducing memory usage and latency, and effectively meeting the needs of real-time pedestrian retrieval; the cross-modal weight adaptor transforms text semantics into visual encoder weight increments, enabling text target perception capabilities in the image encoding stage; the adaptive sampling fusion unit samples key local regions of the target pedestrian under the guidance of the detection query vector, improving the ability to express fine-grained attributes such as clothing, shoes, and backpacks; the training stage adopts multi-layer retrieval supervision, and the inference stage only uses the last layer feature path, balancing robustness and inference efficiency. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart of an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks in parallel, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the integrated network structure of an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks in parallel according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the cross-modal weighted adaptor structure of an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks in parallel according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the adaptive sampling fusion device structure of an integrated cross-modal pedestrian retrieval method that performs detection and retrieval tasks in parallel according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the parallel multi-level supervision and joint training mechanism of an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be used to implement the present invention. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of the present invention disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of the present invention.

[0019] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0020] To illustrate the technical solution described in this invention, specific embodiments are described below, showing only the parts related to the embodiments of this invention.

[0021] Example: like Figure 1 , Figure 2 As shown, the present invention provides an integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks, including the following steps: S100: Acquire the entire image to be retrieved. and text query The visual feature map is obtained through processing. (In this embodiment, visual encoder processing is preferred) and text retrieval features Visual feature map Text retrieval features Input a cross-modal weight adaptor to generate dynamic weight increments for the attention projection layer in the visual encoder. Dynamic weight increment By adjusting the attention projection matrix with relevant low-rank increments, the model's adaptability to context is significantly improved with almost no increase in inference overhead. Dynamic weight increments are generated through a cross-modal weight adaptor. , , In this embodiment, the attention projection weights of the visual encoder are dynamically compensated based on text semantics. Furthermore, the cross-modal weight adaptor can also drive the visual encoder to encode images. Obtain visual features S200: Dynamic weight increment At least one of the query projection weights, key projection weights, and value projection weights applied to the multi-head self-attention module in the visual encoder can also be extended to feedforward networks, cross-attention modules, or other linear projection layers. This transforms fixed visual encoding into text-guided visual encoding, converting the passive image reception and fixed feature extraction unidirectional process into a bidirectional interactive mechanism where text commands control visual attention in real time. By dynamically adjusting the focus, granularity, and semantic level of feature extraction, it significantly improves the model's adaptability to task requirements. S300: Towards The language adaptive visual coding layer, composed of a layer transformer encoder, takes the query vector as input. This layer serves as a shared coding layer for the detection and retrieval branches. The shared coding layer outputs the first... Layer regression query vector For each level i The language-adaptive visual coding layer will fuse text and image information and output the regression query vector of the current layer. Based on this regression query vector The adaptive sampling fusion unit samples and aggregates local discriminative information to generate image retrieval features at the corresponding level. Call for submissions F In the middle. The detection branch is based on the last layer of regression query vector. (In this embodiment, the regression query vector is at layer 6) Predict the bounding box of the target pedestrian. The bounding box can be represented as the coordinates of its center point and its width and height. S400: Based on the regression query vector. Search branch prediction Two-dimensional offset of each sampling point and sampling weights In visual feature map Local feature sampling is performed (preferably bilinear interpolation sampling in this embodiment, a mathematical technique that generates new data by estimating unknown points between known data points, which significantly improves visual quality compared to nearest neighbor interpolation), and then fused by a lightweight transform encoder (a lightweight Transformer encoder in this embodiment) to obtain image retrieval features. S500: During the inference phase, the image retrieval features output from the last layer of the retrieval branch are used. Text retrieval features Normalization and similarity calculation are performed to obtain the text query. The matching score between the candidate target and the target pedestrian bounding box is used to determine the matching score. Association, in the corresponding entire image to be retrieved The system locates and marks target pedestrians, outputting the target bounding box, retrieval score, and ranking result. S600: During the training phase, based on the target bounding box, retrieval score, and ranking result, the detection branch and retrieval branch are simultaneously optimized and jointly trained with loss weighting to obtain an integrated cross-modal pedestrian retrieval model algorithm. Pedestrian retrieval is then performed based on this algorithm. In this embodiment, the detection and retrieval branches share image-text fusion features, avoiding redundant extraction of visual features in the two-stage method, reducing memory usage and latency, and effectively meeting the real-time pedestrian retrieval requirements. The cross-modal weighted adaptor transforms text semantics into visual encoder weight increments, enabling text target perception capabilities during the image encoding stage. The adaptive sampling fusion unit samples key local regions of the target pedestrian under the guidance of the detection query vector, improving the expression of fine-grained attributes such as clothing, shoes, and backpacks. The training phase employs multi-layer retrieval supervision, while the inference phase uses only the last layer of feature paths, balancing robustness and inference efficiency. This embodiment achieves an AP (Average Precision, mainly used to evaluate the performance of pedestrian detection tasks, measures the model's ability to accurately outline pedestrians in a given image) of 44.11% and a retrieval mAP (mean Average Precision, used to evaluate the performance of pedestrian re-identification and retrieval, measures the model's ability to accurately match and retrieve the target pedestrian in an image database containing distractors) of 46.12% on the CUHK-SYSU-TBPS dataset (a large-scale benchmark dataset designed specifically for pedestrian detection and re-identification, mainly derived from urban street snapshots and film clips, aiming to cover a wide range of environmental changes and challenges, such as viewpoint changes, lighting conditions, occlusion, etc.; data scale: a total of 18,184 images, 96,143 pedestrian bounding boxes, and 8,432 unique pedestrian identifiers; data partitioning: the training set contains 11,206 images and 5,532 query persons, and the test set contains 6,978 images and 2,900 query persons). After introducing a cross-modal weighted adaptor, the detection AP improved from 41.90% to 44.11%, and the retrieval mAP improved from 44.63% to 46.12%. When the adaptive sampling point count was 81, the retrieval mAP reached 46.12%, which is better than the settings of 49, 64, and 100 sampling points. Compared with the semantic-driven proposal generation (SDPG, which uses high-level semantic information such as text description, category prior, or contextual relationships to guide the generation of candidate regions), the embodiment of this invention has 121.5M parameters and an inference latency of 7.33ms, significantly reducing model size and latency while maintaining good retrieval accuracy.

[0022] As an optional implementation, in step S100, text query Used to describe the appearance attributes of a target pedestrian, such as clothing color, pants, shoes, backpack, hairstyle, posture, etc. Text queries... Input any one of the following: general text encoder, BERT model, RoBERTa model, or CLIP text encoder, to obtain text retrieval features. General-purpose text encoders (such as Word2Vec) generate static word vectors that are independent of context. BERT / RoBERTa are context-dependent dynamic encoders designed specifically for pure text semantic understanding. The CLIP text encoder, on the other hand, serves cross-modal alignment, mapping text to a semantic space shared with images. It is not directly used for fine-tuning text tasks, thus facilitating adaptation to different training objectives and application scenarios, and improving the adaptability of this method. Text query When inputting to the general text encoder, a pre-trained language model is used. The pre-trained language model significantly reduces the dependence of downstream tasks on labeled data and uniformly solves various natural language processing problems. It significantly improves intelligent completion, context-aware editing, and document understanding capabilities. The global semantic vector corresponding to the first classification label of the sequence is taken (because the pre-training task forces the model to compress the global semantics to this position through backpropagation, rather than the model naturally possessing global information). This vector is then passed through a fully connected layer to obtain text retrieval features. .

[0023] As an optional implementation, in step S100, the entire image is... The image is divided into blocks and positional encoding is added to facilitate the conversion of 2D images into serialized inputs that can be processed by Transformers. Simultaneously, spatial structure information is explicitly injected through positional encoding, compensating for the Transformer's lack of positional awareness. Inputs can be ViT visual encoders (suitable for image classification tasks pre-trained on large datasets, tasks requiring strong global contextual understanding), SwinTransformer visual encoders (achieving a balance between efficiency and accuracy), hybrid backbones of convolutional networks and Transformers (reducing data dependency, more suitable for edge device deployment), or lightweight visual backbones (faster inference speed than ViT, suitable for embedded devices) to obtain visual feature maps. The appropriate method can be selected based on task requirements and usage scenarios to balance global modeling capabilities, computational efficiency, and hardware compatibility, thereby improving the adaptability of this method.

[0024] As an optional implementation, in step S100, such as Figure 3 As shown, the dynamic weight increments of the attention projection layer are generated using the following formula. and output hidden features : , ,in, To utilize the hidden features obtained after superimposing dynamic weight increments, These are the pre-trained static weights in the visual encoder. For dynamic weight increments, The input feature vector of the attention projection layer, and As a low-rank adapting substrate, These are the dynamic coefficients generated from the current text and image input. For the number of low-rank adapting bases, is the ordinal number of the low-rank basis.

[0025] As an optional implementation, in step S100, the visual feature map is... Text retrieval features The specific operation of the input cross-modal weighted adaptor is as follows: mapping text features and visual features to a shared latent space: ; Calculate the spatial correlation weight map : Based on this relevance weight graph, text-related visual embeddings are aggregated. E : E Dynamic coefficients are generated jointly by text vectors and visual feedback. : ;in, For text queries The global text semantic vector obtained after linear mapping and feature pooling For feature pooling operations, For text queries, Visual feature map The original visual feature matrix after flattening This is the shared latent space visual feature matrix obtained after linear mapping. This is a spatial correlation weighting diagram. It is the Sigmoid activation function. For text-related visual embedding, For bias terms, , , It is a learnable mapping matrix.

[0026] As an optional implementation, in step S200, such as Figure 3 As shown, the formula for the dynamic weight increment ΔW simultaneously acting on the query projection weight, key projection weight, and value projection weight of the multi-head self-attention module is: ;in, This represents the visual feature matrix input to the multi-head self-attention module; , and These represent the query projection weight matrix, the key projection weight matrix, and the value projection weight matrix, respectively. , and These represent the dynamic weight increments of the query, key, and value projections generated by the cross-modal weighted adaptor based on the current text and image input; , and These represent the query matrix, key matrix, and value matrix formed after dynamic compensation, respectively. Through the above calculations, the same image will generate different query vectors, key vectors, and value vectors when faced with different text queries, thus enabling the visual encoder to highlight the regions related to the text description during the feature extraction stage.

[0027] As an optional implementation, in step S400, such as Figure 4 As shown, image retrieval features The expression is: ,in, As a reference point, For the first Each sampling point offset, For sampling weights, For the number of sampling points, the preferred method is... , Represents visual feature maps. Represents the sampling function. The sampled local features are fed into a lightweight Transformer encoder for fusion. The lightweight Transformer encoder solves the computational bottleneck of Transformer in real-time detection and edge deployment by using designs such as alternating attention, channel dimensionality reduction, CNN hybrid architecture, and dynamic pruning, while retaining the ability to model global context.

[0028] As an optional implementation, in step S600, the loss function expression for the loss-weighted joint training of the detection branch and the retrieval branch is: , , ,in, The preferred value for the weighting coefficient used to detect loss is 0.92. This represents the loss function for the detection branch. This represents the Smooth L1 loss. The Smooth L1 loss is a piecewise regression loss function. When the prediction error is small, the L2 loss (squared term) is used to ensure gradient smoothness. When the error is large, it degenerates into the L1 loss (linear term) to suppress the influence of outliers and optimize the coordinate offset between the predicted bounding box and the ground truth box. GIOU represents the loss function. By introducing the area of ​​the minimum bounding rectangle as a penalty term, the loss function can still provide an effective gradient in non-overlapping scenarios. It is used to solve the defect that non-overlapping samples cannot be optimized in bounding box regression. When they are completely overlapping, GIOU=1. When they are non-overlapping and infinitely far apart, the value of GIOU tends to -1. This represents the loss function for the retrieval branch. This represents the similarity distribution matching loss. By modeling the matching relationship of sample pairs as a probability distribution rather than a single similarity score, it forces the model to learn a globally consistent semantic alignment structure, thus addressing the problem of traditional point-to-point loss ignoring the relative relationships of samples within a batch. The identity classification loss is a representation learning method that treats each pedestrian ID as an independent category and uses cross-entropy loss to supervise feature learning, enabling the features extracted by the model to accurately distinguish different identities. In this example, SDM loss, ID loss, triplet loss, contrastive learning loss, or a combination thereof can also be used. ID loss is the basic supervision signal for classification tasks, triplet loss and contrastive learning loss belong to the category of metric learning (improving discriminability by constraining the distance in the feature space), while SDM loss specifically refers to the dynamically weighted loss designed for scale sensitivity in infrared small target detection.

[0029] As an optional implementation, in step S600, such as Figure 5 As shown, during the joint training phase of the integrated cross-modal pedestrian retrieval model algorithm, the outputs of encoders from layers 1 to 6 are all fed into the retrieval branch for supervision; during the inference phase of the integrated cross-modal pedestrian retrieval model, only layer 6 of the encoder is used for output. This mechanism of multi-layered retrieval supervision during the training phase and rapid output from the last layer during the inference phase balances model learning sufficiency and inference efficiency.

[0030] The embodiment is merely a specific example and does not indicate that this is the only way to implement the present invention.

[0031] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. An integrated cross-modal pedestrian retrieval method that combines detection and retrieval tasks, characterized in that, Includes the following steps: S100: Obtain the entire image to be retrieved. and text query The visual feature map is obtained through processing. and text retrieval features Visual feature map Text retrieval features Input a cross-modal weight adaptor to generate dynamic weight increments for the attention projection layer in the visual encoder. ; S200: Dynamic Weight Increment At least one of the query projection weight, key projection weight, and value projection weight applied to the multi-head self-attention module converts fixed visual encoding into text-guided visual encoding. S300: Towards The language adaptive visual coding layer, composed of a layer transformer encoder, takes the query vector as input. This layer serves as a shared coding layer for the detection and retrieval branches. The shared coding layer outputs the first... Layer regression query vector The detection branch is based on the last layer of regression query vector. Predict target pedestrian bounding box ; S400: Based on Regression Query Vector Search branch prediction Two-dimensional offset of each sampling point and sampling weights In visual feature map Local feature sampling is performed on the image, and the resulting image retrieval features are obtained by fusing them using a lightweight transform encoder. ; S500: During the inference phase, the image retrieval features output from the last layer of the retrieval branch will be used. Text retrieval features Normalization and similarity calculation are performed to obtain the text query. The matching score between the candidate target and the target pedestrian bounding box is used to determine the matching score. Association, in the corresponding entire image to be retrieved Locate and mark the target pedestrian in the center, and output the target bounding box, search score and ranking results; S600: During the training phase, based on the target box, retrieval score, and ranking results, the detection branch and retrieval branch are simultaneously optimized and jointly trained with loss weighting to obtain an integrated cross-modal pedestrian retrieval model algorithm. Pedestrian retrieval is then performed based on the integrated cross-modal pedestrian retrieval model algorithm.

2. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S100, text query Used to describe the appearance attributes of the target pedestrian, text query Input any one of the following: general text encoder, BERT model, RoBERTa model, or CLIP text encoder, to obtain text retrieval features. .

3. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S100, text query When inputting a general text encoder, a pre-trained language model is used, and the global semantic vector corresponding to the first classification label of the sequence is taken and passed through a fully connected layer to obtain text retrieval features. .

4. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S100, the entire image The image is divided into blocks and positional encoding is added. These blocks are then input into a ViT visual encoder, a Swing Transformer visual encoder, a hybrid backbone of a convolutional network and a Transformer, or a lightweight visual backbone to obtain visual feature maps. .

5. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S100, the dynamic weight increment of the attention projection layer is generated using the following formula. and output hidden features : ， ， in, To utilize the hidden features obtained after superimposing dynamic weight increments, These are the pre-trained static weights in the visual encoder. For dynamic weight increments, The input feature vector of the attention projection layer, and As a low-rank adapting substrate, These are the dynamic coefficients generated from the current text and image input. For the number of low-rank adapting bases, is the ordinal number of the low-rank basis.

6. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S100, the visual feature map Text retrieval features The specific steps for inputting the cross-modal weighted adaptor are as follows: Mapping textual and visual features to a shared latent space: ; Calculate the spatial correlation weight graph : ; Based on this correlation weight map, text-related visual embeddings are aggregated. E : E ; Dynamic coefficients are generated jointly by text vectors and visual feedback. : ; in, For text queries The global text semantic vector obtained after linear mapping and feature pooling For feature pooling operations, For text queries, Visual feature map The original visual feature matrix after flattening This is the shared latent space visual feature matrix obtained after linear mapping. This is a spatial correlation weighting diagram. It is the Sigmoid activation function. For text-related visual embedding, For bias terms, , , It is a learnable mapping matrix.

7. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S200, the formula for the dynamic weight increment ΔW simultaneously acting on the query projection weight, key projection weight, and value projection weight of the multi-head self-attention module is as follows: ; in, This represents the visual feature matrix input to the multi-head self-attention module; , and These represent the query projection weight matrix, the key projection weight matrix, and the value projection weight matrix, respectively. , and These represent the dynamic weight increments of the query, key, and value projections generated by the cross-modal weighted adaptor based on the current text and image input; , and These represent the query matrix, key matrix, and value matrix formed after dynamic compensation, respectively.

8. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S400, image retrieval features The expression is: ， in, As a reference point, For the first Each sampling point offset, For sampling weights, The number of sampling points. Represents visual feature maps. Represents the sampling function. This indicates that the sampled local features are fed into the lightweight Transformer encoder for fusion.

9. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S600, the loss function expression for the loss-weighted joint training of the detection branch and the retrieval branch is: ， ， ， in, Weighting coefficients for detecting loss; This represents the loss function for the detection branch. Indicates SmoothL1 loss. Indicates GIOU loss; This represents the loss function for the retrieval branch. This represents the similarity distribution matching loss. This indicates the loss in identity classification.

10. The integrated cross-modal pedestrian retrieval method with parallel detection and retrieval tasks as described in claim 1, characterized in that, In step S600, during the joint training phase of the integrated cross-modal pedestrian retrieval model algorithm, the outputs of encoders from layer 1 to layer 6 are all incorporated into the retrieval branch for supervision; during the inference phase of the integrated cross-modal pedestrian retrieval model, only layer 6 of the encoder is used for output.

Images