Modality adaptive descriptive query pedestrian re-identification method and system

Abstract

The application discloses a modal self-adaptive descriptive query pedestrian re-identification method and system, and the method comprises the following steps: building a pedestrian re-identification architecture network model of multi-modal feature learning; optimizing a feature extractor, obtaining a pedestrian re-identification architecture network model of the feature extractor after optimization; inputting a training set into the pedestrian re-identification architecture network model of the feature extractor after optimization to perform a retrieval task training, updating network parameters, and obtaining a pedestrian re-identification architecture network model after parameter updating; inputting a to-be-identified modal sample into the pedestrian re-identification architecture network model after parameter updating to perform modal recognition, and obtaining a target pedestrian. The modal self-adaptive descriptive pedestrian re-identification method provided by the application can adaptively process various descriptive modalities, adapt to the modal uncertainty problem in an actual scene, and widen the application scene of the pedestrian re-identification model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition and retrieval, in particular to a modal adaptive sketch-based query person re-identification method and system. BACKGROUND

[0002] Person Re-identification (ReID) aims to realize the identification and retrieval of target pedestrians in a cross-camera scene. This technology is widely used in intelligent video monitoring, intelligent security and other technical fields. Existing researches are mainly based on picture query of person re-identification, that is, using pedestrian photos / videos to retrieve target pedestrian photos / videos under different cameras. However, in many criminal cases, due to the inability to obtain the photos of suspects in time, case investigators obtain information about suspects by collecting eyewitnesses' text descriptions of suspects or drawing sketches of pedestrians (sketch-based modal) according to the descriptions. Therefore, some researchers propose to use the text descriptions of eyewitnesses, sketches of pedestrians or the combination of text descriptions and sketches to retrieve target pedestrian images in a monitoring environment, that is, sketch-based query person re-identification.

[0003] At present, most text-based or sketch-based person re-identification methods rely on one of the data modalities as a query set to complete the retrieval task of pedestrian images. Although text modal data is relatively easy to obtain, it cannot accurately describe the details of visual appearance. Since the training tasks of each modality are independent, the model cannot be generalized from one modality to another unknown modality. For example, a ReID model trained on a text-based dataset is basically ineffective in a sketch-based scenario, and vice versa. Due to the difference in modality types, the generalization ability of the model is poor, which limits the applicability of the model in actual deployment. On the other hand, Zhai et al. proposed a multi-modal fusion person re-identification, that is, using sketches and texts as queries to retrieve target pedestrians, which effectively improves the retrieval accuracy of the model. However, this method uses independent pre-training parameters of text and image for multi-modal representation learning, which has poor universality. In addition, in actual monitoring scenarios, it is difficult to determine whether text or sketch is available, that is, there is a modal missing problem. Due to the independent training of the task, the existing cross-modal or multi-modal ReID methods cannot well solve this problem. An intelligent monitoring system should be able to effectively handle various modal information. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the above background art, and provide a modal adaptive sketch-based query person re-identification method and system.

[0005] In a first aspect, the application provides a modal-adaptive sketchy query pedestrian re-identification method, comprising the following steps:

[0006] S1, a pedestrian re-identification architecture network model for multi-modal feature learning is built, the pedestrian re-identification architecture network model comprising an encoder adopting a ViT network as a feature extractor for sketch and RGB image modalities, and a Transformer network as a feature extractor for a text modality;

[0007] S2, the feature extractors are jointly optimized by using a task-specific loss, a cross-modal interaction loss, and a task-aware dynamic training method to obtain a pedestrian re-identification architecture network model after optimization of the feature extractors;

[0008] S3, a multi-modal dataset is obtained, the multi-modal dataset is divided into a training set and a test set, the training set is input into the pedestrian re-identification architecture network model after optimization of the feature extractors for retrieval task training, and network parameters are updated to obtain a pedestrian re-identification architecture network model after parameter update;

[0009] S4, a to-be-identified modal sample is input into the pedestrian re-identification architecture network model after parameter update for modal recognition to obtain a target pedestrian.

[0010] According to the first aspect, in a first possible implementation manner of the first aspect, the S2, the feature extractors are jointly optimized by using a task-specific loss, a cross-modal interaction loss, and a task-aware dynamic training method to obtain a pedestrian re-identification architecture network model after optimization of the feature extractors, specifically comprising the following steps:

[0011] S21, a task-specific modal learning method is used to obtain a task-specific modal learning loss function;

[0012] S22, each type of retrieval training task is given a modal learning loss function with a dynamically changing weight to obtain a task-specific modal learning loss function after increase of a modulation factor;

[0013] S23, the feature extractors are optimized by using the obtained task-specific modal learning loss function after increase of the modulation factor to obtain a pedestrian re-identification architecture network model after optimization of the feature extractors.

[0014] According to the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the S21, the task-specific modal learning method is used to obtain a task-specific modal learning loss function, specifically comprising the following steps:

[0015] a first retrieval loss function for a sample pair and a second retrieval loss function for a sample pair are obtained;

[0016] According to the first retrieval loss function and the second retrieval loss function of the obtained sample pair, a task-specific modal learning loss function is obtained.

[0017] According to the first possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the task-specific modal learning loss function of each retrieval training task is given a dynamically changing weight, and the step of obtaining the task-specific modal learning loss function with the added modulation factor specifically includes the following steps:

[0018] The prediction confidence of the sketch retrieval RGB (R: Red; G: Green; B: Blue, English full name and Chinese interpretation of RGB) task and the prediction confidence of the text retrieval RGB task are obtained.

[0019] According to the prediction confidence of the sketch retrieval RGB task and the prediction confidence of the text retrieval RGB task, a modulation factor is obtained.

[0020] According to the obtained modulation factor, a task-specific modal learning loss function with the added modulation factor is obtained.

[0021] According to the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the step of obtaining the prediction confidence of the sketch retrieval RGB task and the prediction confidence of the text retrieval RGB task specifically includes the following steps:

[0022] According to the modal loss of a specific task, the prediction confidence of the sketch retrieval RGB task p SR (i) and the prediction confidence of the text retrieval RGB task p TR (i) is as follows:

[0023]

[0024]

[0025] According to the third possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the step of obtaining the modulation factor according to the prediction confidence of the sketch retrieval RGB task p SR (i) and the prediction confidence of the text retrieval RGB task p TR (i) specifically includes the following steps:

[0026] The confidence harmonic mean of the sketch retrieval RGB task and the text retrieval RGB task is obtained.

[0027] The obtained confidence harmonic mean is multiplied by the prediction confidence of the text retrieval RGB task to obtain the modulation factor.

[0028] In a sixth possible implementation manner of the first aspect, according to the third possible implementation manner of the first aspect, the step of obtaining the task-specific modal learning loss function with the increased modulation factor according to the obtained modulation factor specifically comprises the following steps:

[0029] The obtained modulation factor is subjected to numerical change according to the modulation factor, and the task-specific modal learning loss function with the increased modulation factor is obtained:

[0030]

[0031]

[0032] wherein, α t and γ are a first hyperparameter and a second hyperparameter for controlling the downward trend of the loss curve, is a modal loss function of the sketch retrieval RGB task, is a modal loss function of the text retrieval RGB task.

[0033] In a seventh possible implementation manner of the first aspect, according to the third possible implementation manner of the first aspect, after the step of obtaining the multi-modal data set, dividing the multi-modal data set into a training set and a test set, inputting the training set into the pedestrian re-identification architecture network model optimized by the feature extractor to perform retrieval task training, and updating the network parameters, and obtaining the pedestrian re-identification architecture network model with updated parameters, the following step is further included:

[0034] The test set is input into the pedestrian re-identification architecture network model with updated parameters, and an evaluation index of the pedestrian re-identification architecture network model is obtained.

[0035] In a second aspect, the present application provides a modal self-adaptive sketch query pedestrian re-identification system, comprising:

[0036] An architecture network building module is configured to build a pedestrian re-identification architecture network model for multi-modal feature learning, wherein the pedestrian re-identification architecture network model comprises an encoder adopting a ViT network as a feature extractor for a sketch modal and an RGB image modal, and a Transformer network as a feature extractor for a text modal;

[0037] An optimization module is in communication connection with the architecture network building module and is configured to jointly optimize the feature extractors by using a task-specific loss, a cross-modal interaction loss and a task-aware dynamic training method, and obtain a pedestrian re-identification architecture network model with optimized feature extractors;

[0038] The updating module is connected with the optimization module in communication, configured to obtain a multi-modal data set, divide the multi-modal data set into a training set and a test set, input the training set into the pedestrian re-identification architecture network model optimized by the feature extractor to perform retrieval task training, and update network parameters to obtain the pedestrian re-identification architecture network model with updated parameters.

[0039] The identification module is connected with the optimization module in communication, configured to input the obtained to-be-identified modal sample into the pedestrian re-identification architecture network model with updated parameters to perform modal identification, and obtain a target pedestrian.

[0040] According to a second aspect, in a first possible implementation manner of the second aspect, the optimization module comprises:

[0041] The loss function obtaining unit is configured to obtain a task-specific modal learning loss function by using a task-specific modal learning method.

[0042] The modulation factor obtaining unit is connected with the loss function obtaining unit in communication, configured to assign a dynamically changing weight to the modal learning loss function of each type of retrieval training task, and obtain the task-specific modal learning loss function with the added modulation factor.

[0043] The optimization unit is connected with the modulation factor obtaining unit in communication, configured to optimize the feature extractor by using the task-specific modal learning loss function with the added modulation factor, and obtain the pedestrian re-identification architecture network model optimized by the feature extractor.

[0044] Compared with the prior art, the present application has the following advantages:

[0045] The modal adaptive descriptive query pedestrian re-identification method provided by the present application introduces a general pedestrian re-identification architecture network model, jointly considers cross-modal and multi-modal tasks, and thus adaptively processes multiple descriptive modalities, adapts to the modal uncertainty problem in actual scenes, and widens the application scenarios of the pedestrian re-identification model. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is a method flowchart of the modal adaptive descriptive query pedestrian re-identification method provided by the present application;

[0047] Figure 2 is a schematic diagram of a pedestrian re-identification architecture network model for a descriptive query in the prior art;

[0048] Figure 3 is a schematic diagram of a modal adaptive descriptive pedestrian re-identification architecture network model provided by the present application;

[0049] Figure 4is a functional module block diagram of a modal adaptive descriptive query pedestrian re-identification system provided by the present application. DETAILED DESCRIPTION

[0050] Reference will now be made in detail to the specific embodiments of the present application, examples of which are illustrated in the accompanying drawings. While the present application will be described in conjunction with the specific embodiments, it will be understood that the present application is not intended to be limited to the described embodiments. On the contrary, the present application is intended to cover modifications, alterations and equivalents, which are included within the spirit and scope of the present application, as defined by the appended claims. It should be noted that the method steps described herein can all be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0051] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0052] Note: The examples to be introduced next are only one specific example, and are not as a limitation on the embodiments of the present application must be as follows: specific steps, values, conditions, data, order, etc. Those skilled in the art can use the concept of the present application to construct more embodiments not mentioned in the present specification by reading the present specification.

[0053] The modal adaptive descriptive query pedestrian re-identification method provided by the present application solves the technical problem that the existing pedestrian re-identification method cannot effectively solve the cross-modal or multi-modal retrieval and recognition task.

[0054] In a first aspect, with reference to Figure 1 The present application provides a modal adaptive descriptive query pedestrian re-identification method, comprising the following steps:

[0055] S1, a multi-modal feature learning pedestrian re-identification architecture network model is built, the pedestrian re-identification architecture network model comprises an encoder adopting a ViT network as a sketch modality and an RGB image modality feature extractor, and a Transformer network as a text modality feature extractor, the high-efficiency global level modeling capability of the ViT network architecture is used to mine the global features of the sketch and photo modalities, and the dual-encoder Transformer network architecture is used to mine the text features of the text modality;

[0056] The key to realizing the modal adaptive general pedestrian re-identification is to establish a shared multi-modal feature extractor for multi-modal data. Given a query modal sample, the multi-modal feature extractor can adaptively learn the modal invariant information of the modal to retrieve the target pedestrian. Generally, the RGB, sketch or text description of the pedestrian can be classified as visual or text modal. Therefore, the general pedestrian re-identification architecture mainly models the relationship between visual and text modal and the relationship within the modal. In recent years, in the field of visual language pre-training research, a large number of researchers have explored the semantic correspondence relationship between image and text modal. Among them, the CLIP model uses text as a supervision signal to learn the matching relationship between text and image, thereby forming a transferable model.

[0057] Inspired by the CLIP model, the present application proposes a dual-encoder pedestrian re-identification architecture network model for multi-modal feature learning. For the visual input modal V (RGB is V r , and sketch is V s ), it is first divided into small blocks of equal size, mapped to a fixed-dimensional vector, and the image sequence vector V = {[IMG], v1, v2, …, v m} is obtained. After being combined with the position vector, the image sequence vector is input into the visual transformer to learn the visual context features. Among them, V[IMG] represents the feature representation of the global image. Similarly, for the text description T, it is encoded into a word sequence T = {[SOS], t1, t2, …, t n , [CLS]} using the byte pair encoding method. A text transformer encoder is used to mine the context relationship of the text. T[SOS] and T[CLv] are used to represent the beginning and end of the text sequence feature, respectively. Finally, the feature blocks V r [IMG], V s [IMG] and T[CLS] are used as modal features for cross-modal matching. The final fusion feature block is F[CLS] = V s [IMG] + T[CLS].

[0058] S2, a task-specific loss, a cross-modal interaction loss and a task-aware dynamic training method are used to jointly optimize the feature extractor, and an optimized pedestrian re-identification architecture network model of the feature extractor is obtained.

[0059] S3, a multi-modal data set is obtained, the multi-modal data set is divided into a training set and a test set, the training set is input into the optimized pedestrian re-identification architecture network model of the feature extractor for retrieval task training, and the network parameters of the pedestrian re-identification architecture network model are updated to obtain a parameter updated pedestrian re-identification architecture network model.

[0060] S4, input the obtained to-be-identified modal sample into the updated pedestrian re-identification architecture network model for modal identification, and obtain a target pedestrian.

[0061] The modal-adaptive descriptive query pedestrian re-identification method provided in the application introduces a general pedestrian re-identification architecture network model, jointly considers cross-modal and multi-modal tasks, and balances the retrieval training of multiple retrieval tasks through a task-aware dynamic training method, thereby adaptively processing multiple descriptive modalities, adapting to the modal uncertainty problem in actual scenes, and widening the application scenarios of the pedestrian re-identification architecture network model.

[0062] In an embodiment, in order to implement the retrieval task of different query modalities, it is crucial to mine the modal-invariant features between data. The application provides a task-specific modal learning scheme to independently optimize the distance between different query modal sample and gallery sample. The step of obtaining the pedestrian re-identification architecture network model after optimization of the feature extractor by jointly optimizing the feature extractor through the task-specific loss, the cross-modal interaction loss and the task-aware dynamic training method specifically comprises the following steps:

[0063] S21, a task-specific modal learning method is used to obtain a task-specific modal learning loss function, so as to mine the discriminative feature representation of the visual and text modal processes and implement the retrieval task of cross-modal and multi-modal query modalities.

[0064] S22, the modal learning loss function of each type of retrieval training task is given a dynamically changing weight, a task-specific modal learning loss function with an added modulation factor is obtained, so as to solve the problem that the general ReID also faces the additional challenge of possible overfitting of a single task due to the different difficulty levels of different retrieval training, and the retrieval training method of fixed loss weight addition cannot realize the generalization of different cross-modal retrieval tasks, resulting in insufficient robustness of the model. The task-aware dynamic training method balances the retrieval training of multiple tasks. The principle of the task-aware dynamic training method is to adjust the loss contribution of different retrieval tasks according to the training difficulty of the task, i.e., the predicted confidence. The dynamic training strategy improves the model generalization ability by tending to train difficult tasks. Through joint exploration of multi-modal data learning, mutual enhancement between different retrieval tasks is realized, and the robustness of the model to different query modalities is improved. Specifically, when a cross-modal retrieval task can retrieve the corresponding image with a high prediction confidence, the dynamic training method increases the loss contribution of the remaining cross-modal retrieval tasks through a modulation factor.

[0065] S23, the feature extractor is optimized through the task-specific modal learning loss function with the added modulation factor, and a pedestrian re-identification architecture network model after optimization of the feature extractor is obtained.

[0066] In an embodiment, the S21, the task-specific modal learning loss function is obtained by using a task-specific modal learning method, and specifically includes the following steps:

[0067] The S211, M sample pairs (query, gallery) are obtained by sampling, and the sample pair retrieval loss function is as follows:

[0068]

[0069]

[0070] Wherein, q represents a query modal vector, g represents a gallery sample modal feature, <q i , g i > represents the cosine similarity between q i and g i , τ represents a temperature coefficient, is a first sample pair retrieval loss function, is a second sample pair retrieval loss function.

[0071] S22, the basic principle of the pedestrian re-identification architecture network model of the application is to jointly consider the cross-modal and multi-modal fusion tasks, mine the discriminative feature representation shared by visual and text modalities, and mainly involves three tasks of sketch retrieval image task, text retrieval image task and multi-modal fusion (sketch + text) retrieval image task. Therefore, according to the obtained first sample pair retrieval loss function and second sample pair retrieval loss function, the task-specific modal learning loss function is as follows:

[0072]

[0073]

[0074] Wherein, is a modal learning loss function of the sketch retrieval image task, is a modal learning loss function of the text retrieval image task, is a modal learning loss function of the multi-modal fusion retrieval image task.

[0075] In an embodiment, the S22, the modal learning loss function of each type of retrieval training task is given a dynamically changing weight, and the task-specific modal learning loss function after adding a modulation factor is obtained, and specifically includes the following steps:

[0076] S221, the prediction confidence of the sketch retrieval RGB task and the prediction confidence of the text retrieval RGB task are obtained.

[0077] S222, acquire a modulation factor according to the prediction confidence of sketch retrieval RGB task and the prediction confidence of text retrieval RGB task;

[0078] S223, acquire a task-specific modal learning loss function with the modulation factor added according to the acquired modulation factor, so as to realize that when one cross-modal retrieval task can retrieve the corresponding image with high prediction confidence, the dynamic training strategy will increase the loss contribution of the remaining cross-modal retrieval task through the modulation factor.

[0079] In an embodiment, S221, the step of acquiring the prediction confidence of sketch retrieval RGB task and the prediction confidence of text retrieval RGB task specifically includes the following steps:

[0080] The cosine similarity after the softmax function between the query and the gallery sample in formula (1) determines the prediction confidence of the real match, and the prediction confidence of sketch retrieval RGB task p SR (i) and the prediction confidence of text retrieval RGB task p TR (i) is as follows:

[0081]

[0082]

[0083] In an embodiment, the step of acquiring the modulation factor according to the prediction confidence of sketch retrieval RGB task p SR (i) and the prediction confidence of text retrieval RGB task p TR (i) specifically includes the following steps:

[0084] Acquire the harmonic mean of the confidence of sketch retrieval RGB task and the confidence of text retrieval RGB task;

[0085] Multiply the acquired harmonic mean of the confidence by the prediction confidence of text retrieval RGB task to acquire the modulation factor as follows:

[0086]

[0087]

[0088] In the formula, w SR represents the modulation factor of sketch retrieval RGB task, w TR (i) is the modulation factor of text retrieval RGB task.

[0089] In an embodiment, the step of acquiring a task-specific modal learning loss function with the modulation factor added according to the acquired modulation factor specifically includes the following steps:

[0090] The obtained modulation factor is subjected to numerical change according to the user, and a task-specific modal learning loss function after the modulation factor is increased is obtained:

[0091]

[0092]

[0093] wherein a t and γ are a first hyperparameter and a second hyperparameter for controlling the downward trend of the loss curve, is a modal loss function of the sketch retrieval RGB task, is a modal loss function of the text retrieval RGB task, w SR represents a modulation factor of the sketch retrieval RGB task, w TR represents a modulation factor of the text retrieval RGB task.

[0094] In an embodiment, the step of obtaining a multi-modal data set in S3 specifically comprises the following steps:

[0095] In order to facilitate the research of a general pedestrian re-identification architecture, the present application collects the sketch modal of each identity and the text-based ReID data set (CUHK-PEDES, ICFGPEDES and RSTPReid) to construct a multi-modal data set (Tri-CUHK-PEDES, Tri-ICFGPEDES and Tri-RSTPReid).

[0096] In an embodiment, the step of obtaining the sketch modal information of the corresponding pedestrian specifically comprises the following steps:

[0097] 1. Background erasing, first erase the background of the RGB image through the Aliyun API to obtain the RGB image after removing the background, so as to reduce the influence of background noise. This API returns the foreground pedestrian by identifying the human contour in the input image and separating it from the background.

[0098] 2. Sketch synthesis, on the basis of the RGB image after removing the background, the API of Meitu Company is applied to generate a sketch in the style of hand-drawn sketch of the pedestrian, so as to obtain the sketch modal information of the corresponding pedestrian.

[0099] In an embodiment, after the step of obtaining a multi-modal data set, dividing the multi-modal data set into a training set and a test set, inputting the training set into the pedestrian re-identification architecture network model optimized by the feature extractor to perform retrieval task training, and updating the network parameters, the pedestrian re-identification architecture network model after parameter updating is obtained, the step further comprises the following steps:

[0100] The test set input parameter is updated to the pedestrian re-identification architecture network model, and the evaluation index Rank-k matching precision and average retrieval precision mAP of the pedestrian re-identification architecture network model are obtained, which respectively reflect the correct cross-modal image retrieval probability and the average correct rate.

[0101] In a more specific embodiment, the present application uses the ViT network to extract the features of the visual mode, the Transformer network to extract the features of the text mode, and the pre-training parameters obtained from the CLIP to initialize the network parameters. The batch size is 192, which contains 64 identities, each containing a photo, a sketch and a text. In the training stage, the present application uses random cropping, padding and horizontal flipping as data augmentation. And using the adaptive moment estimation (Adam) optimizer to train the network for retrieval task. In addition, the initial learning rate is set to 1e-5, and the cosine learning rate decay is used. The temperature coefficient τ is set to 0.07.

[0102] In order to verify the effectiveness of the present application, the retrieval results of the present application are compared with the existing text-photo recognition methods, and the existing sketch-photo recognition methods mainly include:

[0103] [1] Zhang, Ying, and Huchuan Lu. "Deep cross-modal projection learning for image-text matching." Proceedings of the European conference on computer vision (ECCV). 2018.

[0104] [2] Sarafianos, Nikolaos, Xiang Xu, and Ioannis A. Kakadiaris. "Adversarial representation learning for text-to-image matching." Proceedings of the IEEE / CVF international conference on computer vision. 2019.

[0105] [3] Jing, Ya, et al. "Pose-guided joint global and attentive local matching network for text-based person search." Association for the Advance of Artificial Intelligence (AAAI) (2020).

[0106] [4] Lee, Kuang-Huei, et al. "Stacked cross attention for image-text matching." Proceedings of the European conference on computer vision (ECCV). 2018.

[0107] [5] Wang, Zijie, et al. "Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifold." Proceedings of the 30th ACM International Conference on Multimedia. 2022.

[0108] [6] Wang, Zijie, et al. "Capturing All-round Information Beyond Color for Text-based Person Retrieval." Proceedings of the 30th ACM International Conference on Multimedia. 2022.

[0109] The test results on the Tri-CUHK-PEDES dataset are shown in Table 1:

[0110] Table 1

[0111] Method Source Rank-1 Rank-5 Rank-10 Document [1] ECCV 18 49.37 - 79.27 Document [2] ICCV 19 54.51 77.56 84.78 Document [3] AAAI 20 54.12 77.56 84.78 Reference method of the present application - 67.13 84.60 90.37 Method of the present application - 68.71 85.35 90.84

[0112] The test results on the Tri-ICFG-PEDES dataset are shown in Table 2:

[0113] Table 2

[0114] Method Source Rank-1 Rank-5 Rank-10 Document [1] ECCV 18 43.51 65.44 74.26 Document [4] ECCV 18 50.05 69.65 77.21 Reference method of the present application - 58.36 75.97 82.32 Method of the present application - 61.28 77.40 83.16

[0115] The test was performed on the Tri-ICFG-PEDES dataset, and the results are shown in Table 3:

[0116] Table 3

[0117] Method Source Rank-1 Rank-5 Rank-10 Document [5] MM 22 45.55 68.20 77.85 Document [6] MM 22 47.35 69.55 79.00 Reference method of the present application - 57.30 78.05 86.10 Method of the present application - 60.25 79.85 87.10

[0118] As can be seen from Table 1, Table 2 and Table 3:

[0119] Compared with other network models, the modal adaptive descriptive query pedestrian re-identification method provided in the application effectively improves the retrieval accuracy of the model under the condition that the test setting remains unchanged.

[0120] The visual and text modal feature learning double encoder provided in the application is more effective than most convolutional neural network-based methods on three datasets.

[0121] The method provided in the application unifies the cross-modal retrieval task and the multi-modal retrieval task in one framework, improving the model performance and practicality.

[0122] After applying the modal fusion method, the retrieval accuracy on three datasets is significantly improved, proving the effectiveness of the modal adaptive descriptive query pedestrian re-identification method provided in the application for cross-modal and multi-modal retrieval.

[0123] To verify the effect of multi-modal fusion, the application based on the Tri-CUHK-PEDES dataset tests three types of retrieval tasks: text retrieval RGB (T->R), sketch retrieval RGB (S->R) and multi-modal fusion retrieval RGB (T+S->R), and the experimental results are shown in Table 4.

[0124] Table 4

[0125]

[0126] As can be seen from Table 4, multi-modal fusion retrieval can effectively improve the model performance, and the modal adaptive descriptive query pedestrian re-identification method provided in the application effectively improves the generalization of the model for the three types of retrieval tasks, achieving adaptive processing of multiple modalities.

[0127] In a second aspect, based on the same inventive concept, the present application provides a modal adaptive sketch query pedestrian re-identification system, comprising an architecture network building module 100, an optimization module 200, an updating module 300 and an identification module 400. The architecture network building module 100 is configured to build a pedestrian re-identification architecture network model for multi-modal feature learning. The pedestrian re-identification architecture network model comprises an encoder adopting a ViT network as a feature extractor for sketch and RGB image modalities, and a feature extractor for text modality adopting a Transformer network. The optimization module 200 is in communication connection with the architecture network building module and is configured to jointly optimize the feature extractors by using a task-specific loss, a cross-modal interaction loss and a task-aware dynamic training method, so as to obtain a pedestrian re-identification architecture network model after optimization of the feature extractors. The updating module 300 is in communication connection with the optimization module and is configured to obtain a multi-modal data set, divide the multi-modal data set into a training set and a test set, input the training set into the pedestrian re-identification architecture network model after optimization of the feature extractors for retrieval task training, update network parameters, and obtain a pedestrian re-identification architecture network model after parameter updating. The identification module 400 is in communication connection with the optimization module and is configured to input a to-be-identified modal sample into the pedestrian re-identification architecture network model after parameter updating for modal identification, and obtain a target pedestrian.

[0128] In an embodiment, the optimization module further comprises a loss function obtaining unit, a modulation factor obtaining unit and an optimization unit. The loss function obtaining unit is configured to obtain a task-specific modal learning loss function by using a task-specific modal learning method. The modulation factor obtaining unit is in communication connection with the loss function obtaining unit and is configured to assign a dynamically changing weight to the modal learning loss function of each type of retrieval training task, and obtain a task-specific modal learning loss function after adding a modulation factor. The optimization unit is in communication connection with the modulation factor obtaining unit and is configured to optimize the feature extractors by using the task-specific modal learning loss function after adding the modulation factor, and obtain a pedestrian re-identification architecture network model after optimization of the feature extractors.

[0129] Based on the same inventive concept, the present application also provides a computer readable storage medium having a computer program stored thereon. The computer program is executed by a processor to implement all or part of the method steps of the above method.

[0130] The application implements all or part of the processes in the above method, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in a jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include an electrical carrier signal and a telecommunication signal.

[0131] Based on the same inventive concept, the embodiments of the application also provide an electronic device, including a memory and a processor, the memory stores a computer program running on the processor, and the processor implements all method steps or part of the method steps in the above method when executing the computer program.

[0132] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0133] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program (such as a sound playing function, an image playing function, etc.) required by a function; and the data storage area can store data (such as audio data, video data, etc.) created according to the use of the mobile phone. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0134] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, a server or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program codes.

[0135] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), servers and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0136] These computer program instructions can also be stored in a computer readable memory capable of guiding a computer or other programmable data processing apparatus to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks. Figure 1 The functions specified in one or more flows and / or blocks.

[0137] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or steps of the functions specified in the flowchart

[0138] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A modality-adaptive descriptive query pedestrian re-identification method, characterized in that, Includes the following steps: A multimodal feature learning-based pedestrian re-identification architecture network model is constructed. The pedestrian re-identification architecture network model includes an encoder using a ViT network as a feature extractor for sketch mode and RGB image mode, and a Transformer network as a feature extractor for text mode. We employ a task-specific loss, cross-modal interaction loss, and task-aware dynamic training method to jointly optimize the feature extractor and obtain a pedestrian re-identification architecture network model with optimized feature extractor. Obtain a multimodal dataset, divide the multimodal dataset into a training set and a test set, input the training set into the optimized pedestrian re-identification architecture network model of the feature extractor for retrieval task training, update the network parameters, and obtain the pedestrian re-identification architecture network model with updated parameters. The pedestrian re-identification architecture network model with updated input parameters of the modal sample to be identified is used for modal identification to obtain the target pedestrian; The step of jointly optimizing the feature extractor using a dynamic training method that employs task-specific loss, cross-modal interaction loss, and task awareness to obtain the optimized pedestrian re-identification architecture network model specifically includes the following steps: A task-specific modality learning method is used to obtain a task-specific modality learning loss function; Assign dynamically changing weights to the modality learning loss function for each type of retrieval training task to obtain the task-specific modality learning loss function after adding a modulation factor; The feature extractor is optimized by obtaining the task-specific modality learning loss function with the modulation factor added, and the optimized pedestrian re-identification architecture network model is obtained. The step of assigning dynamically changing weights to the modality learning loss function for each type of retrieval training task, and obtaining the task-specific modality learning loss function after adding the modulation factor, specifically includes the following steps: Obtain the prediction confidence scores for the RGB task of sketch retrieval and the RGB task of text retrieval; The modulation factor is obtained based on the prediction confidence scores of the RGB task for sketch retrieval and the RGB task for text retrieval. Based on the obtained modulation factor, obtain the task-specific modality learning loss function after adding the modulation factor.

2. The modality-adaptive descriptive query pedestrian re-identification method as described in claim 1, characterized in that, The step of obtaining the task-specific modality learning loss function using a task-specific modality learning method specifically includes the following steps: Obtain the first retrieval loss function and the second retrieval loss function for the sample pairs; Based on the obtained sample pairs, the first retrieval loss function and the second retrieval loss function, the task-specific modality learning loss function is obtained.

3. The modality-adaptive descriptive query pedestrian re-identification method as described in claim 1, characterized in that, The steps for obtaining the prediction confidence scores for the sketch retrieval RGB task and the text retrieval RGB task specifically include the following steps: Based on the modal loss for a specific task, the confidence level of the RGB task in sketch retrieval is predicted. Prediction confidence for RGB text retrieval task The formula is shown below:

4. The modality-adaptive descriptive query pedestrian re-identification method as described in claim 1, characterized in that, The prediction confidence level based on the obtained sketch retrieval RGB task is... Prediction confidence for RGB text retrieval task The steps to obtain the modulation factor specifically include the following: Obtain the harmonic mean confidence score for both the sketch retrieval RGB task and the text retrieval RGB task; The modulation factor is obtained by multiplying the harmonic mean of the obtained confidence scores by the predicted confidence scores of the RGB text retrieval task.

5. The modality-adaptive descriptive query pedestrian re-identification method as described in claim 1, characterized in that, The step of obtaining the task-specific modality learning loss function after adding the modulation factor based on the obtained modulation factor specifically includes the following steps: The obtained modulation factor is numerically modified according to the player's skill level, and the task-specific modality learning loss function after adding the modulation factor is obtained: in, and The first and second hyperparameters are used to control the downward trend of the loss curve. Let be the modal loss function for the RGB retrieval task in sketching. This is the modal loss function for the RGB text retrieval task.

6. The modality-adaptive descriptive query pedestrian re-identification method as described in claim 1, characterized in that, The steps of obtaining the multimodal dataset, dividing the multimodal dataset into training and testing sets, inputting the training set into the optimized person re-identification architecture network model via the feature extractor for retrieval task training, updating the network parameters, and obtaining the parameter-updated person re-identification architecture network model further include the following steps: The pedestrian re-identification architecture network model is updated with the test set input parameters, and the evaluation metric of the pedestrian re-identification architecture network model is obtained.

7. A modality-adaptive descriptive query person re-identification system, used to implement the modality-adaptive descriptive query person re-identification method as described in claim 1, characterized in that, include: The architecture network building module is used to build a multimodal feature learning pedestrian re-identification architecture network model. The pedestrian re-identification architecture network model includes an encoder using a ViT network as a feature extractor for sketch mode and RGB image mode, and a Transformer network as a feature extractor for text mode. The optimization module, which is communicatively connected to the architecture network building module, is used to jointly optimize the feature extractor using task-specific loss, cross-modal interaction loss, and task-aware dynamic training methods to obtain the pedestrian re-identification architecture network model after feature extractor optimization. The update module, which is connected to the optimization module, is used to acquire a multimodal dataset, divide the multimodal dataset into a training set and a test set, input the training set into the optimized pedestrian re-identification architecture network model of the feature extractor for retrieval task training, update the network parameters, and obtain the pedestrian re-identification architecture network model with updated parameters. The identification module, which is communicatively connected to the optimization module, is used to perform modal recognition on the pedestrian re-identification architecture network model updated with the input parameters of the acquired modal sample to be identified, and to obtain the target pedestrian.

8. The modality-adaptive descriptive query pedestrian re-identification system as described in claim 7, characterized in that, The optimization module includes The loss function acquisition unit is used to acquire a task-specific modality learning loss function using a task-specific modality learning method. The modulation factor acquisition unit is communicatively connected to the loss function acquisition unit and is used to assign dynamically changing weights to the modality learning loss function of each type of retrieval training task, and to acquire the task-specific modality learning loss function after the modulation factor has been added. The optimization unit is communicatively connected to the modulation factor acquisition unit and is used to optimize the feature extractor by acquiring the task-specific modality learning loss function with the modulation factor added, and to obtain the pedestrian re-identification architecture network model after the feature extractor is optimized.

Images