Pedestrian re-identification method based on multi-modal information supervision

Abstract

The invention discloses a pedestrian re-recognition method based on multi-modal information supervision, and relates to the technical field of image recognition, and the method comprises the steps: inputting any given two pedestrian images into a trained deep neural network model, outputting two corresponding pedestrian information feature vectors, calculating the similarity between the two pedestrian information feature vectors, and obtaining a pedestrian re-recognition result; pedestrian re-identification is realized; the deep neural network model comprises a pedestrian feature extraction network branch and an auxiliary training branch, the input of the pedestrian feature extraction network branch is any given two pedestrian images, and the auxiliary training branch only participates in the training process of the deep neural network model. The auxiliary training branch carries out loss function optimization supervision operation on pedestrian information feature vectors output by the pedestrian feature extraction network branch so as to train trainable parameters in the pedestrian feature extraction network branch; according to the pedestrian re-identification method, large-scale high-quality sample data can be obtained, and the accuracy and robustness of pedestrian re-identification are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a pedestrian re-identification method based on multimodal information supervision. Background Art

[0002] Person re-identification (ReID) is a computer vision task that aims to identify and match the same person from images taken by different cameras or at different times. ReID technology has been widely used in cross-camera multi-target tracking, smart security, video surveillance and other fields. Person re-identification faces multiple technical challenges. This process needs to overcome the appearance differences of pedestrians due to various changes in posture, perspective, lighting, occlusion, etc., so the robustness of the algorithm is very high.

[0003] In the past 20 years, pedestrian re-identification technology has been a research hotspot in the fields of machine vision and artificial intelligence, and many classic algorithms have emerged. The specific algorithms can be divided into the following categories:

[0004] (1) Person re-identification based on traditional technology. This type of algorithm mainly relies on manually designed feature extraction and metric learning for person re-identification. Commonly used algorithms include color distribution histogram comparison algorithm, texture distribution histogram comparison algorithm, perceptual hashing algorithm, etc. These methods have achieved certain results in person re-identification, but because they use the calculation method of the underlying features of the image, they are easily disturbed by factors such as posture, perspective, lighting, occlusion, etc., and have low robustness.

[0005] (2) Person re-identification based on deep learning technology. In recent years, with the development of deep learning technology, artificial intelligence technology has gradually been applied to the field of image similarity acquisition. With the help of deep learning technologies such as convolutional neural networks, attention neural networks, and transformer networks, significant difference features can be extracted from two local images of pedestrians, and the similarity of pedestrians in the two images can be directly output, which greatly improves the accuracy and stability of pedestrian re-identification. Although the method of pedestrian re-identification based on deep learning technology can achieve good results, the generalization ability of this technology is insufficient. The fundamental reason for this deficiency is that it is impossible to obtain a large amount of well-labeled sample data. Well-labeled sample data needs to consider the diversity of scenes, consistency of labels, and the long tail of data distribution. Based on the above reasons, the labeling cost of large-scale data is very high, which greatly limits the development of pedestrian re-identification technology based on deep learning technology.

[0006] Recently, the visual language big model has made significant progress. With its large parameters, the visual language big model can focus on data of both visual (such as images and videos) and language (such as text) modalities at the same time, and complete multiple complex tasks, such as image description generation, visual question answering, text generation image, etc. At present, the multi-model big model is very complex and requires a lot of computing resources, which is not suitable for practical use. Summary of the invention

[0007] Based on the technical problems existing in the background technology, the present invention proposes a pedestrian re-identification method based on multimodal information supervision. By using the analysis results generated by a large language model as a supervision signal to guide the training of the pedestrian feature extraction network branch, large-scale high-quality sample data can be obtained without human participation, greatly improving the accuracy and robustness of pedestrian re-identification.

[0008] The present invention proposes a pedestrian re-identification method based on multimodal information supervision. Two given pedestrian images are input into a trained deep neural network model, two corresponding pedestrian information feature vectors are output, and the similarity between the two pedestrian information feature vectors is calculated. When the similarity is greater than a set similarity threshold, it is determined that the pedestrians in the two pedestrian images are the same person.

[0009] The deep neural network model includes a pedestrian feature extraction network branch and an auxiliary training branch. The input of the pedestrian feature extraction network branch is any two given pedestrian images. The auxiliary training branch only participates in the training process of the deep neural network model. The auxiliary training branch performs a loss function optimization supervision operation on the pedestrian information feature vector output by the pedestrian feature extraction network branch to train the trainable parameters in the pedestrian feature extraction network branch.

[0010] Furthermore, the pedestrian feature extraction network branch includes a visual encoder imageencode, a maximum pooling layer maxpool, a convolutional layer conv0, and a feature size adjustment layer reshape0 connected in sequence;

[0011] Any two given pedestrian images are sent to the visual encoder image encode as input images, and the feature size adjustment layer reshape0 outputs the pedestrian information feature vectors corresponding to the two images.

[0012] Furthermore, the auxiliary training branch includes a text vectorization module word embedding, a feature size adjustment layer reshape1, a concatenation layer concat0, a concatenation layer concat1, a language large model module LLM and a fully connected layer fc;

[0013] The question prompt text is input into the text vectorization module word embedding, and the two feature vectors output by the visual encoder image encode are sent to the feature size adjustment layer reshape1.

[0014] The two feature vectors output by the feature size adjustment layer reshape1 and the text vector output by the text vectorization module wordembedding are sent to the concatenation layer concat1. The output of the concatenation layer concat1 is connected to the input of the language large model module LLM.

[0015] The two pedestrian information feature vectors output by the feature size adjustment layer reshape0 are input into the concatenation layer concat0, and the output of the concatenation layer concat0 passes through the fully connected layer fc. The fully connected layer fc outputs the prediction result of the classification task. Based on the prediction result of the classification task and the answer output by the language large model module LLM, a loss function layer is constructed to train the pedestrian feature extraction network branch.

[0016] Furthermore, the fully connected layer fc includes fully connected layers fc_1, ..., and fully connected layers fc_n, and each fully connected layer represents a prediction result of any classification task.

[0017] Furthermore, the number of loss function layers corresponds one-to-one to the number of fully connected layers, and a cross entropy loss function is set in the loss function layer.

[0018] Furthermore, the training process of the deep neural network model is as follows:

[0019] Collect multiple pedestrian images and preprocess them to construct a sample image training set; design a question prompt text set, which is used to describe the problem to be judged in the current input image pair, corresponding to the classification task represented by the fully connected layer fc;

[0020] Construct the target loss function of the deep neural network model based on cross entropy loss;

[0021] Select two pedestrian images from the image training set and select one question prompt text from the question prompt text set to form a training sample data pair, and input the training sample data pair into the deep neural network model;

[0022] Fine-tune the language model module LLM: ignore the maximum pooling layer maxpool, convolution layer conv0, feature size adjustment layer reshape0, feature size adjustment layer reshape1, concatenation layer concat0, concatenation layer concat1 and fully connected layer fc, freeze the text vectorization module word embedding, fine-tune the visual encoder image encode and the language model module LLM;

[0023] Training the pedestrian feature extraction network branch: freeze the text vectorization module word embedding, visual encoder image encode and language large model module LLM, and supervise the training of the maximum pooling layer maxpool, convolution layer conv0, feature size adjustment layer reshape0, feature size adjustment layer reshape1, splicing layer concat0, splicing layer concat1 and fully connected layer fc based on the answer output by the language large model module LLM in the auxiliary training branch.

[0024] A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned pedestrian re-identification method when executing the computer program;

[0025] A computer-readable storage medium is characterized in that a plurality of classification programs are stored on the computer-readable storage medium, and the plurality of classification programs are used to be called by a processor and execute the pedestrian re-identification method as described above.

[0026] The advantage of a pedestrian re-identification method based on multimodal information supervision provided by the present invention is that: the pedestrian re-identification method based on multimodal information supervision provided in the structure of the present invention,. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 It is a structural diagram of a deep neural network model;

[0029] The letters and numbers next to each module graphic represent the output feature map size of the current module, that is, the number of feature map channels × feature map width × feature map height. DETAILED DESCRIPTION

[0030] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0031] This embodiment uses a convolutional neural network (CNN) structure and a transformer structure to design a deep neural network model. In order to facilitate the description of this embodiment, some terms are defined: feature map resolution refers to feature map width × feature map height, feature map size refers to feature map channel number × feature map width × feature map height, kernel size refers to kernel width × kernel height, span refers to width direction span × height direction span, and each convolution layer is followed by a batch normalization layer and a nonlinear activation layer.

[0032] like Figure 1 and 2 As shown, the present invention proposes a pedestrian re-identification method based on multimodal information supervision, comprising the following steps:

[0033] S1. Design a deep neural network model. The deep neural network model is the core of artificial intelligence technology. Its main function is to extract key details and features of input pedestrian images. The network structure of the deep neural network model designed in this embodiment is as follows: Figure 2 As shown, the deep neural network model includes a pedestrian feature extraction network branch and an auxiliary training branch. The input of the pedestrian feature extraction network branch is any two given pedestrian images. The auxiliary training branch only participates in the training process of the deep neural network model. The auxiliary training branch performs loss function optimization supervision operation on the pedestrian information feature vector output by the pedestrian feature extraction network branch to train the trainable parameters in the pedestrian feature extraction network branch.

[0034] The pedestrian feature extraction network branch includes a visual encoder image encode, a maximum pooling layer maxpool, a convolutional layer conv0, and a feature size adjustment layer reshape0 connected in sequence; any two given pedestrian images are sent to the visual encoder image encode as input images, and the feature size adjustment layer reshape0 outputs the pedestrian information feature vectors corresponding to the two images.

[0035] The auxiliary training branch includes the text vectorization module word embedding, the feature size adjustment layer reshape1, the concatenation layer concat0, the concatenation layer concat1, the language large model module LLM and the fully connected layer fc; the question prompt text is input into the text vectorization module word embedding, the two feature vectors output by the visual encoder image encode are sent to the feature size adjustment layer reshape1, the two feature vectors output by the feature size adjustment layer reshape1 and the text vector output by the text vectorization module word embedding are all sent to the concatenation layer concat1, and the output of the concatenation layer concat1 is connected to the input of the language large model module LLM; the two pedestrian information feature vectors output by the feature size adjustment layer reshape0 are input into the concatenation layer concat0, and the output of the concatenation layer concat0 passes through the fully connected layer fc, the fully connected layer fc outputs the prediction result of the classification task, and the loss function layer is constructed based on the prediction result of the classification task and the answer output by the language large model module LLM to train the pedestrian feature extraction network branch.

[0036] Auxiliary training branch correspondence Figure 2 In the area shown by the dotted box, the auxiliary training branch only participates in training. Its main function is to understand the input visual features and question prompt text information features with the help of the powerful expression ability of the language large model module LLM, and generate the correct answer to the question. The answer serves as a supervision signal to guide the training of the pedestrian feature extraction network branch.

[0037] Specifically, person1 layer and person2 layer represent two 3-channel RGB images with the same resolution, representing two partial images of pedestrians. For the convenience of description, the resolution of the input image is 64x128 in this embodiment; text prompt represents the question prompt text, which is mainly used to describe the problem to be judged for the current input image. The description content of the question prompt text can be arbitrary, and it only needs to correspond to the specific classification tasks of the subsequent fc_1 layer, fc_2 layer, fc_3 layer, ..., fc_n-1 layer, fc_n layer and other network layers; the image encoder module is a visual encoder used to extract deep features of the input image. The module can be any commonly used network structure, such as the resnet network structure, the visual encoder module of the multimodal encoder CLIP (Learning Transferable Visual Models From Natural Language Supervision), and it only needs to ensure that its output feature map is a 16-fold downsampled feature map of the input feature map; vec1 and vec2 represent the input layer person1 layer and person2 layer after the visual encoder image The feature vector output after encoding; the wordembedding module is a text vectorization module for inputting question prompt text, which mainly converts the input text into a multi-dimensional feature vector of fixed length. This module can be any commonly used text vectorization module, such as the vectorization module used by the text encoder in CLIP. T represents the dimension or sequence length of the text vector; maxpool is a maximum value pooling layer with a kernel size of 2x2 and a span of 2x2; the conv0 layer is a convolutional layer with a kernel size of 3x3 and a span of 2x2; the reshape0 layer and the reshape1 layer are both feature size adjustment layers, which are used to adjust the size of the input feature map; F1 and F2 represent the pedestrian information feature vectors output after the input layers person1 and person2 pass through the pedestrian feature extraction network branch.

[0038] The concat0 layer and the concat1 layer are both concatenation layers, which are used to merge multiple input feature maps into an output feature map according to the channel dimension; the LLM module is a language large model module, which can be any commonly used large model structure, such as the Llama network, the Qwen network, etc.; Answer: Yes / No represents the answer to the question output by the LLM module; the fc_1 layer, the fc_2 layer, the fc_3 layer, ..., the fc_n-1 layer, and the fc_n layer are all fully connected layers, and each fully connected layer represents the predicted output of any classification task. For example, in this embodiment, the fc_1 layer represents whether the color and texture of the upper body clothes of the pedestrians in the two input images are the same, the fc_2 layer represents whether the color and texture of the lower body clothes of the pedestrians in the two input images are the same, the fc_3 layer represents whether the outlines of the pedestrians in the two input images are the same, the fc_n-1 layer represents whether the genders of the pedestrians in the two input images are the same, and the fc_n layer represents whether the pedestrians in the two input images are the same person. The loss1 layer, loss2 layer, loss3 layer, ..., lossn-1 layer, and lossn layer are all supervision function layers, which correspond one-to-one to the fc_1 layer, fc_2 layer, fc_3 layer, ..., fc_n-1 layer, and fc_n layer.

[0039] S2. Training the deep neural network model mainly involves optimizing the parameters of the deep neural network model through a large amount of training sample data of paired images with known answers, so that the deep neural network model can accurately obtain the pedestrian features in the input image. The specific steps are as follows:

[0040] S21. Obtain a sample image training set, which mainly collects pedestrian images in various scenes, various light sources, and various angles. Then, the pedestrian position is obtained through the existing human shape detection model. Then, the local image of the pedestrian is cropped, and finally the size of the local image of the pedestrian is normalized.

[0041] S22. Design a question prompt text set. The question prompt text in the question prompt text set is mainly used to describe the problem to be judged in the current input image pair. The description content of the question prompt text can be arbitrary, and it only needs to correspond to the specific task of the subsequent classification network. For example, in this embodiment, the question text input prompts used include: "Are the colors and textures of the upper body clothes of the pedestrians in the two input images the same?", "Are the colors and textures of the lower body clothes of the pedestrians in the two input images the same?", "Are the outlines of the pedestrians in the two input images the same?", "Are the genders of the pedestrians in the two input images the same?", "Are the pedestrians in the two input images the same person?".

[0042] S23. Design a target loss function for the deep neural network model. The loss function used in the deep neural network designed in this embodiment is a cross entropy loss function.

[0043] S24, training the deep neural network model, mainly by sending the pedestrian image pair and the corresponding question prompt text into the defined deep neural network model to learn the relevant model parameters. The specific training strategy is as follows:

[0044] S241, obtaining a training sample data pair, mainly by randomly selecting two pedestrian partial images from a training sample image set and selecting a question prompt text from predefined question prompt texts, to form a training sample data pair together.

[0045] S242, fine-tuning the language large model module LLM. The language large model cannot be trained from scratch because of its huge number of parameters. This embodiment adopts a fine-tuning training mode based on LORA (LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS) technology, and performs fine-tuning related to pedestrian attributes based on the existing language large model base to adapt to this embodiment. The specific fine-tuning method is to first ignore the maximum pooling layer maxpool, the convolution layer conv0, the feature size adjustment layer reshape0, the feature size adjustment layer reshape1, the splicing layer concat0, the splicing layer concat1, and the fully connected layer fc and other network layers, freeze the text vectorization module word embedding, and fine-tune the visual encoder image encode and the language large model module LLM.

[0046] S243. Train the pedestrian feature extraction network branch. First, the text vectorization module word embedding, the visual encoder image encode and the language model module LLM are used to supervise the training of the maximum pooling layer maxpool, the convolution layer conv0, the feature size adjustment layer reshape0, the feature size adjustment layer reshape1, the concatenation layer concat0, the concatenation layer concat1 and the fully connected layer fc based on the answer output by the language model module LLM in the auxiliary training branch. It should be noted that for each input training sample data pair, only the corresponding classification task layer is selected for optimization, and the optimization of irrelevant task layers is ignored.

[0047] S244, joint training, mainly alternately performs the training of step S242 and step S243. In this embodiment, the training steps are switched after every 10 algorithm iterations.

[0048] S3. Using a deep neural network model, for any two given pedestrian images, the corresponding pedestrian information feature vectors are obtained through the pedestrian feature extraction network branch respectively, and then the similarity between the two pedestrian information feature vectors is calculated based on a distance measurement benchmark, such as cosine similarity, Euclidean distance, etc., to achieve pedestrian identity re-identification. When the similarity is greater than the set similarity threshold, it is judged that the pedestrians in the two pedestrian images are the same person.

[0049] Through steps S1 to S3, this embodiment, with the help of the powerful expressive power of the large language model, comprehensively perceives and generates correct answers to questions based on visual features and question prompt text, and uses them as supervisory signals to train the pedestrian feature extraction network branch; with the help of auxiliary training branches, the model's perception of the detailed features of the input pedestrian image can be greatly improved without increasing the model prediction time, thereby improving the accuracy of the pedestrian similarity score; with the help of the training strategy of automatically generating supervisory signals with the help of the large language model, large-scale training sample data can be obtained without the need for manual annotation of data, thereby greatly improving the learning ability of the pedestrian feature extraction network; the entire network model has strong versatility, fast recognition speed, and strong robustness for pedestrian re-identification in various scenarios.

[0050] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A pedestrian re-identification method based on multimodal information supervision, characterized in that: Input any two pedestrian images into the trained deep neural network model, output two corresponding pedestrian information feature vectors, calculate the similarity between the two pedestrian information feature vectors, and when the similarity is greater than the set similarity threshold, it is judged that the pedestrians in the two pedestrian images are the same person; The deep neural network model includes a pedestrian feature extraction network branch and an auxiliary training branch. The input of the pedestrian feature extraction network branch is any two given pedestrian images. The auxiliary training branch only participates in the training process of the deep neural network model. The auxiliary training branch performs a loss function optimization supervision operation on the pedestrian information feature vector output by the pedestrian feature extraction network branch to train the trainable parameters in the pedestrian feature extraction network branch.

2. The pedestrian re-identification method based on multimodal information supervision according to claim 1 is characterized in that: The pedestrian feature extraction network branch includes a visual encoder image encode, a maximum pooling layer maxpool, a convolutional layer conv0 and a feature size adjustment layer reshape0 connected in sequence; Any two given pedestrian images are sent to the visual encoder image encode as input images, and the feature size adjustment layer reshape0 outputs the pedestrian information feature vectors corresponding to the two images.

3. The pedestrian re-identification method based on multimodal information supervision according to claim 2 is characterized in that: The auxiliary training branch includes a text vectorization module word embedding, a feature size adjustment layer reshape1, a concatenation layer concat0, a concatenation layer concat1, a language large model module LLM, and a fully connected layer fc; The question prompt text is input into the text vectorization module word embedding, and the two feature vectors output by the visual encoder image encode are sent to the feature size adjustment layer reshape1. The two feature vectors output by the feature size adjustment layer reshape1 and the text vector output by the text vectorization module word embedding are sent to the concatenation layer concat1. The output of the concatenation layer concat1 is connected to the input of the language large model module LLM. The two pedestrian information feature vectors output by the feature size adjustment layer reshape0 are input into the concatenation layer concat0, and the output of the concatenation layer concat0 passes through the fully connected layer fc. The fully connected layer fc outputs the prediction result of the classification task. Based on the prediction result of the classification task and the answer output by the language large model module LLM, a loss function layer is constructed to train the pedestrian feature extraction network branch.

4. The pedestrian re-identification method based on multimodal information supervision according to claim 3 is characterized in that: The fully connected layer fc includes fully connected layers fc_1, ..., and fully connected layers fc_n, and each fully connected layer represents the prediction result of any classification task.

5. The pedestrian re-identification method based on multimodal information supervision according to claim 4 is characterized in that: The number of loss function layers corresponds one to one to the number of fully connected layers, and the cross entropy loss function is set in the loss function layer.

6. The pedestrian re-identification method based on multimodal information supervision according to claim 3 is characterized in that: The training process of the deep neural network model is as follows: Collect multiple pedestrian images and preprocess them to construct a sample image training set; design a question prompt text set, which is used to describe the problem to be judged in the current input image pair, corresponding to the classification task represented by the fully connected layer fc; Construct the target loss function of the deep neural network model based on cross entropy loss; Select two pedestrian images from the image training set and select one question prompt text from the question prompt text set to form a training sample data pair, and input the training sample data pair into the deep neural network model; Fine-tune the language model module LLM: ignore the maximum pooling layer maxpool, convolution layer conv0, feature size adjustment layer reshape0, feature size adjustment layer reshape1, concatenation layer concat0, concatenation layer concat1 and fully connected layer fc, freeze the text vectorization module word embedding, fine-tune the visual encoder image encode and the language model module LLM; Training the pedestrian feature extraction network branch: freeze the text vectorization module word embedding, the visual encoder imageencode and the language large model module LLM, and supervise the training of the maximum pooling layer maxpool, the convolution layer conv0, the feature size adjustment layer reshape0, the feature size adjustment layer reshape1, the splicing layer concat0, the splicing layer concat1 and the fully connected layer fc based on the answer output by the language large model module LLM in the auxiliary training branch.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the pedestrian re-identification method according to any one of claims 1 to 6 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the pedestrian re-identification method according to any one of claims 1 to 6.

Images