Text and pedestrian video retrieval method based on cross-modal learning

Abstract

The application provides a text and pedestrian video retrieval method based on cross-modal learning, and the method comprises the following steps: carrying out gray processing and mask processing on a plurality of videos to be identified respectively, and carrying out filtering processing on the text to be identified; processing each video through a visual feature extraction network to obtain three groups of video global features, and processing the text through a text encoder to obtain two groups of text global features; capturing fine-grained information by using a trained video retrieval model; obtaining the similarity between each video and text based on the global features and fine-grained features; and sorting the similarity between each video and text to obtain the video with the highest similarity as the retrieval result. The application improves the interference caused by the insufficient details of the pedestrian features in the video and the color dependence in the current video retrieval method, selects the combination of gray and color video frames and the combination of global and local, refines the pedestrian features and the text features, and improves the retrieval accuracy of the model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular to a method for retrieving text and pedestrian videos based on cross-modal learning. Background Technology

[0002] The purpose of pedestrian video retrieval is to search for and locate target pedestrians in a video surveillance database based on verbal descriptions provided by witnesses.

[0003] While image and text retrieval technologies have matured significantly, greatly improving both speed and accuracy, video retrieval technology is still in its nascent stage and is more challenging than image retrieval, yet it remains essential. Videos possess temporal characteristics compared to images; a video clip can reveal the continuous behavior of a pedestrian over a specific period, containing richer action features and continuous motion changes not found in images.

[0004] In the process of video retrieval, the extraction of pedestrian features in the video is often affected by factors such as background and color dependence, resulting in the loss of some fine-grained features and failing to achieve good retrieval results.

[0005] Removing distracting information from the video and shielding the retrieval results from the influence of color information dependence are urgent problems that need to be solved. Summary of the Invention

[0006] The purpose of this invention is to address the problem of decreased accuracy in pedestrian video retrieval caused by background interference and color dependence, and to propose a text and pedestrian video retrieval method based on cross-modal learning.

[0007] The technical solution of this invention is:

[0008] This invention provides a method for retrieving text and pedestrian videos based on cross-modal learning, the method comprising the following steps:

[0009] Step 1: Perform grayscale and masking processing on multiple videos to be identified to obtain grayscale video frame sequences and masked video frame sequences, and perform grayscale filtering processing on the text to obtain grayscale text;

[0010] Step 2: Process each video after Step 1 using a visual feature extraction network to obtain three sets of global video features for each video, including: original video global features, grayscale video global features, and masked video global features; train the text after Step 1 using a text encoder to obtain two sets of global text features, including original text global features and grayscale text global features.

[0011] Step 3: Use the trained video retrieval model to capture fine-grained information from the grayscale video, masked video, and original text corresponding to each video. After processing, obtain the fine-grained features of the original text, and the fine-grained features of each grayscale video and masked video. Obtain the similarity between each video and the text based on the global features and fine-grained features. Sort the similarity between each video and the text, and obtain the video with the highest similarity as the retrieval result.

[0012] Furthermore, the grayscale conversion of the video in step 1 includes:

[0013] First, the video is processed by frame extraction to obtain the video frame sequence V. O ={v1,…,v N};

[0014] Using the grayscale function V in OpenCV G The video frames are converted to obtain a grayscale video frame sequence V. G ={v G1 ,…,v GN}

[0015] Furthermore, in step 1, the video masking process employs a visual encoder, which uses the Vision Transformer model and includes:

[0016] Video frame sequence V O ={v1,…,v N The input is processed by a visual encoder to obtain the self-attention map A of the video frame. Partial image blocks are masked according to a preset masking rate to obtain the masked video frame sequence V. M ={v M1 ,…,v MN}

[0017] Furthermore, the text filtering process in step 1 includes:

[0018] The NLTK vocabulary tagger is used to filter out adjectives in the text, and adjectives representing colors are further filtered. The positions of the adjectives representing colors are replaced with placeholders [MASK] to obtain the filtered text.

[0019] Furthermore, the video processing in step 2 specifically involves: using the ResNet-50 visual feature extraction network to process the original video frame sequence V... O Grayscale video frame sequence V G and masked video frame sequence V M The process yields three sets of features and corresponding self-attention maps, including global features from the original video. Global features of grayscale video Global features of masked video Original video self-attention map Grayscale video self-attention map Masked video self-attention graph A VM :

[0020]

[0021]

[0022]

[0023] Furthermore, in step 2, the text encoder includes one BERT model and one Bi-LSTM, which encode the original text and the grayscale text respectively, obtaining two sets of global feature vectors and corresponding self-attention maps, including the global features of the original text. Global features of grayscale text Original text self-attention graph Grayscale text self-attention graph

[0024]

[0025]

[0026] Among them, T O and T G These are the original text and grayscale text descriptions, respectively.

[0027] Furthermore, in step 3, the steps for obtaining fine-grained features are as follows:

[0028] Step 31: Use the following formula to analyze the global features of the grayscale video. Global features of masked video and global features of the original text The process yields grayscale video, masked video, and a self-attention map containing fine-grained information from the original text.

[0029]

[0030]

[0031]

[0032] in: This represents the feature dimension adjustment coefficient, used to adjust the feature dimension relative to the desired dimension. Consistent; ω1, ω2, and ω3 are the weight ratios of the weighted sum. These are, respectively, a grayscale video self-attention map, a masked video self-attention map, and a raw text self-attention map;

[0033] Step 32: Apply the softmax function to the self-attention map containing fine-grained information. The process involves ranking the information content of each token in its respective attention graph, and then selecting the tokens with the highest information content as the corresponding fine-grained features of the grayscale video according to a preset percentage. Fine-grained features of masked videos and fine-grained features of the original text

[0034] Furthermore, in step 3, the steps for obtaining the similarity between the video and the text are as follows:

[0035] Step 33: Calculate the cosine similarity of the three sets of video text features for any given video:

[0036]

[0037]

[0038]

[0039] Where: S1 is the global feature of the original video frame. Global features of the original text The cosine similarity between them, S2 is the fine-grained feature of grayscale video. Global features of grayscale text The cosine similarity between them, and S3 is the fine-grained feature of the masked video. Fine-grained features of the original text Cosine similarity;

[0040] Step 34: Use the softmax function to process the three sets of similarities, and use the results as the weights of the features for each set.

[0041]

[0042] Step 35: Based on the weights of each set of features, perform a weighted sum of the three sets of global video features to obtain the effective video features F. V :

[0043]

[0044] Step 36: Calculate the effective video features F V Fine-grained features of the original text The cosine similarity is used as the final similarity S between the video and the text:

[0045]

[0046] Furthermore, the video retrieval model employs cross-entropy loss L during training. ID The cross-modal projection matching loss (CMPM) and cross-modal projection classification loss (CMPC) are trained, and the final loss is obtained using the following formula:

[0047] Loss = L ID +(L CMPM +L CMPC ).

[0048] Furthermore, the cross-entropy loss L id The following formula is used to obtain it:

[0049]

[0050] Where M is the total number of samples, i represents the sample number, a sample includes a video and its corresponding text, ID represents the video number, and y i This represents the actual video ID corresponding to sample number i; This represents the predicted video number corresponding to sample number i.

[0051] The beneficial effects of this invention are:

[0052] This invention combines global features with local fine-grained features, integrating grayscale and color to extract features from video and text from multiple perspectives. This solves the problems of color dependence and interference from video background information on pedestrian features, enhancing the model's ability to extract fine-grained features.

[0053] Before extracting each feature, this invention first processes the video and text to obtain video and text versions containing different feature information, making full use of the information presented at different levels of the video and text, thereby improving the accuracy of the retrieval results.

[0054] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0055] The above and other objects, features and advantages of the present invention will become more apparent from the more detailed description of exemplary embodiments of the invention in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the invention.

[0056] Figure 1 This is a flowchart of the video retrieval method of the present invention.

[0057] Figure 2 This is a schematic diagram of video global feature extraction according to the present invention.

[0058] Figure 3 This is a schematic diagram of the global text feature extraction of the present invention.

[0059] Figure 4 This is a schematic diagram of fine-grained feature filtering for video and text according to the present invention. Detailed Implementation

[0060] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0061] like Figure 1 As shown, a text and pedestrian video retrieval method based on cross-modal learning includes the following steps:

[0062] Step 1: Perform grayscale and masking processing on multiple videos to be identified to obtain grayscale video frame sequences and masked video frame sequences, and perform grayscale filtering processing on the text to obtain grayscale text;

[0063] like Figure 2 As shown, grayscale conversion of a video includes:

[0064] First, the video is processed by frame extraction to obtain the video frame sequence V. O ={v1,…,v N};

[0065] Using the grayscale function V in OpenCV G The video frames are converted to obtain a grayscale video frame sequence V. G ={v G1 ,…,v GN}

[0066] Video masking is performed using a visual encoder, which employs the Vision Transformer model and includes:

[0067] Video frame sequence V O ={v1,…,v N The input is processed by a visual encoder to obtain the self-attention map A of the video frame. Partial image blocks are masked according to a preset masking rate to obtain the masked video frame sequence V. M ={v M1 ,…,v MN}

[0068] like Figure 3As shown, the text filtering process includes: using the vocabulary tagger in NLTK to filter out adjectives in the text, filtering the adjectives representing colors, replacing the positions of the adjectives representing colors with placeholders [MASK], and obtaining the filtered text.

[0069] Step 2: Process each video after Step 1 using a visual feature extraction network (RestNet-50) to obtain three sets of global video features for each video, including: original video global features, grayscale video global features, and masked video global features; train the text after Step 1 using a text encoder to obtain two sets of global text features, including original text global features and grayscale text global features.

[0070] like Figure 2 As shown, the video processing specifically involves using a ResNet-50 visual feature extraction network to process the original video frame sequence V. O Grayscale video frame sequence V G and masked video frame sequence V M The process yields three sets of features and corresponding self-attention maps, including global features from the original video. Global features of grayscale video Global features of masked video Original video self-attention map Grayscale video self-attention map Self-attention graph of masked video

[0071]

[0072]

[0073]

[0074] like Figure 3 As shown, the text encoder includes one BERT model and one Bi-LSTM, which encode the original text and the grayscale text respectively, resulting in two sets of global feature vectors and corresponding self-attention maps, including the global features of the original text. Global features of grayscale text Original text self-attention graph Grayscale text self-attention graph

[0075]

[0076]

[0077] Among them, T O and T GThese are the original text and grayscale text descriptions, respectively.

[0078] Step 3: Use the trained video retrieval model to capture fine-grained information from the grayscale video, masked video, and original text corresponding to each video. After processing, obtain the fine-grained features of the original text, and the fine-grained features of each grayscale video and masked video. Obtain the similarity between each video and the text based on the global features and fine-grained features. Sort the similarity between each video and the text, and obtain the video with the highest similarity as the retrieval result.

[0079] like Figure 4 As shown, by combining video text features with the sigmoid function and self-attention maps, fine-grained features that are easily overlooked in videos and texts are captured. The steps for obtaining fine-grained features are as follows:

[0080] Step 31: Use the following formula to analyze the global features of the grayscale video. Global features of masked video and global features of the original text The process yields grayscale video, masked video, and a self-attention map containing fine-grained information from the original text.

[0081]

[0082]

[0083]

[0084] in: This represents the feature dimension adjustment coefficient, used to adjust the feature dimension relative to the desired dimension. Consistent; ω1, ω2, and ω3 are the weight ratios of the weighted sum. These are, respectively, a grayscale video self-attention map, a masked video self-attention map, and a raw text self-attention map;

[0085] Step 32: Apply the softmax function to the self-attention map containing fine-grained information. The process involves ranking the information content of each token in its respective attention graph, and then selecting the tokens with the highest information content as the corresponding fine-grained features of the grayscale video according to a preset percentage. Fine-grained features of masked videos and fine-grained features of the original text

[0086]

[0087]

[0088]

[0089] Step 33: Calculate the cosine similarity of the three sets of video text features for any given video:

[0090]

[0091]

[0092]

[0093] Where: S1 is the global feature of the original video frame. Global features of the original text The cosine similarity between them, S2 is the fine-grained feature of grayscale video. Global features of grayscale text The cosine similarity between them, and S3 is the fine-grained feature of the masked video. Fine-grained features of the original text Cosine similarity;

[0094] Step 34: Use the softmax function to process the three sets of similarities, and use the results as the weights of the features for each set.

[0095]

[0096] Step 35: Based on the weights of each set of features, perform a weighted sum of the three sets of global video features to obtain the effective video features F. V :

[0097]

[0098] Step 36: Calculate the effective video features F V Fine-grained features of the original text The cosine similarity is used as the final similarity S between the video and the text:

[0099]

[0100] In this embodiment, by combining global features with local fine-grained features and combining grayscale and color, features of video and text are extracted from multiple perspectives. This can filter out interfering information in the video and shield the interference of color information dependence on the retrieval effect, resulting in more accurate retrieval results.

[0101] Furthermore, the video retrieval model employs cross-entropy loss L during training. ID The training process includes cross-modal projection matching loss (CMPM) and cross-modal projection classification loss (CMPC); the cross-entropy loss L... id The following formula is used to obtain it:

[0102]

[0103] Where M is the total number of samples, i represents the sample number, a sample includes a video and its corresponding text, ID represents the video number, and y i This represents the actual video ID corresponding to sample number i; This represents the predicted video number corresponding to sample number i.

[0104] The final loss is obtained using the following formula:

[0105] Loss = L ID +(L CMPM +L CMPC ).

[0106] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A text and pedestrian video retrieval method based on cross-modal learning, characterized in that, The method includes the following steps: Step 1: Perform grayscale and masking processing on multiple videos to be identified to obtain grayscale video frame sequences and masked video frame sequences, and perform grayscale filtering processing on the text to obtain grayscale text; Step 2: Process each video after Step 1 using a visual feature extraction network to obtain three sets of global video features for each video, including: original video global features, grayscale video global features, and masked video global features; train the text after Step 1 using a text encoder to obtain two sets of global text features, including original text global features and grayscale text global features. Step 3: Use the trained video retrieval model to capture fine-grained information from the grayscale video, masked video, and original text corresponding to each video. After processing, obtain the fine-grained features of the original text, and the fine-grained features of each grayscale video and masked video. Obtain the similarity between each video and the text based on the global features and fine-grained features. Sort the similarity between each video and the text, and obtain the video with the highest similarity as the retrieval result. The video processing in step 2 specifically involves using the ResNet-50 visual feature extraction network to process the original video frame sequence. Grayscale video frame sequence and masked video frame sequence The process yields three sets of features and corresponding self-attention maps, including global features from the original video. Global features of grayscale video Global features of masked video Original video self-attention map Grayscale video self-attention map Self-attention map of masked video : ； ； ； Step 2 involves a text encoder consisting of one BERT model and one Bi-LSTM, which encode the original text and the grayscale text respectively, resulting in two sets of global feature vectors and corresponding self-attention maps, including the global features of the original text. Global features of grayscale text Original text self-attention graph Grayscale text self-attention graph : ； ； in, and These are the original text and grayscale text descriptions, respectively. The steps for obtaining fine-grained features in step 3 are as follows: Step 31: Use the following formula to analyze the global features of the grayscale video. Global features of masked video and global features of the original text The process yields grayscale video, masked video, and a self-attention map containing fine-grained information from the original text. , ; ； ； ； in: , , This represents the feature dimension adjustment coefficient, used to adjust the feature dimension relative to the desired dimension. , Consistent; , , The weights are the proportions of the weighted sum. , , These are, respectively, a grayscale video self-attention map, a masked video self-attention map, and a raw text self-attention map; Step 32: Apply the softmax function to the self-attention map containing fine-grained information. , The process involves ranking the information content of each token in its respective attention graph, and then selecting the tokens with the highest information content as the corresponding fine-grained features of the grayscale video according to a preset percentage. Fine-grained features of masked videos and fine-grained features of the original text .

2. The text and pedestrian video retrieval method based on cross-modal learning according to claim 1, characterized in that, Step 1, which involves converting the video to grayscale, includes: First, the video is processed by frame extraction to obtain a video frame sequence. ; Using the grayscale function in OpenCV The video frames are converted to obtain a grayscale video frame sequence. .

3. The text and pedestrian video retrieval method based on cross-modal learning according to claim 2, characterized in that, Step 1 involves masking the video using a visual encoder, which employs the Vision Transformer model and includes: video frame sequence The self-attention maps of video frames are obtained from the input visual encoder. By masking part of the image blocks according to a preset mask rate, a masked video frame sequence is obtained. .

4. The text and pedestrian video retrieval method based on cross-modal learning according to claim 1, characterized in that, Step 1, the text filtering process, includes: Use NLTK's vocabulary tagger to filter out adjectives in the text, and then filter adjectives representing colors using placeholders. Replace the positions of the adjectives representing the colors to obtain the filtered text.

5. The text and pedestrian video retrieval method based on cross-modal learning according to claim 1, characterized in that... In step 3, the steps for obtaining the similarity between the video and the text are as follows: Step 33: Calculate the cosine similarity of the three sets of video text features for any given video: ； ； ； in: Global features of the original video frame Global features of the original text Cosine similarity between them Fine-grained features for grayscale videos Global features of grayscale text Cosine similarity between them Fine-grained features for masked videos Fine-grained features of the original text Cosine similarity; Step 34: Use the softmax function to process the three sets of similarities, and use the results as the weights of the features for each set. : ； Step 35: Based on the weights of each set of features, perform a weighted sum of the three sets of global video features to obtain effective video features. : ； Step 36: Calculate valid video features Fine-grained features of the original text The cosine similarity is used as the final similarity between the video and the text. : 。 6. The text and pedestrian video retrieval method based on cross-modal learning according to claim 1, characterized in that, The video retrieval model is trained using cross-entropy loss. The cross-modal projection matching loss (CMPM) and cross-modal projection classification loss (CMPC) are trained, and the final loss is obtained using the following formula: 。 7. The text and pedestrian video retrieval method based on cross-modal learning according to claim 6, characterized in that, The cross-entropy loss The following formula is used to obtain it: ； in, The total number of samples, This indicates the sample ID. A sample consists of a video and corresponding text. The ID represents the video ID. Indicates the number is The actual video ID corresponding to the sample; Indicates the number is The predicted video ID corresponding to the sample.

Images