Text video pedestrian re-identification method based on visual semantic clustering optimization

Abstract

A text video pedestrian re-identification method based on visual semantic clustering optimization comprises the following steps: firstly, extracting frames from a video, generating feature vectors by using a visual encoder, and clustering the feature vectors in combination with a visual semantic optimization strategy; meanwhile, a motion coding module is adopted to code motion information in the video to generate corresponding motion features. Then, the visual features and the motion features are fused to generate video features containing rich spatio-temporal information. Besides, the text description is encoded through a text encoder, text features are generated, the text features and the video features are projected to a common embedding space, and finally similarity measurement is performed by using cosine similarity. And finally, by optimizing a loss function, embedding of related texts and videos is closer, embedding of unrelated texts and videos is closer, and effective retrieval of texts and pedestrian videos is realized. According to the method, the efficiency and the accuracy of an algorithm are remarkably improved in a text video cross-modal pedestrian re-identification task.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and specifically is a text video pedestrian re-identification method based on visual semantic clustering optimization. Background Art

[0002] Cross-modal person re-identification technology, which uses text descriptions to retrieve specific pedestrians in videos, is an important means to achieve intelligent analysis in video surveillance and multimedia retrieval systems. With the surge in the amount of video data, traditional image- and text-based retrieval methods face new challenges, which require not only processing static images, but also analyzing and understanding video sequences containing rich spatiotemporal data.

[0003] Models based on the Transformer architecture have achieved remarkable success in visual and textual tasks. However, when processing video data, due to the content redundancy between video frames, the traditional visual tokenization process generates a large number of homogeneous tags, which form dense clusters in the embedding space, resulting in repeated calculations and redundant operations, significantly reducing computational efficiency. Therefore, how to effectively screen appropriate visual tags to avoid redundant calculations caused by homogeneous tags is a key issue that needs to be solved urgently.

[0004] The present application is different from the prior art as follows:

[0005] Patent CN118072349A proposes a method for collecting visible light monitoring videos of pedestrians with an optical camera, performing frame-by-frame pedestrian detection and size normalization preprocessing on the videos, and then performing data enhancement by random horizontal flipping to obtain a batch of pedestrian image training sets. This method belongs to the recognition method of pedestrian images; this patent proposes a method for segmenting and extracting features from pedestrian videos, which belongs to the recognition and retrieval method of pedestrian videos. There are essential differences in the media and modality selection of the two.

[0006] Patent CN118072349A proposes a comprehensive model including a feature extraction network, a token clustering learning module, a feature alignment learning module, and a prediction module; this patent proposes modules such as a visual encoder, a motion encoding module, and a text encoder, focusing on the extraction and fusion of visual features and motion features. Both are multimodal learning frameworks, but there are substantial differences in specific module design and goals.

[0007] Patent CN118072349A uses a clustering alignment mechanism for pedestrian features and text features, and achieves network training and optimization by optimizing the corresponding loss function; this patent adopts a visual semantic optimization strategy, using k-medoids clustering algorithm optimization and cosine similarity calculation to improve the similarity between text and video features. There is an essential difference between the two in the specific strategies for achieving similarity measurement.

[0008] Patent CN118072349A uses the Sinkhorn algorithm to update the similarity matrix between text query and image features in the optimization of the retrieval process for the final image retrieval; this patent generates video features based on the joint weighted sum of features to achieve retrieval. The two are fundamentally different in the implementation of the retrieval mechanism and the algorithms used.

[0009] Technical comparison with patent CN118711217A "A cross-modal pedestrian re-identification method based on common semantic learning between modalities"

[0010] Patent CN118711217A uses Swin Transformer as an image encoder to process pedestrian images, and uses BERT as a text encoder to extract corresponding features; while your patent uses a visual encoder and a motion encoding module to extract visual features and motion features of the video respectively. Although both use modern deep learning models for feature extraction, there are essential differences in modality selection and application strategies.

[0011] Patent CN118711217A builds a complex framework including text semantic clustering module and common semantic collaboration module to enhance semantic consistency between modalities; this patent realizes pedestrian re-identification by integrating visual features with motion features. The network structure and working principle of the two are essentially different. The former emphasizes the interaction of common semantics, while the latter focuses on the combination of spatiotemporal features.

[0012] In the verification process, patent CN118711217A generates query text through a fine-tuned text decoder and calculates the cosine similarity of features to obtain the matching degree between image and text; this patent constructs a mechanism to generate retrieval features through feature fusion and weighted summation. This result shows that the specific strategies for implementing retrieval are essentially different between the two. Summary of the invention

[0013] To address the problem of decreased computational efficiency caused by redundant visual markers and repeated calculations, the present invention proposes a text-video person re-identification method based on visual semantic clustering optimization. This method significantly improves the efficiency and accuracy of the algorithm in the text-video cross-modal person re-identification task.

[0014] To achieve the above object, the technical solution adopted by the present invention is:

[0015] A text-video person re-identification method based on visual semantic clustering optimization includes the following steps:

[0016] Step 1: After segmenting the video, extract frame images from each segment and input these frame images into the visual encoder to generate visual tags;

[0017] Step 2: Cluster the visual tags generated by the frames through visual semantic optimization strategy to reduce redundant features;

[0018] Step 3: Encode the motion information extracted from the video through a motion coding module to generate motion features;

[0019] Step 4: Fuse the visual features with the motion features to form a video feature representation rich in spatiotemporal information;

[0020] Step 5: Encode the text description using a text encoder to generate text features;

[0021] Step 6: Map the text features and video features into a common embedding space and use cosine similarity to measure the similarity;

[0022] Step 7: By optimizing the loss function, relevant text and video embeddings are made closer, while irrelevant embeddings are made farther apart, thereby achieving efficient retrieval of text and pedestrian videos.

[0023] As a further improvement of the present invention, the segmentation processing of the video frame in step 1 includes:

[0024] Step 11. First, convert the video i Perform segmentation processing to obtain video segment sequence Where N represents the number of video segments. This process is done by setting a fixed time interval or dynamically adjusting according to the changes in video content to ensure that each segment can effectively represent the key content of the video.

[0025] Step 12: In each paragraph Select N representative frames within represents the nth frame in the tth segment of the ith video, where the sth segment of the ith video;

[0026] Step 13: In each segment, for each selected frame Decompose it into a series of K non-overlapping patches e i,t,n,k , represents the kth patch of the nth frame in the tth segment of the i-th video;

[0027] Step 14: In each frame, embed the time position of each patch one by one Patches and spatial position embeddings for distinguishing different spatial locations in the same frame Used to distinguish patches at the same spatial location but in different frames and add a learnable embedding at the beginning Transformed into a one-dimensional vector sequence through a two-dimensional linear layer It is called visual mark

[0028] As a further improvement of the present invention, the visual encoder used in step 1 includes a Vision Transformer model, which is as follows:

[0029] Step 15: To reduce redundancy and optimize visual semantic processing, apply the k-medoids clustering algorithm to the K×N visual markers in each segment to perform visual semantic optimization strategy, and obtain a new sequence with K×N visual markers. The center mark selected in the clustering algorithm

[0030] Step 16: The visual encoder applies a multi-head attention mechanism to the center tag sequence C and further processes it through a multi-layer perceptron.

[0031] Step 17: Extract the hidden representation of the special [CLS] tag in the last block output of the visual encoder to obtain the visual feature VIS (v i )=Visual encoder

[0032] As a further improvement of the present invention, the visual semantic optimization strategy used in step 2 includes:

[0033] Step 21: Use k-means algorithm to mark the visual Initially divided into K clusters, the centroids are μ1,μ2,...,μ K A visual marker for the middle frame of each video segment;

[0034] Step 22: For each mark m i Assign to its nearest centroid μ h , which aims to minimize the distance between the marker and the centroid

[0035] By calculating the centroid of each cluster, it is updated with the average position of all the markers in the cluster. This process involves an indicator function, whose result is equal to when the condition in the brackets is true.

[0036] Step 23: Pass The centroid of each cluster is calculated to update the average position of all markers in the cluster. This process involves the indicator function 1{·}, whose result takes the value of 1 when the condition in the brackets is met;

[0037] Step 24, continuously repeating step 22 and step 23 until the convergence criterion is reached, that is, the updated change of the centroid is very small or the preset number of iterations is reached;

[0038] Step 25: After completing the clustering operation within each paragraph, select all the central tags from the clustering results of each paragraph, and then discard the non-central tags to form a new sequence of K×N visual tags.

[0039] As a further improvement of the present invention, the motion coding module used in step 3 includes:

[0040] Step 31: Integrate the pre-trained S3D model into the motion coding module as a feature extractor. i , the S3D model generates a series of time series features MO1, MO2, ..., MO W These characteristics MO w Respectively represent the motion characteristics of the video at each time step w;

[0041] Step 32: For each time step MO w Provides a unique temporal context embedding T(v i ) to obtain the time when the action occurs and obtain the action sequence

[0042] Step 34: Use the Transformer structure to process the action sequence The multi-head attention in Transformer further enhances the motion capture capability, learns information in parallel in space, and generates the final motion feature Motion (v i )=Motion encoder(E(v i )).

[0043] As a further improvement of the present invention, the feature fusion in step 4 includes:

[0044] Step 41: Visual feature VIS (v i )pass Perform maximum-minimum normalization so that all eigenvalues ​​are scaled to the interval [0,1];

[0045] Step 42, motion feature Motion (v i )pass Perform maximum-minimum normalization so that all eigenvalues ​​are scaled to the interval [0,1];

[0046] Step 43: Share the same dimension d for visual features and motion features Fusion , after the linear layer transformation and Map visual and motion features into a unified dimensional space to achieve consistency between the two dimensions;

[0047] Step 44: Connect the visual features and motion features to obtain a joint feature representation F joint =Φ(VIS(v i ),Motion(v i ), Φ is a simple splicing operation;

[0048] Step 45: construct a weight generation network G. The joint feature representation is processed by the weight generation network G to generate a pre-fusion weight α pre =G(F joint Θ G ), Θ G represents the learnable parameters of the weight generation network;

[0049] Step 46: Use the Softmax function to normalize the fusion weights to obtain the final weights of the visual and motion features α = Softmax (α pre );

[0050] Step 47: Perform weighted summation of visual and motion features based on standardized weights to generate fused video features F video =α visual VIS(v i )+α motion Motion(v i ), where these weights correspond to visual and motion features, respectively.

[0051] As a further improvement of the present invention, the text encoder in step 5 includes:

[0052] Step 51: Segment the text information and add a special [CLS] token at the beginning to form an input sequence y = [y1, y2, ..., y M ];

[0053] Step 52: Input the input sequence y into the BERT model to generate a series of hidden states hs = [hs1, hs2, ..., hs M ], each hidden state hs j is the corresponding input token h j The vector representation of ;

[0054] Step 53: By extracting the output of the special [CLS] token at the beginning of the input sequence y, the text query representation CAP(y) can be obtained.

[0055] As a further improvement of the present invention, the text-pedestrian video embedding space used in step 6 includes:

[0056] Step 61: define a mapping function for projecting text and video features into the embedding space LP (F caption(q j ),F video (v i ));

[0057] Step 62: To measure the similarity between text and video in the embedding space, cosine similarity is used as a metric in the embedding space.

[0058] T represents the transpose of the vector, and |·|2 represents the L2 norm, i.e., the Euclidean length of the vector.

[0059] As a further improvement of the present invention, the loss function optimized in step 7 includes:

[0060] Step 71: The goal of the training method is to make related text and video embeddings closer in a specified batch, and irrelevant ones farther apart, so the loss function is defined as

[0061] v i and q j Represent the features of the i-th video and j-th text description of batch size B respectively;

[0062] Step 72: Use back propagation and gradient descent algorithms to optimize model parameters. The optimization goal is to minimize the loss function so that the model can more accurately match text descriptions and video features in the embedding space.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] The present invention proposes a text-video cross-modal pedestrian re-identification method based on visual semantic clustering optimization. The method first extracts frames from the video and generates feature vectors using a visual encoder, and clusters the feature vectors in combination with a visual semantic optimization strategy to reduce the influence of redundant features. At the same time, a motion encoding module is used to encode the motion information in the video to generate corresponding motion features. Subsequently, the visual features are fused with the motion features to generate video features containing rich spatiotemporal information. In addition, the text description is encoded by a text encoder to generate text features, and the text features and video features are projected into a common embedding space, and finally the cosine similarity is used to measure the similarity. Finally, by optimizing the loss function, the relevant text and video embeddings are made closer, and the irrelevant ones are made more distant, so as to achieve effective retrieval of text and pedestrian videos. The visual encoding module adopts the Vision Transformer model, processes the video blocks through convolution, and flattens them to obtain visual tags, and learns the spatiotemporal position embedding at the same time, so that the visual encoding module can more accurately understand the spatiotemporal background of each block. The motion coding module uses a pre-trained S3D model to extract dynamic information from the video and combines it with time embedding to generate motion features containing temporal information. This method significantly improves the efficiency and accuracy of the algorithm in the text-video cross-modal person re-identification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flow chart of the present invention;

[0066] Figure 2 Schematic diagram of clustering algorithm of the present invention;

[0067] Figure 3 This is the overall network framework diagram of the present invention. DETAILED DESCRIPTION

[0068] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0069] Flowchart as Figure 1 As shown, the schematic diagram of the clustering algorithm is as follows Figure 2 As shown, the overall network framework is as follows Figure 3 As shown, the present invention proposes a text video pedestrian re-identification method based on visual semantic clustering optimization, comprising the following steps:

[0070] The technical solution of the present invention is:

[0071] A text-video cross-modal person re-identification method based on visual semantic clustering optimization is proposed. The method specifically includes the following steps:

[0072] Step 1: After the video is segmented, frame images are extracted from each segment and input into the visual encoder to generate visual tags.

[0073] Furthermore, firstly, the video v i Perform segmentation processing to obtain video segment sequence Where N represents the number of video segments. This process can be adjusted dynamically by setting a fixed time interval or according to the changes in the video content to ensure that each segment can effectively represent the key content of the video. This segmentation strategy takes into account the contextual information of the video, making subsequent feature extraction more efficient;

[0074] Furthermore, in each paragraph Select N representative frames from (s segment of the i-th video) represents the nth frame in the tth segment of the ith video.

[0075] Furthermore, within each segment, for each selected frame Decompose it into a series of K non-overlapping patches e i,t,n,k , represents the kth patch of the nth frame in the tth segment of the i-th video;

[0076] Furthermore, in each frame, temporal position embedding is added to each of these patches Patches and spatial position embeddings for distinguishing different spatial locations in the same frame Used to distinguish patches at the same spatial location but in different frames and add a learnable embedding at the beginning Transformed into a one-dimensional vector sequence through a two-dimensional linear layer It is called visual mark

[0077] Step 2: Cluster the visual tags generated by the frames through visual semantic optimization strategy to reduce redundant features;

[0078] Furthermore, in order to reduce redundancy and optimize visual semantic processing, the k-medoids clustering algorithm is applied to the K×N visual markers in each segment to perform visual semantic optimization strategy, and a new sequence with K×N visual markers is obtained. The center mark selected in the clustering algorithm

[0079] Furthermore, the k-means algorithm is used to transform the visual markers Initially divided into K clusters, the centroids are μ1,μ2,...,μ K A visual marker for the middle frame of each video segment;

[0080] Further, each marker mi is assigned to its nearest centroid μh , which aims to minimize the distance between the marker and the centroid

[0081] By calculating the centroid of each cluster, it is updated with the average position of all the markers in the cluster. This process involves an indicator function, whose result is equal to when the condition in the brackets is true.

[0082] Further, through The centroid of each cluster is calculated to update the average position of all markers in the cluster. This process involves the indicator function 1{·}, whose result takes the value of 1 when the condition in the brackets is met;

[0083] Further, the calculation is repeated until the convergence criterion is reached, that is, the updated change of the centroid is very small or the preset number of iterations is reached;

[0084] Furthermore, after completing the clustering operation in each paragraph, all central tags are selected from the clustering results of each paragraph, and then the non-central tags are discarded to form a new sequence C = {C i 1,C i 2,...,C i S};

[0085] Furthermore, the visual encoder applies a multi-head attention mechanism to the center tag sequence C and further processes it through a multi-layer perceptron;

[0086] Furthermore, the hidden representation of the special [CLS] tag in the last block output of the visual encoder is extracted to obtain the visual features

[0087] Step 3: Encode the motion information extracted from the video through a motion coding module to generate motion features;

[0088] Furthermore, the pre-trained S3D model is integrated into the motion coding module as a feature extractor. i , the S3D model generates a series of time series features MO1, MO2, ..., MO W These characteristics MO w Respectively represent the motion characteristics of the video at each time step w;

[0089] Furthermore, for each time step MO w Provides a unique temporal context embedding T(v i ) to obtain the time when the action occurs and obtain the action sequence

[0090] Furthermore, the Transformer structure is used to process the action sequence The multi-head attention in Transformer further enhances the motion capture capability, learns information in parallel in space, and generates the final motion feature Motion (v i )=Motion encoder(E(v i ))

[0091] Step 4: Fuse the visual features with the motion features to form a video feature representation rich in spatiotemporal information;

[0092] Furthermore, the visual feature VIS (v i )pass Perform maximum-minimum normalization so that all eigenvalues ​​are scaled to the interval [0,1];

[0093] Furthermore, the motion feature Motion(v i )pass Perform maximum-minimum normalization so that all eigenvalues ​​are scaled to the interval [0,1];

[0094] Furthermore, the visual features and motion features share the same dimension d Fus i on , after the linear layer transformation and Map visual and motion features into a unified dimensional space to achieve consistency between the two dimensions;

[0095] Furthermore, the visual features and motion features are connected to obtain a joint feature representation

[0096] F joint =Φ(VIS(v i ),Motion(v i ), Φ is a simple splicing operation;

[0097] Furthermore, a weight generation network G is constructed. The joint feature representation is processed by the weight generation network G to generate a pre-fusion weight α pre =G(F joint Θ G ), Θ G represents the learnable parameters of the weight generation network;

[0098] Furthermore, the Softmax function is used to normalize the fusion weights to obtain the final weights of the visual and motion features α = Softmax (α pre );

[0099] Furthermore, the visual and motion features are weighted and summed according to the normalized weights to generate the fused video feature F video =α visual VIS(vi )+α motion Motion(v i )α visual and α motion are the weights corresponding to visual and motion features;

[0100] Step 5: Encode the text description using a text encoder to generate text features;

[0101] Furthermore, the text information is segmented and a special [CLS] token is added at the beginning to form the input sequence y = [y1, y2, ..., y M ];

[0102] Furthermore, the input sequence y is input into the BERT model to generate a series of hidden states hs = [hs1, hs2, ..., hs M ], each hidden state hs j is the corresponding input token h j The vector representation of ;

[0103] Furthermore, by extracting the output of the special [CLS] token at the beginning of the input sequence y, the text query representation CAP(y) can be obtained;

[0104] Step 6: Map text features and video features into a common embedding space and use cosine similarity to measure similarity

[0105] Furthermore, a mapping function is defined to project text and video features into the embedding space function LP (F caption (q j ),F video (v i ));

[0106] Furthermore, in order to measure the similarity between text and video in the embedding space, cosine similarity is used as a metric in the embedding space.

[0107] T represents the transpose of the vector, and |·|2 represents the L2 norm, i.e., the Euclidean length of the vector;

[0108] Step 7: By optimizing the loss function, relevant text and video embeddings are made closer and irrelevant ones are made more distant, so as to achieve effective retrieval of text and pedestrian videos.

[0109] Furthermore, the training method aims to make related text and video embeddings closer and irrelevant ones farther apart in a specified batch, so the loss function is defined as

[0110] v i and qj Represent the features of the i-th video and j-th text description of batch size B respectively;

[0111] Furthermore, back-propagation and gradient descent algorithms are used to optimize the model parameters. The optimization goal is to minimize the loss function, so that the model can more accurately match the text description and video features in the embedding space.

[0112] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A text-to-video person re-identification method based on visual semantic clustering optimization, characterized by: The steps include: Step 1: After segmenting the video, extract frame images from each segment and input these frame images into the visual encoder to generate visual tags; Step 2: Cluster the visual tags generated by the frames through visual semantic optimization strategy to reduce redundant features; Step 3: Encode the motion information extracted from the video through a motion coding module to generate motion features; Step 4: Fuse the visual features with the motion features to form a video feature representation rich in spatiotemporal information; Step 5: Encode the text description using a text encoder to generate text features; Step 6: Map the text features and video features into a common embedding space and use cosine similarity to measure the similarity; Step 7: By optimizing the loss function, relevant text and video embeddings are made closer, while irrelevant embeddings are made farther apart, thereby achieving efficient retrieval of text and pedestrian videos.

2. The method for person re-identification in text and video based on visual semantic clustering optimization according to claim 1, characterized in that: The segmentation process of the video frame in step 1 includes: Step 11. First, convert the video i Perform segmentation processing to obtain video segment sequence Where N represents the number of video segments. This process is done by setting a fixed time interval or dynamically adjusting according to the changes in video content to ensure that each segment can effectively represent the key content of the video. Step 12: In each paragraph Select N representative frames within represents the nth frame in the tth segment of the ith video, where the sth segment of the ith video; Step 13: In each segment, for each selected frame Decompose it into a series of K non-overlapping patches e i,t,n,k , represents the kth patch of the nth frame in the tth segment of the i-th video; Step 14: In each frame, embed the time position of each patch one by one Patches and spatial position embeddings for distinguishing different spatial locations in the same frame Used to distinguish patches at the same spatial location but in different frames and add a learnable embedding at the beginning Transformed into a one-dimensional vector sequence through a two-dimensional linear layer It is called visual mark 3. The text-to-video person re-identification method based on visual semantic clustering optimization according to claim 1, characterized in that: The visual encoder used in step 1 includes the Vision Transformer model, as follows: Step 15: To reduce redundancy and optimize visual semantic processing, apply the k-medoids clustering algorithm to the K×N visual markers in each segment to perform visual semantic optimization strategy, and obtain a new sequence with K×N visual markers. The center mark selected in the clustering algorithm Step 16: The visual encoder applies a multi-head attention mechanism to the center tag sequence C and further processes it through a multi-layer perceptron. Step 17: Extract the hidden representation of the special [CLS] tag in the last block output of the visual encoder to obtain the visual features 4. The method for person re-identification in text and video based on visual semantic clustering optimization according to claim 1, characterized in that: The visual semantic optimization strategies used in step 2 include: Step 21: Use k-means algorithm to mark the visual Initially divided into K clusters, the centroids are μ1,μ2,...,μ K A visual marker for the middle frame of each video segment; Step 22: For each mark m i Assign to its nearest centroid μ h , which aims to minimize the distance between the marker and the centroid By calculating the centroid of each cluster, it is updated with the average position of all the markers in the cluster. This process involves an indicator function, whose result is equal to when the condition in the brackets is true. Step 23: Pass The centroid of each cluster is calculated to update the average position of all markers in the cluster. This process involves the indicator function 1{·}, whose result takes the value of 1 when the condition in the brackets is met; Step 24, continuously repeating step 22 and step 23 until the convergence criterion is reached, that is, the updated change of the centroid is very small or the preset number of iterations is reached; Step 25: After completing the clustering operation within each paragraph, select all the central tags from the clustering results of each paragraph, and then discard the non-central tags to form a new sequence of K×N visual tags.

5. The method for person re-identification in text and video based on visual semantic clustering optimization according to claim 1, characterized in that: The motion coding modules used in step 3 include: Step 31: Integrate the pre-trained S3D model into the motion coding module as a feature extractor. i , the S3D model generates a series of time series features MO1, MO2, ..., MO W These characteristics MO w Respectively represent the motion characteristics of the video at each time step w; Step 32: For each time step MO w Provides a unique temporal context embedding T(v i ) to obtain the time when the action occurs and obtain the action sequence Step 34: Use the Transformer structure to process the action sequence The multi-head attention in Transformer further enhances the motion capture capability, learns information in parallel in space, and generates the final motion feature Motion (v i )=Motion encoder(E(v i )) .

6. The method for person re-identification in text and video based on visual semantic clustering optimization according to claim 1, characterized in that: The feature fusion in step 4 includes: Step 41: Visual feature VIS (v i )pass Perform maximum-minimum normalization so that all eigenvalues ​​are scaled to the interval [0,1]; Step 42, motion feature Motion (v i )pass Perform maximum-minimum normalization so that all eigenvalues ​​are scaled to the interval [0,1]; Step 43: Share the same dimension d for visual features and motion features Fusion , after the linear layer transformation and Map visual and motion features into a unified dimensional space to achieve consistency between the two dimensions; Step 44: Connect the visual features and motion features to obtain a joint feature representation F joint =Φ(VIS(v i ),Motion(v i ), Φ is a simple splicing operation; Step 45: construct a weight generation network G. The joint feature representation is processed by the weight generation network G to generate a pre-fusion weight α pre =G(F joint Θ G ), Θ G represents the learnable parameters of the weight generation network; Step 46: Use the Softmax function to normalize the fusion weights to obtain the final weights of the visual and motion features α = Softmax (α pre ); Step 47: Perform weighted summation of visual and motion features based on standardized weights to generate fused video features F video =α visual VIS(v i )+α motion Motion(v i ), where these weights correspond to visual and motion features, respectively.

7. The method for person re-identification in text and video based on visual semantic clustering optimization according to claim 1, characterized in that: The text encoder in step 5 includes: Step 51: Segment the text information and add a special [CLS] token at the beginning to form an input sequence y = [y1, y2, ..., y M ]; Step 52: Input the input sequence y into the BERT model to generate a series of hidden states hs = [hs1, hs2, ..., hs M ], each hidden state hs j is the corresponding input token h j The vector representation of ; Step 53: By extracting the output of the special [CLS] token at the beginning of the input sequence y, the text query representation CAP(y) can be obtained.

8. The method for person re-identification in text and video based on visual semantic clustering optimization according to claim 1, characterized in that: The text-to-person video embedding space used in step 6 includes: Step 61: define a mapping function for projecting text and video features into the embedding space LP (F caption (q j ),F video (v i )); Step 62: To measure the similarity between text and video in the embedding space, cosine similarity is used as a metric in the embedding space. T represents the transpose of the vector, and |·|2 represents the L2 norm, i.e., the Euclidean length of the vector.

9. The method for person re-identification in text and video based on visual semantic clustering optimization according to claim 1, characterized in that: The loss functions optimized in step 7 include: Step 71: The goal of the training method is to make related text and video embeddings closer in a specified batch, and irrelevant ones farther apart, so the loss function is defined as v i and q j Represent the features of the i-th video and j-th text description of batch size B respectively; Step 72: Use back propagation and gradient descent algorithms to optimize model parameters. The optimization goal is to minimize the loss function so that the model can more accurately match text descriptions and video features in the embedding space.

Images