Video text retrieval character trajectory generation method fusing multi-modal text features

Abstract

The invention discloses a multi-modal text feature fused video text retrieval character trajectory generation method, which comprises the following steps of: S1, inputting a video and detecting a target to obtain coordinate positions of target frames (x1, y1, x2 and y2) of all N characters in an image; s2, feature extraction; s3, the figure coordinates (x1, y1, x2, y2), the text vector and the image feature vector are sent to a DeepSort model, then feature matching is carried out, and matched features are stored in a MySQL database; and S4, track analysis and display. According to the invention, text description of figure clothing, wearing, riding tools and the like in the video is realized, and a target action track is positioned through database matching; in addition, a multi-modal text description feature is added in original target tracking, word vectorization of text description is realized through a Bert model, and text description vector loss is increased in a Hungary matching link in target tracking, so that the problem of TrackID exchange caused by shielding or overlapping of multiple target persons is avoided.

Description

Technical Field

[0001] This invention relates to the field of computer vision analysis technology, and in particular to a method for generating person trajectories in video text retrieval by fusing multimodal text features. Background Technology

[0002] Currently, person trajectory generation requires the person to have corresponding wearable or mobile devices, and location information must be obtained through GPS or BeiDou Navigation Satellite System (BDS) to generate the trajectory. Furthermore, video tracking often only identifies the target's location information and image features, then uses Hungarian matching to track the corresponding target. If the target to be identified is occluded or multiple targets overlap, Trackid swapping can easily occur, leading to errors in target trajectory generation. Moreover, it is impossible to locate the target person through text search; manual review of the video is required to find the target person. Therefore, this solution proposes a video text retrieval method for generating person trajectories that integrates multimodal text features. Summary of the Invention

[0003] The purpose of this invention is to provide a method for generating character trajectories in video text retrieval by fusing multimodal text features, in order to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for generating person trajectories in video text retrieval by fusing multimodal text features, comprising the following steps:

[0005] S1, Video Input and Object Detection: For each frame of video data, a YOLO model with pre-loaded weights is used. In the YOLO model, the label number of a person is 0. By filtering out the prediction results with CLS of 0, the object is identified and the coordinates of the bounding boxes (x1, y1, x2, y2) of all N people in the image are obtained.

[0006] S2, Feature extraction: Based on the target bounding box of the person obtained in the previous step, OpenCV is used to crop the image. The cropped image is then processed by a multimodal model to predict the text description and extract the image features. Then, the BERT model is used to vectorize the predicted text description and the ReID model is used to vectorize the cropped image.

[0007] S3, Target Tracking and Data Storage: The character coordinates (x1, y1, x2, y2), text vector, and image feature vector are fed into the DeepSort model, and then Hungarian matching is performed to obtain the corresponding TrackID. Finally, the detected character coordinates (x1, y1, x2, y2), character description text, image feature vector, timestamp, and character ID are stored in the MySQL database. The character ID value is the TrackID corresponding to the Hungarian matching.

[0008] S4, Trajectory Analysis and Display: Use text search to find the person description text field in the MySQL database, extract the corresponding movement trajectory of the person to be searched, and then generate a person movement trajectory map.

[0009] Preferably, S1 includes the following specific operations:

[0010] S101, First, obtain the video stream from the specified path and the real-time camera;

[0011] S102, splits the video stream into individual frames, setting to extract 30-35 frames per second;

[0012] S103, Load the pre-trained YOLO model, apply the YOLO model to each frame of the image to perform object detection, and obtain the prediction results;

[0013] S104, filter out the prediction results with label number 0, and extract the target box coordinates.

[0014] Preferably, step S2 includes the following specific operations:

[0015] S201, first use OpenCV to extract the image of the person based on the coordinates of the target bounding box;

[0016] S202 uses a multimodal model to analyze the captured images of people and generate text descriptions about their clothing, accessories, and riding tools.

[0017] S203, input the generated text description into the BERT model for text sentence vectorization to obtain text sentence vectors;

[0018] S204 uses the ReID model to extract features from the cropped image of the person to obtain the image feature vector.

[0019] Preferably, the character coordinates (x1, y1, x2, y2) in S3 are obtained in S1, and the text vector and image feature vector are obtained in S2.

[0020] Preferably, step S3 includes the following specific operations:

[0021] S301 integrates the detected person coordinates, text vectors, and image feature vectors into a data structure;

[0022] S302, input the integrated data into the DeepSort model for target tracking;

[0023] S303, calculate the cosine similarity between the N feature vectors of the Track object and the M detected image feature vectors of the figures, generate an N×M cosine similarity matrix, and then use the 1-cosine similarity matrix to represent the image feature loss matrix.

[0024] S304, calculate the Euclidean distance between the N Track object positions predicted by the Kalman filter and the M detected object positions, and generate the distance loss matrix;

[0025] S305, calculate the cosine similarity between the N person text sentence vectors in the Track object and the detected M person text sentence vectors to obtain an N×M cosine similarity matrix, and then use the 1-cosine similarity matrix to represent the text feature loss matrix.

[0026] S306, add the three matrices together and use the Hungarian algorithm to perform TrackID matching;

[0027] S307 stores the coordinates, description text, image feature vector, timestamp, and corresponding TrackID of the matched person in a MySQL database. The ID value of the person is the TrackID corresponding to the Hungarian match.

[0028] Preferably, the TrackID matching involves assigning a TrackID to a corresponding person.

[0029] Preferably, TrackID represents the character ID value after Hungarian matching.

[0030] Preferably, step S4 includes the following specific operations:

[0031] S401, First, the user searches the MySQL database by entering a text description;

[0032] S402, calculate the Levenshtein distance based on the input text, determine the record with the highest matching degree to the input text, and obtain the corresponding person ID;

[0033] S403: Based on the found person ID, extract all records for that ID from the database and sort them by timestamp;

[0034] S404, calculate the center point of each target box and draw these center points into a trajectory according to the timestamp order;

[0035] S405 uses visualization tools to draw trajectories on images, showing the movement path of a person.

[0036] Preferably, the center point of the target box is calculated as ((x1+x2) / 2, (y1+y2) / 2).

[0037] Preferably, the visualization tool is either Matplotlib or OpenCV.

[0038] The technical effects and advantages of this invention are as follows:

[0039] This invention designs and implements text descriptions of clothing, accessories, and riding tools used by individuals in videos, and locates the target's movement trajectory by matching these descriptions to a database. Furthermore, it adds multimodal text description features to the original target tracking process, using a BERT model to convert the text descriptions into word vectors, and incorporating text description vector loss in the Hungarian matching stage of target tracking. This avoids Trackid swapping issues caused by occlusion or overlapping multiple targets, thus enhancing the stability of target tracking. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the overall operation of the present invention.

[0041] Figure 2 This is a flowchart illustrating the operation of video input and target detection in this invention.

[0042] Figure 3 This is a flowchart illustrating the feature extraction process of the present invention.

[0043] Figure 4 This is a flowchart illustrating the target tracking and data storage operations of this invention.

[0044] Figure 5 This is a flowchart illustrating the operation of trajectory analysis and demonstration in this invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] This invention provides, for example Figure 1 The video text retrieval method for generating person trajectories by fusing multimodal text features, as shown, includes the following steps:

[0047] S1, Video Input and Object Detection: For each frame of video data, a YOLO model with pre-loaded weights is used. In the YOLO model, the label number of a person is 0. By filtering out the prediction results with CLS of 0, the object is identified and the coordinates of the bounding boxes (x1, y1, x2, y2) of all N people in the image are obtained.

[0048] For details, please refer to Figure 2 As shown, S1 includes the following specific operations:

[0049] S101, firstly, the video stream is obtained from the specified path and the real-time camera, and the obtained video stream is ensured to meet the processing requirements in terms of video format and resolution;

[0050] S102 splits the video stream into individual frames, setting it to extract 30-35 frames per second to reduce processing burden and avoid repeated processing of duplicate frames;

[0051] S103, Load the pre-trained YOLO model, apply the YOLO model to each frame of the image to perform object detection, and obtain the prediction results;

[0052] S104, filter out the prediction results with label number 0, and extract the target box coordinates.

[0053] It's important to note that YOLO is a real-time object detection model that simultaneously identifies multiple objects in a single image and locates their bounding boxes. YOLO's core feature is treating object detection as a regression problem, directly mapping image pixels to bounding box coordinates and class probabilities, thus achieving efficient detection. The YOLO model divides the input image into an S×S grid, with each grid responsible for detecting objects (if the object's center is within that grid). Each grid predicts a fixed number of bounding boxes (often called "anchor boxes"), along with a confidence score (representing the probability that an object is inside the box) and a class probability (representing the class the object belongs to). Each grid outputs information including the bounding box coordinates (usually its position relative to the grid), a confidence score, and a class probability. The final output is the predictions from all grids, processed by non-maximum suppression (NMS) to obtain the final detection result.

[0054] S2, Feature extraction: Based on the target bounding box of the person obtained in the previous step, OpenCV is used to crop the image. The cropped image is then processed by a multimodal model to predict the text description and extract the image features. Then, the BERT model is used to vectorize the predicted text description and the ReID model is used to vectorize the cropped image.

[0055] For details, please refer to Figure 3 As shown, S2 includes the following specific operations:

[0056] S201, first use OpenCV to extract the image of the person based on the coordinates of the target bounding box;

[0057] S202, a multimodal model is used to analyze the captured images of people and generate text descriptions of their clothing, accessories, and riding tools. Through this step, text descriptions of the clothing, accessories, and riding tools of people in the video are generated, and the target's movement trajectory is located by matching the database.

[0058] S203, input the generated text description into the BERT model for text sentence vectorization to obtain text sentence vectors;

[0059] S204 uses the ReID model to extract features from the cropped image of the person to obtain the image feature vector.

[0060] It should be noted that the process of using OpenCV to crop a person's image based on the bounding box coordinates includes reading the image, cropping according to the bounding box coordinates, and then saving or displaying the cropped image. The process involves: using OpenCV to read the original image; defining the coordinates of the bounding box based on the output of the YOLO model (usually the coordinates of the top-left and bottom-right corners (x1, y1) and (x2, y2)); using NumPy's array slicing function to crop the person's image based on the bounding box coordinates; and then choosing to display the cropped image or save it to a file.

[0061] S3, Target Tracking and Data Storage: The character coordinates (x1, y1, x2, y2), text vector, and image feature vector are fed into the DeepSort model, and then Hungarian matching is performed to obtain the corresponding TrackID. Finally, the detected character coordinates (x1, y1, x2, y2), character description text, image feature vector, timestamp, and character ID are stored in the MySQL database. The character ID value is the TrackID corresponding to the Hungarian matching.

[0062] For details, please refer to Figure 4 As shown, the character coordinates (x1, y1, x2, y2) in S3 are obtained from S1, and the text vector and image feature vector are obtained from S2.

[0063] S3 includes the following specific operations:

[0064] S301 integrates the detected person coordinates, text vectors, and image feature vectors into a data structure (list);

[0065] S302, input the integrated data into the DeepSort model for target tracking;

[0066] S303, calculate the cosine similarity between the N feature vectors of the Track object and the M detected image feature vectors of the figures, generate an N×M cosine similarity matrix, and then use the 1-cosine similarity matrix to represent the image feature loss matrix.

[0067] S304, calculate the Euclidean distance between the N Track object positions predicted by the Kalman filter and the M detected object positions, and generate the distance loss matrix;

[0068] S305, calculate the cosine similarity between the N person text sentence vectors in the Track object and the detected M person text sentence vectors to obtain an N×M cosine similarity matrix, and then use the 1-cosine similarity matrix to represent the text feature loss matrix.

[0069] S306. Add the three matrices together and use the Hungarian algorithm to perform TrackID matching. TrackID matching is to assign TrackID to the corresponding person, and it is necessary to ensure that the correct TrackID is assigned to the corresponding person.

[0070] S307 stores the coordinates, description text, image feature vector, timestamp, and corresponding TrackID of the matched person in a MySQL database. The ID value of the person is the TrackID corresponding to the Hungarian match, ensuring the atomicity and consistency of the data storage process and avoiding data loss.

[0071] It should be noted that in DeepSort's cascaded matching, the cosine similarity between the N character feature vectors in the Track object and the M detected character image feature vectors is calculated, resulting in an N×M cosine similarity matrix. This matrix is ​​then used to represent the image feature loss matrix as a 1-cosine similarity matrix. Simultaneously, the Euclidean distance loss matrix is ​​calculated between the Kalman-filtered predicted character position coordinates of the N Track objects and the detected M character position coordinates. Finally, the N character text descriptions in the Track object and the M detected character description texts are processed through a BERT model to obtain corresponding text sentence vectors, and then cosine similarity is calculated, resulting in a new N×M cosine similarity matrix. This matrix is ​​then used to represent the text feature loss matrix as a 1-cosine similarity matrix. The three N*M loss matrices are summed and then fed into Hungarian matching to obtain the corresponding TrackID match. Finally, the detected character coordinates (x1, y1, x2, y2), character description text, image feature vector, timestamp, and character ID are stored in a MySQL database, where the character ID is the TrackID obtained after Hungarian matching.

[0072] S4, Trajectory Analysis and Display: Use text search to find the person description text field in the MySQL database, extract the corresponding movement trajectory of the person to be searched, and then generate a person movement trajectory map.

[0073] For details, please refer to Figure 5 As shown, S4 includes the following specific operations:

[0074] S401, First, the user searches the MySQL database by entering a text description. This operation requires a user interface so that the user can enter the text description.

[0075] S402, calculate the Levenshtein distance from the input text, determine the record with the highest matching degree to the input text, and obtain the corresponding person ID, which is the TrackID;

[0076] S403: Based on the found person ID, extract all records for that ID from the database and sort them by timestamp;

[0077] S404, calculate the center point of each target box and draw these center points into a trajectory according to the timestamp order. The center point of the target box is calculated as ((x1+x2) / 2, (y1+y2) / 2);

[0078] S405 uses visualization tools to draw trajectories on images, showing the movement path of a person. The visualization tool can be either Matplotlib or OpenCV, and it provides an export function to save the trajectory data and visualization results as a report or image file.

[0079] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for generating person trajectories in video text retrieval by integrating multimodal text features, characterized in that: The generation method includes the following steps: S1, Video Input and Object Detection: For each frame of video data, a YOLO model with pre-loaded weights is used. In the YOLO model, the label number of a person is 0. By filtering out the prediction results with CLS of 0, the object is identified and the coordinates of the bounding boxes (x1, y1, x2, y2) of all N people in the image are obtained. S2, Feature extraction: Based on the target bounding box of the person obtained in the previous step, OpenCV is used to crop the image. The cropped image is then processed by a multimodal model to predict the text description and extract the image features. Then, the BERT model is used to vectorize the predicted text description and the ReID model is used to vectorize the cropped image. S3, Target Tracking and Data Storage: The character coordinates (x1, y1, x2, y2), text vector, and image feature vector are fed into the DeepSort model, and then Hungarian matching is performed to obtain the corresponding TrackID. Finally, the detected character coordinates (x1, y1, x2, y2), character description text, image feature vector, timestamp, and character ID are stored in the MySQL database. The character ID value is the TrackID corresponding to the Hungarian matching. S4, Trajectory Analysis and Display: Use text search to find the person description text field in the MySQL database, extract the corresponding movement trajectory of the person to be searched, and then generate a person movement trajectory map.

2. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 1, characterized in that, S1 includes the following specific operations: S101, First, obtain the video stream from the specified path and the real-time camera; S102, splits the video stream into individual frames, setting to extract 30-35 frames per second; S103, Load the pre-trained YOLO model, apply the YOLO model to each frame of the image to perform object detection, and obtain the prediction results; S104, filter out the prediction results with label number 0, and extract the target box coordinates.

3. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 1, characterized in that, S2 includes the following specific operations: S201, first use OpenCV to extract the image of the person based on the coordinates of the target bounding box; S202 uses a multimodal model to analyze the captured images of people and generate text descriptions about their clothing, accessories, and riding tools. S203, input the generated text description into the BERT model for text sentence vectorization to obtain text sentence vectors; S204 uses the ReID model to extract features from the cropped image of the person to obtain the image feature vector.

4. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 1, characterized in that, The character coordinates (x1, y1, x2, y2) in S3 are obtained from S1, and the text vector and image feature vector are obtained from S2.

5. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 4, characterized in that, S3 includes the following specific operations: S301 integrates the detected person coordinates, text vectors, and image feature vectors into a data structure; S302, input the integrated data into the DeepSort model for target tracking; S303, calculate the cosine similarity between the N feature vectors of the Track object and the M detected image feature vectors of the figures, generate an N×M cosine similarity matrix, and then use the 1-cosine similarity matrix to represent the image feature loss matrix. S304, calculate the Euclidean distance between the N Track object positions predicted by the Kalman filter and the M detected object positions, and generate the distance loss matrix; S305, calculate the cosine similarity between the N person text sentence vectors in the Track object and the detected M person text sentence vectors to obtain an N×M cosine similarity matrix, and then use the 1-cosine similarity matrix to represent the text feature loss matrix. S306, add the three matrices together and use the Hungarian algorithm to perform TrackID matching; S307 stores the coordinates, description text, image feature vector, timestamp, and corresponding TrackID of the matched person in a MySQL database, where the person's ID value is the TrackID corresponding to the Hungarian match.

6. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 5, characterized in that, The TrackID matching process involves assigning a TrackID to a corresponding person.

7. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 6, characterized in that, The TrackID represents the character ID value obtained after Hungarian matching.

8. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 1, characterized in that, S4 includes the following specific operations: S401, First, the user searches the MySQL database by entering a text description; S402, calculate the Levenshtein distance based on the input text, determine the record with the highest matching degree to the input text, and obtain the corresponding person ID; S403: Based on the found person ID, extract all records for that ID from the database and sort them by timestamp; S404, calculate the center point of each target box and draw these center points into a trajectory according to the timestamp order; S405 uses visualization tools to draw trajectories on images, showing the movement path of a person.

9. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 8, characterized in that, The center point of the target box is calculated as ((x1+x2) / 2, (y1+y2) / 2).

10. The video text retrieval method for generating character trajectories by fusing multimodal text features according to claim 8, characterized in that, The visualization tool can be either Matplotlib or OpenCV.

Images