A multi-modal and vector database based multi-target pedestrian re-identification system

Abstract

The application discloses a multi-target pedestrian re-identification system based on a multi-modal and vector database, relates to the network communication and positioning technical field, and solves the problem of cross-target and cross-modal track association in a complex multi-camera scene. The multi-target pedestrian re-identification system comprises a monocular tracking module, a multi-modal extraction module, a track generation module, a multi-view matching module and a global retrieval module. The multi-modal features and the multi-modal multi-path recall strategy are organically combined, the accuracy and applicability of cross-modal pedestrian re-identification are significantly improved, the track-level feature generation and storage design is used, the modeling capability for the dynamic characteristics of a target in a complex scene is enhanced, and the spatiotemporal constraint mechanism and the multi-modal features are cooperatively designed, so that the logical consistency and global optimality of the track association of a target person are ensured.

Description

Technical Field

[0001] This invention relates to the field of network communication and positioning technology, and in particular to a multi-target pedestrian re-identification system based on multimodal and vector databases. Background Technology

[0002] In intelligent monitoring systems, the demand for multi-target multi-camera (MTMC) pedestrian re-identification technology is increasing in indoor scenarios, such as intelligent building management, shopping mall behavior analysis, and office security.

[0003] In indoor scenes, due to complex lighting conditions, significant differences in viewing angles, and frequent occlusion, relying solely on single modal features (such as visual appearance features or motion trajectory features) is insufficient to comprehensively and stably represent the characteristics of target pedestrians, leading to a decrease in recognition accuracy and robustness. Existing methods lack sufficient support for the diversity and complementarity of feature representations, limiting their adaptability in complex environments.

[0004] In indoor scenes with multiple cameras and non-overlapping fields of view, the appearance features of a pedestrian often vary significantly under different cameras, and some features may even be lost or inconsistent. Traditional pedestrian re-identification technologies rely more on appearance similarity for matching, which is difficult to handle complex situations such as pedestrians with similar appearances or incomplete features, resulting in limited cross-camera association performance.

[0005] Although the number of target pedestrians in indoor scenes is limited, the dense deployment of cameras, the large amount of data collected, and the diversity of target features mean that existing methods have bottlenecks in multimodal feature fusion and retrieval efficiency, and cannot meet the actual needs of high real-time performance and scalability.

[0006] Patent No. CN202411711350.9 discloses a method and system for personalized recommendation based on multi-feature fusion face recognition, aiming to provide real-time, efficient, and accurate recommendation services. The method includes the following steps: real-time acquisition of facial images of the target audience using a camera device, and preprocessing the images to extract facial regions; running a face recognition algorithm on an edge computing device to extract multimodal features including age, gender, appearance, and style features; generating a comprehensive feature vector using a Transformer model through a feature fusion module to achieve style and appearance classification; generating personalized recommendation content based on cosine similarity matching of user feature vectors and a content database, and dynamically optimizing the classification model weights based on user feedback. This invention accurately captures user dynamic behavior and appearance features through time series analysis, dynamic style analysis, and deep appearance analysis modules; and comprehensively generates user feature descriptions by combining visual, speech, and text multimodal information.

[0007] Patent No. CN202311203484.5 discloses an object recognition method based on multimodal features. The method includes: acquiring a gait image sequence set BT from the video to be detected; obtaining an optimal frame image list set BTY based on BT; extracting features from BT and BTY to obtain a feature vector set TZ; obtaining an image quality score list set ZL based on BT and BTY; acquiring a feature vector list MZ for the target object; obtaining a matching degree list set P based on MZ and TZ; obtaining a comprehensive matching degree set PY based on ZL and P; if the maximum comprehensive matching degree in PY is greater than a preset matching degree threshold, the object to be detected corresponding to the maximum comprehensive matching degree in PY is marked as a key object. This application comprehensively considers three factors—facial features, body features, and gait features—of the object to be detected, and determines the weights based on the corresponding quality scores to obtain the comprehensive matching degree. Compared to identity recognition relying solely on gait features, this method is more accurate.

[0008] However, the above-mentioned patents and existing systems face many challenges in complex indoor environments, making it difficult to achieve high-precision, high-robustness, and high-real-time recognition and cross-camera matching of target pedestrians. Summary of the Invention

[0009] The purpose of this invention is to provide a multi-target pedestrian re-identification system based on multimodal and vector databases. This system can significantly improve the accuracy and applicability of cross-modal pedestrian re-identification by organically combining multimodal features, vector databases, and multimodal multi-path recall strategies. Through trajectory-level feature generation and storage design, it enhances the ability to model the dynamic characteristics of targets in complex scenarios. Through the collaborative design of spatiotemporal constraint mechanisms and multimodal features, it ensures the logical consistency and global optimality of the trajectory association between targets and people.

[0010] This invention utilizes the following technical solution:

[0011] A multi-target pedestrian re-identification system based on a multimodal and vector database includes a monocular tracking module, a multimodal extraction module, a trajectory generation module, a multi-view matching module, and a global retrieval module; wherein,

[0012] The monocular tracking module is used to locate and track the target person in each frame of the original image based on the fuzzy dynamic decision-making mechanism and the image enhancement judgment model, and to generate multimodal information.

[0013] The multimodal extraction module is used to process multimodal information according to the quality reconstruction judgment mechanism, extract and fuse multimodal features, and store them in the vector database;

[0014] The trajectory generation module is used to extract trajectory-level feature vectors of multimodal features based on the temporal movement trajectory generation mechanism.

[0015] The multi-view matching module is used to match the trajectory-level feature vectors of different camera devices according to the spatiotemporal constraint mechanism to generate the trajectory of the target person;

[0016] The global retrieval module is used to retrieve and optimize data within the vector database based on a multi-mode, multi-path recall strategy and the characteristics of new target trajectories.

[0017] Preferably, the multimodal information includes target trajectory data, boundary detection boxes, and segmentation masks. The monocular tracking module uses a decomposition and extraction algorithm to convert the video captured by all cameras into several frames of original images. Based on a fuzzy dynamic decision-making mechanism, each frame of original image undergoes fuzziness detection and quantization, and is judged against a preset fuzziness threshold to obtain clear and blurred images. The blurred images are then enhanced to obtain enhanced images. A target detector is used to detect each target person in the enhanced and clear images, obtaining boundary detection boxes and confidence scores. An instance segmentation algorithm is used to segment the boundary detection boxes, generating a segmentation mask. A target tracking algorithm combines the boundary detection boxes and the segmentation mask to associate and track the target person's trajectory in consecutive frames of original images, obtaining target trajectory data and assigning a unique identifier.

[0018] Preferably, the operation flow of the fuzzy dynamic decision-making mechanism is as follows:

[0019] The blur level of each frame of the original image is quantified by using gradient analysis algorithm combined with frequency domain analysis algorithm to obtain the image blur level;

[0020] The image blur is compared with a preset blur threshold. If the image blur is less than the blur threshold, the current image is determined to be a clear image, and target detection, target segmentation and target tracking operations are performed directly.

[0021] If the image blurriness is greater than or equal to the blur threshold, the current image is determined to be a blurry image. The quantization decision layer of the input image enhancement decision model performs feasibility prediction, obtains the feasibility quantization value, and compares it with the preset quantization threshold.

[0022] If the feasibility quantization value is less than the quantization threshold, the image completion layer of the input image enhancement judgment model combines the unique identifier and timestamp, and uses a generative adversarial network to enhance the current image to obtain a clear image for target detection, target segmentation and target tracking operations, while discarding the current image.

[0023] If the feasibility quantization value is greater than or equal to the quantization threshold, the feature extraction layer of the input image enhancement judgment model is used to extract features from the blurred image by combining the principal component analysis algorithm to obtain the global fuzzy matrix.

[0024] The enhancement feedback layer of the image enhancement judgment model combines the PID algorithm to perform resolution adaptation on the global blur matrix and generate a super-resolution feedback matrix.

[0025] The iterative training layer of the image enhancement judgment model combines the ESPCN network and the FSRCNN network to perform several iterations of training on the super-resolution feedback matrix to obtain the full-image super-resolution weight matrix.

[0026] The prediction decision layer of the image enhancement judgment model uses the divergence-cross-entropy loss function to update and optimize the super-resolution weight matrix of the entire image, while outputting the enhanced image.

[0027] Preferably, the multimodal extraction module preprocesses the multimodal information according to the quality reconstruction judgment mechanism:

[0028] A fuzz detector is used to detect the sharpness of the bounding boxes and segmentation masks, and the results are compared with a preset sharpness threshold.

[0029] If the sharpness of the bounding box or segmentation mask is less than the sharpness threshold, the EDSR network combined with the ESRGAN network is used to perform local super-resolution reconstruction of the current bounding box or segmentation mask.

[0030] If the sharpness of the bounding box or segmentation mask is greater than or equal to the sharpness threshold, then the edge detection algorithm is used to perform integrity checks on the bounding box and segmentation mask.

[0031] If the boundary detection box is incomplete, the actual boundary detection box is estimated based on the neighboring frame images or target trajectory data, combined with the target motion direction and trajectory prediction, according to the timestamp and background space.

[0032] If the segmentation mask is incomplete, a mask completion algorithm is used to restore and complete the current segmentation mask based on human key points and the actual bounding box, thereby obtaining a complete mask and completing the preprocessing of multimodal information.

[0033] Preferably, the multi-modal extraction module uses a visual transformer combined with actual bounding boxes to extract global visual features of the target person to obtain an overall appearance vector; the overall appearance vector includes clothing, posture, height, body shape, skin color, hairstyle and texture;

[0034] Simultaneously, a human pose estimator combined with a complete mask is used to extract local visual features of the target person, resulting in component-level feature vectors; the component-level feature vectors include the head, torso, and limbs.

[0035] The face recognition algorithm is combined with the actual bounding box to detect the face of the target person, and obtain the facial feature vector and target attribute information; the target attribute information includes gender, age group and expression.

[0036] An image-language model is used to describe the actual bounding box in language to obtain language feature information; and a language embedding model is used to combine the language feature information with target attribute information to obtain a semantic vector.

[0037] A multimodal fusion algorithm is used to jointly optimize the semantic vector with the overall appearance vector, component-level feature vector, and facial feature vector to obtain a multimodal feature vector;

[0038] The spatial position of the target person entering and leaving the camera's field of view is extracted to obtain the center coordinates of the target detection box. Combined with a monocular ranging algorithm, the coordinates are converted into three-dimensional world coordinates. At the same time, the appearance timestamp and disappearance timestamp are recorded.

[0039] Simultaneously, the trajectory points of the target person within the field of view of the camera device are captured to obtain a sequence of trajectory points, and then the motion characteristics of the target person are calculated; the motion characteristics include speed, direction and acceleration;

[0040] The multimodal feature vectors are fused and appended to the trajectory point sequence respectively, and then stored in the corresponding vector database.

[0041] Preferably, the trajectory generation module, based on the temporal motion trajectory generation mechanism, uses a sliding window moving average algorithm to locally denoise the multimodal feature vectors of each frame to obtain frame-level smooth feature vectors; it then performs temporal clustering of the frame-level smooth feature vectors based on the appearance and disappearance timestamps to generate several modal clusters; it then averages and fuses each modal cluster to generate a cluster-level feature vector; finally, it weights and fuses the cluster-level feature vectors according to the number of cluster frames to generate trajectory-level feature vectors; and finally, it stores the cluster-level feature vectors and trajectory-level feature vectors in a vector database according to the trajectory point sequence.

[0042] Preferably, the multi-view matching module judges and counts the adjacent cameras of each camera device according to the deployment position and field of view of different camera devices, and obtains an adjacency table of each camera device; it calculates the field of view intersection between camera devices based on the three-dimensional world coordinates of each camera device; and it performs trajectory matching and fusion of the target person by combining the trajectory-level feature vector of each camera device in the vector database with a spatiotemporal constraint mechanism.

[0043] If N trajectory-level feature vectors have the same timestamp and are located in the field of view of the same camera device, then each trajectory-level feature vector belongs to a different target person and cannot be merged into the trajectory of the same target person.

[0044] If N trajectory-level feature vectors have the same timestamp, but are located in the field of view of different camera devices and have no field of view intersection, then each trajectory-level feature vector belongs to a different target person and cannot be merged into the trajectory of the same target person.

[0045] If N trajectory-level feature vectors have the same timestamp, but are located in the field of view of different camera devices and have overlapping fields of view, then calculate the Euclidean distance between the trajectory-level feature vectors and compare it with a preset trajectory distance threshold for judgment:

[0046] If the Euclidean distance is less than the trajectory distance threshold, the current trajectory-level feature vector is determined to belong to the same target person and is merged into the trajectory of the same target person.

[0047] If the Euclidean distance is greater than or equal to the trajectory distance threshold, then the current trajectory-level feature vector is determined to belong to different target persons and cannot be merged into the trajectory of the same target person.

[0048] If the timestamps of the N trajectory-level feature vectors are different, and the different camera devices are all located in the adjacency table, then calculate the modality comprehensive matching degree of the multimodal feature vectors in different field-of-view spaces, and compare it with the preset matching threshold:

[0049] If the modal comprehensive matching degree is greater than or equal to the matching threshold, the current trajectory-level feature vector is determined to belong to the same target person, and is merged into the trajectory of the same target person. The unmatched trajectory-level feature vector and the last segment of the target person's trajectory are marked to obtain the candidate trajectory to be matched.

[0050] If the modality matching degree is less than the matching threshold, the current trajectory-level feature vector is determined to belong to different target persons and cannot be merged into the trajectory of the same target person.

[0051] Preferably, the global retrieval module performs multimodal feature extraction on the new target trajectory to obtain a new frame-level smoothed feature vector, and generates a new cluster-level feature vector and a new trajectory-level feature vector through a moving average algorithm and a temporal aggregation algorithm; the process of retrieving the trajectory-level feature vector from the vector database according to the multimodal multi-path recall strategy is as follows:

[0052] The new trajectory-level feature vector of each modality is decomposed to obtain several trajectory element points. At the same time, the candidate trajectories to be matched are parsed and matched and sorted with several trajectory element points to obtain the trajectory matching scores of different modalities.

[0053] The trajectory matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence trajectory matching score;

[0054] The new trajectory-level feature vector is subjected to spatiotemporal constraint verification based on the total confidence trajectory matching score: if the new trajectory-level feature vector successfully matches the unmatched trajectory-level feature vector, the two trajectory-level feature vectors are merged into the trajectory of the same target person;

[0055] If the new trajectory-level feature vector successfully matches the last segment of the target person's trajectory, the new trajectory-level feature vector is incorporated into the matched target person's trajectory; if no match is found, the search proceeds to the next level of cluster-level feature vectors.

[0056] The process of the multi-mode, multi-path recall strategy for hierarchical retrieval of cluster-level feature vectors and frame-level smooth feature vectors in the vector database is as follows:

[0057] The unmatched trajectory-level feature vectors and the last segment of the target person's trajectory are locally aggregated to obtain cluster-level features to be matched;

[0058] The new cluster-level feature vector of each modality is decomposed to obtain several cluster-level element points. At the same time, the cluster-level features to be matched are parsed and matched and sorted with several cluster-level element points to obtain the cluster-level matching scores of different modalities.

[0059] The cluster-level matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence cluster-level matching score;

[0060] Based on the total confidence cluster-level matching score, the new cluster-level feature vector is subjected to spatiotemporal constraint verification: if the new cluster-level feature vector successfully matches the unmatched cluster-level feature vector, the two cluster-level feature vectors are merged into the trajectory of the same target person;

[0061] If the new cluster-level feature vector successfully matches the last segment of the target person's cluster-level feature vector, then the new cluster-level feature vector is incorporated into the matched target person's cluster-level feature vector.

[0062] If no match is found, the computing power resource status is estimated: if the computing power resource is idle, the process proceeds to the next level of frame-level feature vector retrieval; if the computing power resource is busy, the new cluster-level feature vector is saved and marked until the computing power resource is idle, at which point frame-level feature vector retrieval is performed.

[0063] The cluster-level features to be matched are retrieved frame by frame to obtain the frame-level features to be matched.

[0064] The new frame-level feature vector of each modality is decomposed to obtain several frame-level element points. At the same time, the frame-level features to be matched are parsed and matched and sorted with several frame-level element points to obtain the frame-level matching scores of different modalities.

[0065] The frame-level matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence frame-level matching score;

[0066] The new frame-level feature vector is subjected to spatiotemporal constraint verification based on the total confidence frame-level matching score: if the new frame-level feature vector successfully matches the unmatched frame-level feature vector, the two frame-level feature vectors are merged into the trajectory of the same target person.

[0067] If the new frame-level feature vector successfully matches the last segment of the target person's frame-level feature, then the new frame-level feature vector is incorporated into the matched target person's frame-level feature.

[0068] If no match is found, the new frame-level feature vector is saved and marked as an unmatched trajectory.

[0069] Preferably, the operation flow of the global retrieval module further includes the following steps:

[0070] The time intervals between frame-level feature vectors, cluster-level feature vectors, and trajectory-level feature vectors are calculated based on the timestamps and compared with preset time thresholds.

[0071] If the time interval is greater than the time threshold, it is determined that the time connection between the two trajectory feature vectors is unreasonable, and the association is canceled.

[0072] If the time interval is less than or equal to the time threshold, then the two trajectory feature vectors are marked as trajectories to be completed.

[0073] Based on the view space and homography matrix, the spatial distances between frame-level feature vectors, cluster-level feature vectors, and trajectory-level feature vectors are calculated and compared with preset distance thresholds.

[0074] If the spatial distance is greater than the distance threshold, the spatial movement of the feature vectors of the two trajectories is determined to be unreasonable, and the association is canceled.

[0075] If the spatial distance is less than or equal to the distance threshold, then the two trajectory feature vectors are marked as trajectories to be completed.

[0076] The trajectory to be completed is analyzed according to time and space to obtain temporal and spatial feature parameters;

[0077] The time characteristic parameters include the trajectory start time, trajectory end time, and time interval; the spatial characteristic parameters include the start coordinates, end coordinates, and spatial distance.

[0078] Based on temporal and spatial feature parameters, combined with time and distance thresholds, the similarity of each segment of the trajectory of all target persons in the vector database is calculated and compared with a preset similarity threshold.

[0079] If the similarity is greater than or equal to the similarity threshold, the pedestrian re-identification algorithm is used to match the start and end points of the completed segment.

[0080] If the similarity is less than the similarity threshold, a smooth trajectory completion segment is generated using the Bézier curve interpolation algorithm to fill in the gaps between the segments of the target person's trajectory.

[0081] This invention significantly improves the accuracy and applicability of cross-modal person re-identification by organically combining multimodal features with a multimodal multi-path recall strategy; enhances the ability to model the dynamic characteristics of targets in complex scenarios through trajectory-level feature generation and storage design; and ensures the logical consistency and global optimality of target person trajectory association through the collaborative design of spatiotemporal constraint mechanisms and multimodal features. Attached Figure Description

[0082] To more clearly illustrate the technical solutions in the embodiments of the present invention or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0083] Figure 1 Schematic diagram of a multi-target pedestrian re-identification system;

[0084] Figure 2 This is a schematic diagram illustrating the principle of the image enhancement determination model.

[0085] Figure 3 Flowchart for multimodal feature extraction and storage;

[0086] Figure 4 This is a flowchart of a multi-mode, multi-path recall strategy. Detailed Implementation

[0087] The present invention will now be described in detail with reference to the accompanying drawings and embodiments:

[0088] like Figures 1 to 4 As shown, the multi-target pedestrian re-identification system based on multimodal and vector databases of the present invention includes a monocular tracking module, a multimodal extraction module, a trajectory generation module, a multi-view matching module, and a global retrieval module; wherein,

[0089] The monocular tracking module is used to locate and track the target person in each frame of the original image based on the fuzzy dynamic decision-making mechanism and the image enhancement judgment model, and to generate multimodal information.

[0090] The multimodal extraction module is used to process multimodal information according to the quality reconstruction judgment mechanism, extract and fuse multimodal features, and store them in the vector database;

[0091] The trajectory generation module is used to extract trajectory-level feature vectors of multimodal features based on the temporal movement trajectory generation mechanism.

[0092] The multi-view matching module is used to match the trajectory-level feature vectors of different camera devices according to the spatiotemporal constraint mechanism to generate the trajectory of the target person;

[0093] The global retrieval module is used to retrieve and optimize data within the vector database based on a multi-mode, multi-path recall strategy and the characteristics of new target trajectories.

[0094] In this invention, multimodal information includes target trajectory data, boundary detection boxes, and segmentation masks. The monocular tracking module uses a decomposition and extraction algorithm to convert the video captured by all cameras into several frames of original images. Based on a fuzzy dynamic decision-making mechanism, each frame of the original image undergoes fuzziness detection and quantization, and is judged against a preset fuzziness threshold to obtain clear and blurred images. The blurred images are then enhanced to obtain enhanced images. A target detector is used to detect each target person in the enhanced and clear images, obtaining boundary detection boxes and confidence scores. An instance segmentation algorithm is used to segment the boundary detection boxes, generating a segmentation mask. A target tracking algorithm combines the boundary detection boxes and the segmentation mask to associate and track the target person's trajectory in consecutive frames of original images, obtaining target trajectory data and assigning it a unique identifier.

[0095] In this invention, the operation flow of the fuzzy dynamic decision-making mechanism is as follows:

[0096] The blur level of each frame of the original image is quantified by using gradient analysis algorithm combined with frequency domain analysis algorithm to obtain the image blur level;

[0097] The image blur is compared with a preset blur threshold. If the image blur is less than the blur threshold, the current image is determined to be a clear image, and target detection, target segmentation and target tracking operations are performed directly.

[0098] If the image blurriness B(I) is greater than or equal to the blurriness threshold Tb, then the current image I is determined to be a blurred image. The quantization decision layer fq of the image enhancement decision model is input for feasibility prediction to obtain the feasibility quantization value Q, which is then compared with the preset quantization threshold Tq.

[0099] In this embodiment, Where F represents the set of blurred images; Q = fq(I; θq), Where θq represents the quantization parameter, θG represents the completion parameter, and T represents the transpose sign;

[0100] If the feasibility quantization value is less than the quantization threshold, the image completion layer of the input image enhancement judgment model combines the unique identifier ID and timestamp t, and uses the generative adversarial network G to enhance the current image to obtain a clear image Ien, which is then used for target detection, target segmentation and target tracking operations, while discarding the current image.

[0101] If the feasibility quantization value is greater than or equal to the quantization threshold, the feature extraction layer of the input image enhancement judgment model is used to extract features φ from the blurred image by combining the principal component analysis algorithm P, and the global blur matrix Mg is obtained.

[0102] The enhancement feedback layer of the image enhancement judgment model combines the PID algorithm to perform resolution adaptation on the global blur matrix and generate a super-resolution feedback matrix Mf.

[0103] In this embodiment, Where e(t) = ||Mg-Mtarget||2, Kp, Ki, and Kd are all PID coefficients, e(t) represents the matrix difference, Mtarget represents the preset resolution matrix, and τ represents the number of samplings;

[0104] The iterative training layer of the image enhancement judgment model combines the ESPCN and FSRCNN networks to perform several iterations of training on the super-resolution feedback matrix, thereby obtaining the full-image super-resolution weight matrix.

[0105] In this embodiment, ⊕ represents the network fusion operation, and k is the number of iterations;

[0106] The prediction decision layer of the image enhancement judgment model uses the divergence-cross-entropy loss function L to update and optimize the super-resolution weight matrix of the entire image, and outputs the enhanced image Ienh.

[0107] In this embodiment,

[0108]

[0109] Where α represents the weighting coefficient; D KL Let represent the divergence function, p represent the ideal probability vector, q represent the predicted probability vector, H represent the cross-entropy function, and y represent the predicted label. Let W represent the true label and W represent the set of super-resolution weight matrices for the entire image.

[0110] In this invention, the multimodal extraction module preprocesses the multimodal information according to the quality reconstruction judgment mechanism:

[0111] A fuzz detector is used to detect the sharpness of the bounding boxes and segmentation masks, and the results are compared with a preset sharpness threshold.

[0112] If the sharpness of the bounding box or segmentation mask is less than the sharpness threshold, the EDSR network combined with the ESRGAN network is used to perform local super-resolution reconstruction of the current bounding box or segmentation mask.

[0113] If the sharpness of the bounding box or segmentation mask is greater than or equal to the sharpness threshold, then the edge detection algorithm is used to perform integrity checks on the bounding box and segmentation mask.

[0114] If the boundary detection box is incomplete, the actual boundary detection box is estimated based on the neighboring frame images or target trajectory data, combined with the target motion direction and trajectory prediction, according to the timestamp and background space.

[0115] If the segmentation mask is incomplete, a mask completion algorithm is used to restore and complete the current segmentation mask based on human key points and the actual bounding box, thereby obtaining a complete mask and completing the preprocessing of multimodal information.

[0116] In this invention, the multi-modal extraction module uses a visual transformer combined with actual bounding boxes to extract global visual features of the target person, resulting in an overall appearance vector; the overall appearance vector includes clothing, posture, height, body shape, skin color, hairstyle, and texture.

[0117] Simultaneously, a human pose estimator combined with a complete mask is used to extract local visual features of the target person, resulting in component-level feature vectors; the component-level feature vectors include the head, torso, and limbs.

[0118] The face recognition algorithm is combined with the actual bounding box to detect the face of the target person, and obtain the facial feature vector and target attribute information; the target attribute information includes gender, age group and expression.

[0119] An image-language model is used to describe the actual bounding box in language to obtain language feature information; and a language embedding model is used to combine the language feature information with target attribute information to obtain a semantic vector.

[0120] A multimodal fusion algorithm is used to jointly optimize the semantic vector with the overall appearance vector, component-level feature vector, and facial feature vector to obtain a multimodal feature vector;

[0121] The spatial position of the target person entering and leaving the camera's field of view is extracted to obtain the center coordinates of the target detection box. Combined with a monocular ranging algorithm, the coordinates are converted into three-dimensional world coordinates. At the same time, the appearance timestamp and disappearance timestamp are recorded.

[0122] Simultaneously, the trajectory points of the target person within the field of view of the camera device are captured to obtain a sequence of trajectory points, and then the motion characteristics of the target person are calculated; the motion characteristics include speed, direction and acceleration;

[0123] The multimodal feature vectors are fused and appended to the trajectory point sequence respectively, and then stored in the corresponding vector database.

[0124] In this invention, the trajectory generation module, based on the temporal motion trajectory generation mechanism, uses a sliding window moving average algorithm to locally denoise the multimodal feature vectors of each frame, obtaining frame-level smooth feature vectors; it then performs temporal clustering of the frame-level smooth feature vectors based on the appearance and disappearance timestamps, generating several modal clusters; it then averages and fuses each modal cluster to generate a cluster-level feature vector; finally, it weights and fuses the cluster-level feature vectors according to the number of cluster frames to generate trajectory-level feature vectors; and finally, it stores the cluster-level feature vectors and trajectory-level feature vectors in a vector database according to the trajectory point sequence.

[0125] In this invention, the multi-view matching module judges and counts the adjacent cameras of each camera device according to the deployment position and field of view of different camera devices, and obtains an adjacency table of each camera device; it calculates the field of view intersection between camera devices based on the three-dimensional world coordinates of each camera device; and it performs trajectory matching and fusion of the target person by combining the trajectory-level feature vector of each camera device in the vector database with a spatiotemporal constraint mechanism.

[0126] If N trajectory-level feature vectors have the same timestamp and are located in the field of view of the same camera device, then each trajectory-level feature vector belongs to a different target person and cannot be merged into the trajectory of the same target person.

[0127] If N trajectory-level feature vectors have the same timestamp, but are located in the field of view of different camera devices and have no field of view intersection, then each trajectory-level feature vector belongs to a different target person and cannot be merged into the trajectory of the same target person.

[0128] If N trajectory-level feature vectors have the same timestamp, but are located in the field of view of different camera devices and have overlapping fields of view, then calculate the Euclidean distance between the trajectory-level feature vectors and compare it with a preset trajectory distance threshold for judgment:

[0129] If the Euclidean distance is less than the trajectory distance threshold, the current trajectory-level feature vector is determined to belong to the same target person and is merged into the trajectory of the same target person.

[0130] If the Euclidean distance is greater than or equal to the trajectory distance threshold, then the current trajectory-level feature vector is determined to belong to different target persons and cannot be merged into the trajectory of the same target person.

[0131] If the timestamps of the N trajectory-level feature vectors are different, and the different camera devices are all located in the adjacency table, then calculate the modality comprehensive matching degree of the multimodal feature vectors in different field-of-view spaces, and compare it with the preset matching threshold:

[0132] If the modal comprehensive matching degree is greater than or equal to the matching threshold, the current trajectory-level feature vector is determined to belong to the same target person, and is merged into the trajectory of the same target person. The unmatched trajectory-level feature vector and the last segment of the target person's trajectory are marked to obtain the candidate trajectory to be matched.

[0133] If the modality matching degree is less than the matching threshold, the current trajectory-level feature vector is determined to belong to different target persons and cannot be merged into the trajectory of the same target person.

[0134] In this invention, the global retrieval module performs multimodal feature extraction on the new target trajectory to obtain a new frame-level smoothed feature vector, and generates a new cluster-level feature vector and a new trajectory-level feature vector through a moving average algorithm and a temporal aggregation algorithm. The process of retrieving the trajectory-level feature vector from the vector database according to the multimodal multi-path recall strategy is as follows:

[0135] The new trajectory-level feature vector of each modality is decomposed to obtain several trajectory element points. At the same time, the candidate trajectories to be matched are parsed and matched and sorted with several trajectory element points to obtain the trajectory matching scores of different modalities.

[0136] The trajectory matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence trajectory matching score;

[0137] The new trajectory-level feature vector is subjected to spatiotemporal constraint verification based on the total confidence trajectory matching score: if the new trajectory-level feature vector successfully matches the unmatched trajectory-level feature vector, the two trajectory-level feature vectors are merged into the trajectory of the same target person;

[0138] If the new trajectory-level feature vector successfully matches the last segment of the target person's trajectory, the new trajectory-level feature vector is incorporated into the matched target person's trajectory; if no match is found, the search proceeds to the next level of cluster-level feature vectors.

[0139] The process of the multi-mode, multi-path recall strategy for hierarchical retrieval of cluster-level feature vectors and frame-level smooth feature vectors in the vector database is as follows:

[0140] The unmatched trajectory-level feature vectors and the last segment of the target person's trajectory are locally aggregated to obtain cluster-level features to be matched;

[0141] The new cluster-level feature vector of each modality is decomposed to obtain several cluster-level element points. At the same time, the cluster-level features to be matched are parsed and matched and sorted with several cluster-level element points to obtain the cluster-level matching scores of different modalities.

[0142] The cluster-level matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence cluster-level matching score;

[0143] Based on the total confidence cluster-level matching score, the new cluster-level feature vector is subjected to spatiotemporal constraint verification: if the new cluster-level feature vector successfully matches the unmatched cluster-level feature vector, the two cluster-level feature vectors are merged into the trajectory of the same target person;

[0144] If the new cluster-level feature vector successfully matches the last segment of the target person's cluster-level feature vector, then the new cluster-level feature vector is incorporated into the matched target person's cluster-level feature vector.

[0145] If no match is found, the computing power resource status is estimated: if the computing power resource is idle, the process proceeds to the next level of frame-level feature vector retrieval; if the computing power resource is busy, the new cluster-level feature vector is saved and marked until the computing power resource is idle, at which point frame-level feature vector retrieval is performed.

[0146] The cluster-level features to be matched are retrieved frame by frame to obtain the frame-level features to be matched.

[0147] The new frame-level feature vector of each modality is decomposed to obtain several frame-level element points. At the same time, the frame-level features to be matched are parsed and matched and sorted with several frame-level element points to obtain the frame-level matching scores of different modalities.

[0148] The frame-level matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence frame-level matching score;

[0149] The new frame-level feature vector is subjected to spatiotemporal constraint verification based on the total confidence frame-level matching score: if the new frame-level feature vector successfully matches the unmatched frame-level feature vector, the two frame-level feature vectors are merged into the trajectory of the same target person.

[0150] If the new frame-level feature vector successfully matches the last segment of the target person's frame-level feature, then the new frame-level feature vector is incorporated into the matched target person's frame-level feature.

[0151] If no match is found, the new frame-level feature vector is saved and marked as an unmatched trajectory.

[0152] In this invention, the operation flow of the global retrieval module further includes the following steps:

[0153] The time intervals between frame-level feature vectors, cluster-level feature vectors, and trajectory-level feature vectors are calculated based on the timestamps and compared with preset time thresholds.

[0154] If the time interval is greater than the time threshold, it is determined that the time connection between the two trajectory feature vectors is unreasonable, and the association is canceled.

[0155] If the time interval is less than or equal to the time threshold, then the two trajectory feature vectors are marked as trajectories to be completed.

[0156] Based on the view space and homography matrix, the spatial distances between frame-level feature vectors, cluster-level feature vectors, and trajectory-level feature vectors are calculated and compared with preset distance thresholds.

[0157] If the spatial distance is greater than the distance threshold, the spatial movement of the feature vectors of the two trajectories is determined to be unreasonable, and the association is canceled.

[0158] If the spatial distance is less than or equal to the distance threshold, then the two trajectory feature vectors are marked as trajectories to be completed.

[0159] The trajectory to be completed is analyzed according to time and space to obtain temporal and spatial feature parameters;

[0160] The time characteristic parameters include the trajectory start time, trajectory end time, and time interval; the spatial characteristic parameters include the start coordinates, end coordinates, and spatial distance.

[0161] Based on temporal and spatial feature parameters, combined with time and distance thresholds, the similarity of each segment of the trajectory of all target persons in the vector database is calculated and compared with a preset similarity threshold.

[0162] If the similarity is greater than or equal to the similarity threshold, the pedestrian re-identification algorithm is used to match the start and end points of the completed segment.

[0163] If the similarity is less than the similarity threshold, a smooth trajectory completion segment is generated using the Bézier curve interpolation algorithm to fill in the gaps between the segments of the target person's trajectory.

[0164] Example:

[0165] In traditional single-camera tracking workflows, input image quality issues (such as blurriness, out-of-focus images, or motion blur) often lead to a decline in target detection and tracking performance. To address this issue, this module designs a fuzzy dynamic decision-making mechanism combined with an image enhancement judgment model:

[0166] The system quickly quantifies the degree of blur in an image to determine if additional enhancement processing is needed. Since modern mainstream surveillance cameras typically have high resolution, gradient analysis (Laplace transform variance) and frequency domain analysis (Fast Fourier Transform) are used to ensure processing speed.

[0167] Based on the blurred detection results, it is determined whether super-resolution enhancement is needed to improve the image, and whether the enhancement could significantly improve detection performance. This is achieved using a deep learning prediction model: a dedicated prediction model is trained, taking a blurred image as input and outputting the image's predicted detection performance in both "direct detection" and "super-resolution post-detection" scenarios.

[0168] Clear and slightly blurred images: directly fed into subsequent object detection, segmentation, and tracking modules without additional processing.

[0169] Moderately blurred images: Proceed to the super-resolution feasibility prediction process. For images where super-resolution is expected to significantly improve detection performance, perform full-image super-resolution processing using a lightweight model (ESPCN, FSRCNN), and then feed them into the subsequent processes. Images with poor super-resolution prediction results are discarded.

[0170] In each frame of the image, an object detector is used to obtain the bounding box location and confidence score of pedestrian targets. An instance segmentation algorithm is applied to each detected bounding box to generate a corresponding segmentation mask. The segmentation mask can further refine the pixel-level region of the target, especially when there is overlap between pedestrian targets, effectively distinguishing the region range of different targets.

[0171] By combining detection boxes with segmentation masks, target tracking algorithms (such as multi-target tracking based on Kalman filtering and the Hungarian algorithm) are used to associate the trajectory of targets in consecutive frames. Trajectory smoothing, re-association of lost targets, and dynamic updates to the segmentation mask improve the integrity and accuracy of the trajectory.

[0172] The segmented pedestrian images are provided to the multimodal feature extraction module (visual features, facial features, language descriptions, etc.) to extract global visual features of pedestrians. When pedestrian targets overlap or the background is complex, the segmentation mask can refine the effective pixel area within the detection box, improving the accuracy of visual feature extraction. Simultaneously, it provides higher-quality input to the pose estimation module, helping the system extract local information more accurately in occluded or overlapping pedestrian scenes. It also provides temporal and spatial information on pedestrian motion to the spatiotemporal feature modeling module, enabling the analysis of pedestrian motion patterns and spatiotemporal constraints related to cross-camera connections.

[0173] In the single-camera target tracking module, although global blur level has been detected, the target bounding box and segmentation mask may still be blurry due to issues such as resolution, motion blur, and lighting conditions. Directly extracting features would reduce the system's recognition accuracy, thus requiring a secondary sharpness assessment. Since the resolution of the target bounding box and segmentation mask has been significantly reduced, a blur detector based on a convolutional network can be used for sharpness evaluation. When the sharpness is below a set threshold, the target bounding box is marked as "blurred," and the system enters the super-resolution processing stage.

[0174] For blurred or low-resolution bounding boxes, super-resolution techniques can restore the sharpness of the target, enabling the model to extract higher-quality features. Unlike the lightweight models used in full-image super-resolution, high-quality models (such as EDSR and ESRGAN) are used for target super-resolution reconstruction to enhance the resolution of the detection boxes and segmentation masks. For bounding boxes combined with segmentation masks, super-resolution reconstruction is performed only on the region within the mask area to avoid unnecessary computation of background regions.

[0175] In cases where occlusion or the target being located at an image edge results in incomplete bounding boxes or segmentation masks, recovering the occluded target region or estimating the true boundary position of the target is particularly important for improving the completeness of feature extraction. Analyzing the contextual region of the target bounding box (such as neighboring frames or trajectory information) and combining it with target motion direction and trajectory prediction allows for the inference of the target's actual boundary through spatiotemporal features.

[0176] Based on human structure modeling (such as human keypoint detection) or contextual reasoning (such as GAN-based mask completion techniques), mask completion is performed on occluded or truncated regions. Combining existing segmentation masks and estimated target bounding boxes, a complete mask region is generated. The restored complete bounding boxes and segmentation masks are used for subsequent feature extraction. For targets that are not fully displayed, features of the occluded regions are recovered as much as possible.

[0177] Feature extraction is performed on the overall appearance of pedestrians. Deep learning models (such as ResNet or VisionTransformer) are used to extract global feature vectors containing information such as color, texture, and clothing style. Global features provide overall appearance information of pedestrians and are suitable for situations where the target is not occluded or has minimal overlap. A segmentation mask generated by an instance segmentation model is applied to extract features from the effective pixel regions within the target bounding box. Human pose estimation (such as HRNet) is combined to extract part-level feature vectors of pedestrians (including head, torso, limbs, etc.) to enhance the ability to capture local differences. Local features are particularly important in occluded or overlapping scenarios, as they can avoid interference from background noise.

[0178] The system detects pedestrian facial regions within the bounding box and extracts high-precision facial feature vectors using face recognition models (RetinaFace, ArcFace). Facial features are highly discriminative and provide reliable identity information when the target is visible from the front. Based on the facial regions, pedestrian attribute information (including gender, age group, and expression) is further extracted. This attribute information serves as auxiliary features and is combined with other modal features to enhance the system's robustness.

[0179] A natural language description of the pedestrian (e.g., "male in a red coat") is generated from the bounding box using an image-language model (BLIP). The language description provides high-level semantic information about the visual features, especially when the target appearance is blurred or occluded; language features can effectively supplement the system's feature representation capabilities. The generated language description is merged with the facial attributes generated in the previous step (this step aims to prevent the image-language model from failing to extract information such as gender and age group), and then input into a language embedding model (e.g., BERT, Transformer) to extract the corresponding semantic vector. The semantic vector is jointly optimized with visual and facial features to improve the performance of cross-modal matching.

[0180] Extract the timestamps of the target's first appearance and last disappearance, extract the spatial position of the target entering and leaving the field of view, represent it as the pixel coordinates of the target box center, and combine it with the camera's monocular ranging algorithm to convert it into world coordinates.

[0181] The target tracking module acquires a sequence of trajectory points within the camera's field of view, with each point containing its temporal and spatial location. It captures the target's complete motion path, supporting cross-camera correlation and behavior analysis. Based on the trajectory point sequence, it calculates the target's motion characteristics (such as velocity, direction, and acceleration). All feature vectors are flattened and stored in a corresponding vector database set. Each vector is appended with trajectory metadata constructed using spatiotemporal features, fully leveraging the performance advantages of the vector database while preserving information for each trajectory segment to facilitate intra-trajectory clustering and cross-camera aggregation. The following is an example of storing frame-level features of the target trajectory in a vector database set:

[0182] {"trajectory_id":"T001",#trajectory ID; "camera_id":"C001",#camera ID; "time":"15:30:10.033",#frame timestamp; "position":[100,200],#center position of the detection box; "global_feature":[0.12,0.45,0.67,...,0.89]#feature vector;}

[0183] {"trajectory_id":"T001",#trajectory ID; "camera_id":"C001",#camera ID; "time":"15:30:10.033",#frame timestamp; "local_features":[f_head_1,...,f_head_256,f_torso_1,...,f_torso_256,f_legs_1,...,f_legs_256],#feature vectors

[0184] "body_info":{"head":[0,255],#Index range of head features; "torso":[256,511],#Index range of torso features; "legs":[512,767]#Index range of leg features;}}

[0185] {"trajectory_id":"T001",#trajectory ID; "camera_id":"C001",#camera ID; "time":"15:30:10.033",#frame timestamp; "face_feature":[0.23,0.45,0.67,...,0.89],#feature vector; "gender":"male",#gender; "age_group":"youth",#age group; "expression":"smile"#expression}

[0186] {"trajectory_id":"T001",#trajectory ID; "camera_id":"C001",#camera ID; "time":"15:30:10.033",#frame timestamp; "text":"male, young man, wearing red clothes and sunglasses, smiling",#text information; "text_vector":[0.11,0.22,0.33,...,0.99]#feature vector};

[0187] For each frame of the trajectory, a sliding window moving average method is used to eliminate local noise in the feature sequence and enhance feature stability for each modal feature (such as facial features, overall visual features, and local features). Temporal clustering is performed on the smooth features across multiple frames of the trajectory to capture the multi-segment dynamic characteristics of the trajectory. The clustering results reflect the feature changes of the target at different stages (such as the target entering an occlusion area, changes in movement direction, and changes in lighting conditions). Each cluster is averaged and fused to generate cluster-level features. The cluster-level features are then weighted and fused according to the number of frames in each cluster to generate a trajectory-level feature vector.

[0188] The cluster-level features after clustering and the final fused trajectory-level features are both stored in the vector database in the same way as the single-frame features. It should be noted that, since the feature fluctuations of different modalities are not consistent, the cluster lengths of each modality may not be exactly the same.

[0189] Frame overlap constraint of the same camera: If two trajectories have frame overlap in the field of view of the same camera (i.e. they are captured by the camera at the same time for a part of the time), then it can be determined that the two trajectories are different targets and cannot be merged into the same trajectory.

[0190] Frame overlap constraint for non-intersecting camera pairs: If the fields of view of two cameras do not overlap, they cannot see the same target within the same time period. Therefore, if trajectories from different cameras appear within the same time period, they likely belong to different targets and cannot be easily merged.

[0191] Intersecting camera pair positional overlap constraint: This assesses whether the spatial positions of different trajectories from two cameras with overlapping fields of view are sufficiently close within the same time period. This constraint determines whether the two trajectories belong to the same person by calculating the position of the trajectories in the world coordinate system.

[0192] Neighbor constraint: This analyzes whether a target can move from the field of view of one camera to the field of view of another without being captured by other cameras in between. Using an adjacency table between cameras, the system can determine whether two trajectories might belong to the same target.

[0193] A multi-modal, multi-path recall strategy is employed: Multi-modal features (such as global visual features, local features, facial features, and linguistic description features) are used for layer-by-layer retrieval and matching. Each modal feature is retrieved independently, and a final matching score is calculated using a fusion strategy, validated against spatiotemporal constraints. The basic process is as follows:

[0194] New trajectories are entered into the database: multimodal frame-level features of new trajectories are extracted, and cluster-level and trajectory-level features are generated through moving average and temporal aggregation.

[0195] Multimodal hierarchical retrieval: Retrieval is performed layer by layer according to the priority of trajectory-level features → cluster-level features → frame-level features.

[0196] Spatiotemporal constraint verification: Perform spatiotemporal constraint verification on the matching results to ensure the logical rationality of the matching.

[0197] Trajectory-level feature retrieval rapidly discovers potential matching global trajectories: In the vector database, trajectory-level features of the single-camera trajectory to be matched are used to retrieve two types of candidate trajectories: a. all single-camera trajectories not matched into the cross-camera trajectory; b. the last single-camera trajectory among all matched cross-camera trajectories. Trajectory-level features for each modality are retrieved separately, and corresponding matching scores are obtained. The matching scores for each modality are weighted and fused. If facial features are present, they are given higher weights; if no facial features are present, the weights are adjusted according to the scene (e.g., increasing the weight of language features in occluded scenes). The fused total matching scores are sorted from high to low. Results with higher confidence scores are then subjected to spatiotemporal constraint verification in sequence. If a match is successful with the aforementioned type a trajectory, the two single-camera trajectories are merged into a new cross-camera trajectory. If a match is successful with the aforementioned type b trajectory, the current single-camera trajectory is incorporated into the matched cross-camera trajectory. Once a match is successful, the process ends; if a match fails, the process proceeds to the next layer of cluster-level feature retrieval.

[0198] Cluster-level feature retrieval models and matches local characteristics in the trajectory, suitable for situations where trajectory-level features cannot be matched: In the vector database, cluster-level features of the single-camera trajectory to be matched are used sequentially to retrieve all cluster-level features of the aforementioned two types of trajectories (a and b). Cluster-level features for each modality are retrieved separately, and corresponding matching scores are obtained, then fused and sorted as described above. In order, results with higher confidence scores are subjected to spatiotemporal constraint verification. As mentioned above, once a match is successful, the process ends; if a match fails, three strategies are selected based on the availability of current computing resources: 1. Proceed to the next layer, frame-level feature retrieval. 2. Save the existing results and perform frame-level feature retrieval when computing power is idle. 3. Mark the current single-camera trajectory as unmatched, awaiting subsequent association.

[0199] Frame-level feature retrieval is the final retrieval layer, used to find matching candidate trajectories frame by frame. In the vector database, the frame-level features of the single-camera trajectory to be matched are used sequentially to retrieve all frame-level features from the aforementioned two categories of trajectories (a and b). Frame-level features for each modality are retrieved separately, and corresponding matching scores are obtained, then fused and sorted as described above. In order, results with higher confidence scores are subjected to spatiotemporal constraint verification. As previously stated, once a match is successful, the process ends; if frame-by-frame searching still cannot match the current trajectory, the current single-camera trajectory is marked as an unmatched trajectory, awaiting subsequent association.

[0200] Spatiotemporal consistency check: Cross-camera trajectory association must meet temporal and spatial consistency requirements. The spatiotemporal consistency check is the first step in spatiotemporal optimization, used to verify whether the current trajectory association results are logically consistent and to filter out unreasonable associations.

[0201] Temporal continuity check: Ensure the target trajectory has a reasonable temporal connection between different cameras. Calculate the time interval between two trajectories. If the interval is too long (adjusted according to the scene), the temporal relationship between the two trajectories is considered unreasonable, and the association is canceled. If there is a long gap in the target's time but it is still reasonable (e.g., the target is briefly occluded or leaves and then re-enters another camera), it is marked as "trace to be completed" and enters the trajectory completion stage.

[0202] Spatial rationality check: Ensure that the spatial movement of the target trajectory between different cameras conforms to physical laws. The pixel coordinates of the trajectory are transformed to a unified world coordinate system using a homography matrix, and the actual spatial distance of the trajectory is calculated. If the distance is greater than a specified threshold (depending on the scene), the spatial movement of the two trajectories is considered unreasonable, and the association is canceled.

[0203] Spatiotemporal trajectory completion: Due to limitations in camera coverage, target occlusion, or detection failure, there may be gaps in time or space between trajectories. Spatiotemporal trajectory completion aims to fill these gaps by generating smooth trajectory completion segments through interpolation methods. It involves finding temporally and spatially reasonable candidate trajectories for the end points of missing trajectories. Trajectories with similar times and reasonable spatial distances are retrieved from the database as candidates: a. The time interval between the start time of the candidate trajectory and the end time of the current trajectory is within a threshold; b. The spatial distance between the start point of the candidate trajectory and the end point of the current trajectory is less than a threshold. Subsequently, feature consistency detection is performed, using ReID feature vectors to match the start and end points of the completion segment. If the feature consistency is low, the completion is abandoned.

[0204] Trajectory interpolation: If the existing trajectory cannot be completed, a smooth trajectory completion segment is generated to fill the gaps between trajectories. Linear interpolation or Bézier curve interpolation can be used.

Claims

1. A multi-target pedestrian re-identification system based on multimodal and vector databases, characterized in that, include: The monocular tracking module is used to locate and track the target person in each frame of the original image based on the fuzzy dynamic decision-making mechanism and the image enhancement judgment model, and to generate multimodal information. The multimodal extraction module is used to process multimodal information according to the quality reconstruction judgment mechanism, extract and fuse multimodal features, and store them in the vector database; The trajectory generation module is used to extract trajectory-level feature vectors of multimodal features based on the temporal movement trajectory generation mechanism. The multi-view matching module is used to match the trajectory-level feature vectors of different camera devices according to the spatiotemporal constraint mechanism to generate the trajectory of the target person; The global retrieval module is used to retrieve and optimize data in the vector database based on a multi-mode, multi-path recall strategy and the characteristics of new target trajectories. In the monocular tracking module, the operation flow of the fuzzy dynamic decision-making mechanism is as follows: The blur level of each frame of the original image is quantified by using gradient analysis algorithm combined with frequency domain analysis algorithm to obtain the image blur level; The image blur is compared with a preset blur threshold. If the image blur is less than the blur threshold, the current image is determined to be a clear image, and target detection, target segmentation and target tracking operations are performed directly. If the image blurriness is greater than or equal to the blur threshold, the current image is determined to be a blurry image. The quantization decision layer of the input image enhancement decision model performs feasibility prediction, obtains the feasibility quantization value, and compares it with the preset quantization threshold. If the feasibility quantization value is less than the quantization threshold, the image completion layer of the input image enhancement judgment model combines the unique identifier and timestamp, and uses a generative adversarial network to enhance the current image to obtain a clear image for target detection, target segmentation and target tracking operations, while discarding the current image. If the feasibility quantization value is greater than or equal to the quantization threshold, the feature extraction layer of the input image enhancement judgment model is used to extract features from the blurred image by combining the principal component analysis algorithm to obtain the global fuzzy matrix. The enhancement feedback layer of the image enhancement judgment model combines the PID algorithm to perform resolution adaptation on the global blur matrix and generate a super-resolution feedback matrix. The iterative training layer of the image enhancement judgment model combines the ESPCN network and the FSRCNN network to perform several iterations of training on the super-resolution feedback matrix to obtain the full-image super-resolution weight matrix. The prediction decision layer of the image enhancement judgment model uses the divergence-cross-entropy loss function to update and optimize the super-resolution weight matrix of the whole image, and outputs the enhanced image. The global retrieval module performs multimodal feature extraction on the new target trajectory to obtain a new frame-level smooth feature vector, and generates a new cluster-level feature vector and a new trajectory-level feature vector. The multimodal multi-path recall strategy refers to hierarchical retrieval of trajectory-level feature vectors, cluster-level feature vectors and frame-level smooth feature vectors in the vector database.

2. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 1, characterized in that: The multimodal information includes target trajectory data, boundary detection boxes, and segmentation masks; the monocular tracking module converts the video captured by all cameras into several frames of original images; according to the fuzzy dynamic decision mechanism, the fuzziness of each frame of original image is detected and quantized, and judged against a preset fuzziness threshold to obtain clear images and fuzzy images, and then the quality of the fuzzy images is enhanced to obtain enhanced images; An object detector is used to detect each target person in the enhanced and clear images, and boundary detection boxes and confidence scores are obtained. The boundary detection boxes are segmented to generate a segmentation mask; the boundary detection boxes and the segmentation mask are combined to associate and track the trajectory of the target person in consecutive frames of the original image, obtain the target trajectory data, and assign a unique identifier.

3. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 1, characterized in that: The multimodal extraction module preprocesses the multimodal information according to the quality reconstruction judgment mechanism: The sharpness of the boundary detection box and segmentation mask is detected and compared with a preset sharpness threshold. If the sharpness of the bounding box or segmentation mask is less than the sharpness threshold, then local super-resolution reconstruction is performed on the current bounding box or segmentation mask. If the sharpness of the bounding box or segmentation mask is greater than or equal to the sharpness threshold, then integrity checks are performed on the bounding box and segmentation mask. If the boundary detection box is incomplete, the actual boundary detection box is estimated based on the neighboring frame images or target trajectory data, combined with the target motion direction and trajectory prediction, according to the timestamp and background space. If the segmentation mask is incomplete, the current segmentation mask is restored and completed based on the human body key points and the actual bounding box to obtain a complete mask, thereby completing the multimodal information preprocessing.

4. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 1, characterized in that: The multi-modal extraction module uses a visual transformer combined with an actual bounding box to extract global visual features of the target person and obtain an overall appearance vector. Simultaneously, a human pose estimator combined with a complete mask is used to extract local visual features of the target person, resulting in component-level feature vectors. The face recognition algorithm is combined with the actual bounding box to detect the face of the target person and obtain the facial feature vector and target attribute information; An image-language model is used to describe the actual bounding box in language, thereby obtaining language feature information; The language feature information is merged by combining the language embedding model with the target attribute information to obtain the semantic vector; A multimodal fusion algorithm is used to jointly optimize the semantic vector with the overall appearance vector, component-level feature vector, and facial feature vector to obtain a multimodal feature vector; The spatial position of the target person entering and leaving the camera's field of view is extracted to obtain the center coordinates of the target detection box. Combined with a monocular ranging algorithm, the coordinates are converted into three-dimensional world coordinates. At the same time, the appearance timestamp and disappearance timestamp are recorded. Simultaneously, the trajectory points of the target person within the field of view of the camera device are captured to obtain a sequence of trajectory points, and then the motion characteristics of the target person are calculated; the motion characteristics include speed, direction and acceleration; The multimodal feature vectors are fused and appended to the trajectory point sequence respectively, and then stored in the corresponding vector database.

5. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 1, characterized in that: The trajectory generation module uses a sliding window moving average algorithm to locally denoise the multimodal feature vector of each frame according to the temporal motion trajectory generation mechanism, and obtains a frame-level smooth feature vector. Based on the occurrence and disappearance timestamps, the frame-level smooth feature vectors are temporally clustered to generate several modality clusters; each modality cluster is then averaged and fused to generate a cluster-level feature vector. Cluster-level feature vectors are weighted and fused according to the number of cluster frames to generate trajectory-level feature vectors; Cluster-level feature vectors and trajectory-level feature vectors are stored in a vector database based on the trajectory point sequence.

6. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 1, characterized in that: The multi-view matching module, based on the deployment location and field of view of different camera devices, judges and counts the adjacent camera devices of each camera device to obtain an adjacency table for each camera device; it calculates the field of view intersection between camera devices based on the three-dimensional world coordinates of each camera device; and it performs trajectory matching and fusion of the target person by combining the trajectory-level feature vectors of each camera device in the vector database with a spatiotemporal constraint mechanism. If N trajectory-level feature vectors have the same timestamp and are located in the field of view of the same camera device, then each trajectory-level feature vector belongs to a different target person and cannot be merged into the trajectory of the same target person. If N trajectory-level feature vectors have the same timestamp, but are located in the field of view of different camera devices and have no field of view intersection, then each trajectory-level feature vector belongs to a different target person and cannot be merged into the trajectory of the same target person. If N trajectory-level feature vectors have the same timestamp, but are located in the field of view of different camera devices and have overlapping fields of view, then calculate the Euclidean distance between the trajectory-level feature vectors and compare it with a preset trajectory distance threshold for judgment: If the Euclidean distance is less than the trajectory distance threshold, the current trajectory-level feature vector is determined to belong to the same target person and is merged into the trajectory of the same target person. If the Euclidean distance is greater than or equal to the trajectory distance threshold, then the current trajectory-level feature vector is determined to belong to different target persons and cannot be merged into the trajectory of the same target person. If the timestamps of the N trajectory-level feature vectors are different, and the different camera devices are all located in the adjacency table, then calculate the modality comprehensive matching degree of the multimodal feature vectors in different field-of-view spaces, and compare it with the preset matching threshold: If the modal comprehensive matching degree is greater than or equal to the matching threshold, the current trajectory-level feature vector is determined to belong to the same target person, and is merged into the trajectory of the same target person. The unmatched trajectory-level feature vector and the last segment of the target person's trajectory are marked to obtain the candidate trajectory to be matched. If the modal comprehensive matching degree is less than the matching threshold, the current trajectory-level feature vector is determined to belong to different target persons and cannot be merged into the trajectory of the same target person.

7. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 1, characterized in that: The process of retrieving trajectory-level feature vectors from a vector database based on a multi-modal, multi-path recall strategy is as follows: The new trajectory-level feature vector of each modality is decomposed to obtain several trajectory element points. At the same time, the candidate trajectories to be matched are parsed and matched and sorted with several trajectory element points to obtain the trajectory matching scores of different modalities. The trajectory matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence trajectory matching score; The new trajectory-level feature vector is subjected to spatiotemporal constraint verification based on the total confidence trajectory matching score: if the new trajectory-level feature vector successfully matches the unmatched trajectory-level feature vector, the two trajectory-level feature vectors are merged into the trajectory of the same target person; If the new trajectory-level feature vector successfully matches the last segment of the target person's trajectory, the new trajectory-level feature vector is incorporated into the matched target person's trajectory; if no match is found, the search proceeds to the next level of cluster-level feature vector retrieval.

8. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 7, characterized in that: The process of the multi-mode multi-path recall strategy for hierarchical retrieval of cluster-level feature vectors and frame-level smooth feature vectors in the vector database is as follows: The unmatched trajectory-level feature vectors and the last segment of the target person's trajectory are locally aggregated to obtain cluster-level features to be matched; The new cluster-level feature vector of each modality is decomposed to obtain several cluster-level element points. At the same time, the cluster-level features to be matched are parsed and matched and sorted with several cluster-level element points to obtain the cluster-level matching scores of different modalities. The cluster-level matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence cluster-level matching score; Based on the total confidence cluster-level matching score, the new cluster-level feature vector is subjected to spatiotemporal constraint verification: if the new cluster-level feature vector successfully matches the unmatched cluster-level feature vector, the two cluster-level feature vectors are merged into the trajectory of the same target person; If the new cluster-level feature vector successfully matches the last segment of the target person's cluster-level feature vector, then the new cluster-level feature vector is incorporated into the matched target person's cluster-level feature vector. If no match is found, the computing power resource status is estimated: if the computing power resource is idle, the process proceeds to the next level of frame-level feature vector retrieval. If computing resources are busy, new cluster-level feature vectors are saved and marked until frame-level feature vector retrieval is performed when computing resources are idle. The cluster-level features to be matched are retrieved frame by frame to obtain the frame-level features to be matched. The new frame-level feature vector of each modality is decomposed to obtain several frame-level element points. At the same time, the frame-level features to be matched are parsed and matched and sorted with several frame-level element points to obtain the frame-level matching scores of different modalities. The frame-level matching scores of each modality are weighted and fused based on the boundary detection boxes to obtain the total confidence frame-level matching score; The new frame-level feature vector is subjected to spatiotemporal constraint verification based on the total confidence frame-level matching score: if the new frame-level feature vector successfully matches the unmatched frame-level feature vector, the two frame-level feature vectors are merged into the trajectory of the same target person. If the new frame-level feature vector successfully matches the last segment of the target person's frame-level feature, then the new frame-level feature vector is incorporated into the matched target person's frame-level feature. If no match is found, the new frame-level feature vector is saved and marked as an unmatched trajectory.

9. The multi-target pedestrian re-identification system based on multimodal and vector database according to claim 8, characterized in that: The operation process of the global retrieval module also includes the following steps: The time intervals between frame-level feature vectors, cluster-level feature vectors, and trajectory-level feature vectors are calculated based on the timestamps and compared with preset time thresholds. If the time interval is greater than the time threshold, it is determined that the time connection between the two trajectory feature vectors is unreasonable, and the association is canceled. If the time interval is less than or equal to the time threshold, then the two trajectory feature vectors are marked as trajectories to be completed. Based on the view space and homography matrix, the spatial distances between frame-level feature vectors, cluster-level feature vectors, and trajectory-level feature vectors are calculated and compared with preset distance thresholds. If the spatial distance is greater than the distance threshold, the spatial movement of the feature vectors of the two trajectories is determined to be unreasonable, and the association is canceled. If the spatial distance is less than or equal to the distance threshold, then the two trajectory feature vectors are marked as trajectories to be completed. The trajectory to be completed is analyzed according to time and space to obtain temporal and spatial feature parameters; Based on temporal and spatial feature parameters, combined with time and distance thresholds, the similarity of each segment of the trajectory of all target persons in the vector database is calculated and compared with a preset similarity threshold. If the similarity is greater than or equal to the similarity threshold, the pedestrian re-identification algorithm is used to match the start and end points of the completed segment. If the similarity is less than the similarity threshold, a smooth trajectory completion segment is generated to fill in the gaps between the segments of the target person's trajectory.

Images