Multimodal vector retrieval optimization method, system, computer device and storage medium
Patent Information
- Application Number
- CN202610468092.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-09
- Publication Date
- 2026-09-01
AI Technical Summary
[0005]在本实施例中提供了一种多模态向量检索优化方法、系统、计算机设备和存储介质,以解决相关技术中存在动态场景下检索精度低的问题
[0035]与相关技术相比,在本实施例中提供的多模态向量检索优化方法、系统、计算机设备和存储介质,通过对获取的长视频进行关键帧提取,得到关键帧图像;对关键帧图像进行目标检测和实例分割,得到纯化目标区域;对纯化目标区域进行特征提取,得到多维度特征;多维度特征包括静态特征和动态时序特征;动态时序特征包括运动轨迹特征和连续帧行为特征;将多维度特征输入至预训练的动态特征融合网络,生成时空融合向量;并将时空融合向量推送至向量数据库;动态特征融合网络的融合权重,根据目标质量评分、运动强度和目标占比进行动态分配;基于向量数据库中的时空融合向量,对多模态查询内容进行向量检索,得到检索结果。通过提取运动轨迹与连续帧行为等动态时序特征,并结合目标质量与运动强度动态分配融合权重生成高辨识度时空融合向量。解决了相关技术中存在动态场景下检索精度低的问题,实现了提升动态场景下检索精度。
Smart Images

Figure CN122673389A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision, and in particular to multimodal vector retrieval optimization methods, systems, computer devices, and storage media. Background Technology
[0002] Multimodal vector retrieval in video surveillance scenarios is significantly impacted by dynamic scene conditions (nighttime, rain, and occlusion), posing a serious challenge to high-precision retrieval. Therefore, an efficient and reliable multimodal vector retrieval optimization technique is crucial for improving retrieval accuracy in dynamic scenes.
[0003] Existing multimodal vector retrieval methods typically employ object detection algorithms such as YOLO to obtain target bounding box regions, then use models like ResNet to extract global appearance features from a single frame, and finally perform similarity matching in a vector database. The drawback is that it only extracts static features from a single frame, ignoring dynamic and environmental information, leading to reduced retrieval accuracy in dynamic scenes (such as nighttime, rainy days, and occlusion).
[0004] There is currently no effective solution to the problem of low retrieval accuracy in dynamic scenarios in related technologies. Summary of the Invention
[0005] This embodiment provides a multimodal vector retrieval optimization method, system, computer device, and storage medium to address the problem of low retrieval accuracy in dynamic scenarios in related technologies.
[0006] Firstly, this embodiment provides a multimodal vector retrieval optimization method, including:
[0007] Keyframes are extracted from the acquired long video to obtain keyframe images;
[0008] Target detection and instance segmentation are performed on the keyframe image to obtain the purified target region;
[0009] Feature extraction is performed on the purification target region to obtain multi-dimensional features; the multi-dimensional features include static features and dynamic temporal features; the dynamic temporal features include motion trajectory features and continuous frame behavior features.
[0010] The multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector; the spatiotemporal fusion vector is then pushed into a vector database; the fusion weights of the dynamic feature fusion network are dynamically allocated based on the target quality score, motion intensity, and target proportion.
[0011] Based on the spatiotemporal fusion vector in the vector database, vector retrieval is performed on the multimodal query content to obtain the retrieval results.
[0012] In some embodiments, the keyframe extraction from the acquired long video to obtain keyframe images includes:
[0013] Based on a preset motion entropy value, keyframes are extracted from the long video to obtain the keyframe images; the motion entropy value is used to adjust the extraction interval of the keyframe images.
[0014] In some embodiments, the step of performing target detection and instance segmentation on the keyframe image to obtain a purified target region includes:
[0015] Target detection and instance segmentation are performed on the keyframe image to obtain pixel-level target mask regions; and the target mask regions are then scored for quality.
[0016] The target mask regions with quality scores below a preset threshold are filtered out to obtain the purified target regions.
[0017] In some embodiments, the static features include target subject features, key detail features, and environmental association features;
[0018] The target subject features are extracted using a pre-trained visual transformer model;
[0019] The key detail features are extracted using an attention mechanism combined with a pre-trained residual network;
[0020] The environmental features are extracted using a pre-trained lightweight convolutional neural network.
[0021] In some of these embodiments, the motion trajectory features are calculated using optical flow.
[0022] The continuous frame behavioral features are extracted using a pre-trained long short-term memory network.
[0023] In some of these embodiments, the vector database employs a hierarchical storage structure;
[0024] The hierarchical storage structure includes a first partition and a second partition. The first partition is divided according to the target category, and the second partition is divided according to time information and camera identification.
[0025] In some embodiments, the step of performing vector retrieval on multimodal query content based on the spatiotemporal fusion vector in the vector database to obtain retrieval results includes:
[0026] If the multimodal query content is an image query, then the target feature vector of the image is extracted; and in the vector database, the target feature vector is used for vector retrieval to obtain the retrieval result;
[0027] If the multimodal query content is a text query, then the attribute feature vectors of the text are extracted through a pre-built dedicated word vector library; and vector retrieval is performed on the attribute feature vectors in the vector database to obtain the retrieval results.
[0028] Secondly, this embodiment provides a multimodal vector retrieval optimization system, including: a preprocessing layer, a feature extraction layer, a vector library storage layer, and a retrieval engine layer;
[0029] The preprocessing layer is used to extract keyframes from the acquired long video to obtain keyframe images; and to perform target detection and instance segmentation on the keyframe images to obtain purified target regions.
[0030] The feature extraction layer is used to extract features from the purified target region to obtain multi-dimensional features. The multi-dimensional features include static features and dynamic temporal features. The dynamic temporal features include motion trajectory features and continuous frame behavior features. The multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector. The spatiotemporal fusion vector is then pushed to a vector database. The fusion weights of the dynamic feature fusion network are dynamically allocated based on the target quality score, motion intensity, and target proportion.
[0031] The vector library storage layer is used to store the spatiotemporal fusion vector;
[0032] The retrieval engine layer is used to perform vector retrieval on multimodal query content based on the spatiotemporal fusion vector in the vector database to obtain retrieval results.
[0033] Thirdly, this embodiment provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal vector retrieval optimization method described in the first aspect above.
[0034] Fourthly, this embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal vector retrieval optimization method described in the first aspect above.
[0035] Compared with related technologies, the multimodal vector retrieval optimization method, system, computer equipment, and storage medium provided in this embodiment extract keyframes from the acquired long video to obtain keyframe images; perform target detection and instance segmentation on the keyframe images to obtain purified target regions; extract features from the purified target regions to obtain multi-dimensional features; the multi-dimensional features include static features and dynamic temporal features; the dynamic temporal features include motion trajectory features and continuous frame behavior features; input the multi-dimensional features into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector; and push the spatiotemporal fusion vector to a vector database; the fusion weights of the dynamic feature fusion network are dynamically allocated according to the target quality score, motion intensity, and target proportion; based on the spatiotemporal fusion vector in the vector database, vector retrieval is performed on the multimodal query content to obtain retrieval results. By extracting dynamic temporal features such as motion trajectory and continuous frame behavior, and combining them with target quality and motion intensity to dynamically allocate fusion weights, a highly recognizable spatiotemporal fusion vector is generated. This solves the problem of low retrieval accuracy in dynamic scenes in related technologies and improves retrieval accuracy in dynamic scenes.
[0036] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0037] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0038] Figure 1 This is a hardware structure block diagram of a terminal device for a multimodal vector retrieval optimization method provided in an embodiment of this application;
[0039] Figure 2 This is a flowchart of a multimodal vector retrieval optimization method provided in an embodiment of this application;
[0040] Figure 3 This is a flowchart of obtaining a purified target region provided in an embodiment of this application;
[0041] Figure 4 This is a flowchart of vector retrieval based on multimodal query content provided in an embodiment of this application;
[0042] Figure 5 This is a schematic diagram of the architecture of a multimodal vector retrieval optimization method provided in an embodiment of this application;
[0043] Figure 6 This is a structural block diagram of a multimodal vector retrieval optimization system provided in an embodiment of this application.
[0044] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 210, preprocessing layer; 220, feature extraction layer; 230, vector library storage layer; 240, retrieval engine layer. Detailed Implementation
[0045] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.
[0046] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.
[0047] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the multimodal vector retrieval optimization method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.
[0048] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the multimodal vector retrieval optimization method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0049] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0050] This embodiment provides a multimodal vector retrieval optimization method. Figure 2 This is a flowchart of the multimodal vector retrieval optimization method in this embodiment, as follows: Figure 2 As shown, the process includes the following steps:
[0051] Step S210: Extract keyframes from the acquired long video to obtain keyframe images. Perform object detection and instance segmentation on the keyframe images to obtain purified target regions.
[0052] Specifically, keyframe extraction is performed on the acquired long video. The extraction interval can be dynamically adjusted by calculating the motion entropy between video frames; alternatively, keyframes can be extracted at fixed time intervals (e.g., one frame per second); or keyframes can be selected based on the importance score of the object detection results. The specific method of keyframe extraction is not limited here. Next, object detection and instance segmentation are performed on the keyframe images, outputting pixel-level object mask regions and filtering out background redundancy and low-quality objects to obtain purified target regions. Object detection and instance segmentation involve using algorithms to identify specific objects (pedestrians, vehicles, etc.) in the image and accurately segment the target regions.
[0053] Step S220: Extract features from the purification target region to obtain multi-dimensional features; the multi-dimensional features include static features and dynamic temporal features; the dynamic temporal features include motion trajectory features and continuous frame behavior features;
[0054] Multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector; the spatiotemporal fusion vector is then pushed into a vector database; the fusion weights of the dynamic feature fusion network are dynamically allocated based on the target quality score, motion intensity, and target proportion.
[0055] Specifically, multi-dimensional feature extraction is performed on the purification target region to obtain static and dynamic features; the dynamic temporal features include motion trajectory features and continuous frame behavior features. Motion trajectory features can be obtained through optical flow, Kalman filtering, and particle filtering algorithms; the method of obtaining motion trajectory features is not restricted here. Continuous frame behavior features can be obtained through long short-term memory networks, graph convolutional networks, or two-stream networks; the method of obtaining continuous frame behavior features is not restricted here.
[0056] Multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector. The dynamic feature fusion network can employ a Transformer network, an attention mechanism network, or a recurrent neural network; the type of dynamic feature fusion network is not restricted here. The fusion weights of the dynamic feature fusion network can be dynamically allocated based on the target quality score, target motion direction, motion intensity, background complexity, and target proportion. The generated spatiotemporal fusion vector is pushed to a vector database, which can be done asynchronously via message queues, conditionally triggered pushes, or event-driven pushes, along with timestamps, camera identifiers, and target category labels. The method of vector push is not restricted here.
[0057] Step S230: Based on the spatiotemporal fusion vector in the vector database, perform vector retrieval on the multimodal query content to obtain the retrieval results.
[0058] Specifically, based on spatiotemporal fusion vectors in the vector database, vector retrieval is performed on multimodal query content to obtain retrieval results. This can be achieved by calculating the similarity between the feature vectors obtained after feature extraction from the input image and the spatiotemporal fusion vectors in the vector database, returning the Top N retrieval results with the highest similarity; alternatively, by matching the feature vectors obtained after parsing the input text with the spatiotemporal fusion vectors in the vector database, returning the Top N retrieval results with the highest matching degree; or by accepting both images and text as joint queries, extracting feature vectors from each, performing weighted fusion, and then retrieving them from the vector database. No restrictions are placed on the specific type of multimodal query.
[0059] The above steps first extract keyframe images from the acquired long video, and then perform target detection and pixel-level instance segmentation on these keyframe images to filter out low-quality targets, thereby obtaining purified target regions. Second, feature extraction is performed on the purified target regions to obtain multi-dimensional features. Based on this, the multi-dimensional features are input into a dynamic feature fusion network. This network dynamically allocates fusion weights based on target quality scores, motion intensity, and target proportion, thereby adaptively highlighting highly discriminative features and suppressing heavily interfered features, generating a highly recognizable spatiotemporal fusion vector. Finally, multimodal retrieval matching is performed based on this highly recognizable spatiotemporal fusion vector. By improving the recognizableness of the spatiotemporal fusion vector, it is possible to effectively distinguish vectors in different dynamic scenes such as nighttime, rainy days, and occlusion. Therefore, during vector retrieval, similar targets can be identified based on the highly recognizable spatiotemporal fusion vector. This solves the problem of low retrieval accuracy in dynamic scenes in related technologies.
[0060] The steps described above are explained in detail below:
[0061] In some embodiments, step S210, which involves extracting keyframes from the acquired long video to obtain keyframe images, includes the following steps:
[0062] Based on a preset motion entropy value, keyframes are extracted from long videos to obtain keyframe images; the motion entropy value is used to adjust the extraction interval of keyframe images.
[0063] Specifically, motion entropy can be calculated based on grayscale images or luminance channels, or it can be combined with temporal smoothing of motion history images. The specific calculation method for motion entropy is not limited here. One method is to calculate the ratio of the difference between keyframes to the total number of pixels using a mathematical formula. If the formula method is used, the formula for calculating motion entropy is:
[0064] ;
[0065] Where S is the motion entropy value, H is the difference between the current frame and the previous frame, and T is the total number of pixels. When the motion entropy value is greater than the preset dynamic threshold (e.g., 0.15), the current scene is determined to be a dynamic scene (such as dense traffic or moving crowds). At this time, the keyframe extraction interval is shortened to 1 frame every 10 frames to ensure that fast-moving targets are not missed. When the motion entropy value is less than the preset static threshold (e.g., 0.02), it is determined to be a static scene (such as an empty parking lot at night). The extraction interval is extended to 1 frame every 300 frames to significantly reduce redundant keyframes. When the motion entropy value is between the two, a medium extraction interval (e.g., 1 frame every 50 frames) is used.
[0066] It is understood that the specific values of the threshold and frame interval extracted above can be configured and adjusted according to the actual application scenario, and are not limited to the fixed values listed.
[0067] This embodiment dynamically adjusts the keyframe extraction interval based on motion entropy values, avoiding the problem of generating a large number of redundant frames in dynamic scenes using fixed intervals or simple frame differences in existing technologies, thus reducing redundant targets by 60%. This improves the semantic effectiveness of keyframes and provides high-quality data input for high-precision retrieval in dynamic scenes.
[0068] In some embodiments, step S210 involves target detection and instance segmentation of the keyframe image to obtain a purified target region, such as... Figure 3 As shown, it includes the following steps:
[0069] Step S211: Perform target detection and instance segmentation on the keyframe image to obtain pixel-level target mask regions; and score the quality of the target mask regions.
[0070] Step S212: Filter target mask regions with quality scores below a preset threshold to obtain purified target regions.
[0071] Specifically, keyframe images are input into an improved object detection network (e.g., YOLOv9), which outputs bounding boxes and category labels (e.g., pedestrians, vehicles, non-motorized vehicles). For each bounding box, an instance segmentation network (e.g., Mask R-CNN) performs pixel-level segmentation to generate a mask region for the object. This mask accurately distinguishes target pixels from background pixels, thus removing background redundancy such as ground and walls within the bounding box. Each generated target mask region is given a quality score, calculated by weighting a sharpness score and a completeness score, with a sharpness score weight of 0.7 and a completeness score weight of 0.3. Target mask regions with quality scores below a preset threshold (e.g., 0.6) are classified as low-quality targets (e.g., severely blurred, occluded areas exceeding half, or severely truncated) and filtered out. Target mask regions with quality scores greater than or equal to the preset threshold are retained, resulting in pixel-level purified target regions with category labels. The object detection network can be a YOLOv8 network or an RT-DETR network, and the instance segmentation network can be a SOLOv2 network or a YOLACT network. There are no restrictions on the specific types of object detection networks and instance segmentation networks.
[0072] It is understood that the above thresholds and weight values can be configured and adjusted according to the actual application scenario, and are not limited to the fixed values listed.
[0073] This embodiment performs target detection, pixel-level instance segmentation, and quality scoring filtering sequentially on keyframe images, achieving high-quality extraction of purified target regions from the original image. This ensures that only clear, complete, high-quality targets enter the retrieval process, reducing the risk of false matching.
[0074] In some embodiments, the static features in step S220 include target subject features, key detail features, and environmental association features;
[0075] The target subject features are extracted using a pre-trained visual transformer model;
[0076] Key detail features are extracted using an attention mechanism combined with a pre-trained residual network;
[0077] Environmental features are extracted using a pre-trained lightweight convolutional neural network.
[0078] Specifically, first, the purified target region is input into a pre-trained visual transformer model (e.g., ViT-L / 16), outputting 768-dimensional target subject features. Second, within the purified target region, an attention mechanism is used to automatically focus on discriminative regions such as faces, license plates, and clothing textures. The attention-weighted region is then input into a pre-trained residual network (e.g., ResNet50) to extract 512-dimensional key detail features. Finally, the environmental context region is extracted by expanding outwards by a predetermined percentage (e.g., 30%) from the bounding box of the purified target region. This extracted region is then input into a pre-trained lightweight convolutional neural network (e.g., MobileNet), outputting 256-dimensional environmental association features. These three features can be extracted independently in parallel or share shallow features from the backbone network to reduce computational cost. The visual transformer model can also be ViT-H / 14 or Swin-V2, the residual network can also be ResNet18 or ResNet101, and the lightweight convolutional neural network can also be ShuffleNetV2 or EfficientNetV2. There are no restrictions on the types of visual transformer models, residual networks, and lightweight convolutional neural networks.
[0079] In this embodiment, static features are subdivided into three dimensions: target subject, key details, and environmental association. Each dimension is extracted using an adapted network, enabling the subsequent dynamic fusion network to flexibly adjust the weights of features according to the needs of the scenario, thereby enhancing the robustness of the retrieval.
[0080] In some embodiments, the motion trajectory features in step S220 are calculated using optical flow.
[0081] Behavioral features of consecutive frames are extracted using a pre-trained long short-term memory network.
[0082] Specifically, a sequence of multiple consecutive frames of purified target region images is acquired, and optical flow is used to calculate the motion trajectory, resulting in 128-dimensional motion trajectory features. The optical flow method can be either the dense optical flow Farneback algorithm or the Lucas-Kanade pyramid optical flow algorithm; the specific type of optical flow method is not limited here. The sequence of multiple consecutive frames (e.g., 5 frames) of purified target region images is input into a pre-trained Long Short-Term Memory (LSTM) network, which outputs 256-dimensional continuous frame behavioral features.
[0083] The extracted target features, key detail features, environmental association features, and dynamic temporal features are input into a pre-trained dynamic feature fusion network for fusion, generating a 1024-dimensional spatiotemporal fusion vector with time and location labels. This spatiotemporal fusion vector is then pushed to a vector database. The dynamic feature fusion network can be a Gated Recurrent Unit (GRU) network, a Long Short-Term Memory (LSTM) network, or a bidirectional LSM network; the type of dynamic feature fusion network is not restricted here.
[0084] The fusion weights of the dynamic feature fusion network are dynamically allocated according to the following methods: target quality score (subject feature weight = 0.5 + 0.2 × quality score), motion intensity (temporal feature weight = 0.3 × motion intensity, activated when motion intensity > 0.4), and target proportion (environment feature weight = 0.2 - 0.1 × target proportion).
[0085] In this embodiment, optical flow directly calculates pixel-level motion vectors, capturing subtle displacements and local deformations of the target between frames, providing continuous motion information for motion trajectory features. LSTM, on the other hand, learns the temporal dependencies of features between frames to perform high-level abstraction of the target's behavior patterns. The combination of these two methods ensures that dynamic temporal features contain both precise geometric information of the low-level motion trajectory and semantic information of the high-level behavior patterns, providing crucial discriminative information for high-precision retrieval in dynamic scenes.
[0086] In some of these embodiments, the vector database in step S220 employs a hierarchical storage structure;
[0087] The tiered storage structure includes a first partition and a second partition. The first partition is divided according to the target category, and the second partition is divided according to time information and camera identification.
[0088] Specifically, the vector database employs a two-tiered hierarchical storage architecture. The first partition is divided according to target categories, for example, establishing independent vector sub-databases based on the category labels output during the target detection phase (pedestrians, vehicles, non-motorized vehicles, animals, etc.). The second partition (secondary partition) further subdivides each target category sub-database according to time information and camera identifiers. The vector database supports a hybrid indexing strategy combining Inverted File with Product Quantization (IVF_PQ) and Hierarchical Navigable Small World (HNSW) nearest neighbor search to balance retrieval speed and accuracy. Furthermore, the vector database can receive newly generated spatiotemporal fusion vectors in real time and optimize historical indexes every morning. After receiving a retrieval request, the vector database returns a candidate vector set as required.
[0089] Furthermore, to ensure the security of sensitive data, privacy protection can be achieved through methods such as federated learning to train encrypted models, homomorphic encryption retrieval, or data anonymization. No specific methods for protecting sensitive data are limited here. Federated learning to train encrypted models involves training feature encryption models (based on federated averaging algorithms) for each region using local sensitive data (faces, license plates). The model parameters only synchronize gradient information (excluding the original data), thus ensuring that the model does not leak sensitive information. Homomorphic encryption retrieval uses partial homomorphic encryption technology on sensitive feature vectors such as faces and license plates, calculating vector similarity in the encrypted state (without decryption), and decrypting the thumbnail (not the original features) only when the final result is returned. Data anonymization involves automatically blurring non-target sensitive information (such as blurring other pedestrian faces in the background) in the search results uploaded to the client, retaining only the clear features of the search target.
[0090] In this embodiment, the vector database adopts a hierarchical storage structure, which significantly improves retrieval efficiency compared to existing technologies that simply partition data by category or use a single global index. Simultaneously, the introduction of a hybrid indexing strategy achieves an optimal balance between speed and accuracy with hundreds of millions of data points, thus supporting hierarchical storage, rapid indexing, and incremental updates for large-scale vectors.
[0091] In some embodiments, step S230 involves performing vector retrieval on the multimodal query content based on spatiotemporal fusion vectors in a vector database to obtain retrieval results, such as... Figure 4 As shown, it includes the following steps:
[0092] Step S231: If the multimodal query content is an image query, then extract the target feature vector of the image; and perform vector retrieval on the target feature vector in the vector database to obtain the retrieval result;
[0093] Step S232: If the multimodal query content is a text query, then the attribute feature vector of the text is extracted through the pre-built dedicated word vector library; and vector retrieval is performed on the attribute feature vector in the vector database to obtain the retrieval results.
[0094] Specifically, for image queries, the system extracts the target feature vector from the input query image, calculates the cosine similarity or Euclidean distance between the target feature vector and vectors in the vector database, and returns the Top N search results (e.g., Top 10) with the highest similarity. For text queries, the system first compares the user-input text with a pre-built dedicated word vector library, which contains over 3000 professional terms (such as electric tricycle, unlicensed vehicle, and worker wearing a safety helmet), and encodes the text using the BERT-Base model. The query text is then decomposed using the dedicated word vector library to obtain attribute feature vectors. Based on these attribute feature vectors, the system first performs attribute filtering in the hierarchical storage of the vector database to narrow the search scope. Then, the similarity between the attribute feature vectors and vectors in the vector database is calculated, and optimization is performed based on time sequence or scene, returning the Top N search results. Search results include, but are not limited to, target screenshots, time, camera location, and similarity scores; there are no restrictions on the data types included in the search results.
[0095] Meanwhile, the retrieval process supports online distillation and updating of fusion weights based on user feedback, and also supports scene recognition to adjust fusion weights (e.g., increasing the weight of edge features in nighttime scenes).
[0096] This embodiment designs differentiated retrieval paths for different modalities of query content, fully leveraging the information advantages of each modality. This embodiment achieves unified and efficient retrieval of multimodal queries, significantly improving the accuracy and efficiency of image-based and text-based image searches in dynamic scenarios.
[0097] In some embodiments, step S230 involves performing vector retrieval on the multimodal query content based on the spatiotemporal fusion vector in the vector database to obtain the retrieval results.
[0098] Specifically, first, it is necessary to identify the type of query content. If the multimodal query content is an image query, the input image will be preprocessed, object detection and instance segmentation will be performed, quality score filtering will be performed, multidimensional feature extraction will be performed, and dynamic feature fusion will be performed to generate the target feature vector of the query image. If the multimodal query content is a text query, the text will be semantically encoded using a pre-built word vector library specifically for the monitoring domain, and the target category (such as pedestrians, vehicles), core attributes (such as color, vehicle type, clothing) and scene behavior description (such as location, direction of movement) will be extracted from the query text to generate a structured attribute feature vector.
[0099] Secondly, a three-level matching module is used for retrieval: The first level is attribute filtering, which quickly locates the corresponding partition subset in the hierarchical storage of the vector database based on the parsed target category, time range, camera ID, and other attribute conditions, significantly narrowing the search scope; the second level is vector similarity matching, which uses a hybrid index (IVF_PQ+HNSW) within the candidate partition to calculate the cosine similarity or Euclidean distance between the vector and the spatiotemporal fusion vector in the database, obtaining a preliminary similarity ranking; the third level is time series and scene optimization, which re-ranks the candidate results based on the time series information (such as "9 AM") or scene constraints (such as "crossroads") that may be carried in the query, for example, prioritizing results that match timestamps or camera locations. Finally, a preset number (such as Top 10) of search results are returned according to the similarity score from high to low, and each result includes metadata information such as target screenshot, timestamp, camera location, and similarity score.
[0100] This embodiment unifies the mapping of multimodal query content to the same vector space for retrieval, realizing an integrated architecture for image-based and text-based image searches. It systematically addresses the issue of low retrieval accuracy in dynamic scenarios from three dimensions: multimodal parsing, hierarchical retrieval, and dynamic optimization, significantly improving the robustness of image-based and text-based image searches in real-world monitoring environments.
[0101] The present embodiment will now be described and illustrated through preferred embodiments.
[0102] This embodiment involves four types of core network element devices. The types, forms, and functions of each device are shown in Table 1, which is a network element device information table:
[0103] Table 1
[0104]
[0105] The preprocessing server consists of GPU servers and is mainly used for keyframe image extraction, object detection, and instance segmentation; the feature extraction server consists of GPU servers and is mainly used for feature extraction and dynamic feature fusion; the vector database server consists of 10 distributed servers and is mainly used for hierarchical storage and index building; the retrieval engine server consists of 16-core 64G CPU servers and is mainly used for multimodal retrieval matching and result ranking.
[0106] Figure 5 The diagram illustrates the architecture of the multimodal vector retrieval optimization method, as shown below. Figure 5As shown, text, images, or videos are uploaded, and preprocessing steps such as object detection and instance segmentation are performed on the uploaded data to obtain purified target regions. Features are extracted from the purified target regions to obtain multi-dimensional features, which are then fused using a dynamic feature fusion network to generate a spatiotemporal fusion vector. This spatiotemporal fusion vector is then pushed to a vector database for subsequent vector retrieval. Retrieval can employ various methods such as view retrieval, intelligent question answering, intelligent tagging, and personalized recommendations. For example, if the user inputs text "white SUV turning right at a crossroads at 9 AM," attribute parsing breaks it down into: target category (vehicle), core attributes (white, SUV), scene description (crossroads), and behavior description (turning right). Based on these attributes, the system first performs attribute pre-filtering in the hierarchical storage of the vector database, retrieving only cameras associated with the corresponding attributes, significantly narrowing the search scope. Then, similarity is calculated, and Top N results are returned.
[0107] This embodiment also provides a multimodal vector retrieval optimization system, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "layer," "subunit," etc., used below can refer to combinations of software and / or hardware that achieve a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0108] Figure 6 This is a block diagram of the multimodal vector retrieval optimization system in this embodiment, as shown below. Figure 6 As shown, the system includes: a preprocessing layer 210, a feature extraction layer 220, a vector library storage layer 230, and a retrieval engine layer 240.
[0109] The preprocessing layer 210 is used to extract keyframes from the acquired long video to obtain keyframe images; target detection and instance segmentation are performed on the keyframe images to obtain purified target regions;
[0110] Feature extraction layer 220 is used to extract features from the purified target region to obtain multi-dimensional features. The multi-dimensional features include static features and dynamic temporal features. The dynamic temporal features include motion trajectory features and continuous frame behavior features. The multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector. The spatiotemporal fusion vector is then pushed to a vector database. The fusion weights of the dynamic feature fusion network are dynamically allocated based on the target quality score, motion intensity, and target proportion.
[0111] Vector library storage layer 230 is used to store spatiotemporal fusion vectors;
[0112] The retrieval engine layer 240 is used to perform vector retrieval on multimodal query content based on spatiotemporal fusion vectors in the vector database to obtain retrieval results.
[0113] The above system solves the problem of low retrieval accuracy in dynamic scenarios in related technologies, and improves retrieval accuracy in dynamic scenarios.
[0114] In some embodiments, the preprocessing layer 210 is also used to extract keyframes from a long video based on a preset motion entropy value to obtain keyframe images; the motion entropy value is used to adjust the extraction interval of the keyframe images.
[0115] In some embodiments, the preprocessing layer 210 is also used to perform target detection and instance segmentation on the keyframe image to obtain pixel-level target mask regions; and to perform quality scoring on the target mask regions;
[0116] The target mask region with a filter quality score lower than a preset threshold is obtained to obtain the purified target region.
[0117] In some of these embodiments, static features include target subject features, key detail features, and environmental association features;
[0118] The target subject features are extracted using a pre-trained visual transformer model;
[0119] Key detail features are extracted using an attention mechanism combined with a pre-trained residual network;
[0120] Environmental features are extracted using a pre-trained lightweight convolutional neural network.
[0121] In some of these embodiments, motion trajectory features are calculated using optical flow; continuous frame behavior features are extracted using a pre-trained long short-term memory network.
[0122] In some of these embodiments, the vector database employs a hierarchical storage structure;
[0123] The tiered storage structure includes a first partition and a second partition. The first partition is divided according to the target category, and the second partition is divided according to time information and camera identification.
[0124] In some embodiments, the retrieval engine layer 240 is also used to extract the target feature vector of the image if the multimodal query content is an image query; and to perform vector retrieval on the target feature vector in the vector database to obtain the retrieval result;
[0125] If the multimodal query is a text query, the attribute feature vectors of the text are extracted using a pre-built dedicated word vector library; and vector retrieval is performed on the attribute feature vectors in the vector database to obtain the retrieval results.
[0126] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.
[0127] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0128] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0129] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:
[0130] S1. Keyframe extraction is performed on the acquired long video to obtain keyframe images. Object detection and instance segmentation are then performed on the keyframe images to obtain purified target regions.
[0131] S2, feature extraction is performed on the purification target region to obtain multi-dimensional features; multi-dimensional features include static features and dynamic temporal features; dynamic temporal features include motion trajectory features and continuous frame behavior features;
[0132] Multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector; the spatiotemporal fusion vector is then pushed into a vector database; the fusion weights of the dynamic feature fusion network are dynamically allocated based on the target quality score, motion intensity, and target proportion.
[0133] S3, based on the spatiotemporal fusion vector in the vector database, performs vector retrieval on the multimodal query content to obtain the retrieval results.
[0134] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.
[0135] Furthermore, in conjunction with the multimodal vector retrieval optimization method provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the multimodal vector retrieval optimization methods described in the above embodiments.
[0136] It should be noted that all information and data involved in this application are authorized by the user or fully authorized by all parties and will be used legally.
[0137] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.
[0138] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.
[0139] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0140] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.
Claims
1. A multimodal vector retrieval optimization method, characterized in that, include: Keyframes are extracted from the acquired long video to obtain keyframe images; Target detection and instance segmentation are performed on the keyframe image to obtain the purified target region; Feature extraction is performed on the purification target region to obtain multi-dimensional features; the multi-dimensional features include static features and dynamic temporal features; the dynamic temporal features include motion trajectory features and continuous frame behavior features. The multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector; the spatiotemporal fusion vector is then pushed into a vector database; the fusion weights of the dynamic feature fusion network are dynamically allocated based on the target quality score, motion intensity, and target proportion. Based on the spatiotemporal fusion vector in the vector database, vector retrieval is performed on the multimodal query content to obtain the retrieval results.
2. The multimodal vector retrieval optimization method according to claim 1, characterized in that, The step of extracting keyframes from the acquired long video to obtain keyframe images includes: Based on a preset motion entropy value, keyframes are extracted from the long video to obtain the keyframe images; the motion entropy value is used to adjust the extraction interval of the keyframe images.
3. The multimodal vector retrieval optimization method according to claim 1, characterized in that, The step of performing target detection and instance segmentation on the keyframe image to obtain a purified target region includes: Target detection and instance segmentation are performed on the keyframe image to obtain pixel-level target mask regions; and the target mask regions are then scored for quality. The target mask regions with quality scores below a preset threshold are filtered out to obtain the purified target regions.
4. The multimodal vector retrieval optimization method according to claim 1, characterized in that, The static features include target subject features, key detail features, and environmental related features; The target subject features are extracted using a pre-trained visual transformer model; The key detail features are extracted using an attention mechanism combined with a pre-trained residual network; The environmental features are extracted using a pre-trained lightweight convolutional neural network.
5. The multimodal vector retrieval optimization method according to claim 1, characterized in that, The motion trajectory features were calculated using the optical flow method. The continuous frame behavioral features are extracted using a pre-trained long short-term memory network.
6. The multimodal vector retrieval optimization method according to claim 1, characterized in that, The vector database adopts a hierarchical storage structure; The hierarchical storage structure includes a first partition and a second partition. The first partition is divided according to the target category, and the second partition is divided according to time information and camera identification.
7. The multimodal vector retrieval optimization method according to claim 1, characterized in that, The method of performing vector retrieval on multimodal query content based on the spatiotemporal fusion vector in the vector database to obtain retrieval results includes: If the multimodal query content is an image query, then the target feature vector of the image is extracted; and in the vector database, the target feature vector is used for vector retrieval to obtain the retrieval result; If the multimodal query content is a text query, then the attribute feature vectors of the text are extracted through a pre-built dedicated word vector library; and vector retrieval is performed on the attribute feature vectors in the vector database to obtain the retrieval results.
8. A multimodal vector retrieval optimization system, characterized in that, include: Preprocessing layer, feature extraction layer, vector library storage layer, and retrieval engine layer; The preprocessing layer is used to extract keyframes from the acquired long video to obtain keyframe images. Target detection and instance segmentation are performed on the keyframe image to obtain the purified target region; The feature extraction layer is used to extract features from the purified target region to obtain multi-dimensional features. The multi-dimensional features include static features and dynamic temporal features. The dynamic temporal features include motion trajectory features and continuous frame behavior features. The multi-dimensional features are input into a pre-trained dynamic feature fusion network to generate a spatiotemporal fusion vector. The spatiotemporal fusion vector is then pushed to a vector database. The fusion weights of the dynamic feature fusion network are dynamically allocated based on the target quality score, motion intensity, and target proportion. The vector library storage layer is used to store the spatiotemporal fusion vector; The retrieval engine layer is used to perform vector retrieval on multimodal query content based on the spatiotemporal fusion vector in the vector database to obtain retrieval results.
9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the multimodal vector retrieval optimization method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the multimodal vector retrieval optimization method according to any one of claims 1 to 7.
