Distributed multi-modal retrieval method and system based on edge video feature preprocessing, equipment and medium

Abstract

The present application relates to the technical field of video data processing and information retrieval, in particular to a distributed multi-modal retrieval method and system based on edge video feature preprocessing, equipment and medium, by using the method provided by the present application, the localization feature extraction and retrieval of the edge device is adopted, the cloud data transmission and computing pressure are reduced, meanwhile, the cross-device task scheduling and result fusion are realized by using the cloud, the efficiency and global coverage are balanced. Timestamp association storage: the feature vector is bound with the video timestamp, the target event can be quickly positioned, and the redundant calculation caused by the full video playback is avoided. Edge multi-modal feature vector query: the edge receives the matching of the multi-modal feature vector of the cloud, the large amount of pre-trained knowledge of the multi-modal model is effectively utilized, and diversified intelligent query mode is provided. Retrieval efficiency is improved: the response time of the edge device local retrieval is reduced from minutes to seconds, which is especially suitable for real-time event tracing in public security scenarios.

Description

Technical Field

[0001] This invention relates to the field of video data processing and information retrieval technology, and more specifically, to a distributed multimodal retrieval method, system, device, and medium based on edge video feature preprocessing. Background Technology

[0002] Traditional video retrieval technologies are mainly divided into text-based retrieval (relying on metadata or manual annotation) and content-based retrieval (CBVR). CBVR achieves retrieval by analyzing visual (keyframes, color / texture features), motion (optical flow, trajectory), or audio (MFCC) features, and uses clustering, classification, or hashing encoding for matching. Traditional methods rely on handcrafted features (such as SIFT, LBP).

[0003] Traditional video retrieval technology relies on centralized cloud processing. Existing technologies require video data to be uploaded to the cloud for feature extraction and retrieval, resulting in high network bandwidth pressure, high latency, difficulty in meeting real-time requirements, and high computing and storage costs. Centralized cloud processing of massive amounts of video data consumes a lot of computing power and is very costly. Summary of the Invention

[0004] The purpose of this invention is to provide a distributed multimodal retrieval method, system, device, and medium based on edge video feature preprocessing to solve the above-mentioned problems in the prior art.

[0005] This invention is achieved through the following technical solution: A distributed multimodal retrieval method based on edge video feature preprocessing includes: Keyframe features of historical videos are obtained, target vectors are generated through a pre-trained multimodal encoder, feature vectors with timestamps are generated from the target vectors, a list of time points is established, and historical videos are stored in the list of time points according to the timestamps of the feature vectors of each historical video. The feature vector is stored in a local or regionally distributed database on the edge device. The current input retrieval data is obtained, and the retrieval task is sent to several edge devices. The matching retrieval is performed in the database, and the similarity is calculated to obtain the target feature vector with the highest similarity to the retrieval data. Based on the timestamp of the current target feature vector, locate the position of the target feature vector in the list of time points, and upload the historical video corresponding to the current target feature vector to the cloud.

[0006] Preferably, the generation of the target vector through the pre-trained multimodal encoder includes: The extracted keyframe features are preprocessed and adjusted to the set size of the pre-trained model within the multimodal encoder. Select a multimodal encoder and perform multimodal extension; The keyframe features include visual features and text features. The visual features and text features are concatenated and input into the fully connected layer of the pre-trained model. A cross-modal attention layer is set up. Visual features and text features interact through the cross-modal attention layer. The weights of each modality are dynamically calculated. Based on the weights, the visual features and text features are weighted and fused, and the fused features are output. The target vector is generated based on the fused features using a multimodal encoder.

[0007] Preferably, the step of generating a timestamped feature vector from the target vector includes: Construct a time encoder to generate the current timestamp encoding vector; The Transformer model is used to fuse the timestamp encoded vector and the target vector, and output a feature vector with timestamp marking.

[0008] Preferably, obtaining the currently input search data includes: The retrieved data can be classified as text data, image data, or video clip data. If the data is text, LLM is used to perform intent understanding, keyword extraction, or query expansion on the natural language text input by the user. The processed text is then input into the same multimodal encoder as the edge to generate query vectors. If the data is image data, the user-uploaded image is preprocessed and input into the multimodal encoder to generate a query vector; If the data is video clips, keyframe sampling is performed on the uploaded video clips, the feature vector of each keyframe is extracted, and the feature vectors of multiple keyframes are aggregated to generate multiple query vectors representing the video clip.

[0009] Preferably, the similarity calculation includes: Perform vector space alignment, setting the feature vectors stored in the database at the edge and the query vector in the same semantic space; The similarity between the query vector and the feature vector in the database is calculated at the edge:

[0010] In the formula, To score the similarity, For query vector, These are the eigenvectors.

[0011] Preferably, obtaining the target feature vector with the highest similarity to the retrieved data includes: Determine if the current target feature vector is a single vector; if it is, output the target feature vector normally. If there is more than one, then obtain several feature vectors with the same similarity, and obtain the original visual features and text features of the current feature vector; Visual features and textual features are used to generate a first feature vector and a second feature vector, respectively, through a multimodal encoder; A similarity comparison model is established, and a comprehensive similarity is calculated based on the first feature vector, the second feature vector, and the query vector. The feature vector corresponding to the highest similarity value is taken as the target feature vector. If there are still multiple highest similarity values, then the feature vectors corresponding to all the highest similarity values ​​are taken as the target feature vector.

[0012] Preferably, the establishment of the similarity comparison model includes:

[0013] In the formula, To assess overall similarity, and All of these are calculations of weights. For visual vectors, For text vectors, when the original data of the query vector is a video or image, then... ,but When the original data of the query vector is text, then ,but , Greater than 1.

[0014] Secondly, the present invention provides a distributed multimodal retrieval system based on edge-end video feature preprocessing, comprising: The data acquisition module is configured to acquire keyframe features of historical videos, generate target vectors through a pre-trained multimodal encoder, generate feature vectors with timestamps from the target vectors, establish a time point list, and store historical videos in the time point list according to the timestamps of the feature vectors of each historical video. The matching module is configured to store feature vectors in a local or regionally distributed database on the edge device, obtain the currently input retrieval data, distribute the retrieval task to several edge devices, perform matching retrieval in the database, calculate similarity, and obtain the target feature vector with the highest similarity to the retrieval data. The output module is configured to find the position of the target feature vector in the time point list based on the timestamp of the current target feature vector, and upload the historical video corresponding to the current target feature vector to the cloud. The main control module, connected to the data acquisition module, matching module, and output module, is used to execute the above-described distributed multimodal retrieval method based on edge video feature preprocessing.

[0015] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described distributed multimodal retrieval method based on edge video feature preprocessing.

[0016] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the distributed multimodal retrieval method based on edge video feature preprocessing as described above.

[0017] The technical solution of the present invention has at least the following advantages and beneficial effects: The method provided by this invention reduces the data transmission and computing pressure in the cloud by extracting and retrieving local features on edge devices, while using the cloud to achieve cross-device task scheduling and result fusion, thus balancing efficiency and global coverage.

[0018] Timestamp-associated storage: Binding feature vectors to video timestamps enables rapid location of target events and avoids redundant calculations during full video playback.

[0019] Edge multimodal feature vector query: The edge receives and matches multimodal feature vectors from the cloud, effectively utilizing the rich pre-trained knowledge of the multimodal model to provide diverse and intelligent query methods.

[0020] Improved search efficiency: Local search on edge devices reduces response time from minutes to seconds, making it particularly suitable for real-time event tracing in public safety scenarios.

[0021] Reduced computing power costs: Edge devices handle more than 90% of the feature extraction and retrieval load, while the cloud only needs to handle lightweight scheduling and sorting tasks such as user input; the edge only needs to deploy a visual encoder and does not need to deploy an LLM, which significantly reduces the computing power consumption at the edge. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1This is a schematic diagram of the process of the present invention; Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] The module division in this application is a logical division. In actual application, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not executed.

[0026] The independently described modules or sub-modules may or may not be physically separated; they may be implemented in software or hardware, and some modules or sub-modules may be implemented in software, with the processor calling the software to implement the function of these modules or sub-modules, while other modules or sub-modules may be implemented in hardware, such as through hardware circuits. Furthermore, some or all of the modules can be selected to achieve the purpose of this application's solution according to actual needs.

[0027] Please refer to Figures 1-2 A distributed multimodal retrieval method based on edge video feature preprocessing includes: S101: Obtain keyframe features from historical videos, generate target vectors through a pre-trained multimodal encoder, generate time-stamped feature vectors from the target vectors, establish a time point list, and store historical videos in the time point list according to the timestamp of the feature vector of each historical video. In real time, keyframe features of video are extracted from edge hardware (such as camera terminals), and high-dimensional vectors are generated using a pre-trained multimodal encoder to generate compact feature vectors with timestamps.

[0028] S102: Store the feature vector in a local or regionally distributed database on the edge device, obtain the currently input retrieval data, distribute the retrieval task to several edge devices, perform matching retrieval in the database, and calculate the similarity to obtain the target feature vector with the highest similarity to the retrieval data. Storing feature vectors locally on edge devices or in regionally distributed databases reduces data redundancy in the cloud. It supports text, images, and video clips as retrieval inputs, and quickly matches the edge-stored feature vectors through cross-modal similarity calculation (such as cosine similarity) to return a list of time points of relevant video clips.

[0029] S103: Based on the timestamp of the current target feature vector, find the position of the target feature vector in the time point list, and upload the historical video corresponding to the current target feature vector to the cloud.

[0030] The cloud system distributes retrieval tasks to multiple edge devices (device selection can be dynamically generated by the Agent program based on user input), aggregates the results returned by each device, and sorts them to achieve efficient global retrieval.

[0031] The method provided by this invention reduces the data transmission and computing pressure in the cloud by extracting and retrieving local features on edge devices, while using the cloud to achieve cross-device task scheduling and result fusion, thus balancing efficiency and global coverage.

[0032] Timestamp-associated storage: Binding feature vectors to video timestamps enables rapid location of target events and avoids redundant calculations during full video playback.

[0033] Edge multimodal feature vector query: The edge receives and matches multimodal feature vectors from the cloud, effectively utilizing the rich pre-trained knowledge of the multimodal model to provide diverse and intelligent query methods.

[0034] Improved search efficiency: Local search on edge devices reduces response time from minutes to seconds, making it particularly suitable for real-time event tracing in public safety scenarios.

[0035] Reduced computing power costs: Edge devices handle more than 90% of the feature extraction and retrieval load, while the cloud only needs to handle lightweight scheduling and sorting tasks such as user input; the edge only needs to deploy a visual encoder and does not need to deploy an LLM, which significantly reduces the computing power consumption at the edge.

[0036] In one exemplary embodiment of the present invention, generating the target vector through a pre-trained multimodal encoder includes: S201: Preprocess the extracted keyframe features and adjust the keyframe features to the set size of the pre-trained model in the multimodal encoder; Use tools such as OpenCV to detect and extract keyframes at fixed intervals (e.g., every 5 seconds) or during dynamic scene changes to avoid redundancy; and uniformly adjust the frames to the size required by the pre-trained model (e.g., CLIP needs to be 224×224 pixels) and perform normalization processing.

[0037] S202: Select a multimodal encoder and perform multimodal extension; The visual encoder can be CLIP-ViT-B / 32: based on the Vision Transformer, it processes images in blocks (32×32 pixels) and outputs a 512-dimensional vector. Alternatively, it can choose ViT-G / 14: a higher-precision model (such as the one used by VideoPoet), which supports 4K resolution block processing and outputs high-dimensional visual features.

[0038] Multimodal extensions include cross-modal alignment: such as the UNITER model, which fuses text and image features through a Transformer to generate a joint embedding vector. It also includes dynamic gating mechanisms: VideoPoet's fusion tower structure, which uses cross-attention to control the weights of text / image / audio features.

[0039] S203: The keyframe features include visual features and text features. The visual features and text features are spliced ​​together and input into the fully connected layer of the pre-trained model. Among them, visual features: visual encoder (keyframe) → high-dimensional vector (such as ViT output 4096 dimensions or CLIP output 512 dimensions).

[0040] Text features: Use a Sentence2Vector or Transformer encoder to generate text vectors (to assist cross-modal alignment).

[0041] S204: Set up a cross-modal attention layer, interact visual features and text features through the cross-modal attention layer, dynamically calculate the weights of each modality, and output the fused features after weighted fusion of visual features and text features based on the weights; Early fusion: combining visual and textual features, inputting to a fully connected layer; Hierarchical fusion: such as hierarchical encoders, which interact visual and textual features through cross-modal attention layers; Gated weighting: Dynamically calculate the weights of each modality (such as the sigmoid generation ratio s) and weighted fuse features.

[0042] S205: Generate a target vector based on the fused features using a multimodal encoder.

[0043] In one exemplary embodiment of the present invention, generating a timestamped feature vector from a target vector includes: Construct a time encoder to generate the current timestamp encoding vector; The Transformer model is used to fuse the timestamp encoded vector and the target vector, and output a feature vector with timestamp marking.

[0044] Specifically, this includes preprocessing the input vector, unifying the vector space of different modalities, performing temporal position encoding enhancement, injecting temporal sequence information, setting a feature fusion gating mechanism, and adaptively adjusting the modality contribution weights.

[0045] In one exemplary embodiment of the present invention, obtaining the currently input search data includes: The retrieved data can be classified as text data, image data, or video clip data. If the data is text, LLM is used to perform intent understanding, keyword extraction, or query expansion on the natural language text input by the user. The processed text is then input into the same multimodal encoder as the edge to generate query vectors. If the data is image data, the user-uploaded image is preprocessed and input into the multimodal encoder to generate a query vector; If the data is video clips, keyframe sampling is performed on the uploaded video clips, the feature vector of each keyframe is extracted, and the feature vectors of multiple keyframes are aggregated to generate multiple query vectors representing the video clip.

[0046] In one exemplary embodiment of the present invention, the similarity calculation includes: The core of cross-modal similarity calculation lies in mapping inputs from different modalities (such as text, images, and video clips) to the same semantic vector space, and achieving cross-modal retrieval through vector similarity matching.

[0047] Perform vector space alignment, setting the feature vectors stored in the database at the edge and the query vector in the same semantic space; The similarity between the query vector and the feature vector in the database is calculated at the edge:

[0048] In the formula, To score the similarity, For query vector, These are the eigenvectors.

[0049] For example, if the cosine similarity between a visual vector in the edge database and the text vector of "red coat" is 0.92 (the highest score), then the video clip with the corresponding timestamp will be returned.

[0050] Results generation: Returns the Top-K similarity score and corresponding timestamp (e.g., [timestamp: 15:03:21, score: 0.92]).

[0051] The above multimodal encoders (text encoder and visual encoder) dynamically select the dedicated encoder to use for processing based on the input modality (text or image) and generate vectors in the same vector space (same dimension); the alignment of output vectors of different modalities needs to be ensured by extensive image and text training in advance; the actual implementation uses publicly available pre-trained Clip model encoders.

[0052] In one exemplary embodiment of the present invention, obtaining the target feature vector with the highest similarity to the retrieved data includes: Determine if the current target feature vector is a single vector; if it is, output the target feature vector normally. If there is more than one, then obtain several feature vectors with the same similarity, and obtain the original visual features and text features of the current feature vector; Visual features and textual features are used to generate a first feature vector and a second feature vector, respectively, through a multimodal encoder; A similarity comparison model is established, and a comprehensive similarity is calculated based on the first feature vector, the second feature vector, and the query vector. The feature vector corresponding to the highest similarity value is taken as the target feature vector. If there are still multiple highest similarity values, then the feature vectors corresponding to all the highest similarity values ​​are taken as the target feature vector.

[0053] Specifically, establishing a similarity comparison model includes:

[0054] In the formula, To assess overall similarity, and All of these are calculations of weights. For visual vectors, For text vectors, when the original data of the query vector is a video or image, then... ,but When the original data of the query vector is text, then ,but , Greater than 1.

[0055] The above method was used to further analyze cases where there was more than one maximum value. By querying the attributes of the vector, multiple feature vectors were further identified, making the final search results more accurate.

[0056] Secondly, the present invention provides a distributed multimodal retrieval system based on edge-end video feature preprocessing, comprising: The data acquisition module is configured to acquire keyframe features of historical videos, generate target vectors through a pre-trained multimodal encoder, generate feature vectors with timestamps from the target vectors, establish a time point list, and store historical videos in the time point list according to the timestamps of the feature vectors of each historical video. The matching module is configured to store feature vectors in a local or regionally distributed database on the edge device, obtain the currently input retrieval data, distribute the retrieval task to several edge devices, perform matching retrieval in the database, calculate similarity, and obtain the target feature vector with the highest similarity to the retrieval data. The output module is configured to find the position of the target feature vector in the time point list based on the timestamp of the current target feature vector, and upload the historical video corresponding to the current target feature vector to the cloud.

[0057] The steps for implementing this method in conjunction with a specific hardware system are as follows: 1. Hardware environment preparation: Edge Computing: Deploy edge computing nodes with some computing power (such as embedded devices equipped with NPU / GPUs or edge servers). Each node connects to one or more cameras. Ensure that the nodes have a stable network connection (wired or wireless).

[0058] Cloud: Prepare cloud server resources, including compute instances, storage, and network bandwidth.

[0059] Network configuration: Ensure unimpeded and secure network communication between edge nodes and cloud servers, as well as between edge nodes themselves (if necessary).

[0060] 2. Software environment deployment: At the edge: Install the operating system (such as a Linux distribution); deploy video stream access modules (such as using RTSP or ONVIF protocols to access camera video streams); deploy video frame processing libraries (such as OpenCV); deploy pre-trained multimodal encoder models (semantically aligned visual encoders with MLP); deploy local vector databases (such as a standalone or lightweight implementation of Milvus); deploy an agent that communicates with the cloud, responsible for receiving tasks, uploading results, and monitoring heartbeat status.

[0061] In the cloud: Deploy a central control service responsible for managing edge node registration, status monitoring, and task distribution. Deploy a user interface service (Web UI or API) to receive user search requests; deploy an Agent query processing module (with predefined prompts and available tools) to interact with the multimodal model to generate the final list of distributed devices and query vectors; deploy the multimodal model, including the same multimodal encoder (for converting user image or video input) and LLM (for query understanding or expansion) as the edge device; and deploy a result aggregation and sorting module.

[0062] 3. Model Loading and Initialization: Edge: Load the pre-trained multimodal encoder model into memory or GPU memory. Initialize the local vector database and create the index structure.

[0063] Cloud: Load the corresponding multimodal model and LLM.

[0064] 4. Edge node registration: After the edge node is started, it sends a registration request to the cloud central control service through the agent program, carrying its own ID, geographical location (optional), and metadata such as the coverage camera information.

[0065] The cloud service authenticates and records information about active edge nodes.

[0066] 5. Video stream access and keyframe sampling: Edge nodes acquire video streams in real time from connected cameras via a video stream access module; sampling at fixed time intervals: for example, every second, a frame is decoded and extracted from the video stream as a keyframe; adaptive sampling based on content changes: monitoring inter-frame differences and extracting keyframes only when the scene changes significantly (such as the appearance of moving objects or scene switching) to reduce redundant calculations.

[0067] 6. Keyframe preprocessing: Perform necessary preprocessing on the extracted keyframes, such as resizing them to the input size required by the model and normalizing the pixel values.

[0068] 7. Visual Feature Vector Extraction: The preprocessed keyframe image is input into the loaded multimodal encoder (which has been semantically aligned). The encoder outputs a high-dimensional visual feature vector for the keyframe.

[0069] 8. Timestamp association: Get the precise timestamp of the current keyframe in the original video stream.

[0070] 9. Feature vector and timestamp storage: The visual feature vector generated in step 7 and the timestamp obtained in step 8 are used as a data pair.

[0071] This data pair is inserted into the local vector database of the edge node. The vector database will then index the vector for fast similarity searching. 10. Status Reporting: Edge nodes periodically send heartbeat packets to the cloud via a program to report their operational status (such as processing frame rate, storage usage, network conditions, etc.).

[0072] 11 users submitted search requests: Users submit search requests through user interfaces provided in the cloud (such as web pages, mobile apps, or APIs).

[0073] The request can be: Text description: For example, "Find the person wearing a red coat who appeared at the entrance of Area A around 3 PM."

[0074] Example image: Upload an image containing the target object or scene.

[0075] Video clip: Upload a short video clip containing the target event.

[0076] 12. Cloud query preprocessing: The cloud service receives a user request and, through the agent program, transforms the original user request into a structure that the model can input. The agent program also provides predefined prompts and tools (e.g., edge device information query tools, other available tools). Based on the intent, it generates a specific list of devices for task distribution (which can be achieved by calling tools, such as retrieving relevant devices at the entrance of area A) and a query vector, as follows. Text processing: Using LLM, intent understanding, keyword extraction, or query expansion are performed on the natural language text input by the user. The processed text (or the original text) is then input into the same multimodal encoder as the edge processor to generate query vectors.

[0077] Image processing: Preprocessing the user-uploaded image. Inputting the image into a multimodal encoder to generate a query vector.

[0078] Video clip processing: Keyframe sampling is performed on the uploaded video clips, feature vectors of each keyframe are extracted, and feature vectors of multiple keyframes are aggregated to generate multiple query vectors representing the video clip.

[0079] 13. Target edge node selection and task distribution: The cloud-based central control service, based on the spatiotemporal information (such as the specified time range and geographical region) and / or global context in the query request, filters the target edge devices from the registered edge node list to perform this retrieval task. If there are no spatiotemporal restrictions or a global search is required, the task can be distributed to all relevant or all active edge nodes. The query vector generated in step 12, along with other retrieval parameters (such as the required number of Top-K results and time range restrictions), is packaged into a retrieval task. The retrieval task is then distributed to one or more selected edge nodes via the network.

[0080] 14. Perform similarity retrieval at the edge: The target edge node receives a retrieval task from the cloud; it parses the query vector and retrieval parameters from the task; it performs a similarity search in the local vector database: it calculates the similarity (such as cosine similarity) between the query vector and all (or within a specified time range) feature vectors stored in the database; based on the similarity score, it finds the top-K most similar feature vectors; and it obtains the timestamps corresponding to these top-K feature vectors.

[0081] 15. Edge-end results returned: Edge nodes bind the retrieved Top-K results (usually a list of (timestamp, similarity score) results). These results are then uploaded to the cloud-based central control service via a proxy program.

[0082] 16. Cloud-based result aggregation and sorting: The cloud service collects the results returned by all edge nodes that participated in the retrieval.

[0083] Merged Results: Results from different edge devices are aggregated into a single list. Each result needs to include the source device ID, timestamp, and similarity score.

[0084] Global sorting: Sort all results globally in descending order based on similarity scores.

[0085] Deduplication and Filtering (Optional): If there is overlap in coverage areas across devices or redundant results, perform necessary deduplication. Filtering can be based on timestamps and device location information.

[0086] Generate the final global search results list.

[0087] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer software product, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0089] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed multimodal retrieval method based on edge video feature preprocessing, characterized in that, include: Keyframe features of historical videos are obtained, target vectors are generated through a pre-trained multimodal encoder, feature vectors with timestamps are generated from the target vectors, a list of time points is established, and historical videos are stored in the list of time points according to the timestamps of the feature vectors of each historical video. The feature vector is stored in a local or regionally distributed database on the edge device. The current input retrieval data is obtained, and the retrieval task is sent to several edge devices. The matching retrieval is performed in the database, and the similarity is calculated to obtain the target feature vector with the highest similarity to the retrieval data. Find the position of the target feature vector in the time point list based on the timestamp of the current target feature vector, and upload the historical video corresponding to the current target feature vector to the cloud. The target feature vector that has the highest similarity to the retrieved data includes: Determine if the current target feature vector is a single vector; if it is, output the target feature vector normally. If there is more than one, then obtain several feature vectors with the same current similarity, and obtain the original visual features and text features of the current feature vector; Visual features and textual features are used to generate a first feature vector and a second feature vector, respectively, through a multimodal encoder; A similarity comparison model is established, and a comprehensive similarity is calculated based on the first feature vector, the second feature vector, and the query vector. Take the feature vector corresponding to the highest similarity value as the target feature vector. If there are still multiple highest similarity values, then take the feature vectors corresponding to all the highest similarity values ​​as the target feature vector. The establishment of the similarity comparison model includes: In the formula, To assess overall similarity, and All of these are calculations of weights. For query vector, For visual vectors, For text vectors, when the original data of the query vector is a video or image, then... ,but When the original data of the query vector is text, then ,but , Greater than 1.

2. The distributed multimodal retrieval method based on edge-end video feature preprocessing according to claim 1, characterized in that, The generation of the target vector through the pre-trained multimodal encoder includes: The extracted keyframe features are preprocessed and adjusted to the set size of the pre-trained model within the multimodal encoder. Select a multimodal encoder and perform multimodal extension; The keyframe features include visual features and text features. The visual features and text features are concatenated and input into the fully connected layer of the pre-trained model. A cross-modal attention layer is set up. Visual features and text features interact through the cross-modal attention layer. The weights of each modality are dynamically calculated. Based on the weights, the visual features and text features are weighted and fused, and the fused features are output. The target vector is generated based on the fused features using a multimodal encoder.

3. The distributed multimodal retrieval method based on edge video feature preprocessing according to claim 2, characterized in that, The process of generating a timestamp-marked feature vector from a target vector includes: Construct a time encoder to generate the current timestamp encoding vector; The Transformer model is used to fuse the timestamp encoded vector and the target vector, and output a feature vector with timestamp marking.

4. The distributed multimodal retrieval method based on edge video feature preprocessing according to claim 3, characterized in that, The process of obtaining the currently input search data includes: The retrieved data can be classified as text data, image data, or video clip data. If the data is text, LLM is used to perform intent understanding, keyword extraction, or query expansion on the natural language text input by the user. The processed text is then input into the same multimodal encoder as the edge to generate query vectors. If the data is image data, the user-uploaded image is preprocessed and input into the multimodal encoder to generate a query vector; If the data is video clips, keyframe sampling is performed on the uploaded video clips, the feature vector of each keyframe is extracted, and the feature vectors of multiple keyframes are aggregated to generate multiple query vectors representing the video clip.

5. The distributed multimodal retrieval method based on edge-end video feature preprocessing according to claim 4, characterized in that, The similarity calculation includes: Perform vector space alignment, setting the feature vectors stored in the database at the edge and the query vector in the same semantic space; The similarity between the query vector and the feature vector in the database is calculated at the edge: In the formula, To score the similarity, For query vector, These are the eigenvectors.

6. A distributed multimodal retrieval system based on edge video feature preprocessing, characterized in that, include: The data acquisition module is configured to acquire keyframe features of historical videos, generate target vectors through a pre-trained multimodal encoder, generate feature vectors with timestamps from the target vectors, establish a time point list, and store historical videos in the time point list according to the timestamps of the feature vectors of each historical video. The matching module is configured to store feature vectors in a local or regionally distributed database on the edge device, obtain the currently input retrieval data, distribute the retrieval task to several edge devices, perform matching retrieval in the database, calculate similarity, and obtain the target feature vector with the highest similarity to the retrieval data. The output module is configured to find the position of the target feature vector in the time point list based on the timestamp of the current target feature vector, and upload the historical video corresponding to the current target feature vector to the cloud; The main control module, connected to the data acquisition module, matching module, and output module, is used to execute the distributed multimodal retrieval method based on edge video feature preprocessing as described in any one of claims 1-5.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the distributed multimodal retrieval method based on edge video feature preprocessing as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the distributed multimodal retrieval method based on edge video feature preprocessing as described in any one of claims 1-5.

Images