A video query method based on multi-modal embedding, electronic device, medium

Abstract

The application discloses a video query method based on multi-modal embedding, an electronic device and a medium. The method first filters a video stream to obtain an anchor frame set, and stores anchor frame visual embedding, question text embedding and a first detection result processed by an adapter into a query cache. When responding to a user question, a candidate frame set is constructed by selecting a relevant anchor frame subset and a time neighborhood thereof, and a current frame and text embedding are distributed to a corresponding adapter through a query route to obtain a second detection result. Finally, the second detection result is subjected to deviation inspection by using adjacent anchor frame data in the cache, and the result after the deviation inspection is taken as a final result. Through anchor frame clipping of a search space, the application realizes efficient, accurate and time-consistent processing of various video query tasks in combination with an adapter and a deviation inspection mechanism.

Description

Technical Field

[0001] This invention belongs to the field of data query technology, and particularly relates to a video query method, electronic device, and medium based on multimodal embedding. Background Technology

[0002] With the rapid growth of video data in fields such as smart cities, security monitoring, and healthcare, the demand for analysis of large-scale video data is increasing rapidly. In real-world business scenarios, video queries exhibit highly diverse forms: one type is structured SQL-style predicate queries, such as filtering based on object categories (e.g., "find frames containing buses"), statistics and aggregation (e.g., "count the number of vehicles passing through an intersection in a certain time period"), and Top-k searches; the other type is natural language queries, such as "find road segments where both buses and trucks appear" or "retrieve clips of pedestrians suddenly running."

[0003] Currently, video database systems (VDBMS) face core challenges in supporting the aforementioned various query types, namely, enormous computational overhead and fragmented query formats. On the one hand, complex semantic analysis often relies on large deep neural networks (DNNs) for detection and tracking. Frame-by-frame inference in massive video and high frame rate scenarios consumes a large amount of GPU resources, making it difficult to meet the latency requirements of online queries. On the other hand, traditional systems typically design independent pipelines and index structures for different tasks such as "counting," "trajectory," or "retrieval," resulting in complex system architectures, difficulty in sharing intermediate results between different queries, and the need to retrain models or reconstruct plans when adding new query types, leading to high maintenance costs.

[0004] To address the aforementioned problems, existing technologies have primarily attempted to implement the following three methods, but all of them suffer from varying degrees of technical shortcomings: The first category is approximate query processing methods based on surrogate models. Typical examples include systems like NoScope, BlazeIt, and Tasti, which approximate expensive object detection models by training lightweight surrogate models (such as lightweight CNNs / MLPs). Their approach involves offline annotation of partial frames using heavy detectors to train query-specific surrogate models, which are then prioritized for online query filtering. However, these methods require designing independent surrogate models for each task, resulting in poor reusability; they also rely heavily on task-specific labeled data, leading to high training costs; and they have high adaptation costs when query types change (e.g., from "Is there a car?" to "Cars and pedestrians co-occur"), while not natively supporting natural language queries and fine-grained semantic retrieval.

[0005] The second category is video analysis methods based on trajectory or data warehouses. These methods (such as MIRIS, OTIF, and LEAP) typically perform offline detection and multi-object tracking on the entire video database, building an object trajectory data warehouse. During queries, the system performs calculations at the trajectory layer rather than the pixel layer. The main drawbacks of this method are the extremely high preprocessing costs of offline detection and tracking; it also typically assumes a static camera viewpoint and easy target tracking, making it sensitive to occlusion and viewpoint changes, thus limiting its versatility; furthermore, because its core lies in motion and spatial topology, it struggles to directly support queries involving fine-grained semantics or open-ended natural language, such as vehicle brands or complex movements.

[0006] The third category is video analysis methods based on unified multimodal embedding. Systems such as Zelda have attempted to use pre-trained multimodal models such as CLIP to map video frames and text queries to a unified semantic space and retrieve relevant frames by calculating text-image similarity. Although this method provides a new approach to unified queries, there are obvious defects in directly using "native embedding": (1) Weak quantity perception: The contrastive learning objective of models such as CLIP emphasizes semantic similarity and does not model cardinality information, resulting in huge errors when predicting counting tasks such as "how many cars". (2) Insufficient recognition of multi-target co-occurrence: The model tends to focus on the most visually prominent single target. In scenarios of multi-target co-occurrence (such as "both buses and trucks are present at the same time"), the multi-label recognition performance is poor and unstable. (3) Insufficient fine-grained semantic discrimination ability: For fine-grained classification tasks such as different models of vehicles of the same brand, the discrimination ability of native embedding is significantly reduced. (4) Insufficient frame filtering efficiency: Existing systems mainly rely on text-image similarity for frame cropping and do not make full use of temporal structure, motion information and video encoding format (such as I-frames / P-frames), resulting in the need to process a large number of invalid frames. (5) Lack of task-aware adaptation: Using the same unadapted embedding for different query families such as counting, multi-label, and retrieval makes it difficult to control query latency and resource overhead while ensuring accuracy.

[0007] In summary, existing technologies are still unable to simultaneously ensure the accuracy of various query types, the controllability of computational and storage overhead, and the fine-grained modeling of quantity, multiple objectives, and fine-grained semantics within a unified multimodal embedding framework. Summary of the Invention

[0008] To address the shortcomings of existing technologies, embodiments of the present invention provide a video query method, electronic device, and medium based on multimodal embedding.

[0009] In a first aspect, embodiments of the present invention provide a video query method based on multimodal embedding, the method comprising the following steps: Each video stream in the video database is decoded frame by frame, and all video frames are filtered to obtain a set of anchor frames; Each type of query question is encoded to obtain a text embedding vector dataset. Encode all anchor frames to obtain anchor frame visual embedding vectors; obtain the text embedding vectors and anchor frame visual embedding vectors corresponding to each type of user question, and the first detection result after adapter processing; store the text embedding vectors, anchor frame visual embedding vectors, and the first detection result as data pairs in the query cache. In response to a user question, a text embedding vector corresponding to the current user question is selected from the text embedding vector dataset, and a subset of anchor frames is selected from the anchor frame set. Based on the anchor frame subset, each anchor frame and its temporal neighborhood video frames are selected as a candidate frame set. Each video frame in the candidate frame set is traversed, and the current video frame to be processed is encoded to obtain the visual embedding vector of the current video frame. The type of user question is determined by querying the route, and the text embedding vector and the visual embedding vector of the current video frame are assigned to the corresponding adapters for processing to obtain the second detection result corresponding to the current video frame. Based on the text embedding vector corresponding to the current user question and the visual embedding vector of the anchor frame adjacent to the visual embedding vector of the current video frame, the corresponding second detection result is searched in the query cache. The first detection result is then subjected to a deviation test based on the second detection result. If the test passes, the second detection result is used as the video query result for the current user question.

[0010] In a second aspect, embodiments of the present invention provide an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the video query method based on multimodal embedding described above.

[0011] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described video query method based on multimodal embedding.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention obtains a set of anchor frames by filtering the video stream, and constructs a set of candidate frames based on the subset of anchor frames and their temporal neighborhood when responding to user questions. This avoids the full traversal of massive video data and only processes the most representative local neighborhood frames, thereby significantly reducing the amount of computation and response latency during online queries while ensuring query coverage.

[0013] (2) Through the query routing mechanism, this invention can identify different types of user questions and assign text embedding vectors and visual embedding vectors to the corresponding adapters for processing. This enables a single system to adaptively adjust feature representations for different tasks such as counting, retrieval, or classification within a unified multimodal embedding space, solving the problem of insufficient accuracy of native embeddings on specific tasks and realizing an architecture that supports multiple queries.

[0014] (3) This invention introduces a query caching and deviation verification mechanism. By pre-storing the first detection result of the anchor frame in the cache and using this result to perform deviation verification on the second detection result of the current video frame during online query, the system can effectively utilize the reference value of the anchor frame to correct the prediction deviation of the current frame. This not only filters out instantaneous noise interference but also ensures the continuity and stability of the video query results in the time dimension. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart of a video query method based on multimodal embedding provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating a video query method based on multimodal embedding provided in an embodiment of the present invention; Figure 3 A schematic diagram of a filtering anchor frame provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0019] like Figure 1 and Figure 2As shown, this embodiment of the invention provides a video query method based on multimodal embedding, the method comprising the following steps: Step S1: Decode each video stream in the video database frame by frame, filter all video frames, and obtain a number of anchor frames.

[0020] Specifically, the process of filtering all video frames to obtain a number of anchor frames includes: Iterate through all video frames of each video stream and calculate the value based on the current video frame. and video frames recently selected as anchor frames Calculate the motion saliency, semantic drift, time interval, and coding priority of the current video frame. Then, perform a weighted sum of these factors to obtain the score for the current video frame; the expression is as follows: In the formula, Indicates the current video frame The corresponding motion saliency, Indicates the current video frame The corresponding semantic drift, Indicates the current video frame The corresponding time interval, Indicates the current video frame The corresponding encoding priority, , , , All are weighting coefficients. ; Set an anchor frame budget for each video stream; select the video frames corresponding to the top K scores as anchor frames based on the anchor frame budget to obtain the anchor frame set.

[0021] Among them, motion saliency The calculation process includes: using the Mixture-of-Gaussian (MOG2) algorithm to calculate the video frame most recently selected as the anchor frame. Compared to the current video frame The change in the percentage of pixels in the image.

[0022] Among them, semantic drift The calculation process includes: using the CLIP model to calculate the video frame most recently selected as the anchor frame. Corresponding semantic vector With the current video frame Corresponding semantic vector Cosine similarity between them.

[0023] Among them, time interval For: the current video frame The timestamp t and the video frame most recently selected as the anchor frame The difference between timestamps k.

[0024] Among them, encoding priority This includes: if the current video frame's encoding type is I-frame, then its encoding priority is set to 1; if the current video frame's encoding type is P-frame, then its encoding priority is set to 0.

[0025] Step S2: Encode each type of query question to obtain a text embedding vector dataset.

[0026] Step S3: Encode all anchor frames to obtain anchor frame visual embedding vectors; obtain the text embedding vectors corresponding to each type of user question, the first detection result after the anchor frame visual embedding vectors are processed by the adapter, and store the text embedding vectors, anchor frame visual embedding vectors, and the first detection result as data pairs in the query cache.

[0027] Step S4: In response to the user question, select the text embedding vector corresponding to the current user question from the text embedding vector dataset, and select a subset of anchor frames from the anchor frame set; based on the anchor frame subset, select each anchor frame and its temporal neighborhood video frames as a candidate frame set; traverse each video frame in the candidate frame set, encode the current video frame to be processed, and obtain the visual embedding vector of the current video frame; determine the type of user question by querying the route, and assign the text embedding vector and the visual embedding vector of the current video frame to the corresponding adapter for processing, and obtain the second detection result corresponding to the current video frame.

[0028] Specifically, if the current user's question is an aggregation query / selection query, the text embedding vector and visual embedding vector are assigned to the first adapter; if the current user's question is a retrieval query, the text embedding vector and visual embedding vector are assigned to the second adapter; and if the current user's question is an action query, the text embedding vector and visual embedding vector are assigned to the third adapter.

[0029] It should be noted that this invention, based on the Unified Multimodal Embedding Space and the Unified Embedding Adaptive Module (UEAM) framework, can simultaneously support predicate selection queries (such as object filtering), aggregation counting queries (such as vehicle statistics), instance / fine-grained retrieval (such as specific vehicle models), and natural language queries within the same system. This avoids the pipeline fragmentation problem of traditional solutions that design independent proxy models and data structures for different query types, significantly reducing the complexity of the system architecture and the development and maintenance costs. Furthermore, if the current user query is an aggregation and / or selection query, the process of assigning the text embedding vector to the first adapter includes: For the current video frame Background frame Encode the data to obtain the embedding vector of the current video frame. Background frame embedding vector ; Embed the current video frame into a vector Background frame embedding vector After transformation by a multi-layer perceptron (MLP), the difference is taken to obtain the residual embedding vector ΔE; the expression is as follows: In the formula, Represents the background frame embedding vector Embedded vectors processed by a multilayer perceptron This represents the embedding vector of the current frame after processing by a multilayer perceptron. The residual embedding vector ΔE is compared with the text embedding vector. By concatenating the features, a hybrid feature representation is obtained. ; Mixed features This means that the input is fed into the target recognition network, and the output is the detection result for each category of target. The target header is a target header configured for each category based on different categories of queries in the current user question.

[0030] The training process for the first adapter includes: Set a first loss function; use the anchor frame set as training samples and combine it with the first loss function to train the first adapter, as shown in the following expression: In the formula, N represents the sample size. This represents the target header corresponding to the i-th sample. This represents the mixed feature representation corresponding to the i-th sample. This represents the embedding vector of the current video frame corresponding to the i-th sample. Let represent the text embedding vector corresponding to the i-th sample.

[0031] Furthermore, if the current user's question is a retrieval query, the text embedding vector is assigned to the second adapter; Obtain target domain data; the target domain data is selected from the Stanford Cars dataset; StanfordCars is a fine-grained visual classification (FGVC) dataset in the field of computer vision. This dataset was built by the Stanford AI Lab and is mainly used to test the performance of algorithms in distinguishing extremely similar object categories (i.e., cars of different models and years).

[0032] Feature extraction is performed on the target domain data to obtain the first feature vector. The first feature vector is transformed by a multilayer perceptron to obtain the second feature vector. The first and second eigenvectors are linearly interpolated according to their weights to obtain the third eigenvector. The expression is as follows: In the formula, Indicates the weighting coefficient; For the current video frame Encode the current video frame to obtain its embedding vector. ; the third feature vector Current video frame embedding vector After splicing ( , After transformation by a multilayer perceptron, the updated image features are obtained; The similarity between the updated image features and the text embedding vector t is calculated to obtain the similarity corresponding to each detection category. After processing by the softmax activation function, the confidence corresponding to each detection category is obtained. The category corresponding to the highest confidence is selected as the final detection result.

[0033] The training process for the second adapter includes: Set a second loss function; use the anchor frame set as training samples and combine it with the second loss function to train the first adapter, as shown in the following expression: In the formula, and All represent weight coefficients, O represents the predicted label output, and y represents the true label. This represents the predicted text embedding vector output by the model. The text embedding vector representing the actual label.

[0034] It should be noted that, for counting tasks, this invention combines a counting modeling method based on background residuals, utilizing the embedding difference between the current frame and the background frame to highlight local object changes, effectively compensating for the CLIP model's lack of quantitative information. For fine-grained retrieval such as vehicle brand and action category, by introducing Few-shot feature caching and contrastive loss, the ability to distinguish complex predicates and fine-grained categories is significantly improved, outperforming commonly used systems.

[0035] Furthermore, the third adapter is implemented based on the CLIP model, using CLIP's pre-trained weights. Based on this model, the user-described action query is encoded as a text vector `emb_txt`. For the video frame to be queried, it is first encoded as a visual vector `emb_img`, then cosine similarity is calculated using the text vector encodings such as "blurry image," "bad weather," and "empty image" and the visual vector `emb_img`. For video frames with high matching scores (similarity greater than a threshold), the similarity is then calculated. If the filtered video frames are not similar to the text vector emb_txt, then they are directly filtered. Next, for each filtered video frame, the similarity between the filtered frame and the text vector emb_txt is calculated, and the top-K results are returned.

[0036] Step S5: Based on the text embedding vector corresponding to the current user question and the visual embedding vector of the anchor frame adjacent to the visual embedding vector of the current video frame, search for the second detection result in the query cache. Perform a deviation test on the first detection result based on the second detection result. If the test passes, use the second detection result as the video query result for the current user question.

[0037] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the video query method based on multimodal embedding as described above. Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities where the video query method based on multimodal embedding provided in this embodiment of the invention is located, except... Figure 4 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0038] Accordingly, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the video query method based on multimodal embedding as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0039] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims

1. A video query method based on multimodal embedding, characterized in that, The method includes: Each video stream in the video database is decoded frame by frame, and all video frames are filtered to obtain a set of anchor frames; Each type of query question is encoded to obtain a text embedding vector dataset. Encode all anchor frames to obtain anchor frame visual embedding vectors; obtain the text embedding vectors and anchor frame visual embedding vectors corresponding to each type of user question, and the first detection result after adapter processing; store the text embedding vectors, anchor frame visual embedding vectors, and the first detection result as data pairs in the query cache. In response to a user question, a text embedding vector corresponding to the current user question is selected from the text embedding vector dataset, and a subset of anchor frames is selected from the anchor frame set. Based on the anchor frame subset, each anchor frame and its temporal neighborhood video frames are selected as a candidate frame set. Each video frame in the candidate frame set is traversed, and the current video frame to be processed is encoded to obtain the visual embedding vector of the current video frame. The type of user question is determined by querying the route, and the text embedding vector and the visual embedding vector of the current video frame are assigned to the corresponding adapters for processing to obtain the second detection result corresponding to the current video frame. Based on the text embedding vector corresponding to the current user question and the visual embedding vector of the anchor frame adjacent to the visual embedding vector of the current video frame, the corresponding second detection result is searched in the query cache. The first detection result is then subjected to a deviation test based on the second detection result. If the test passes, the second detection result is used as the video query result for the current user question.

2. The video query method based on multimodal embedding according to claim 1, characterized in that, The process of filtering all video frames to obtain a number of anchor frames includes: Iterate through all video frames of each video stream, calculate the motion saliency, semantic drift, time interval, and coding priority of the current video frame based on the current video frame and the video frame most recently selected as the anchor point, and perform a weighted sum of motion saliency, semantic drift, time interval, and coding priority to obtain the score of the current video frame; Set an anchor frame budget for each video stream; select the video frames corresponding to the top K scores as anchor frames based on the anchor frame budget to obtain the anchor frame set.

3. The video query method based on multimodal embedding according to claim 2, characterized in that, The calculation process for motion saliency includes: calculating the video frame most recently selected as the anchor frame. Compared to the current video frame The percentage of pixels that change in the image; The semantic drift calculation process includes: calculating the video frame that was most recently selected as the anchor frame. Corresponding semantic vector With the current video frame Corresponding semantic vector Similarity between them; The time interval is: the current video frame. The timestamp t and the video frame most recently selected as the anchor frame The difference between timestamps k; Encoding priority includes: if the current video frame's encoding type is I-frame, then its encoding priority is set to 1; if the current video frame's encoding type is P-frame, then its encoding priority is set to 0.

4. The video query method based on multimodal embedding according to claim 1, characterized in that, The process of determining the type of user question by querying the route and assigning the text embedding vector and visual embedding vector to the corresponding adapter for processing includes: If the current user's query is an aggregation and / or selection query, assign the text embedding vector and the visual embedding vector to the first adapter; If the current user's question is a search query, assign the text embedding vector and the visual embedding vector to the second adapter; If the current user's question is an action query, assign the text embedding vector and the visual embedding vector to the third adapter.

5. The video query method based on multimodal embedding according to claim 4, characterized in that, If the current user query is an aggregation and / or selection query, the process of assigning the text embedding vector to the first adapter includes: Encode the current video frame and the background frame to obtain the current video frame embedding vector and the background frame embedding vector, respectively. The residual embedding vector is obtained by subtracting the current video frame embedding vector and the background frame embedding vector after they have been transformed by a multi-layer perceptron layer. The residual embedding vector is concatenated with the text embedding vector to obtain a hybrid feature representation; The hybrid feature representation is input into the target recognition network, and the detection result is output through the target head corresponding to each category; wherein, the target head is a target head configured for each category according to the different categories of queries in the current user question.

6. The video query method based on multimodal embedding according to claim 4, characterized in that, If the current user's question is a search query, assign the text embedding vector to the second adapter; Obtain target domain data; perform feature extraction on the target domain data to obtain the first feature vector; The first feature vector is transformed by a multilayer perceptron to obtain the second feature vector. The first and second feature vectors are then linearly interpolated according to weight coefficients to obtain the third feature vector. The current video frame is encoded to obtain the current video frame embedding vector; the third feature vector and the current video frame embedding vector are concatenated and then transformed by a multilayer perceptron to obtain the updated image features; The similarity between the updated image features and the text embedding vector is calculated to obtain the similarity corresponding to each detection category. After processing by the softmax activation function, the confidence corresponding to each detection category is obtained. The category corresponding to the highest confidence is selected as the final detection result.

7. A video query method based on multimodal embedding according to claim 1 or 5, characterized in that, The training process for the first adapter includes: Set a first loss function; use the anchor frame set as training samples and combine it with the first loss function to train the first adapter, as shown in the following expression: ； In the formula, N represents the sample size. This represents the target header corresponding to the i-th sample. This represents the mixed feature representation corresponding to the i-th sample. This represents the embedding vector of the current video frame corresponding to the i-th sample. Let represent the text embedding vector corresponding to the i-th sample.

8. A video query method based on multimodal embedding according to claim 1 or 6, characterized in that, The training process for the second adapter includes: Set a second loss function; use the anchor frame set as training samples and combine it with the second loss function to train the first adapter, as shown in the following expression: ； In the formula, and All represent weight coefficients, O represents the predicted label output, and y represents the true label. This represents the predicted text embedding vector output by the model. The text embedding vector representing the actual label.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the video query method based on multimodal embedding as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video query method based on multimodal embedding as described in any one of claims 1-8.

Images