Multi-mode stock video efficient retrieval method and device based on space-time fusion

Abstract

The invention provides a multi-mode stock video efficient retrieval method and device based on space-time fusion, and the method comprises the steps: calculating the inter-frame similarity of a to-be-retrieved video, and deleting redundant frames of the to-be-retrieved video based on the inter-frame similarity to obtain a target to-be-retrieved video; screening the target to-be-retrieved video based on the query text to obtain a plurality of target to-be-retrieved video clips; respectively carrying out feature extraction processing on the query text and the plurality of target to-be-retrieved video clips to obtain a text feature and a plurality of video features; calling a video clip retrieval model based on multi-modal fusion to perform modal interaction prediction based on the text features and the plurality of video features to obtain a prediction result, and determining a target video clip corresponding to the query text in the plurality of target to-be-retrieved video clips based on the prediction result, by deleting the redundant frames and hierarchically searching the target video clips, the target video clips can be quickly screened from massive videos, and the video screening efficiency is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for efficient retrieval of multimodal stock videos based on spatiotemporal fusion. Background Art

[0002] With the acceleration of the pace of smart city construction in my country, video surveillance systems have been laid in most public areas of cities, counties and villages over the past decade. At the same time, in order to protect their own lives and property safety, most families have also installed home surveillance cameras, which has generated massive amounts of video surveillance data.

[0003] The multimodal large model can obtain excellent video spatiotemporal perception capabilities and can accurately identify the start and end times of events related to the query text in the video. However, when faced with massive surveillance videos, the reasoning speed of this method is relatively slow and it is difficult to meet the efficiency requirements of practical applications.

[0004] It can be seen that the video retrieval method in the related art has a technical problem of low retrieval efficiency in massive videos. Summary of the invention

[0005] The present invention provides a method and device for efficient retrieval of multimodal stock videos based on spatiotemporal fusion, which is used to solve the defect of low efficiency in searching for target videos in massive videos in the prior art, and achieve the effect of quickly screening out target videos in massive videos according to usage requirements.

[0006] The present invention provides a method for efficient retrieval of multimodal stock videos based on spatiotemporal fusion, comprising the following steps.

[0007] Acquire a video to be retrieved and a query text; calculate the inter-frame similarity of the video to be retrieved, and delete redundant frames of the video to be retrieved based on the inter-frame similarity to obtain a target video to be retrieved; screen the target video to be retrieved based on the query text to obtain a plurality of target video clips to be retrieved; perform feature extraction on the query text and the plurality of target video clips to be retrieved respectively to obtain text features and a plurality of video features; call a video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the plurality of video features to obtain a prediction result, and determine a target video clip corresponding to the query text among the plurality of target video clips to be retrieved based on the prediction result.

[0008] According to a method for efficient retrieval of multimodal stock videos based on spatiotemporal fusion provided by the present invention, the method calls a video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the multiple video features to obtain a prediction result, and determines a target video clip corresponding to the query text among the multiple target video clips to be retrieved based on the prediction result, including: performing feature mapping processing on the text features and the multiple video features respectively to obtain a target text feature and multiple target video features in the same feature space; performing feature fusion processing on the target text feature and each of the target video features respectively to obtain multiple fusion features; inputting the target text feature and the multiple fusion features into a video clip retrieval model based on multimodal fusion to perform modal interaction prediction to obtain the start and end time points of the target video clip to be retrieved corresponding to the fusion feature that best matches the target text feature, and determining the target video clip according to the start and end time points.

[0009] According to a method for efficient retrieval of multimodal stock videos based on spatiotemporal fusion provided by the present invention, the inter-frame similarity of the video to be retrieved is calculated, and redundant frames of the video to be retrieved are deleted based on the inter-frame similarity to obtain a target video to be retrieved, including: calculating the inter-frame similarity of every two adjacent frames of the video to be retrieved; when the inter-frame similarity is greater than a similarity threshold, deleting any one of the two adjacent frames; and combining all retained frames into the target video to be retrieved according to the original time sequence.

[0010] According to a method for efficient retrieval of multimodal stock videos based on spatiotemporal fusion provided by the present invention, the target video to be retrieved is screened based on the query text to obtain multiple target video segments to be retrieved, including: obtaining multiple key frames in the target video to be retrieved and keywords in the query text, and calculating image features of the key frames; performing feature matching processing on the keywords and the image features, and taking video segments corresponding to the key frames that meet preset matching conditions as the multiple target video segments to be retrieved.

[0011] According to a method for efficient retrieval of multimodal stock videos based on spatiotemporal fusion provided by the present invention, the method also includes: obtaining a new query text; screening the target video to be retrieved based on the new query text to obtain a plurality of new target video clips to be retrieved; performing feature extraction processing on the new query text and the plurality of new target video clips to be retrieved respectively to obtain new text features and a plurality of new video features; calling the video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the new text features and the plurality of new video features to obtain a new prediction result, and determining a new target video clip corresponding to the new query text among the plurality of new target video clips to be retrieved based on the new prediction result.

[0012] According to a method for efficient retrieval of multimodal stock videos based on spatiotemporal fusion provided by the present invention, the method for obtaining the video to be retrieved includes any one of the following: obtaining the video to be retrieved from a video recording device through a real-time streaming protocol, obtaining the video to be retrieved from the video recording device through a software development kit corresponding to the video recording device, and reading the video to be retrieved from a storage medium of the video recording device.

[0013] The present invention also provides an efficient retrieval device for multimodal stock videos based on spatiotemporal fusion, comprising the following modules: an acquisition module, used to acquire a video to be retrieved and a query text; a redundant frame deletion module, used to calculate the inter-frame similarity of the video to be retrieved, and delete the redundant frames of the video to be retrieved based on the inter-frame similarity to obtain a target video to be retrieved; a preliminary screening module, used to screen the target video to be retrieved based on the query text to obtain a plurality of target video clips to be retrieved; a feature extraction module, used to perform feature extraction on the query text and the plurality of target video clips to be retrieved, respectively, to obtain text features and a plurality of video features; an accurate matching module, used to call a video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the plurality of video features to obtain a prediction result, and determine the target video clip corresponding to the query text among the plurality of target video clips to be retrieved based on the prediction result.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an efficient retrieval method for multimodal stock videos based on spatiotemporal fusion as described in any one of the above is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for efficiently retrieving multimodal stock videos based on spatiotemporal fusion as described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for efficient retrieval of multimodal stock videos based on spatiotemporal fusion.

[0017] The present invention provides a method and device for efficient retrieval of multimodal stock videos based on spatiotemporal fusion, which obtains the video to be retrieved and the query text, calculates the inter-frame similarity of the video to be retrieved, and deletes the redundant frames of the video to be retrieved based on the inter-frame similarity to obtain the target video to be retrieved, thereby reducing the amount of calculation during subsequent video screening and improving the calculation efficiency; then the target video to be retrieved is screened based on the query text to obtain multiple target video clips to be retrieved, thereby achieving preliminary screening of the video to be retrieved; finally, feature matching is performed on the text features and multiple video features corresponding to the query text and the multiple target video clips to be retrieved, and the target video clip corresponding to the query text in the multiple target video clips to be retrieved is determined based on the matching results, thereby achieving accurate matching of the video to be retrieved, and through the two steps of preliminary screening and accurate matching, the target video clip can be quickly screened out from massive videos, thereby improving the video screening efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the efficient retrieval method for multimodal stock videos based on spatiotemporal fusion provided by the present invention.

[0020] Figure 2 This is the second flow chart of the efficient retrieval method for multimodal stock videos based on spatiotemporal fusion provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the multi-modal stock video efficient retrieval device based on time-space fusion provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] With the acceleration of the construction of smart cities in my country, video surveillance systems have been laid in most public areas of cities, counties and villages in the past decade. At the same time, in order to protect their own lives and property safety, most families have also installed home surveillance cameras, which has generated a huge amount of video surveillance data. How to better understand the video and quickly find relevant time-series segments in the massive video according to the given query statement has become a problem that needs to be solved urgently.

[0025] In related technologies, the introduction of multimodal visual large models provides a better solution for visual-textual modality alignment. VTimeLLM aligns visual features with the semantic space of the language model (LargeLanguage Model, LLM) by learning image-text pair datasets (such as LCS-558K) so that the LLM can understand the visual information in the video; using automatically annotated video datasets containing multiple events (such as InternVid-10M-FLT), the video content is converted into single-round and multi-round question-answering formats to cultivate the model's temporal reasoning ability. Through these training methods, the model's generalization ability and ability to accurately recognize video time segments have been significantly improved.

[0026] However, although VTimeLLM has obtained excellent video spatiotemporal perception capabilities through a large multimodal model and can accurately identify the start and end time of events related to the query text in the video, when faced with massive surveillance videos, the reasoning speed of this method is relatively slow and it is difficult to meet the efficiency requirements of practical applications.

[0027] A video retrieval method is provided in the related technology, which pre-processes the video files to be retrieved; extracts the voice in the video clip, and uses a speech recognition model to convert the voice into text as the text representation of the auditory element of the video clip; uses a multimodal large model to convert the image information of each frame image in the frame group corresponding to the video clip into text as the text representation of the visual element of the video clip; calculates the vector of the text representation of each video clip through the SBERT model, and stores the obtained video clip encoding vector in a database; uses the inverse sort fusion to comprehensively sort the vector retrieval results and the full-text retrieval results to give the final retrieval results.

[0028] When performing video retrieval, it is necessary to extract the speech in the video clip and use the speech recognition model to convert the speech into text as the text representation of the auditory elements of the video clip. However, in some sensitive places where surveillance equipment is prohibited from picking up sound, the obtained text representation of the video clip is incomplete. At the same time, the final retrieval result obtained by this method is a few sorted videos, and it does not have the ability to perceive the temporal sequence of events in the video that are related to the query text.

[0029] According to the present invention, a multimodal stock video efficient retrieval method based on spatiotemporal fusion can understand finer-grained video clips and identify more accurately.

[0030] Combine the following Figure 1-Figure 2 The present invention describes an efficient retrieval method for multimodal stock videos based on spatiotemporal fusion.

[0031] Optionally, the efficient retrieval method for multimodal existing videos based on time-space fusion of the embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the efficient retrieval method for multimodal existing videos based on time-space fusion in this embodiment being executed by a server.

[0032] Figure 1 This is one of the flow charts of the efficient retrieval method of multimodal stock videos based on spatiotemporal fusion provided by the present invention, such as Figure 1 As shown, the method includes the following steps 101 to 105.

[0033] Step 101: Obtain the video to be retrieved and the query text.

[0034] In some embodiments, the query text can be a simple descriptive text, such as "find clips where cars appear in the video", "retrieve video clips with beach scenes", "retrieve clips containing dance moves", etc., or it can be a complex semantic text, such as "retrieve clips of goals in football matches", "retrieve clips of cars driving fast on the streets at night with flashing lights", etc.

[0035] According to an efficient retrieval method for multimodal stock videos based on spatiotemporal fusion provided by the present invention, the method for obtaining the video to be retrieved includes any one of the following: obtaining the video to be retrieved from a video recording device through a real-time streaming protocol, obtaining the video to be retrieved from the video recording device through a software development kit corresponding to the video recording device, and reading the video to be retrieved from a storage medium of the video recording device.

[0036] Specifically, there are usually three main ways to obtain video data from video recording devices such as surveillance hard disk recorders. Each method has its own characteristics and is suitable for different application scenarios: the first is to obtain video data from real-time streaming protocol (RTSP), that is, to obtain video data streams directly from video recording devices such as video recorders through real-time streaming protocol. The advantage of RTSP is that the video stream can be accessed at any time and decoded in real time, but its bandwidth occupancy is high and it is suitable for scenarios with good network environment; the second is to obtain it through the software development kit (SDK) corresponding to the video recording device, that is, to access and process the video data of the video recording device by using the software development kit adapted by the video recording device such as the video recording device. This method is usually more comprehensive and supports real-time viewing, playback control, screenshots, alarm linkage and other functions of the video. The third is to directly read the storage medium of the video recording device, such as a hard disk, floppy disk, etc., that is, directly take out the hard disk of the video recording device such as the video recording device and connect it to the computer for data reading. This method is usually used in scenarios where data recovery is required or when the hard disk is taken out for data recovery when the video recording device fails.

[0037] By obtaining the to-be-retrieved videos through the above three common methods, it is possible to flexibly select the method of obtaining the to-be-retrieved videos in different scenarios, which greatly improves the efficiency of obtaining the to-be-retrieved videos.

[0038] Step 102: Calculate the inter-frame similarity of the video to be retrieved, and delete redundant frames of the video to be retrieved based on the inter-frame similarity to obtain the target video to be retrieved.

[0039] According to an efficient multimodal stock video retrieval method based on spatiotemporal fusion provided by the present invention, the inter-frame similarity of the video to be retrieved is calculated, and redundant frames of the video to be retrieved are deleted based on the inter-frame similarity to obtain the target video to be retrieved, including: Calculate the inter-frame similarity between every two adjacent frames of the video to be retrieved; When the inter-frame similarity is greater than the similarity threshold, any frame of the two adjacent frames is deleted; All the retained frames are combined into the target video to be retrieved according to the original time sequence.

[0040] Specifically, the present invention uses inter-frame similarity calculation in the redundant frame deletion process to determine whether the similarity of adjacent frames exceeds a preset similarity threshold, thereby screening out key frames, reducing unnecessary frame data volume, and reducing subsequent processing and calculation requirements. The specific algorithm is as follows: in Representation frame and frame The similarity distance between and Represents frames and frame No. Features, is the dimension of the feature. When it is greater than the set threshold, the frame is judged as a redundant frame and deleted.

[0041] In other embodiments, the inter-frame similarity of adjacent frames of the video to be retrieved can also be calculated by a pixel difference method, for example, the difference in grayscale value or RGB value of corresponding pixels of two frames is calculated, and the absolute value of the difference is added to obtain a total difference value. If the difference value is less than a certain set threshold, it means that the contents of the two frames are very similar, and any frame can be regarded as a redundant frame and deleted.

[0042] In addition, the similarity between frames can be calculated in a feature-based manner, such as extracting the Scale Invariant Feature Transform (SIFT) or Speeded Up Robust Features (SURF) of the frame, and then comparing the differences between these features. If the feature matching degree of two adjacent frames is greater than the similarity threshold, it means that the two frames are similar, and either frame can be deleted as a redundant frame.

[0043] In an embodiment of the present invention, by calculating the similarity of adjacent frames, retaining key frames and deleting redundant frames, not only can the storage requirements of redundant information be reduced in the data preprocessing stage, but also the task processing amount and calculation time during subsequent video retrieval can be effectively reduced, the computing resource overhead can be reduced, and the efficiency of subsequent video retrieval processing can be improved.

[0044] Step 103: Screen the target videos to be retrieved based on the query text to obtain multiple target video clips to be retrieved.

[0045] According to an efficient multi-modal stock video retrieval method based on spatiotemporal fusion provided by the present invention, target videos to be retrieved are screened based on query text to obtain multiple target video clips to be retrieved, including: Obtain multiple key frames in the target video to be retrieved and keywords in the query text, and calculate the image features of the key frames; The keywords and image features are matched, and the video clips corresponding to the key frames that meet the preset matching conditions are used as multiple target video clips to be retrieved.

[0046] In the early days of text-based retrieval of video clips, the amount of data is often very large. If detailed feature analysis and matching are performed on each video clip, it will consume a lot of time and computing resources. Therefore, the initial retrieval stage mainly uses coarse-grained screening methods to quickly narrow the search scope and locate the approximate area where the video clips that may contain the content required by the user are located.

[0047] Specifically, the key frames of the target video to be retrieved and the keywords in the query text are first extracted. For example, when the query text is "retrieve video segments of beach scenes", the keywords can be "beach" or "beach scenes", and then the image features of the key frames are calculated. The image features here are simple image features, such as color histograms or simple texture features. The color histogram can describe the distribution of different colors in a frame of an image. For example, for a query text about the beach, the system will look for key frames with a high proportion of blue (representing the sky and sea water) and yellow (representing the beach) in the color histogram; texture features can be obtained by calculating the gradient of the image and other methods to distinguish different types of scenes, such as rough textures may represent forest scenes (tree branches, etc.), and smooth textures may represent water surfaces, etc. Then, feature matching is performed on the keywords and image features, and the video clips corresponding to the key frames that meet the matching conditions are used as multiple target video clips to be retrieved. For example, when the keyword is "beach", the multiple video clips corresponding to the key frames whose proportions of blue and yellow in the color histogram reach a preset value are used as multiple target video clips to be retrieved.

[0048] In other embodiments, the coarse-grained preliminary screening of the videos to be retrieved can also be performed by a screening method based on video metadata. Since videos usually carry some metadata, such as video titles, tags, descriptions, etc., these metadata can be simply matched with text. For example, if the user's query text is "highlights of football games", videos containing "football game" related words in the video titles or tags can be first searched. This method is fast and can quickly exclude a large number of obviously irrelevant videos.

[0049] In addition, it is also possible to use a simple motion feature recognition method, that is, for some query texts involving motion, such as "dance performance", simple motion patterns can be detected by analyzing the differences between video frames. For example, the displacement and motion direction of objects between adjacent frames can be calculated. If it is detected that there are many objects moving rhythmically in consecutive frames, it is possible that the video clips are related to dance performances, and thus they can be initially screened out.

[0050] In the embodiment of the present application, by using a coarse-grained screening method to perform preliminary screening on the videos to be retrieved, the search scope can be quickly narrowed down to locate the approximate area where the video clips that may contain the content required by the user are located, thereby effectively improving the video retrieval efficiency.

[0051] Step 104 : performing feature extraction processing on the query text and the multiple target video clips to be retrieved respectively to obtain text features and multiple video features.

[0052] In some embodiments, feature extraction is performed on the query text and multiple target video clips to be retrieved respectively. A parallel feature extraction method can be used to extract corresponding video features and text features using a video encoder and a text encoder in a hierarchical encoder (such as a HERO module) pre-trained with a video-language full representation. When using the video encoder to extract video features, it is necessary to first perform preprocessing such as frame rate adjustment on the target video clip to be retrieved, and then input the preprocessed video into the video encoder. The video encoder performs local feature extraction, spatiotemporal feature fusion and other processing on the video frames through convolutional layers and pooling layers, and finally generates a global feature vector, i.e., video features, through a series of fully connected layers.

[0053] When using a text encoder to extract text features, it is also necessary to perform preprocessing such as word segmentation and tokenization on the query text, and then input the preprocessed query text into the text encoder. The text encoder will first perform a word vector embedding operation on each token, converting each token into a word vector of fixed dimension. These word vectors can initially represent the semantic information of each token. For example, different words will have different word vectors, and the word vectors of words with similar semantics will be relatively close in the vector space. After the word vector is embedded, the text encoder will widely use the self-attention mechanism. The self-attention mechanism enables the text encoder to pay attention to the relationship between different parts of the text and adjust the weight of each token according to the semantics and context of the text. Through a series of self-attention mechanism applications and other network operations, the text encoder finally generates a feature vector that can represent the entire text, that is, the text feature.

[0054] In an embodiment of the present invention, an encoder combining the SlowFast architecture and the CLIP architecture can be used to extract video features, and an encoder of the CLIP architecture can be used to extract text features. Among them, SlowFast is a dual-path neural network architecture designed specifically for video understanding, in which the Slow path samples the video at a lower frame rate (e.g., 2-5 frames per second) to capture slow-changing information in the video, and the Fast path samples the video at a higher frame rate (e.g., 15-30 frames per second) to capture fast-changing information in the video; CLIP (Contrastive Language-Image Pre-training) is mainly composed of a text encoder and an image encoder. For the text part, the text encoder of CLIP is a neural network based on the Transformer architecture, which inputs the text into the network after pre-processing the text by segmentation, tokenization, etc. Using the self-attention mechanism of the Transformer, the text encoder can learn the grammatical, semantic, and contextual information of the text and convert the text into a semantic feature vector of a fixed dimension.

[0055] SlowFast focuses on the extraction of spatiotemporal structure and dynamic information of videos, while CLIP focuses on capturing the semantic information of video frames when used for video frames. Combining the two can make full use of their respective advantages to obtain more comprehensive and semantically rich video features. For example, the dynamic action information extracted by SlowFast and the object category information extracted by CLIP can together form a more accurate description of the video content for matching with text features. Since CLIP provides a unified feature space for text and video, combined with SlowFast's supplement to video features, better video-text matching can be performed. In video retrieval tasks, this combination can more accurately find video clips that match the text description, improving the accuracy and efficiency of retrieval.

[0056] Step 105: Call the video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on text features and multiple video features to obtain a prediction result, and determine the target video clip corresponding to the query text among the multiple target video clips to be retrieved based on the prediction result.

[0057] According to an efficient multimodal stock video retrieval method based on spatiotemporal fusion provided by the present invention, a video clip retrieval model based on multimodal fusion is called to perform modal interaction prediction based on text features and multiple video features to obtain a prediction result, and a target video clip corresponding to a query text among multiple target video clips to be retrieved is determined based on the prediction result, including: Perform feature mapping processing on the text features and the multiple video features respectively to obtain target text features and multiple target video features in the same feature space; Performing feature fusion processing on the target text feature and each target video feature respectively to obtain multiple fused features; The target text feature and the multiple fusion features are input into a video clip retrieval model based on multimodal fusion for modal interaction prediction, the start and end time points of the target video clip to be retrieved corresponding to the fusion feature that best matches the target text feature are obtained, and the target video clip is determined based on the start and end time points.

[0058] In the embodiment of the present invention, the video segment retrieval model based on multimodal fusion may be a video segment retrieval model based on DERT (Detection Transformer).

[0059] Specifically, after obtaining the text features and video features in step 104, since CLIP creates a relatively unified feature space for the text features and video features, the text features and video features can be directly mapped to the same feature space through CLIP to obtain the target text features and target video features. Then, the target text features and each target video feature are subjected to feature fusion processing respectively to obtain multiple fusion features. The target text features and multiple fusion features are input into the DERT (Detection Transformer) model for modal interaction prediction. The DERT model can explicitly inject the context of the text query into the video representation, so that the model can better use the query information to understand the video content. Based on the target text features and fusion features, the DERT model regresses a center point and width (a representation of the start and end time of the video), that is, the start and end time points of the target video segment to be retrieved corresponding to the fusion feature that best matches the target text features. Finally, the target video segment can be determined based on the start and end time points.

[0060] In the embodiment of the present invention, by designing two levels of preliminary search and precise feature matching, i.e., hierarchical search in stages, preliminary search quickly screens out potentially relevant video clips, ensuring the high efficiency of search; while precise feature matching further improves the accuracy of search results through deep feature comparison. Hierarchical search ensures high accuracy and relevance of returned results while ensuring search speed.

[0061] In some embodiments, the efficient retrieval method for multimodal existing videos based on spatiotemporal fusion provided by the present invention also includes: obtaining a new query text; screening the target videos to be retrieved based on the new query text to obtain multiple new target video clips to be retrieved; performing feature extraction processing on the new query text and the multiple new target video clips to be retrieved respectively to obtain new text features and multiple new video features; calling a video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the new text features and the multiple new video features to obtain a new prediction result, and determining a new target video clip corresponding to the new query text in the multiple new target video clips to be retrieved based on the new prediction result.

[0062] Specifically, after obtaining the target video clip corresponding to the query text from the video to be retrieved, the user can determine whether the retrieval is completed according to actual needs. If other content needs to be retrieved, the user can continue to enter a new query text and re-execute steps 103 to 105 until relevant retrieval results are obtained. There is no need to perform redundant frame deletion and feature extraction on the video to be retrieved, so the efficiency of re-retrieval can be greatly improved.

[0063] Through the above steps of the embodiment of the present invention, by obtaining the video to be retrieved and the query text, calculating the inter-frame similarity of the video to be retrieved, and deleting the redundant frames of the video to be retrieved based on the inter-frame similarity, the target video to be retrieved is obtained, thereby reducing the amount of calculation during subsequent video screening and improving the calculation efficiency; then the target video to be retrieved is screened based on the query text to obtain multiple target video clips to be retrieved, thereby achieving preliminary screening of the video to be retrieved; finally, feature matching is performed on the text features and multiple video features corresponding to the query text and the multiple target video clips to be retrieved, and the target video clip corresponding to the query text in the multiple target video clips to be retrieved is determined based on the matching results, thereby achieving accurate matching of the video to be retrieved. Through the two steps of preliminary screening and accurate matching, the target video clip can be quickly screened out from massive videos, thereby improving the video screening efficiency.

[0064] In some embodiments, see Figure 2 , Figure 2 This is the second flow chart of the efficient retrieval method for multimodal stock videos based on spatiotemporal fusion provided by the present invention.

[0065] The first is the video acquisition step, which is to select the video acquisition method according to user needs, including real-time video stream acquisition, SDK call or direct reading of hard disk files. The next is the redundant frame deletion step. After acquiring the video to be retrieved, the redundant frames of the video to be retrieved are screened. By calculating the similarity of adjacent frames, key frames are retained and redundant frames are deleted, which reduces unnecessary calculations and speeds up subsequent processing efficiency. Then the user enters the query text and performs feature extraction of the video and text. Here, a parallel feature extraction method is used to extract and analyze the features of the video to be retrieved and the query text. Then comes the hierarchical retrieval step. The retrieval process here is divided into two stages: preliminary retrieval and precise matching. Preliminary retrieval is to quickly screen out video clips that may contain target events corresponding to the query text in the database to ensure retrieval efficiency; precise matching means that for the video clips screened by the preliminary retrieval, the similarity between their feature vectors and the query text is further calculated to accurately locate the start and end time of the event, and then the matching results are returned to the user, including relevant information of the video clip, time point and matching degree, etc. Finally, it is determined whether the retrieval is completed based on actual needs. If other content needs to be retrieved, continue to enter the query text and perform the text feature extraction step. Here, there is no need to extract features from the video until the relevant video clips are obtained.

[0066] In the embodiment of the present invention, the amount of data processing is greatly reduced by adopting redundant frame deletion and optimized video sampling strategies, thereby achieving efficient retrieval in massive video data; and the introduction of a hierarchical retrieval mechanism, combined with a two-stage process of preliminary screening and precise matching, ensures rapid response while maintaining high-precision matching of the query text.

[0067] The following is a description of the efficient multimodal stock video retrieval device based on time-space fusion provided by the present invention. The efficient multimodal stock video retrieval device based on time-space fusion described below and the efficient multimodal stock video retrieval method based on time-space fusion described above can be referenced to each other.

[0068] See also Figure 3 , Figure 3 is a schematic diagram of the structure of a multi-modal stock video efficient retrieval device based on spatiotemporal fusion provided by the present invention, such as Figure 3As shown, the multimodal stock video efficient retrieval device based on spatiotemporal fusion provided by the present invention includes an acquisition module 301, a redundant frame deletion module 302, a preliminary screening module 303, a feature extraction module 304 and an accurate matching module 305, wherein the acquisition module 301 is used to acquire the video to be retrieved and the query text; the redundant frame deletion module 302 is used to calculate the inter-frame similarity of the video to be retrieved, and delete the redundant frames of the video to be retrieved based on the inter-frame similarity to obtain the target video to be retrieved; the preliminary screening module 303 is used to screen the target video to be retrieved based on the query text to obtain multiple target video clips to be retrieved; the feature extraction module 304 is used to perform feature extraction processing on the query text and the multiple target video clips to be retrieved respectively to obtain text features and multiple video features; the accurate matching module 305 is used to call the video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the multiple video features to obtain a prediction result, and determine the target video clip corresponding to the query text among the multiple target video clips to be retrieved based on the prediction result.

[0069] In some embodiments, the redundant frame deletion module 302 is also used to calculate the inter-frame similarity between every two adjacent frames of the video to be retrieved; when the inter-frame similarity is greater than a similarity threshold, any one of the two adjacent frames is deleted; and all retained frames are combined into the target video to be retrieved according to the original time sequence.

[0070] In some embodiments, the preliminary screening module 303 is also used to obtain multiple key frames in the target video to be retrieved and keywords in the query text, and calculate the image features of the key frames; perform feature matching processing on the keywords and the image features, and use the video clips corresponding to the key frames that meet the preset matching conditions as the multiple target video clips to be retrieved.

[0071] In some embodiments, the precise matching module 305 is also used to obtain new query text; based on the new query text, the target video to be retrieved is screened to obtain multiple new target video clips to be retrieved; the new query text and the multiple new target video clips to be retrieved are respectively subjected to feature extraction processing to obtain new text features and multiple new video features; the video clip retrieval model based on multimodal fusion is called to perform modal interaction prediction based on the new text features and the multiple new video features to obtain new prediction results, and based on the new prediction results, a new target video clip corresponding to the new query text in the multiple new target video clips to be retrieved is determined.

[0072] In some embodiments, the precise matching module 305 is also used to perform feature mapping processing on the text features and the multiple video features respectively to obtain target text features and multiple target video features in the same feature space; perform feature fusion processing on the target text features and each of the target video features respectively to obtain multiple fusion features; input the target text features and the multiple fusion features into a video clip retrieval model based on multimodal fusion for modal interaction prediction, obtain the start and end time points of the target video clip to be retrieved corresponding to the fusion feature that best matches the target text features, and determine the target video clip based on the start and end time points.

[0073] Specifically, the above-mentioned multimodal existing video efficient retrieval device based on time-space fusion provided by the present invention can implement all the method steps implemented by the above-mentioned multimodal existing video efficient retrieval method embodiment based on time-space fusion, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0074] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 , and the memory 430 communicate with each other through the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute an efficient retrieval method for multimodal existing videos based on spatiotemporal fusion, the method comprising: obtaining a video to be retrieved and a query text; calculating the inter-frame similarity of the video to be retrieved, and deleting redundant frames of the video to be retrieved based on the inter-frame similarity to obtain a target video to be retrieved; screening the target video to be retrieved based on the query text to obtain a plurality of target video clips to be retrieved; performing feature extraction on the query text and the plurality of target video clips to be retrieved respectively to obtain text features and a plurality of video features; calling a video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the plurality of video features to obtain a prediction result, and determining a target video clip corresponding to the query text among the plurality of target video clips to be retrieved based on the prediction result.

[0075] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0076] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal stock video efficient retrieval method based on spatiotemporal fusion provided by the above methods, the method including: obtaining a video to be retrieved and a query text; calculating the inter-frame similarity of the video to be retrieved, and deleting redundant frames of the video to be retrieved based on the inter-frame similarity to obtain a target video to be retrieved; screening the target video to be retrieved based on the query text to obtain a plurality of target video clips to be retrieved; performing feature extraction processing on the query text and the plurality of target video clips to be retrieved respectively to obtain text features and a plurality of video features; calling a video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the plurality of video features to obtain a prediction result, and determining a target video clip corresponding to the query text among the plurality of target video clips to be retrieved based on the prediction result.

[0077] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal stock video efficient retrieval method based on spatiotemporal fusion provided by the above-mentioned methods, the method comprising: obtaining a video to be retrieved and a query text; calculating the inter-frame similarity of the video to be retrieved, and deleting redundant frames of the video to be retrieved based on the inter-frame similarity to obtain a target video to be retrieved; screening the target video to be retrieved based on the query text to obtain a plurality of target video clips to be retrieved; performing feature extraction processing on the query text and the plurality of target video clips to be retrieved, respectively, to obtain text features and a plurality of video features; calling a video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the plurality of video features to obtain a prediction result, and determining a target video clip corresponding to the query text among the plurality of target video clips to be retrieved based on the prediction result.

[0078] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0079] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient retrieval method for multimodal stock videos based on spatiotemporal fusion, characterized in that: include: Get the video to be retrieved and the query text; Calculating the inter-frame similarity of the video to be retrieved, and deleting redundant frames of the video to be retrieved based on the inter-frame similarity to obtain a target video to be retrieved; Screening the target video to be retrieved based on the query text to obtain multiple target video clips to be retrieved; Performing feature extraction processing on the query text and the multiple target video clips to be retrieved respectively to obtain text features and multiple video features; A video clip retrieval model based on multimodal fusion is called to perform modal interaction prediction based on the text features and the multiple video features to obtain a prediction result, and a target video clip corresponding to the query text among the multiple target video clips to be retrieved is determined based on the prediction result.

2. The efficient retrieval method for multimodal stock videos based on spatiotemporal fusion according to claim 1 is characterized in that: The calling of the video segment retrieval model based on multimodal fusion performs modal interaction prediction based on the text feature and the multiple video features to obtain a prediction result, and determines a target video segment corresponding to the query text among the multiple target video segments to be retrieved based on the prediction result, including: Performing feature mapping processing on the text features and the multiple video features respectively to obtain target text features and multiple target video features in the same feature space; Performing feature fusion processing on the target text feature and each target video feature respectively to obtain multiple fused features; The target text feature and the multiple fusion features are input into a video clip retrieval model based on multimodal fusion for modal interaction prediction, the start and end time points of the target video clip to be retrieved corresponding to the fusion feature that best matches the target text feature are obtained, and the target video clip is determined based on the start and end time points.

3. The efficient retrieval method for multimodal stock videos based on spatiotemporal fusion according to claim 1 is characterized in that: The calculating the inter-frame similarity of the video to be retrieved, and deleting redundant frames of the video to be retrieved based on the inter-frame similarity to obtain the target video to be retrieved, includes: Calculating the inter-frame similarity between every two adjacent frames of the video to be retrieved; When the inter-frame similarity is greater than a similarity threshold, deleting any one of the two adjacent frames; All the retained frames are combined into the target video to be retrieved according to the original time sequence.

4. The efficient retrieval method for multimodal stock videos based on spatiotemporal fusion according to claim 1 is characterized in that: The method of screening the target video to be retrieved based on the query text to obtain a plurality of target video clips to be retrieved includes: Acquire multiple key frames in the target video to be retrieved and keywords in the query text, and calculate image features of the key frames; The keywords are matched with the image features, and the video clips corresponding to the key frames that meet the preset matching conditions are used as the multiple target video clips to be retrieved.

5. The efficient retrieval method for multimodal stock videos based on spatiotemporal fusion according to claim 1 is characterized in that: The method further comprises: Get new query text; Based on the new query text, the target video to be retrieved is screened to obtain multiple new target video clips to be retrieved; Performing feature extraction processing on the new query text and the multiple new target video clips to be retrieved respectively to obtain new text features and multiple new video features; The video clip retrieval model based on multimodal fusion is called to perform modal interaction prediction based on the new text features and the multiple new video features to obtain new prediction results, and a new target video clip corresponding to the new query text among the multiple new target video clips to be retrieved is determined based on the new prediction results.

6. The efficient retrieval method for multimodal stock videos based on spatiotemporal fusion according to claim 1 is characterized in that: The method for obtaining the video to be retrieved includes any one of the following: The video to be retrieved is obtained from the video recording device through a real-time streaming protocol, the video to be retrieved is obtained from the video recording device through a software development kit corresponding to the video recording device, and the video to be retrieved is read from a storage medium of the video recording device.

7. An efficient retrieval device for multimodal stock videos based on spatiotemporal fusion, characterized in that: The device comprises: The acquisition module is used to obtain the video to be retrieved and the query text; A redundant frame deletion module, used for calculating the inter-frame similarity of the video to be retrieved, and deleting the redundant frames of the video to be retrieved based on the inter-frame similarity to obtain the target video to be retrieved; A preliminary screening module, used for screening the target video to be retrieved based on the query text to obtain multiple target video clips to be retrieved; A feature extraction module is used to perform feature extraction processing on the query text and the multiple target video clips to be retrieved, respectively, to obtain text features and multiple video features; The precise matching module is used to call the video clip retrieval model based on multimodal fusion to perform modal interaction prediction based on the text features and the multiple video features to obtain a prediction result, and determine the target video clip corresponding to the query text among the multiple target video clips to be retrieved based on the prediction result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the efficient retrieval method for multimodal stock videos based on spatiotemporal fusion as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the efficient retrieval method for multimodal stock videos based on spatiotemporal fusion as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the efficient retrieval method for multimodal stock videos based on spatiotemporal fusion as described in any one of claims 1 to 6 is implemented.

Images