Multimodal video content retrieval method and device based on shot frame sampling, and medium

Abstract

The application discloses a kind of multi-modal video content retrieval methods, equipment and medium based on lens frame sampling.The method comprises the following steps: obtaining original video file, and the original video file is split, and each video shot is obtained;At least two kinds of representation vectors of each video shot are determined respectively, and each representation vector and the metadata of each video shot are stored in target database, and the target video database is obtained;The metadata of each video shot includes: identification information or timestamp information;In response to the input instruction of target retrieval content, the target retrieval content is represented by vector, and the target retrieval vector is obtained;The reference distance of target retrieval vector and each representation vector stored in video database is determined, and the preselected retrieval score is determined based on target weight;Each preselected retrieval score is screened, target video shot is determined, and target video shot is displayed.The scheme of the application can quickly and accurately retrieve relevant video content.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, device and medium for multimodal video content retrieval based on shot frame sampling. Background Technology

[0002] Video retrieval is an information technology that quickly finds relevant video information from a large amount of video data based on the user's search request. It includes content-based video retrieval, compression domain-based video retrieval, semantic-based video retrieval, and dynamic feature extraction, etc. It aims to help users retrieve the video sequences they need from existing databases. Practical applications mainly include digital libraries, news broadcasting, commercial advertising, music videos, distance learning, video archives, and medical applications.

[0003] Currently, video content retrieval is mainly achieved by matching text tags or static keyframes in video files. This method is slow and makes it difficult to integrate different modalities of video file content (such as text content, sentence content, or image content), resulting in reduced retrieval accuracy.

[0004] How to quickly and accurately retrieve relevant video content is a key research issue in the industry. Summary of the Invention

[0005] This invention provides a multimodal video content retrieval method, device, and medium based on shot frame sampling, so as to quickly and accurately retrieve relevant video content.

[0006] According to one aspect of the present invention, a multimodal video content retrieval method based on shot frame sampling is provided, the method comprising:

[0007] Obtain the original video file and segment the original video file to obtain individual video shots;

[0008] At least two representation vectors are determined for each of the video shots, and the representation vectors and metadata of each video shot are stored in the target database to obtain the target video database; wherein, the representation vectors include: text vectors, sentence vectors and image vectors; the metadata of each video shot includes: identification information or timestamp information;

[0009] In response to the input command for the target search content, the target search content is vectorized to obtain the target search vector;

[0010] Determine the reference distances between the target retrieval vector and each representation vector stored in the video database, and determine the pre-selection retrieval score based on the target weight;

[0011] The pre-selected search scores are filtered, the target video shots are determined based on the filtering results, and the target video shots are displayed.

[0012] Optionally, the step of segmenting the original video file to obtain individual video shots includes:

[0013] Extract each video frame image from the original video file, and determine the color histogram and texture features of each video frame image;

[0014] The low-frequency component of the first video frame image is determined, and the Hamming distance between the first video frame image and the second video frame image is determined based on the low-frequency component; wherein the first video frame image and the second video frame image are consecutive frames;

[0015] Determine the color histogram differences and texture feature differences between the first video frame image and the second video frame image;

[0016] The target difference value between the first video frame image and the second video frame image is determined based on the Hamming distance, color histogram difference, and texture feature difference.

[0017] If the target difference value is determined to be greater than a set difference threshold, the first video frame image or the second video frame image is determined as a shot boundary of the original video file;

[0018] The original video file is segmented based on the lens boundaries to obtain individual video shots.

[0019] Optionally, the step of determining at least two representation vectors for each of the video shots and storing each representation vector and the metadata of each video shot in the target database to obtain the target video database includes:

[0020] Determine the motion intensity and content complexity of each video frame image in the target video shot, and perform sparse sampling on the target video shot based on the motion intensity and content complexity of each video frame image to obtain the target video frame image;

[0021] Determine the target key video frame image based on the target video frame image;

[0022] The target video frame image is input into the target visual language model to obtain the text vector, sentence vector, image vector, identification information, and timestamp information of the target video frame; wherein, the identification information includes the identification information of the original video file or the identification information of the target video shot;

[0023] The text vectors, sentence vectors, image vectors, identifier information, and timestamp information are stored according to a preset indexing structure to obtain the target video database.

[0024] Optionally, determining the target key video frame image based on the target video frame image includes:

[0025] Determine the natural language description of the target video frame image, and determine the semantic matching degree between the target video frame image and the natural language description;

[0026] A target heatmap is generated based on each semantic matching degree, and the target key video frame image is determined based on the target heatmap.

[0027] Optionally, the step of responding to the input instruction of the target retrieval content and vectorizing the target retrieval content to obtain the target retrieval vector includes:

[0028] Determine the target text length of the target search content;

[0029] If the length of the target text is determined to be greater than a set length threshold, keywords are extracted from the target search content to obtain target text information that matches the target search content;

[0030] The target text information is vectorized to obtain the target retrieval vector.

[0031] Optionally, determining the reference distances between the target retrieval vector and each representation vector stored in the video database, and determining the pre-selection retrieval score based on the target weight, includes:

[0032] Determine the first similarity between the target retrieval vector and the text vectors stored in the video database, and determine the segment-level text distance based on the first similarity;

[0033] Determine the second similarity between the target retrieval vector and the sentence vectors stored in the video database, and determine the sentence-level text distance based on the second similarity;

[0034] Determine the third similarity between the target retrieval vector and the image vectors stored in the video database, and determine the cross-modal distance between the image and text based on the third similarity;

[0035] The fragment-level text distance, the sentence-level text distance, and the cross-modal distance between the image and text are weighted and fused to obtain the pre-selected retrieval score;

[0036] The weighting parameters are determined by the target weights; the target weights satisfy normalization constraints and are obtained by performing three-dimensional parameter space clustering analysis and performance verification on the labeled test set.

[0037] Optionally, the step of filtering the pre-selected search scores, determining the target video shot based on the filtering results, and displaying the target video shot includes:

[0038] The pre-selected search scores are sorted according to a set order, and at least two reference search scores are determined based on the sorting results.

[0039] Identify the reference video shots corresponding to each of the aforementioned reference searches, and perform content similarity analysis on each of the aforementioned reference video shots;

[0040] If the content similarity result between the first reference video shot and the second reference video shot is greater than the set similarity threshold, then the first reference video shot or the second reference video shot will be filtered out.

[0041] The target video shot is determined from the remaining reference video shots, and the keyframe of the target video shot is determined as the visual cover of the target video shot, and each target video shot is displayed through the target page.

[0042] According to another aspect of the present invention, a multimodal video content retrieval device based on shot frame sampling is provided, the device comprising:

[0043] The video segmentation module is used to acquire the original video file and segment the original video file to obtain individual video shots;

[0044] A video database creation module is used to determine at least two representation vectors for each of the video shots, and store each representation vector and the metadata of each video shot into a target database to obtain a target video database; wherein, the representation vectors include: text vectors, sentence vectors and image vectors; the metadata of each video shot includes: identification information or timestamp information;

[0045] The retrieval vector determination module is used to respond to the input instruction of the target retrieval content, and to vectorize the target retrieval content to obtain the target retrieval vector;

[0046] The pre-selection retrieval score determination module is used to determine the reference distance between the target retrieval vector and each representation vector stored in the video database, and to determine the pre-selection retrieval score based on the target weight;

[0047] The target video determination module is used to filter the pre-selected search scores, determine the target video shots based on the filtering results, and display the target video shots.

[0048] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0049] At least one processor; and

[0050] A memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the multimodal video content retrieval method based on shot frame sampling as described in any embodiment of the present invention.

[0052] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the multimodal video content retrieval method based on shot frame sampling as described in any embodiment of the present invention.

[0053] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the multimodal video content retrieval method based on shot frame sampling as described in any embodiment of the present invention.

[0054] The technical solution of this invention involves acquiring an original video file and segmenting it to obtain individual video shots; determining at least two representation vectors for each video shot and storing the representation vectors and metadata of each video shot in a target database to obtain a target video database; wherein the representation vectors include text vectors, sentence vectors, and image vectors; the metadata of each video shot includes identification information or timestamp information; responding to an input instruction for target retrieval content, the target retrieval content is vectorized to obtain a target retrieval vector; the reference distances between the target retrieval vector and each representation vector stored in the video database are determined, and a pre-selection retrieval score is determined based on the target weight; the pre-selection retrieval scores are filtered, and target video shots are determined based on the filtering results and displayed. This allows for quick and accurate retrieval of relevant video content, improving the user's retrieval experience.

[0055] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of a multimodal video content retrieval method based on shot frame sampling according to Embodiment 1 of the present invention;

[0058] Figure 2 This is a flowchart of a multimodal video content retrieval method based on shot frame sampling according to Embodiment 2 of the present invention;

[0059] Figure 3 This is a flowchart of the creation of a target video database according to Embodiment 2 of the present invention;

[0060] Figure 4 This is a schematic diagram of a video retrieval process according to Embodiment 2 of the present invention;

[0061] Figure 5 This is a schematic diagram of the structure of a multimodal video content retrieval device based on shot frame sampling according to Embodiment 3 of the present invention;

[0062] Figure 6 This is a schematic diagram of the structure of an electronic device that implements the multimodal video content retrieval method based on lens frame sampling according to an embodiment of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0064] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0065] Example 1

[0066] Figure 1 This is a flowchart of a multimodal video content retrieval method based on shot frame sampling according to Embodiment 1 of the present invention. This embodiment is applicable to situations where relevant video shots can be retrieved quickly and accurately based on input content. This method can be executed by a multimodal video content retrieval device based on shot frame sampling, which can be implemented in hardware and / or software and can be configured in electronic devices such as computers, servers, or tablet computers. Figure 1 As shown, the method includes:

[0067] Step 110: Obtain the original video file and segment the original video file to obtain individual video shots.

[0068] The original video file can be any one or more video files, such as news video files, promotional video files, or exciting clips from movies, etc. This embodiment does not limit them.

[0069] Optionally, in this embodiment, each original video file can be obtained from a video library; or it can be obtained from the Internet. In this embodiment, there are no limitations on the method or location of obtaining the original video files.

[0070] In this embodiment, after obtaining the original video file, the original video file can be further segmented to obtain the segmented video shots. In this embodiment, the original video file can be segmented based on shots, scenes, or segments. For example, a video can be segmented into a video shot; a group of related shots can be segmented into a video shot; and the opening, main text, and closing credits can be segmented into video segments.

[0071] In this embodiment, the original video file can be segmented based on shot boundary detection, or it can be segmented based on a semantic field segmentation algorithm. This embodiment does not limit the specific segmentation method.

[0072] Optionally, in this embodiment, segmenting the original video file to obtain video shots may include: extracting each video frame image from the original video file and determining the color histogram and texture features of each video frame image; determining the low-frequency component of the first video frame image and determining the Hamming distance between the first video frame image and the second video frame image based on the low-frequency component; wherein the first video frame image and the second video frame image are consecutive frames; determining the color histogram difference and texture feature difference between the first video frame image and the second video frame image; determining a target difference value between the first video frame image and the second video frame image based on the Hamming distance, color histogram difference, and texture feature difference; if the target difference value is greater than a set difference threshold, determining the first video frame image or the second video frame image as a shot boundary of the original video file; and segmenting the original video file based on the shot boundary to obtain video shots.

[0073] In an optional implementation of this embodiment, after obtaining the original video file, each video frame image in the original video file can be further extracted, and the color histogram and texture features of each video frame image can be determined; furthermore, the low-frequency component of the first video frame image can be determined, and the Hamming distance between the first video frame image and the second video frame image can be determined based on the low-frequency component; wherein, the first video frame image is any video frame image in the original video file, and the first video frame image and the second video frame image are consecutive frames, that is, the second video frame image can be a subsequent frame image of the first video frame image.

[0074] Furthermore, the color histogram difference and texture feature difference between the first video frame image and the second video frame image can be determined; furthermore, a target difference value between the first video frame image and the second video frame image can be determined based on the Hamming distance, color histogram difference, and texture feature difference between them; if the target difference value is determined to be greater than a set difference threshold, then the first video frame image or the second video frame image can be determined as a shot boundary of the original video file; furthermore, the original video file can be segmented based on this shot boundary, for example, the first video frame image can be segmented into a first video shot, and the second video frame image can be segmented into a second video shot.

[0075] In this embodiment, the color histogram (i.e., color histogram) of each video frame image can be determined using the following formula:

[0076] ;

[0077] in, To support different color depth types, calculate the histogram for each frame, using abrupt changes as potential shot cut-off points.

[0078] Texture feature (LBP) 3x3 neighborhood calculation, selecting the center pixel. With its surrounding 8 pixels (Clockwise) Compare the grayscale values ​​of the neighboring pixels with those of the center pixel:

[0079] ;

[0080] Arrange the eight binary results clockwise into an 8-bit binary number, then convert them to decimal as the LBP value for that center point:

[0081] ;

[0082] In this embodiment, texture features are extracted frame by frame from the video sequence to construct spatiotemporal features, which serve as potential features for determining shot switching.

[0083] In this embodiment, the low-frequency components can be obtained by discrete cosine transform. The low-frequency components are concentrated in the upper left corner of the DCT coefficients. After converting to binary representation, the Hamming distance is calculated.

[0084] ;

[0085] Here, A and B are binary strings of the low-frequency DCT components of the same length; the Hamming distance between consecutive frames is calculated for each frame of the video sequence as a potential feature for judging shot switching.

[0086] Furthermore, by combining color space histogram differences, texture features, and low-frequency components, it can be determined whether the intensity change between consecutive frames exceeds a threshold δ.

[0087] ;

[0088] Ultimately, a collection of video shots can be output. .

[0089] Step 120: Determine at least two representation vectors for each of the video shots, and store each representation vector and the metadata of each video shot into the target database to obtain the target video database.

[0090] The representation vectors include: text vectors, sentence vectors, and image vectors; the metadata of each video shot includes: identification information or timestamp information.

[0091] Optionally, in this embodiment, after segmenting the video shots, the representation vectors of different categories for each video shot can be further determined, such as text vectors, sentence vectors, or image vectors. Furthermore, the representation vectors of each video shot, along with identification information or timestamp information, can be added to the target database to obtain the target video database. The identification information can be a unique identifier for each video shot or an identifier for the original video file; this embodiment does not impose any limitations on it.

[0092] Optionally, in this embodiment, determining at least two representation vectors for each video shot and storing each representation vector and the metadata of each video shot in a target database to obtain a target video database may include: determining the motion intensity and content complexity of each video frame image in the target video shot, and performing sparse sampling on the target video shot based on the motion intensity and content complexity of each video frame image to obtain target video frame images; determining target key video frame images based on the target video frame images; inputting the target video frame images into a target visual language model to obtain the text vector, sentence vector, image vector, identification information, and timestamp information of the target video frame; wherein, the identification information includes the identification information of the original video file or the identification information of the target video shot; and storing the text vector, sentence vector, image vector, identification information, and timestamp information according to a preset index structure to obtain the target video database.

[0093] Optionally, in this embodiment, determining the target key video frame image based on the target video frame image may include: determining the natural language description of the target video frame image, and determining the semantic matching degree between the target video frame image and the natural language description; generating a target heatmap based on each semantic matching degree, and determining the target key video frame image based on the target heatmap.

[0094] In this embodiment, the motion intensity of each video frame can be determined based on the following formula:

[0095] ;

[0096] Where N is the size of the sliding window; Let be the image of the (i+k)th frame; W and H are the image width and height, respectively.

[0097] Content complexity can be calculated using the following formula:

[0098] ;

[0099] in, Image entropy; The standard deviation of the gradient; Edge density; .

[0100] Image entropy can be calculated using the following formula: ;in, This represents the probability distribution of the color components.

[0101] Edge density can be calculated based on the following formula: ; ;in, .

[0102] The gradient direction can be calculated based on the following formula: .

[0103] Furthermore, sparse sampling can be performed within each video shot, with enhanced sampling before and after shot transitions to ensure the continuity of shot content and the temporal coherence of the video content. In this embodiment, a dynamic sampling interval strategy is used:

[0104] ;

[0105] in, Let be the sampling interval for the i-th time period; Based on the sampling coefficient (0.5≤ ≤2.0); The basic sampling interval (usually 1 second); The motion sensitivity coefficient (0.2≤ ≤0.8); Complexity sensitivity coefficient (0.1≤ ≤0.6).

[0106] It should be noted that, in this embodiment, the enhanced sampling for lens position changes can effectively ensure the temporal continuity of the perceived frame sequence.

[0107] In an optional implementation of this embodiment, after sparsely sampling the target video shot to obtain the target video frame image, the natural language description of the target video frame image can be further determined, and the semantic matching degree between the target video frame image and the natural language description can be determined; a target heatmap is generated based on each semantic matching degree, and the target key video frame image is determined based on the target heatmap.

[0108] Optionally, in this embodiment, the sparse frame sampling sequence can be input into a pre-fine-tuned large model to generate video content description. Because of the use of a dynamic coefficient frame sampling strategy, the completeness and accuracy of the resulting video content description are significantly improved compared to direct uniform sampling.

[0109] In this embodiment, based on a heatmap, key frames are filtered by recursively segmenting the timeline to ensure a reasonable and uniform distribution of key frames and avoid the loss of time segments. Regions with concentrated scores are selected from the top M frames with the highest matching degree (focused content). Regions with dispersed scores are divided into B regions, and the highest-scoring frame is selected from each region and recursively subdivided until the coverage requirement (globally continuous content) is met.

[0110] Calculation of cumulative scores over an interval (within the interval [a, b]): ;

[0111] Adaptive segmentation point localization: Where t is the timestamp of the first time the cumulative score percentage is ≥50%.

[0112] Multi-objective optimization function (global optimization after recursion): ;

[0113] The first item is the relevance score, and the second item is the coverage score. It is the variance of the keyframe time interval. It is a dynamic adjustment rule:

[0114] ;

[0115] Recursion termination condition:

[0116] ;

[0117] The scheme in this embodiment uses a dynamic trade-off mechanism, introducing a hyperparameter λ to balance the weights of relevance and coverage, and approximates the optimal solution through a clustering recursive algorithm. The optimization objective formula is as follows:

[0118] ;

[0119] in, For the keyframe index set, Indicates query With frames Match score, For hyperparameters, To measure the uniformity of temporal distribution, the specific calculation method is the variance of the time interval between keyframes.

[0120] The threshold (λ) is automatically adjusted based on the video description. For long videos with concentrated events, a smaller λ is used (emphasizing relevance); for events distributed across multiple time points, a larger λ is used (emphasizing coverage). To reduce computational overhead, low-frequency sampling is used to select keyframes, significantly reducing the number of large model calls and improving material processing efficiency.

[0121] Furthermore, the target key video frame images can be input as image sequences for content understanding into the target visual language model (e.g., VLM) to generate video content descriptions (fragment text); sentence-level segmentation is performed based on the fragment text, and frame-text matching is used to achieve a one-to-one correspondence between key frames and sentence-level text, while recording the key frame timestamps; furthermore, video file identification information, scene identification (i.e., video shot identification), fragment text vectors, sentence-level text vectors, key frame feature vectors, and timestamp data can be stored in the target database.

[0122] For example, in this embodiment, the preset index architecture can be as shown in Table 1:

[0123] Table 1

[0124]

[0125] It is understood that the content in the video database can be dynamically added in this embodiment.

[0126] Step 130: In response to the input instruction of the target retrieval content, the target retrieval content is vectorized to obtain the target retrieval vector.

[0127] The target search content can be text content, audio content, or image content, and this embodiment does not limit it.

[0128] In this embodiment, search terms can be entered in the search bar of the target page, which can be the video search page of any website or application.

[0129] Optionally, in this embodiment, after receiving the input instruction for the target search content, the target search content can be converted into a vector form to obtain the target search vector. For example, if the target search content is text content, the text content can be vectorized to obtain the target search vector; if the target search content is speech content, the speech content can be vectorized to obtain the target search vector; if the target search content is image content, the image content can be understood and the understanding result can be vectorized to obtain the target search vector.

[0130] Step 140: Determine the reference distances between the target retrieval vector and each representation vector stored in the video database, and determine the pre-selection retrieval score based on the target weight.

[0131] Optionally, in this embodiment, after obtaining the target retrieval vector, the reference distance between the target retrieval vector and each representation vector stored in the video database can be further determined, and the pre-selection retrieval score can be determined based on the target weight; for example, the similarity between the target retrieval vector and each text vector, sentence vector and image vector can be calculated respectively, and the distance between the target retrieval vector and each database vector can be determined based on the similarity calculation results, which is referred to as the reference distance in this embodiment.

[0132] Furthermore, a pre-selected retrieval score can be obtained based on the target weight, where the target weight can be the weight of the similarity calculation results between the text vector, sentence vector and image vector obtained in advance, and their sum is 1.

[0133] Step 150: Filter the pre-selected search scores, determine the target video shots based on the filtering results, and display the target video shots.

[0134] Optionally, in this embodiment, after calculating each pre-selected search score, the pre-selected search scores can be further filtered, the target video shots can be determined based on the filtering results, and each target video shot can be displayed for the user to select.

[0135] In one optional implementation of this embodiment, after calculating each pre-selected search score, the pre-selected search scores can be sorted in ascending order. Based on the sorting result, a set number of pre-selected search scores (e.g., 10 or 20, which are not limited in this embodiment) are selected from the top, and the video shots corresponding to these pre-selected search scores are determined as target video shots. These target video shots are then displayed on the search results page.

[0136] This embodiment obtains an original video file and segments it to obtain individual video shots. At least two representation vectors are determined for each video shot, and the representation vectors and metadata of each video shot are stored in a target database to obtain a target video database. The representation vectors include text vectors, sentence vectors, and image vectors. The metadata of each video shot includes identification information or timestamp information. In response to an input command for target search content, the target search content is vectorized to obtain a target search vector. Reference distances between the target search vector and each representation vector stored in the video database are determined, and a pre-selection search score is determined based on the target weight. The pre-selection search scores are then filtered, and target video shots are determined based on the filtering results and displayed. This allows for quick and accurate retrieval of relevant video content, improving the user's search experience.

[0137] Example 2

[0138] Figure 2 This is a flowchart of a multimodal video content retrieval method based on shot frame sampling according to Embodiment 2 of the present invention. This embodiment is a further refinement of the above technical solution, and the technical solution in this embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the method includes:

[0139] Step 210: Obtain the original video file and segment the original video file to obtain individual video shots.

[0140] Step 220: Determine at least two representation vectors for each of the video shots, and store each representation vector and the metadata of each video shot into the target database to obtain the target video database.

[0141] Step 230: Determine the target text length of the target retrieval content; if the target text length is greater than a set length threshold, extract keywords from the target retrieval content to obtain target text information that matches the target retrieval content; and vectorize the target text information to obtain the target retrieval vector.

[0142] The length threshold can be set to 50 bytes or 100 bytes, etc., but this embodiment does not limit it.

[0143] Optionally, after receiving the target search content, the text length of the target search content can be further determined, which is referred to as the target text length in this embodiment. Further, if the target text length is determined to be greater than a set length threshold, keyword extraction can be performed on the target search content. For example, the target search content can be input into a large language model to extract key semantic information; thereby obtaining target text information matching the target search content. Further, the target text information can be vectorized to obtain the target search vector.

[0144] In one optional implementation of this embodiment, if the target text is longer than 50 characters, a large language model can be called to extract keywords or condense semantics. This can avoid diluting the semantic vector matching relevance score with excessively long text.

[0145] Step 240: Determine the reference distances between the target retrieval vector and each representation vector stored in the video database, and determine the pre-selection retrieval score based on the target weight.

[0146] Optionally, determining the reference distances between the target retrieval vector and each representation vector stored in the video database, and determining the pre-selected retrieval score based on the target weights, may include: determining a first similarity between the target retrieval vector and the text vectors stored in the video database, and determining a segment-level text distance based on the first similarity; determining a second similarity between the target retrieval vector and the sentence vectors stored in the video database, and determining a sentence-level text distance based on the second similarity; determining a third similarity between the target retrieval vector and the image vectors stored in the video database, and determining an image-text cross-modal distance based on the third similarity; and weighting and fusing the segment-level text distance, the sentence-level text distance, and the image-text cross-modal distance to obtain the pre-selected retrieval score; wherein the weighting parameters are determined by the target weights; the target weights satisfy normalization constraints and are obtained through three-dimensional parameter space clustering analysis and performance verification on the labeled test set.

[0147] In an optional implementation of this embodiment, after obtaining the target retrieval vector, the similarity between the target retrieval vector and the text vectors stored in the video database can be further determined, which is referred to as the first similarity in this embodiment; furthermore, the fragment-level text distance can be determined based on the first similarity.

[0148] Furthermore, the similarity between the target retrieval vector and the sentence vectors stored in the video database can be determined, which is referred to as the second similarity in this embodiment; further, the sentence-level text distance is determined based on the second similarity.

[0149] Furthermore, the similarity between the target retrieval vector and the image vectors stored in the video database can be determined, which is referred to as the third similarity in this embodiment; further, the cross-modal distance between the image and text is determined based on the third similarity.

[0150] Furthermore, the fragment-level text distance, the sentence-level text distance, and the cross-modal distance between the text and image can be weighted and fused to obtain the pre-selected retrieval score.

[0151] In this embodiment, the three-dimensional weighted fusion scores (pre-selection retrieval scores) for fragment distance, sentence-level distance, and cross-modal distance between text and images can be calculated based on the following formulas:

[0152] ;

[0153] in, , , Dynamic optimization is verified through data testing, and the set of data with the best overall performance is selected as the retrieval parameters. In this embodiment, , , These are collectively referred to as target weights.

[0154] In this embodiment, the best matching result is obtained by using the comprehensive score of multi-level and multi-path recall results. This can accurately match specific video materials and corresponding storyboards and timestamps, thus achieving precise jump.

[0155] Step 250: Filter the pre-selected search scores, determine the target video shots based on the filtering results, and display the target video shots.

[0156] Optionally, in this embodiment, filtering the pre-selected search scores, determining target video shots based on the filtering results, and displaying the target video shots may include: sorting the pre-selected search scores in a set order, determining at least two reference search scores based on the sorting results; determining reference video shots corresponding to each reference search, and performing content similarity analysis on each reference video shot; if the content similarity result between the first reference video shot and the second reference video shot is greater than a set similarity threshold, then filtering out either the first reference video shot or the second reference video shot; determining the target video shot from the remaining reference video shots, determining the keyframe of the target video shot as the visual cover of the target video shot, and displaying each target video shot through the target page.

[0157] The order can be set from smallest to largest or from largest to smallest; this embodiment does not limit it.

[0158] In an optional implementation of this embodiment, after sorting the pre-selected search scores, the pre-selected search scores with larger search scores (e.g., the top 20, 30, or 40 in the sequence, which are not limited in this embodiment) can be determined as reference search scores.

[0159] Furthermore, reference video shots corresponding to each reference retrieval score can be identified separately, and content similarity analysis can be performed on each reference video shot; if the content similarity result between the first reference video shot and the second reference video shot is greater than a set similarity threshold (e.g., 0.9 or 0.95), the first reference video shot or the second reference video shot can be filtered out.

[0160] After filtering out reference video shots with high content similarity, target video shots can be determined from the remaining reference video shots. For example, reference video shots with a reference retrieval score greater than a set threshold can be identified as target video shots. Furthermore, keyframes of the target video shots can be extracted and used as visual covers for the target video shots. Each target video shot is then displayed on a target page. The target page can be an input interface for the search content or a separately generated results display page; this embodiment does not limit its scope.

[0161] In this embodiment, the solution responds to the input command for the target search content by vectorizing the target search content to obtain a target search vector; determines the reference distance between the target search vector and each representation vector stored in the video database, and determines a pre-selected search score based on the target weight; filters each of the pre-selected search scores, determines the target video shot based on the filtering results, and displays the target video shot. This approach can accurately determine search results matching the target search content based on a pre-established video database and display the search results, thereby improving the user's search experience.

[0162] To better understand the multimodal video content retrieval method based on shot frame sampling involved in this embodiment, Figure 3 This is a flowchart of the creation process of a target video database according to Embodiment 2 of the present invention, which mainly includes:

[0163] Step 310: Input video file.

[0164] Step 320: Shot segmentation.

[0165] Specifically, the input video is segmented into multiple semantically coherent shots, segments, or scenes.

[0166] Step 330: Dynamic sparse sampling.

[0167] Specifically, in each segmented scene, a dynamic sparse sampling method is used to select keyframes.

[0168] Step 340: Generate prompt text.

[0169] Specifically, based on the keyframes obtained from sampling, descriptive text, i.e. prompts, is generated using a visual language model (VLM).

[0170] Step 350: Generate a heat map.

[0171] Specifically, the matching degree between video frames and accompanying text is calculated to generate a heatmap. The heatmap reflects the relevance of each frame to the text description, helping to identify the keyframes that best represent the video content.

[0172] Step 360: Recursive segmentation sampling.

[0173] Specifically, the sampling process is further optimized through a recursive segmentation algorithm to ensure that the selected timestamps are not only highly relevant but also have broad time coverage. This process may involve multiple iterations until preset conditions are met.

[0174] Step 370: Multimodal feature extraction.

[0175] Specifically, multimodal feature extraction is performed on the selected keyframes and their corresponding text descriptions to generate feature vectors containing both visual and textual information.

[0176] Step 380: Video semantic understanding.

[0177] Specifically, based on the extracted multimodal features, semantic understanding of the video is performed, including but not limited to scene recognition, action recognition, and object detection, in order to deeply analyze the video content.

[0178] Step 390: Data is imported into the database, generating the target video database.

[0179] Specifically, the processed data, including video identifiers, fragment text vectors, sentence-level text vectors, keyframe vectors, timestamps, etc., are stored in an efficient vector database for subsequent rapid retrieval and application.

[0180] Figure 4 This is a schematic diagram of a video retrieval process according to Embodiment 2 of the present invention, which mainly includes:

[0181] Step 410: Text query input.

[0182] Step 420: Generate text vectors.

[0183] Specifically, the input text can be encoded and converted into a form that a computer can process.

[0184] Step 430: Visual similarity matching.

[0185] Step 440: Semantic similarity matching.

[0186] Step 450: Weighted Mixing & Multi-Way Recall.

[0187] Specifically, the system can combine visual and semantic similarity matching results, employing a weighted hybrid strategy to comprehensively consider the contributions of both, and generate a final similarity score. Simultaneously, the system may perform multi-path recall from multiple dimensions (such as scene, action, and object) to ensure the comprehensiveness and accuracy of the retrieval results.

[0188] Step 460: Result sorting, deduplication filtering, and threshold filtering.

[0189] Specifically, search results can be sorted based on comprehensive similarity scores, prioritizing the display of video footage most relevant to the query; duplicate search results can be removed to avoid displaying the same content multiple times, thus improving user experience; a similarity threshold can be set to filter out results below the threshold, ensuring that each returned item has a high degree of relevance.

[0190] Step 470: Video identification, similarity score, and timestamp location.

[0191] The final search results include the following information: video identifier, used to identify specific video files; similarity score, used to reflect the relevance between the query text and the video content; and timestamp location, used to pinpoint the specific time point in the video, allowing users to directly jump to relevant content.

[0192] The above process starts with a text query input, goes through multiple steps such as text processing, multimodal similarity matching, and result optimization, and finally outputs video footage and its detailed information that is highly relevant to the query, achieving accurate retrieval from text to video.

[0193] Example 3

[0194] Figure 5 This is a schematic diagram of a multimodal video content retrieval device based on shot frame sampling according to Embodiment 3 of the present invention. Figure 5 As shown, the device includes: a video segmentation module 510, a video database creation module 520, a retrieval vector determination module 530, a pre-selection retrieval score determination module 540, and a target video determination module 550.

[0195] The video segmentation module 510 is used to acquire the original video file and segment the original video file to obtain various video shots;

[0196] The video database creation module 520 is used to determine at least two representation vectors for each of the video shots, and store each representation vector and the metadata of each video shot into the target database to obtain the target video database; wherein, the representation vectors include: text vectors, sentence vectors and image vectors; the metadata of each video shot includes: identification information or timestamp information;

[0197] The retrieval vector determination module 530 is used to vectorize the target retrieval content in response to the input instruction of the target retrieval content to obtain the target retrieval vector;

[0198] The pre-selection retrieval score determination module 540 is used to determine the reference distance between the target retrieval vector and each representation vector stored in the video database, and to determine the pre-selection retrieval score based on the target weight;

[0199] The target video determination module 550 is used to filter the pre-selected search scores, determine the target video shot based on the filtering results, and display the target video shot.

[0200] In this embodiment, the solution involves obtaining the original video file through a video segmentation module and segmenting it to obtain individual video shots. A video database creation module determines at least two representation vectors for each video shot and stores these representation vectors along with the metadata of each video shot in a target database to obtain a target video database. The representation vectors include text vectors, sentence vectors, and image vectors. The metadata of each video shot includes identification information or timestamp information. A retrieval vector determination module, responding to the input command for the target retrieval content, vectorizes the target retrieval content to obtain a target retrieval vector. A pre-selection retrieval score determination module determines the reference distances between the target retrieval vector and each representation vector stored in the video database and determines a pre-selection retrieval score based on the target weight. A target video determination module filters the pre-selection retrieval scores, determines the target video shots based on the filtering results, and displays the target video shots to quickly and accurately retrieve relevant video content, thus improving the user's search experience.

[0201] In an optional implementation of this embodiment, the video segmentation module 510 is specifically used to extract each video frame image from the original video file and determine the color histogram and texture features of each video frame image.

[0202] The low-frequency component of the first video frame image is determined, and the Hamming distance between the first video frame image and the second video frame image is determined based on the low-frequency component; wherein the first video frame image and the second video frame image are consecutive frames;

[0203] Determine the color histogram differences and texture feature differences between the first video frame image and the second video frame image;

[0204] The target difference value between the first video frame image and the second video frame image is determined based on the Hamming distance, color histogram difference, and texture feature difference.

[0205] If the target difference value is determined to be greater than a set difference threshold, the first video frame image or the second video frame image is determined as a shot boundary of the original video file;

[0206] The original video file is segmented based on the lens boundaries to obtain individual video shots.

[0207] In an optional implementation of this embodiment, the video database creation module 520 is specifically used to determine the motion intensity and content complexity of each video frame image in the target video shot, and to perform sparse sampling on the target video shot based on the motion intensity and content complexity of each video frame image to obtain the target video frame image.

[0208] Determine the target key video frame image based on the target video frame image;

[0209] The target video frame image is input into the target visual language model to obtain the text vector, sentence vector, image vector, identification information, and timestamp information of the target video frame; wherein, the identification information includes the identification information of the original video file or the identification information of the target video shot;

[0210] The text vectors, sentence vectors, image vectors, identifier information, and timestamp information are stored according to a preset indexing structure to obtain the target video database.

[0211] In an optional implementation of this embodiment, the video database creation module 520 includes: a key video frame image determination submodule, used for:

[0212] Determine the natural language description of the target video frame image, and determine the semantic matching degree between the target video frame image and the natural language description;

[0213] A target heatmap is generated based on each semantic matching degree, and the target key video frame image is determined based on the target heatmap.

[0214] In an optional implementation of this embodiment, the retrieval vector determination module 530 is specifically used to determine the target text length of the target retrieval content;

[0215] If the length of the target text is determined to be greater than a set length threshold, keywords are extracted from the target search content to obtain target text information that matches the target search content;

[0216] The target text information is vectorized to obtain the target retrieval vector.

[0217] In an optional implementation of this embodiment, the pre-selection retrieval score determination module 540 is specifically used to determine the first similarity between the target retrieval vector and the text vectors stored in the video database, and to determine the segment-level text distance based on the first similarity;

[0218] Determine the second similarity between the target retrieval vector and the sentence vectors stored in the video database, and determine the sentence-level text distance based on the second similarity;

[0219] Determine the third similarity between the target retrieval vector and the image vectors stored in the video database, and determine the cross-modal distance between the image and text based on the third similarity;

[0220] The fragment-level text distance, the sentence-level text distance, and the cross-modal distance between the image and text are weighted and fused to obtain the pre-selected retrieval score;

[0221] The weighting parameters are determined by the target weights; the target weights satisfy normalization constraints and are obtained by performing three-dimensional parameter space clustering analysis and performance verification on the labeled test set.

[0222] In an optional implementation of this embodiment, the target video determination module 550 is specifically used to sort the pre-selected search scores according to a set order, and determine at least two reference search scores based on the sorting results;

[0223] Identify the reference video shots corresponding to each of the aforementioned reference searches, and perform content similarity analysis on each of the aforementioned reference video shots;

[0224] If the content similarity result between the first reference video shot and the second reference video shot is greater than the set similarity threshold, then the first reference video shot or the second reference video shot will be filtered out.

[0225] The target video shot is determined from the remaining reference video shots, and the keyframe of the target video shot is determined as the visual cover of the target video shot, and each target video shot is displayed through the target page.

[0226] The multimodal video content retrieval device based on shot frame sampling provided in this embodiment of the invention can execute the multimodal video content retrieval method based on shot frame sampling provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0227] The collection, storage, use, processing, transmission, provision, and disclosure of video files involved in the technical solutions of this invention comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0228] Example 4

[0229] Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0230] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0231] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0232] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as multimodal video content retrieval methods based on shot frame sampling.

[0233] In some embodiments, the multimodal video content retrieval method based on shot frame sampling can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the multimodal video content retrieval method based on shot frame sampling described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the multimodal video content retrieval method based on shot frame sampling by any other suitable means (e.g., by means of firmware).

[0234] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0235] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0236] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0237] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0238] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0239] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) in terms of management difficulty and weak business scalability.

[0240] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0241] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

[0242] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements a database detection method as provided in any embodiment of this application.

[0243] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including LANs or WANs—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0244] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0245] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A multi-modal video content retrieval method based on shot frame sampling, characterized in that, The method comprises: acquiring an original video file and splitting the original video file to obtain each video shot; determining at least two representation vectors of each video shot, and storing each representation vector and metadata of each video shot in a target database to obtain a target video database; wherein the representation vectors comprise a text vector, a sentence vector and an image vector; and the metadata of each video shot comprises identification information or timestamp information; in response to an input instruction of target search content, performing vector representation on the target search content to obtain a target search vector; determining a reference distance between the target search vector and each representation vector stored in the video database, and determining a pre-selected search score based on a target weight; screening each pre-selected search score, determining a target video shot based on a screening result, and displaying the target video shot; wherein the determining of the at least two representation vectors of each video shot and the storing of each representation vector and the metadata of each video shot in the target database to obtain the target video database comprises: determining the motion intensity and content complexity of each video frame image in the target video shot, and performing sparse sampling on the target video shot based on the motion intensity and content complexity of each video frame image to obtain a target video frame image; determining a target key video frame image based on the target video frame image; inputting the target key video frame image into a target visual language model to obtain a text vector, a sentence vector, an image vector, identification information and timestamp information of the target video frame; and storing the text vector, the sentence vector, the image vector, the identification information and the timestamp information according to a preset index architecture to obtain the target video database; the determining of the target key video frame image based on the target video frame image comprises: determining a natural language description of the target video frame image, and determining a semantic matching degree between the target video frame image and the natural language description; generating a target heat map based on each semantic matching degree, and determining a target key video frame image based on the target heat map.

2. The multi-modal video content retrieval method based on shot frame sampling according to claim 1, characterized in that, The splitting of the original video file to obtain each video shot comprises: extracting each video frame image in the original video file and determining a color histogram and a texture feature of each video frame image; determining a low-frequency component of a first video frame image, and determining a Hamming distance between the first video frame image and a second video frame image based on the low-frequency component; wherein the first video frame image and the second video frame image are consecutive frames; determining a color histogram difference and a texture feature difference between the first video frame image and the second video frame image; determining a target difference value between the first video frame image and the second video frame image based on the Hamming distance, the color histogram difference and the texture feature difference; in a case where the target difference value is greater than a set difference threshold, determining the first video frame image or the second video frame image as a shot boundary of the original video file; Split the original video file based on the shot boundary to obtain each video shot.

3. The multi-modal video content retrieval method based on shot frame sampling according to claim 1, wherein, The identification information comprises identification information of the original video file or identification information of the target video shot.

4. The multi-modal video content retrieval method based on shot frame sampling according to claim 1, wherein, The target retrieval vector is obtained by performing vectorization representation on the target retrieval content in response to the input instruction of the target retrieval content, and the method comprises: determining a target text length of the target retrieval content; if it is determined that the target text length is greater than a set length threshold, performing keyword extraction on the target retrieval content to obtain target text information matched with the target retrieval content; performing vectorization representation on the target text information to obtain the target retrieval vector.

5. The lens-frame-sampling-based multi-modal video content retrieval method of claim 1, wherein, The target retrieval vector is determined to have a reference distance from each representation vector stored in the video database, and a pre-selected retrieval score is determined based on a target weight, and the method comprises: determining a first similarity between the target retrieval vector and a text vector stored in the video database, and determining a segment-level text distance based on the first similarity; determining a second similarity between the target retrieval vector and a sentence vector stored in the video database, and determining a sentence-level text distance based on the second similarity; determining a third similarity between the target retrieval vector and an image vector stored in the video database, and determining a cross-modal distance between text and image based on the third similarity; performing weighted fusion on the segment-level text distance, the sentence-level text distance, and the cross-modal distance between text and image to obtain the pre-selected retrieval score; wherein the weighting parameters are determined by the target weight; the target weight satisfies a normalization constraint, and is obtained by performing three-dimensional parameter space clustering analysis and performance verification on a labeled test set.

6. The multi-modal video content retrieval method based on shot frame sampling according to claim 1, wherein, The pre-selected retrieval scores are screened, a target video shot is determined based on a screening result, and the target video shot is displayed, and the method comprises: sorting each pre-selected retrieval score in a set order, and determining at least two reference retrieval scores based on a sorting result; determining reference video shots corresponding to each reference retrieval score, and performing content similarity analysis on each reference video shot; if a content similarity result of a first reference video shot and a second reference video shot is greater than a set similarity threshold, performing filtering processing on the first reference video shot or the second reference video shot; determining a target video shot from the remaining reference video shots, determining a key frame of the target video shot as a visual cover of the target video shot, and displaying each target video shot through a target page.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the multi-modal video content retrieval method based on shot frame sampling according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to implement the multi-modal video content retrieval method based on shot frame sampling according to any one of claims 1-6 when executed.

9. A computer program product comprising a computer program which, when executed by a processor, implements the multi-modal video content retrieval method based on shot frame sampling according to any one of claims 1-6.

Images