Multi-stage video clip intelligent retrieval method and system

Abstract

The invention relates to a multi-stage video clip intelligent retrieval method, which comprises the following steps of: analyzing and coding user input content, generating an input picture semantic vector and an input text semantic vector, determining a first candidate clip from a video material library based on the input picture semantic vector, the input text semantic vector and a semantic vector library, performing multi-dimensional fusion scoring on the first candidate fragments, selecting a second candidate fragment from the first candidate fragments according to a multi-dimensional fusion scoring result, and performing deep interaction processing and semantic entity correlation analysis on the second candidate fragment and the user input content to obtain a comprehensive score; and determining a target fragment from the second candidate fragments according to the comprehensive score. Through the method and the device, the problem of inaccurate video clip retrieval is solved, the retrieval range is quickly narrowed based on semantic vector library coarse screening, the screening efficiency is improved, and the screening accuracy is improved through multi-dimensional fusion scoring, deep interaction processing and semantic entity correlation analysis.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method and system for intelligent retrieval of multi-stage video segments. Background Technology

[0002] In the current era of big data, with the rapid development of short video platforms, digital marketing, and AI-generated content, how to quickly and accurately retrieve video clips that meet specific semantic needs from massive video material libraries has become a key technical challenge.

[0003] Existing video retrieval technologies mainly consist of keyword-based retrieval or image-based vector retrieval. Keyword-based retrieval methods rely on video metadata, manually labeled tags, or automatically recognized text information. Their retrieval effectiveness is severely limited by the completeness of the keyword system and the accuracy of the labeling. Furthermore, they cannot understand the deep semantic relationships between the query and the video content, resulting in low recall and poor flexibility of search results. Image-based vector retrieval methods, while improving semantic understanding, only vectorize images, limiting their ability to capture complex query intent and easily leading to imprecise search results. Summary of the Invention

[0004] This application provides a multi-stage intelligent video clip retrieval method, system, electronic device, and storage medium to at least solve the problem of inaccurate video clip retrieval in related technologies.

[0005] In a first aspect, embodiments of this application provide a multi-stage intelligent video segment retrieval method, the method comprising: The user input is parsed and encoded to generate an input screen semantic vector and an input text semantic vector. Based on the input screen semantic vector, the input text semantic vector, and the semantic vector library, a first candidate segment is determined from the video material library, wherein the semantic vector library is constructed based on the video material library. Perform multidimensional fusion scoring on the first candidate segment, and select a preset number of segments as the second candidate segment based on the multidimensional fusion scoring results; The second candidate segment and the user input content are subjected to deep interaction processing and semantic entity correlation analysis respectively to obtain a comprehensive score for each second candidate segment. Based on the comprehensive score, the target segment is determined from the second candidate segments.

[0006] In some embodiments, the semantic vector library includes a visual semantic vector library and a text semantic vector library; determining the first candidate segment from the video material library based on the input visual semantic vector, the input text semantic vector, and the semantic vector library includes: A structured query language is generated based on user input, and the structured query language is matched against an SQLite database to obtain a preliminary set of candidate fragments. The preliminary candidate fragment set is matched with the image semantic vector library and the text semantic vector library respectively to obtain image vector subsets and text vector subsets; Using FAISS, k-nearest neighbor search is performed on the subset of the input image vector and the subset of the input text vector based on the input image semantic vector and the input text semantic vector, and the first candidate segment is determined based on the search results.

[0007] In some embodiments, the multidimensional fusion scoring of the first candidate fragment includes: Based on the input image semantic vector and the image semantic vector of the first candidate segment, determine the image semantic score; Based on the input text semantic vector and the text semantic vector of the first candidate segment, the dialogue semantic score is determined; Based on a preset tag system, the tag similarity between the user input content and the first candidate segment is calculated to obtain a dynamic tag matching score; The image semantic score, dialogue semantic score, and dynamic tag matching score are linearly weighted and fused to obtain a multi-dimensional weighted score.

[0008] In some embodiments, the process of performing deep interaction processing and semantic entity relevance analysis on the second candidate fragment and the user input content, respectively, to obtain a comprehensive score for each second candidate fragment includes: Based on the interactive encoder, the scene description and dialogue of the second candidate segment are deeply interacted with the user input content to obtain the fusion score of each second candidate segment; Extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score; Based on the fusion score and the object overlap score, a comprehensive score is determined for each of the second candidate segments, and a target segment is determined from the second candidate segments based on the comprehensive score.

[0009] In some embodiments, the user input includes user screen queries and user dialogue queries; the fusion score for each of the second candidate segments includes: The user screen query is concatenated with the screen description of the second candidate segment to obtain a screen input pair; The user's dialogue query is concatenated with the dialogue text of the second candidate segment to obtain a dialogue input pair; The image input pair and the dialogue input pair are respectively input into the interactive encoder to obtain the raw image score and the raw dialogue score; The fusion score is obtained based on the original image score and the original dialogue score.

[0010] In some embodiments, obtaining the fusion score based on the original image score and the original dialogue score includes: Based on preset screen weights and dialogue weights, the original screen scores and the original dialogue scores are weighted and fused to obtain a weighted score; The weighted scores are obtained by nonlinearly mapping the weighted scores using the hyperbolic tangent function.

[0011] In some embodiments, calculating the semantic similarity between the first core entity and the second core entity to obtain the object overlap score includes: Extract the first core entity set from the user input content, and obtain the second core entity set corresponding to the second candidate fragment; For each entity in the first core entity set, calculate the cosine similarity between its semantic vector and the semantic vector of each entity in the second core entity set, and take the largest cosine similarity as the best matching score. The overlap score of the objects is obtained by taking the arithmetic mean of the best matching scores of all entities in the first core entity set.

[0012] Secondly, embodiments of this application provide a multi-stage intelligent video segment retrieval system, the system comprising: The first selection module is used to parse and encode the user input content, generate input screen semantic vector and input text semantic vector, and determine the first candidate segment from the video material library based on the input screen semantic vector, the input text semantic vector and the semantic vector library, wherein the semantic vector library is constructed based on the video material library; The second selection module is used to perform multi-dimensional fusion scoring on the first candidate segment, and select a preset number of segments as the second candidate segments based on the multi-dimensional fusion scoring results from the first candidate segments. The third selection module is used to perform deep interactive processing and semantic entity correlation analysis on the second candidate fragment and the user input content, respectively, to obtain a comprehensive score for each second candidate fragment, and to determine the target fragment from the second candidate fragments based on the comprehensive score.

[0013] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multi-stage intelligent video segment retrieval method as described in the first aspect above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-stage intelligent video segment retrieval method as described in the first aspect above.

[0015] Compared to related technologies, the multi-stage intelligent video clip retrieval method provided in this application has the following characteristics: First, it uses a semantic vector library for rapid coarse screening to quickly narrow down the search scope. Second, it employs multi-dimensional fusion scoring for screening, comprehensively considering multiple indicators to avoid bias from a single indicator. Third, it utilizes deep interactive processing and semantic entity relevance analysis to uncover the semantic logical relationship between the input content and the clips, improving the efficiency and accuracy of clip screening and solving the problem of inaccurate video clip retrieval. Furthermore, it avoids missed or false detections through dual matching of visual content and dialogue text. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a multi-stage intelligent video segment retrieval method according to an embodiment of this application; Figure 2 This is a structural block diagram of a multi-stage video segment intelligent retrieval system according to an embodiment of this application; Figure 3 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0018] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0019] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0020] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0021] This embodiment provides a multi-stage intelligent video segment retrieval method. Figure 1 This is a flowchart of a multi-stage intelligent video segment retrieval method according to an embodiment of this application, such as... Figure 1As shown, the process includes the following steps: Step S101: Parse and encode the user input content to generate input screen semantic vector and input text semantic vector. Based on the input screen semantic vector, input text semantic vector and semantic vector library, determine the first candidate segment from the video material library. The semantic vector library is constructed based on the video material library.

[0022] A hybrid retrieval architecture combining structured initial screening and multimodal semantic recall is adopted: First, based on user-input keywords, classification labels, and time range, keyword matching is used to perform efficient filtering on a pre-built SQLite database to generate an initial candidate set; then, a dual-channel vector index of image semantics and text semantics is constructed, and a pre-trained Bi-Encoder model is used to map natural language queries to the corresponding semantic spaces, and k-nearest neighbor recall is performed in parallel in FAISS to achieve high recall matching and obtain the first candidate fragment.

[0023] The image semantic vector is generated by encoding the video's shot descriptions, image types, and various visual tags, while the text semantic vector comes from the video's dialogue or script. User input is parsed into two parts: image description query Q. v With dialogue search Q d Using a pre-trained bidirectional encoder model (Bi-Encoder), such as BAAI / bge-large-zh-v1.5, Q... v With Q d Encoded separately as normalized high-dimensional vectors: Input image semantic vector v q With the input text semantic vector d q .

[0024] In some embodiments, the semantic vector library includes a video semantic vector library and a text semantic vector library; step S101, which determines the first candidate segment from the video material library based on the input video semantic vector, the input text semantic vector, and the semantic vector library, includes: Step S1011: Generate a structured query language based on user input, and match the structured query language against the SQLite database to obtain a preliminary candidate fragment set.

[0025] The system receives deterministic filtering conditions input by the user. In this embodiment, the user input includes, but is not limited to, keywords, duration ranges, and preset category tags. The category tags cover semantic dimensions such as image type and emotional atmosphere. A pre-query is performed on the backend metadata database (SQLite database) using Structured Query Language (SQL). This fully utilizes the database indexing mechanism to quickly filter out video segments that do not meet the basic conditions, generating a preliminary candidate segment set. This stage significantly narrows the candidate range for subsequent semantic calculations, providing a fundamental guarantee for overall system efficiency.

[0026] It should be noted that the keywords support Boolean logical expressions, such as "(A|B) AND C AND NOT D", which can be achieved by combining SQL operators such as LIKE, NOT LIKE, and OR, thereby providing users with flexible and efficient hard filtering capabilities.

[0027] Step S1012: Match the preliminary candidate fragment set with the image semantic vector library and the text semantic vector library respectively to obtain the image vector subset and the text vector subset.

[0028] This embodiment maintains two independent high-dimensional vector index libraries, corresponding to the image semantics and text semantics of video clips, respectively.

[0029] Step S1013: Using FAISS, k-nearest neighbor search is performed in the subset of image vectors and the subset of text vectors based on the input image semantic vector and the input text semantic vector, and the first candidate segment is determined based on the search results.

[0030] Semantic matching is performed on the initially screened candidate segments. FAISS (Facebook AISimilarity Search) is used to perform a k-nearest neighbor search in the corresponding vector index, efficiently recalling the top-K semantically most similar candidate segments based on Euclidean distance.

[0031] Traditional video filtering often relies on single-dimensional retrieval (such as based solely on text tags or solely on image features), which can easily lead to problems such as text matching but image mismatch, or similar images but semantically disconnected text. This embodiment constructs a dual-library architecture of "image semantic vector library + text semantic vector library," simultaneously matching from both visual and semantic dimensions to achieve audio-visual semantic alignment, thereby reducing matching bias at its source.

[0032] In existing technologies, structured queries can only achieve precise keyword matching and cannot handle fuzzy semantics or latent needs (e.g., if a user inputs "romantic seaside atmosphere," keyword matching alone cannot capture the emotional semantics of "romantic"); while simple semantic vector matching lacks the efficiency of structured filtering and is difficult to quickly narrow down the search scope. This solution uses a hierarchical logic of "structured query → preliminary filtering → dual-database semantic matching → precise search." First, it uses SQLite structured queries to quickly filter irrelevant fragments (reducing the amount of data for subsequent semantic matching), and then uses a semantic vector database to mine deep relationships, achieving a combination of "precise filtering + fuzzy semantic matching."

[0033] Furthermore, this embodiment introduces the FAISS (Facebook AI Similarity Search) toolkit to perform k-nearest neighbor search on the "image vector subset + text vector subset". By utilizing FAISS's efficient indexing mechanism (such as IVF and HNSW indexes), the complexity of vector similarity calculation is greatly reduced, enabling fast and accurate retrieval under large data volumes and resolving the contradiction between semantic matching accuracy and retrieval efficiency.

[0034] Step S102: Perform multi-dimensional fusion scoring on the first candidate segment, and select a preset number of segments from the first candidate segment as the second candidate segment based on the multi-dimensional fusion scoring results.

[0035] In some embodiments, step S102 of performing multidimensional fusion scoring on the first candidate fragment includes: Step S1021: Determine the semantic score of the image based on the input image semantic vector and the image semantic vector of the first candidate segment.

[0036] Step S1022: Determine the dialogue semantic score based on the input text semantic vector and the text semantic vector of the first candidate segment.

[0037] Step S1023: Based on the preset tag system, calculate the tag similarity between the user input content and the first candidate segment to obtain the dynamic tag matching score.

[0038] Step S1024: Linear weighted fusion of the image semantic score, dialogue semantic score and dynamic tag matching score is performed to obtain a multi-dimensional weighted score.

[0039] The system performs an initial sorting of the candidate segments recalled by dual-channel retrieval, and integrates three types of correlation signals from image semantics, dialogue semantics, and tag system.

[0040] For any candidate segment C i Let its image vector be v. i The text vector is l iGiven that all vectors have been normalized, its semantic segmentation S... sem (C i This can be represented as:

[0041] Semantic Segmentation of Dialogue dia (C i This can be represented as:

[0042] S sem (C i ) and S dia (C i In mathematics, it is equivalent to cosine similarity and is used to measure the degree of matching between input content and candidate judgments in their respective semantic spaces.

[0043] Calculate the screen description query Q using the Bi-Encoder model v With each tag T in the tagging system j (e.g., "warmth" or "achievement display") semantic similarity Sim(Q) v ,T j The tagging system is divided into several semantic dimensions D. k For example, the emotional atmosphere dimension D1, the content function dimension D2, etc. For candidate segment C... i In dimension D k The set of tags owned by the entity is denoted as Tags(C). i D k The label matching score for semantic dimension k is S. tag (C i D k The similarity between the tags contained in the current segment and the query is defined as the average value multiplied by the preset weight w of that dimension. k ,Right now:

[0044] Final dynamic tag matching score S tag (C i This is obtained by normalizing and weighting the scores across all dimensions to ensure that its value range is comparable to other semantic scores.

[0045] Based on this, candidate fragment C is generated by linearly weighting and fusing the three sub-scores. i Multidimensional weighted score S total (C i ): S total (C i )=α·S sem(C i ) +β·S dia (C i ) +γ·S tag (C i ) Where α, β, and γ are configurable non-negative weight parameters, satisfying α + β + γ = 1. Based on S... total (C i The candidate set is sorted, and the top-N segments (e.g., the top 20) are selected as the second candidate segments and then proceed to the subsequent deep reordering stage.

[0046] Optionally, the initial tag score weight γ is set to 0, at which point the tag score enhancement function is not enabled; upon receiving a tag instruction from the user (e.g., "more emphasis on emotion", "more emphasis on content function", or "more emphasis on presentation techniques"), the tag score enhancement function is enabled in response to the tag instruction, γ is adjusted to a preset non-zero value, and a multi-dimensional weighted score S is calculated. total (C i ).

[0047] Step S103: Perform deep interactive processing and semantic entity correlation analysis on the second candidate fragments and the user input content respectively to obtain the comprehensive score of each second candidate fragment. Based on the comprehensive score, determine the target fragment from the second candidate fragments.

[0048] For candidate segments in the initial screening and recall stages, a re-ranking score is generated through deep interaction using an interactive encoder (Cross-Encoder) model, taking into account both image semantic similarity and dialogue semantic similarity. A semantic entity enhancement module is also introduced to perform fine-grained re-ranking, thereby improving the relevance of the results.

[0049] In some embodiments, step S103 involves performing deep interactive processing and semantic entity relevance analysis on the second candidate fragment and the user input content, respectively, to obtain a comprehensive score for each second candidate fragment, including: Step S1031: Based on the interactive encoder, the scene description and dialogue of the second candidate segment are deeply interacted with the user input content to obtain the fusion score of each second candidate segment.

[0050] Cross-Encoder models, such as cross-encoder / mmarco-mMiniLMv2-L12-H384-v1, can be abstracted into a mapping function f. CE Text pairs A ,text B The correlation score is mapped to a real number.

[0051] In some embodiments, the user input includes user screen query and user dialogue query; step S1031 specifically includes: Step S201: The user screen query and the screen description of the second candidate segment are concatenated to obtain a screen input pair.

[0052] Step S202: Concatenate the user's dialogue query with the dialogue text of the second candidate segment to obtain a dialogue input pair.

[0053] Step S203: Input the screen input pair and the dialogue input pair into the interactive encoder respectively to obtain the original screen score and the original dialogue score.

[0054] Step S204: Obtain the fusion score based on the original image score and the original dialogue score.

[0055] For each candidate fragment C i Construct separate screen input pairs and dialogue input pairs: screen input pairs are generated by the user querying Q on the screen. v Image description D of the candidate clip i Composed of multiple parts, represented as [CLS]Q v [SEP]D i [SEP]; Dialogue input is used by user dialogue query Q d The dialogue text L of the candidate segment i Composed of multiple parts, represented as [CLS]Q d [SEP]L i [SEP]. The two pairs of inputs mentioned above are fed into the Cross-Encoder model to obtain the original image resolution R. v (i)=f CE (P v (i) and the original score of the dialogue R d (i)=f CE (P d (i)).

[0056] In some embodiments, step S204 specifically includes: Step S2041: Based on preset screen weights and dialogue weights, the original screen scores and original dialogue scores are weighted and fused to obtain a weighted score.

[0057] Step S2042: The weighted scores are nonlinearly mapped using the hyperbolic tangent function to obtain the fused scores.

[0058] The original scores for visuals and dialogue are weighted by a preset internal weight λ. v With λ d (For example, 0.8 and 0.2) are combined into a weighted score R. fused (i): R fused(i) = λ v ·R v (i)+λ d ·R d (i) Since the logits score output by the Cross-Encoder has an unpredictable range, which is detrimental to subsequent processing and sorting, this embodiment uses a hyperbolic tangent function for nonlinear mapping and introduces a temperature coefficient τ to adjust the sensitivity of the mapping.

[0059] Mapped fusion score R final (i) is restricted to the interval (-1, 1).

[0060] Step S1032: Extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score.

[0061] In some embodiments, step S1032 specifically includes: Step S301: Extract the first core entity set from the user input content and obtain the second core entity set corresponding to the second candidate fragment.

[0062] Step S302: For each entity in the first core entity set, calculate the cosine similarity between its semantic vector and the semantic vector of each entity in the second core entity set, and take the largest cosine similarity as the best matching score.

[0063] Step S303: Calculate the arithmetic mean of the best matching scores of all entities in the first core entity set to obtain the object overlap score.

[0064] Using natural language processing tools (such as spaCy) to query Q from the user screen v Automatic extraction of the core noun entity set E from the Chinese q (The first core entity), such as "dog," "cell phone," "desk," etc. For each candidate fragment C i Obtain its pre-extracted second core entity set E i Unlike simple keyword matching, this embodiment maps all entities to semantic vectors using a Bi-Encoder model and calculates the semantic similarity between the query entity set and the candidate entity set. Specifically, for each query entity e... j ∈E q Calculate its relationship with the second core entity set E iThe cosine similarity of all entities in the query entity is calculated, and the maximum value is taken as the best matching score for that entity. Then, the arithmetic mean of the best matching scores for all query entities is calculated to obtain the object similarity score (Object_Similarity(C)). i ):

[0065] Step S1033: Determine the comprehensive score of each second candidate segment based on the fusion score and the object overlap score, and determine the target segment from the second candidate segments based on the comprehensive score.

[0066] The fusion score is used as the core score for deep interaction re-ranking, and the object overlap score is used as a multiplicative enhancement factor, applied to the aforementioned fine-ranking score, to form the final score (C). i ) is represented as: Final_Score(C i ) =R fused (i)·(1+w obj ·Object_Similarity(C i )) Among them, w obj This is a non-negative adjustable parameter used to control the strength of the influence of entity matching on the final ranking.

[0067] This mechanism ensures that, given overall semantic fit, explicitly mentioned key objects can significantly improve the final ranking of candidate segments, thereby enhancing the relevance of the results and user satisfaction.

[0068] This embodiment uses an interactive encoder to perform two-way deep interaction between the visual description and dialogue of the second candidate segment and the user's input content. This can uncover the semantic relationship, emotional fit, and logical consistency between the input and the segment, fully restore the user's real needs, and avoid the screening bias caused by misjudgment of the interaction intent.

[0069] By using the logic of "extracting core entities + calculating semantic similarity", the core entities (such as key objects / scenes like "child", "kite", and "seaside") are first extracted from the user input content and candidate fragments. Then, the degree of matching is quantified through semantic similarity algorithm to obtain the object overlap score, thereby achieving accurate and quantifiable entity matching.

[0070] By using a two-dimensional decision-making process of "fusion score + object overlap score", the system considers both the overall interaction fit between the input and the fragment (fusion score) and emphasizes the accurate matching of core entities (object overlap score), forming a dual verification of "overall fit + core focus", which reduces the probability of misselection from the root.

[0071] Through the above steps, the first stage uses a semantic vector library for rapid coarse screening to quickly narrow down the search scope. The second stage uses multi-dimensional fusion scoring for screening, comprehensively considering multiple indicators to avoid bias from a single indicator. The third stage uses deep interactive processing and semantic entity relevance analysis to uncover the semantic logical relationship between the input content and the segment, improving the efficiency and accuracy of segment screening and solving the problem of inaccurate video segment retrieval. Simultaneously, dual matching of visual content and dialogue text avoids missed or false detections.

[0072] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0073] This embodiment also provides a multi-stage intelligent video segment retrieval system, which is used to implement the above embodiments and preferred embodiments, and will not be repeated hereafter. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0074] Figure 2 This is a structural block diagram of a multi-stage intelligent video segment retrieval system according to an embodiment of this application, such as... Figure 2 As shown, the system includes: The first selection module 41 is used to parse and encode the user input content, generate input screen semantic vector and input text semantic vector, and determine the first candidate segment from the video material library based on the input screen semantic vector, the input text semantic vector and the semantic vector library. The semantic vector library is constructed based on the video material library.

[0075] The second selection module 42 is used to perform multi-dimensional fusion scoring on the first candidate segment and select a preset number of segments from the first candidate segment as the second candidate segment based on the multi-dimensional fusion scoring results.

[0076] The third selection module 43 is used to perform deep interactive processing and semantic entity correlation analysis on the second candidate fragments and the user input content, respectively, to obtain the comprehensive score of each second candidate fragment, and to determine the target fragment from the second candidate fragments based on the comprehensive score.

[0077] In some embodiments, the semantic vector library includes a screen semantic vector library and a text semantic vector library; the first selection module 41 includes: The initial screening module is used to generate a structured query language based on user input, and then match the structured query language against the SQLite database to obtain a preliminary set of candidate fragments.

[0078] The matching module is used to match the initial candidate fragment set with the image semantic vector library and the text semantic vector library respectively, to obtain image vector subsets and text vector subsets.

[0079] The selection module is used to perform k-nearest neighbor search in the subsets of image vectors and text vectors based on the input image semantic vector and the input text semantic vector using FAISS, and determine the first candidate segment based on the search results.

[0080] In some embodiments, the second selection module 42 includes: The image scoring module is used to determine the image semantic score based on the input image semantic vector and the image semantic vector of the first candidate segment.

[0081] The dialogue scoring module is used to determine the dialogue semantic score based on the semantic vector of the input text and the semantic vector of the first candidate segment.

[0082] The tag scoring module is used to calculate the tag similarity between the user input content and the first candidate segment based on a preset tag system, and obtain a dynamic tag matching score.

[0083] The comprehensive scoring module is used to linearly weight and fuse the image semantic score, dialogue semantic score, and dynamic tag matching score to obtain a multi-dimensional weighted score.

[0084] In some embodiments, the third selection module 43 includes: The deep interaction analysis module is used to perform deep interaction between the image description and dialogue of the second candidate segment and the user input content based on the interactive encoder, so as to obtain the fusion score of each second candidate segment. The semantic entity analysis module is used to extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score. The target segment determination module is used to determine the comprehensive score of each second candidate segment based on the fusion score and the object overlap score, and to determine the target segment from the second candidate segments based on the comprehensive score.

[0085] In some embodiments, the deep interaction analysis module includes: The image pair generation module is used to concatenate the user's image query with the image description of the second candidate segment to obtain an image input pair.

[0086] The dialogue pair generation module is used to concatenate the user's dialogue query with the dialogue text of the second candidate segment to obtain a dialogue input pair.

[0087] The raw score calculation module is used to input the screen input pairs and dialogue input pairs into the interactive encoder to obtain the screen raw score and dialogue raw score.

[0088] The fusion module is used to obtain a fusion score based on the original image score and the original dialogue score.

[0089] In some embodiments, the fusion module includes: The weighting module is used to weight and fuse the original scores of the visuals and dialogues based on preset visual weights and dialogue weights to obtain a weighted score.

[0090] The mapping module is used to perform a nonlinear mapping on the weighted scores using the hyperbolic tangent function to obtain the fused scores.

[0091] In some embodiments, the semantic entity analysis module includes: The entity acquisition module is used to extract the first core entity set from the user input content and obtain the second core entity set corresponding to the second candidate fragment.

[0092] The similarity calculation module is used to calculate the cosine similarity between the semantic vector of each entity in the first core entity set and the semantic vector of each entity in the second core entity set, and to take the highest cosine similarity as the best matching score.

[0093] The overlap determination module is used to calculate the arithmetic mean of the best matching scores of all entities in the first core entity set to obtain the object overlap score.

[0094] Through the modules described above, the first stage uses a semantic vector library for rapid coarse screening to quickly narrow down the search scope. The second stage uses multi-dimensional fusion scoring for screening, comprehensively considering multiple indicators to avoid bias from a single indicator. The third stage uses deep interactive processing and semantic entity relevance analysis to uncover the semantic logical relationship between the input content and the segments, improving the efficiency and accuracy of segment screening and solving the problem of inaccurate video segment retrieval. Simultaneously, dual matching of visual content and dialogue text avoids missed or false detections.

[0095] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0096] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0097] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0098] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: S1 parses and encodes the user input content to generate input screen semantic vectors and input text semantic vectors. Based on the input screen semantic vectors, input text semantic vectors, and the semantic vector library, the first candidate segment is determined from the video material library. The semantic vector library is constructed based on the video material library.

[0099] S2, perform multi-dimensional fusion scoring on the first candidate segment, and select a preset number of segments from the first candidate segment as the second candidate segment based on the multi-dimensional fusion scoring results.

[0100] S3. Perform deep interactive processing and semantic entity correlation analysis on the second candidate fragments and the user input content respectively to obtain the comprehensive score of each second candidate fragment. Based on the comprehensive score, determine the target fragment from the second candidate fragments.

[0101] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0102] In one embodiment, Figure 3 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 3 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 3 As shown, this electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-stage intelligent video segment retrieval method.

[0103] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0105] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0106] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-stage intelligent video segment retrieval method, characterized in that, The method includes: The user input is parsed and encoded to generate an input screen semantic vector and an input text semantic vector. Based on the input screen semantic vector, the input text semantic vector, and the semantic vector library, a first candidate segment is determined from the video material library, wherein the semantic vector library is constructed based on the video material library. Perform multidimensional fusion scoring on the first candidate segment, and select a preset number of segments as the second candidate segment based on the multidimensional fusion scoring results; The second candidate segment and the user input content are subjected to deep interaction processing and semantic entity correlation analysis respectively to obtain a comprehensive score for each second candidate segment. Based on the comprehensive score, the target segment is determined from the second candidate segments.

2. The method according to claim 1, characterized in that, The semantic vector library includes a visual semantic vector library and a text semantic vector library; the step of determining the first candidate segment from the video material library based on the input visual semantic vector, the input text semantic vector, and the semantic vector library includes: A structured query language is generated based on user input, and the structured query language is matched against an SQLite database to obtain a preliminary set of candidate fragments. The preliminary candidate fragment set is matched with the image semantic vector library and the text semantic vector library respectively to obtain image vector subsets and text vector subsets; Using FAISS, k-nearest neighbor search is performed on the subset of the input image vector and the subset of the input text vector based on the input image semantic vector and the input text semantic vector, and the first candidate segment is determined based on the search results.

3. The method according to claim 1, characterized in that, The multidimensional fusion scoring of the first candidate fragment includes: Based on the input image semantic vector and the image semantic vector of the first candidate segment, determine the image semantic score; Based on the input text semantic vector and the text semantic vector of the first candidate segment, the dialogue semantic score is determined; Based on a preset tag system, the tag similarity between the user input content and the first candidate segment is calculated to obtain a dynamic tag matching score; The image semantic score, dialogue semantic score, and dynamic tag matching score are linearly weighted and fused to obtain a multi-dimensional weighted score.

4. The method according to claim 1, characterized in that, The process of performing deep interactive processing and semantic entity relevance analysis on the second candidate fragment and the user input content, respectively, to obtain a comprehensive score for each second candidate fragment includes: Based on the interactive encoder, the scene description and dialogue of the second candidate segment are deeply interacted with the user input content to obtain the fusion score of each second candidate segment; Extract the first core entity from the user input content and the second core entity from the second candidate fragment, calculate the semantic similarity between the first core entity and the second core entity, and obtain the object overlap score; Based on the fusion score and the object overlap score, a comprehensive score is determined for each of the second candidate segments, and a target segment is determined from the second candidate segments based on the comprehensive score.

5. The method according to claim 4, characterized in that, The user input includes user screen query and user dialogue query; the fusion score obtained for each second candidate segment includes: The user screen query is concatenated with the screen description of the second candidate segment to obtain a screen input pair; The user's dialogue query is concatenated with the dialogue text of the second candidate segment to obtain a dialogue input pair; The image input pair and the dialogue input pair are respectively input into the interactive encoder to obtain the raw image score and the raw dialogue score; The fusion score is obtained based on the original image score and the original dialogue score.

6. The method according to claim 5, characterized in that, The process of obtaining the fusion score based on the original image score and the original dialogue score includes: Based on preset screen weights and dialogue weights, the original screen scores and the original dialogue scores are weighted and fused to obtain a weighted score; The weighted scores are obtained by nonlinearly mapping the weighted scores using the hyperbolic tangent function.

7. The method according to claim 4, characterized in that, The calculation of the semantic similarity between the first core entity and the second core entity to obtain the object overlap score includes: Extract the first core entity set from the user input content, and obtain the second core entity set corresponding to the second candidate fragment; For each entity in the first core entity set, calculate the cosine similarity between its semantic vector and the semantic vector of each entity in the second core entity set, and take the largest cosine similarity as the best matching score. The overlap score of the objects is obtained by taking the arithmetic mean of the best matching scores of all entities in the first core entity set.

8. A multi-stage intelligent video segment retrieval system, characterized in that, The system includes: The first selection module is used to parse and encode the user input content, generate input screen semantic vector and input text semantic vector, and determine the first candidate segment from the video material library based on the input screen semantic vector, the input text semantic vector and the semantic vector library, wherein the semantic vector library is constructed based on the video material library; The second selection module is used to perform multi-dimensional fusion scoring on the first candidate segment, and select a preset number of segments as the second candidate segments based on the multi-dimensional fusion scoring results from the first candidate segments. The third selection module is used to perform deep interactive processing and semantic entity correlation analysis on the second candidate fragment and the user input content, respectively, to obtain a comprehensive score for each second candidate fragment, and to determine the target fragment from the second candidate fragments based on the comprehensive score.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-stage intelligent video segment retrieval method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the multi-stage intelligent video segment retrieval method as described in any one of claims 1 to 7.

Images