A video content pre-analysis method, retrieval method, system, and storage medium based on a combination of large models, embedding, and reranking.

Abstract

This invention discloses a video content pre-analysis method, retrieval method, system, and storage medium based on a combination of large-scale models, embedding, and reranking. The method includes the following steps: calling a large-scale model to perform content description and structured analysis on the video content of offline videos; converting the output text description into vectors to form first vector data; converting the output structured description into a structured format to form a first key information list; converting each element in the first key information list into vectors to form second vector data; storing the first and second vector data in a vector database and establishing a relationship between the first and second vector data. By combining a large-scale model with embedding for video content pre-analysis, subsequent retrieval no longer relies on the large-scale model, reducing the consumption of large-scale model tokens, improving video retrieval accuracy, and reducing response time.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a video content pre-analysis method, retrieval method, system, and storage medium based on a combination of large models, embedding, and reranking. Background Technology

[0002] With the explosive growth of video data, retrieving the required data quickly and accurately from this data has become a major technical challenge. Traditional pure information retrieval typically employs a retrieval method combining large models and embeddings, but this approach suffers from low accuracy, slow retrieval speed, and high retrieval costs due to the consumption of large model tokens. Summary of the Invention

[0003] The purpose of this invention is to provide a video content pre-analysis method, retrieval method, system, and storage medium based on a combination of large models, embedding, and reranking, in order to solve the problems of low accuracy, slow retrieval speed, and high retrieval cost in video data retrieval.

[0004] This invention is achieved through the following technical solution:

[0005] A video content pre-analysis method based on a combination of large models, embedding, and reranking includes the following steps:

[0006] The large model is invoked to perform content description and structured analysis on the video content of offline videos, and outputs a text description of the video content and a structured description of the video content, respectively.

[0007] The Embedding model is used to convert the output text description into vectors, forming the first vector data;

[0008] The output structured description is converted into a structured format to form a first key information list. The first key information list consists of multiple elements, represented as: [first attribute keyword] [second attribute keyword] [third attribute keyword]...;

[0009] The Embedding model is used to convert each element in the first key information list into a vector, forming the second vector data;

[0010] Store the first vector data and the second vector data into the vector database, and establish the association between the first vector data and the second vector data.

[0011] In some embodiments, the [first attribute keyword] is a keyword representing the target, the [second attribute keyword] is a quantifier used to modify the target, and the [third attribute keyword] is an adjective or adverb used to modify the target.

[0012] In some embodiments, a unique ID is configured for the first vector data in the data structure of the first vector data, and a parentId corresponding to the ID of the first vector data is configured in the data structure of the second vector data. The association between the first vector data and the second vector data is established through the correspondence between parentId and ID.

[0013] On the other hand, the present invention also provides a video content retrieval method based on a combination of large models, embedding, and reranking, for retrieving offline videos processed by the aforementioned video content pre-analysis method, comprising the following steps:

[0014] Obtain the search description content input by the user, and call the large model to perform a structured description of the search description content;

[0015] The Embedding model is used to convert the search description into vectors, forming third vector data; the third vector data is then used to query the vector database to retrieve all first vector data that meet the similarity requirements with the third vector data, forming the first result set.

[0016] Retrieve the second vector data associated with each first vector data in the first result set from the vector database to form the second result set;

[0017] The search description content is converted into a structured format to form a second key information list with the same format as the first key information. The second key information list consists of multiple elements, represented as: [first attribute keyword] [second attribute keyword] [third attribute keyword]...;

[0018] The Embedding model is used to convert each element in the second key information list into a vector, forming the fourth vector data;

[0019] The similarity of each element's vector in the fourth vector data is compared with the similarity of each element's vector in the second vector data in the second result set. The second vector data in which the similarity of each element meets the requirements is obtained, and the video content corresponding to the obtained second vector data is output as the search result.

[0020] In some embodiments, the steps of comparing the similarity of the vectors of each element in the fourth vector data with the vectors of each element in the second vector data in the second result set are as follows:

[0021] Step a: Compare the similarity between the vector of the first element in the fourth vector data and the vector of the first element in the first row of the second vector data in the second result set.

[0022] Step b: When the similarity meets the set similarity threshold, continue to compare the similarity between the vector of the second element in the fourth vector data and the vector of the second element in the first row vector data, and then compare the vector similarity between the fourth vector data and other corresponding elements in the first row vector data in turn.

[0023] Step c: When the similarity does not meet the set similarity threshold, skip the similarity comparison of other elements in the first row vector data;

[0024] Step d: If there is a next row vector data in the second vector data, then perform a similarity comparison with the next row vector data according to steps a-c, and then complete the similarity comparison with all row vector data in the second vector data in turn.

[0025] Step e: Following steps a-d, perform similarity comparisons with all second vector data in the second result set.

[0026] In some embodiments, the similarity of each element vector in the second vector data corresponding to the search results is weighted and calculated, and the search results are sorted according to the weighted calculation results.

[0027] On the other hand, the present invention also provides a video content preprocessing system based on a combination of large models, embedding, and reranking, for performing the video content pre-analysis method to analyze the video, including:

[0028] The first module is used to call a large model to perform content description and structured analysis on the video content of offline videos, and outputs a text description of the video content and a structured description of the video content, respectively.

[0029] The second module is used to convert the output text description into vectors using the Embedding model, forming the first vector data;

[0030] The third module is used to convert the output structured description into a structured format to form the first key information list;

[0031] The fourth module is used to convert each element in the first key information list into a vector using the Embedding model, forming the second vector data;

[0032] The fifth module is used to store the first vector data and the second vector data into the vector database and establish the association between the first vector data and the second vector data.

[0033] On the other hand, the present invention also provides a video content retrieval system based on a combination of large models, embedding, and reranking, for performing video retrieval using the video content retrieval method, including:

[0034] The sixth module is used to obtain the search description content input by the user and call the large model to perform a structured description of the search description content;

[0035] The seventh module is used to convert the retrieved description content into vectors using the Embedding model, forming the third vector data;

[0036] The eighth module is used to query the vector database using third vector data, retrieve all first vector data that meet the similarity requirements with the third vector data, and form the first result set;

[0037] The ninth module is used to retrieve the second vector data associated with each first vector data in the first result set from the vector database, forming the second result set;

[0038] The tenth module is used to convert the structured description of the search results into a structured format, forming a list of second key information with the same format as the first key information.

[0039] The eleventh module is used to convert each element in the second key information list into a vector using the Embedding model, forming the fourth vector data;

[0040] The twelfth module is used to compare the similarity of each element's vector in the fourth vector data with the similarity of each element's vector in the second vector data in the second result set, obtain the second vector data in which the similarity of each element meets the requirements, and output the video content corresponding to the obtained second vector data as the search result.

[0041] On the other hand, the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the video content pre-analysis method described above.

[0042] On the other hand, the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the video content retrieval method described above.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] This invention employs a combination of large models and embeddings to pre-analyze video content. It converts the structured description of video content into a specific structured format, forming first and second vector data corresponding to each video content, and establishes a correlation between the first and second vector data. Through the pre-analyzed offline videos, subsequent retrieval no longer needs to rely on the large model, reducing the consumption of large model tokens.

[0045] Based on the pre-analysis of video content, the search is first performed using RAG (Rapid Rank) and then the search results are reranked through structured embedding retrieval and vector similarity comparison. The search results are then presented to the user in sequence, which improves the accuracy of video retrieval and reduces response time. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a video content pre-analysis method based on a combination of large models, embedding, and rerank, as described in an embodiment of the present invention.

[0048] Figure 2 This is a flowchart of a video content retrieval method based on a combination of large models, embedding, and rerank in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0050] When using a large model combined with RAG for retrieval, the typical approach is to first retrieve some initial results using RAG, and then analyze the relevant results using the large model. This retrieval method is time-consuming and consumes a large amount of tokens. The inventors discovered that the reason for this problem is that in the above retrieval method, both the description in the video footage and the vector transformed from the user's search query are sparse. It is difficult to effectively configure the similarity threshold between these two vectors. If the similarity threshold is set too high, some search results that meet the criteria will be filtered out; if it is set too low, a large number of irrelevant results will be retrieved. Therefore, the common approach is to set a small similarity threshold and then use the large model to understand and analyze the retrieved content to determine whether it is the search result the user wants. This results in a long retrieval time and consumes a large amount of tokens.

[0051] RAG (Retrieval Augmented Generation) is a technique that combines information retrieval and text generation to improve the quality of answers generated by large models by retrieving relevant information from external knowledge bases.

[0052] This invention employs a combination of large models, embeddings, and reranking to address the low accuracy of existing retrieval methods combining large models and embeddings, as well as the long response times and high token consumption associated with methods using large models combined with RAG (Research Aggregator). This invention only invokes the large model during video pre-analysis and user question analysis. During retrieval, RAG is performed first, followed by structured embedding retrieval and similarity comparison, and then reranking the retrieval results. The results are then presented to the user sequentially. Subsequent retrievals no longer rely on the large model, thereby improving video retrieval accuracy and reducing response time and token consumption of the large model.

[0053] Large models can be built using existing large CV models.

[0054] Embedding models are machine learning models that transform high-dimensional, discrete data (such as text, images, and audio) into low-dimensional, continuous vector representations. These vectors (embedded vectors) can capture semantic or feature information of the data, making it easier for computers to process and analyze complex relationships.

[0055] Reranking is a technique used in the field of information retrieval to optimize the ranking of search results. It re-evaluates the relevance of selected search results and re-ranks them to ensure that the output is more relevant to the user's question.

[0056] In some embodiments of the present invention, reference is made to Figure 1 A video content pre-analysis method based on a combination of large models, embedding, and reranking includes the following steps:

[0057] S011. By using methods such as frame extraction, the interface of the large model is called to perform content description and structured analysis on the video content of the offline video, and the output is a text description of the video content and a structured description of the video content, respectively.

[0058] S012. The output text description is converted into a vector using the Embedding model to form the first vector data;

[0059] S013. Convert the output structured description into a structured format to form a first key information list. The first key information list consists of multiple elements, represented as: [first attribute keyword] [second attribute keyword] [third attribute keyword]...;

[0060] Among them, the [first attribute keyword] is the keyword that represents the target, the [second attribute keyword] is the quantifier used to modify the target, and the [third attribute keyword] is the adjective, adverb, and other keywords used to modify the target. For example, the first key information list is represented as: [target] [quantifier (modifying content)] [adjective & adverb, etc. (modifying content)];

[0061] S014. Use the Embedding model to convert the contents of each element in the first key information list into vectors to form the second vector data;

[0062] S015. Store the first vector data and the second vector data into the vector database, and establish the association between the first vector data and the second vector data for subsequent retrieval.

[0063] In some embodiments, a unique ID is configured for the first vector data in the data structure of the first vector data, and a parentId corresponding to the ID of the first vector data is configured in the data structure of the second vector data. The association between the first vector data and the second vector data is established through the correspondence between parentId and ID.

[0064] In some embodiments of the present invention, based on the pre-analyzed offline video described above, referring to Figure 2 A video content retrieval method based on a combination of large models, embedding, and reranking includes the following steps:

[0065] S021. The user inputs the description of the video search to be performed;

[0066] S022. Call the interface of the large model to preprocess the search description content input by the user, including completing, correcting and standardizing the search description content, rewriting the search description content to obtain the rewritten search description content, and performing a structured description on the rewritten search description content.

[0067] S023. The Embedding model is used to convert the search description content into vectors to form third vector data; the third vector data is used to query the vector database to retrieve all first vector data whose similarity to the third vector data is greater than the set first similarity threshold, forming the first result set;

[0068] S024. Obtain the second vector data associated with each first vector data in the first result set from the vector database to form the second result set;

[0069] S025. Convert the structured description of the search results into a structured format to form a second key information list with the same format as the first key information. The second key information list is represented as: [First Attribute Keyword] [Second Attribute Keyword] [Third Attribute Keyword]...;

[0070] S026. Use the Embedding model to convert the contents of each element in the second key information list into vectors to form the fourth vector data;

[0071] S027. Compare the similarity of each element's vector in the fourth vector data with the vector of each element in the second vector data in the second result set, obtain the second vector data whose similarity of each element meets the requirements, output the video content corresponding to the second vector data as the search result, and form a search result set.

[0072] S028. Calculate the similarity of each element vector in the second vector data corresponding to each search result in the search result set using weighted calculation, and sort the search results according to the weighted calculation results.

[0073] S029. Extract frames from the video content corresponding to the search results and return them to the user along with the search results.

[0074] In some embodiments, the logic for comparing the similarity of the vectors of each element in the fourth vector data with the vectors of each element in the second vector data in the second result set is as follows:

[0075] Step a: If the user's search description contains the word "target", compare the similarity of the vector of "target" with the vector of "target" in the first row of the second vector data in the second result set, and set the similarity threshold to 90%.

[0076] Step b: When the similarity meets the set similarity threshold, continue to compare the similarity of the [quantifier] vector in the fourth vector data with the [quantifier] vector in the same row of vector data, and then compare the vector similarity of other elements in the fourth vector data with other elements in the first row of vector data in turn.

[0077] Step c: If the similarity does not meet the similarity threshold, skip the similarity comparison of other elements in the first row of the second vector data.

[0078] Step d: If there is a next row of vector data in the second vector data, then perform a similarity comparison with the next row of vector data, and then complete the similarity comparison of all row vector data in the second vector data in turn.

[0079] Step e: Following steps a-d, perform similarity comparisons with all second vector data in the second result set.

[0080] The following detailed description of the method flow for video content pre-analysis and video content retrieval according to the present invention, with reference to specific embodiments, is provided below.

[0081] The process of pre-analyzing video content is as follows:

[0082] Users upload offline videos, and through frame extraction and other methods, a large model is invoked to perform content description and structured analysis on the video content of the offline videos, outputting a textual description of the video content and a structured description of the video content, respectively.

[0083] For example, analyzing two videos, the output would be a textual description of the content of each video:

[0084] Video 1: "The video shows an overpass on the road with a few scattered people on it, including a man dressed in a red dress..."

[0085] Video 2: "In the video, you can see a road with an overpass. On the overpass, there are two middle-aged men wearing red skirts. They..."

[0086] The large model is used to perform frame-by-frame analysis on the two videos, outputting a structured description of the video content. This structured description is then converted into a structured format, resulting in the first list of key information, which is represented as follows:

[0087] Video 1:

[0088] [Overpass] [One] [On the road];

[0089] [A male] [on the overpass, wearing a red dress];

[0090] Video 2:

[0091] [A road];

[0092] [Overpass] [One] [On the road];

[0093] [Man] [Two] [On the overpass, middle-aged, wearing a red dress].

[0094] The text descriptions of Video 1 and Video 2 are converted into vectors using the Embedding model to form the first vector data.

[0095] The Embedding model is used to convert the content of each element in the first key information list of video 1 and video 2 into vectors to form the second vector data.

[0096] The first vector data and the second vector data are stored in the vector database. The storage structure of the first vector data and the second vector data in the vector database is represented as follows:

[0097] Id, parentId, content, contentVector, quantifier, quantifierVector, modify, modifyVetor;

[0098] Taking video 1 as an example, the data structure of its first vector data is represented as follows:

[0099] 1, null, "From the video, we see a pedestrian bridge on the road...", "...(vector of the previous field's content)", null, null, null, null;

[0100] The corresponding data structures for the second vector data are represented as follows:

[0101] 2, 1, “overpass”, “...(vector of the overpass)”, “one”, “...(vector of the one)”, “on the road”, “...(vector of the road)”;

[0102] 3, 1, “male”, “... (vector of [male])”, “one”, “... (vector of [one])”, “on the overpass, wearing a red dress”, “... (vector of [on the overpass, wearing a red dress])”.

[0103] The first vector data and the second vector data of video 1 are associated through the correspondence between Id and parentId, thereby establishing the association between the first vector data and the second vector data to facilitate subsequent video content retrieval.

[0104] The data structures and association relationships of the first vector data and the second vector data of Video 2 follow the same representation as that of Video 1.

[0105] After pre - analyzing the video content based on the above method, the process of retrieving the video content is as follows:

[0106] The retrieval description content input by the user for video retrieval is: "Help me find a man in a red dress on the overpass";

[0107] Call the interface of the large - model to complete, correct, and regularize the retrieval description content, and the re - written retrieval description content obtained is:

[0108] "A man in a red dress on the overpass";

[0109] Perform a structured description on the rewritten retrieval description content, and convert the structured description into a structured format to form a second key information list, expressed as:

[0110]

Man

One

On the overpass, wearing a red dress

[0111] Use the Embedding model to convert the above - mentioned retrieval description content into vectors to form the third vector data; use the third vector data to query in the vector database, and retrieve all the first vector data whose similarity with the third vector data is greater than the first similarity threshold to form the first result set;

[0112] Based on the association relationship, obtain the second vector data associated with each first vector data in the first result set in the vector database to form the second result set.

[0113] For example, through the third vector data corresponding to "A man in a red dress on the overpass", retrieve the first vector data corresponding to "Id = 1" to form the first result set;

[0114] Based on the association relationship, take out the second vector data corresponding to "parentId = 1" in the vector database to form the second result set.

[0115] Use the Embedding model to convert the content in each element of the second key information list into vectors to form the fourth vector data.

[0116] Compare the vectors of each element in the fourth vector data (

Man

One

On the overpass, wearing a red dress

Overpass

One

On the road

Male

One

On the overpass, wearing a red dress

[0117] a) First, compare the vector of the first row of the second vector data with the vector of the first element of the fourth vector data, which is "male". Then compare the vector of the first element of the first row of the fourth vector data, which is "overpass", with the vector of the first element of the first row of the first vector data. If the similarity is less than the set similarity threshold of 90%, then skip the vector similarity comparison of other elements in that row of the vector data and proceed to the similarity comparison with the next row of the vector data.

[0118] b) Compare the similarity between the vector of "Male" and the vector of the first element "Male" in the second row of vector data. If the similarity is greater than the set similarity threshold of 90%, then continue to compare the similarity between the next element.

[0119] c) Compare the similarity between the vector of [one] and the vector of [another], and if the similarity is greater than the set similarity threshold of 85%, then continue to compare the similarity between the next element;

[0120] e) Compare the similarity between the vector of "on the overpass, wearing a red dress" and the vector of "on the overpass, wearing a red dress". If the similarity is greater than the set similarity threshold of 80%.

[0121] Based on similarity comparison, the similarity between each element vector in the fourth vector data and each element vector in the second vector data corresponding to video 1 meets the set threshold requirement. Therefore, video 1 is output as the search result and placed into the search result set.

[0122] The similarity of other second vector data in the second result set is compared sequentially to obtain the retrieval result set.

[0123] Different weights are assigned to the similarity of each element in the second vector data corresponding to each search result in the search result set. The similarity is then weighted and calculated. The search results in the search result set are sorted according to the weighted calculation results. The sorted search results are then returned to the user.

[0124] On the other hand, the present invention also provides a video content preprocessing system based on a combination of large models, embedding, and reranking, for performing the above-mentioned video content pre-analysis methods to analyze videos, including:

[0125] The first module is used to call a large model to perform content description and structured analysis on the video content of offline videos, and outputs a text description of the video content and a structured description of the video content, respectively.

[0126] The second module is used to convert the output text description into vectors using the Embedding model, forming the first vector data;

[0127] The third module is used to convert the output structured description into a structured format to form the first key information list;

[0128] The fourth module is used to convert each element in the first key information list into a vector using the Embedding model, forming the second vector data;

[0129] The fifth module is used to store the first vector data and the second vector data into the vector database and establish the association between the first vector data and the second vector data.

[0130] On the other hand, the present invention also provides a video content retrieval system based on a combination of large models, embedding, and reranking, for performing the above-mentioned video content retrieval methods to retrieve videos, including:

[0131] The sixth module is used to obtain the search description content input by the user and call the large model to perform a structured description of the search description content;

[0132] The seventh module is used to convert the retrieved description content into vectors using the Embedding model, forming the third vector data;

[0133] The eighth module is used to query the vector database using third vector data, retrieve all first vector data that meet the similarity requirements with the third vector data, and form the first result set;

[0134] The ninth module is used to retrieve the second vector data associated with each first vector data in the first result set from the vector database, forming the second result set;

[0135] The tenth module is used to convert the structured description of the search results into a structured format, forming a list of second key information with the same format as the first key information.

[0136] The eleventh module is used to convert each element in the second key information list into a vector using the Embedding model, forming the fourth vector data;

[0137] The twelfth module is used to compare the similarity of each element's vector in the fourth vector data with the similarity of each element's vector in the second vector data in the second result set, obtain the second vector data in which the similarity of each element meets the requirements, and output the video content corresponding to the obtained second vector data as the search result.

[0138] In some embodiments, the video content retrieval system further includes a thirteenth module, which is used to perform weighted calculation on the similarity of each element vector in the second vector data corresponding to the retrieval results, and to sort the retrieval results according to the weighted calculation results.

[0139] On the other hand, the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the video content pre-analysis method described above.

[0140] On the other hand, the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the video content retrieval method described above.

[0141] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A video content retrieval method based on a combination of large models, embedding, and reranking, characterized in that: Includes the following steps: The large model is invoked to perform content description and structured analysis on the video content of offline videos, and outputs a text description of the video content and a structured description of the video content, respectively. The Embedding model is used to convert the output text description into vectors, forming the first vector data; The output structured description is converted into a structured format to form a first key information list. The first key information list consists of multiple elements, represented as: [first attribute keyword, second attribute keyword, third attribute keyword, ...]; the first attribute keyword is a keyword representing the target, the second attribute keyword is a quantifier used to modify the target, and the third attribute keyword is an adjective or adverb used to modify the target. The Embedding model is used to convert each element in the first key information list into a vector, forming the second vector data; Store the first vector data and the second vector data into the vector database, and establish the association between the first vector data and the second vector data; Obtain the search description content input by the user, and call the large model to perform a structured description of the search description content; The Embedding model is used to convert the search description into vectors, forming third vector data; the third vector data is then used to query the vector database to retrieve all first vector data that meet the similarity requirements with the third vector data, forming the first result set. Retrieve the second vector data associated with each first vector data in the first result set from the vector database to form the second result set; The structured description of the search results is converted into a structured format to form a second key information list with the same format as the first key information. The second key information list consists of multiple elements, represented as: [first attribute keyword, second attribute keyword, third attribute keyword, ...]; The Embedding model is used to convert each element in the second key information list into a vector, forming the fourth vector data; The similarity of each element's vector in the fourth vector data is compared with the similarity of each element's vector in the second vector data in the second result set. The second vector data in which the similarity of each element meets the requirements is obtained, and the video content corresponding to the obtained second vector data is output as the search result.

2. The video content retrieval method based on a combination of large model, embedding, and reranking as described in claim 1, is characterized in that, The steps for comparing the similarity of each element's vector in the fourth vector data with the vector of each element in the second vector data in the second result set are as follows: Step a: Compare the similarity between the vector of the first element in the fourth vector data and the vector of the first element in the first row of the second vector data in the second result set. Step b: When the similarity meets the set similarity threshold, continue to compare the similarity between the vector of the second element in the fourth vector data and the vector of the second element in the first row vector data, and then compare the vector similarity between the fourth vector data and other corresponding elements in the first row vector data in turn. Step c: When the similarity does not meet the set similarity threshold, skip the similarity comparison of other elements in the first row vector data; Step d: If there is a next row vector data in the second vector data, then perform a similarity comparison with the next row vector data according to steps a-c, and then complete the similarity comparison with all row vector data in the second vector data in turn. Step e: Following steps a-d, perform similarity comparisons with all second vector data in the second result set.

3. The video content retrieval method based on a combination of large model, embedding, and reranking as described in claim 1, is characterized in that, The similarity of each element vector in the second vector data corresponding to the search results is weighted and calculated, and the search results are sorted according to the weighted calculation results.

4. The video content retrieval method based on a combination of large model, embedding, and reranking as described in claim 1, characterized in that, In the data structure of the first vector data, a unique ID is configured for the first vector data. In the data structure of the second vector data, a parentId corresponding to the ID of the first vector data is configured. The association between the first vector data and the second vector data is established through the correspondence between parentId and ID.

5. A video content retrieval system based on a combination of large models, embedding, and reranking, characterized in that: The method for performing video retrieval according to any one of claims 1-4 includes: The first module is used to call a large model to perform content description and structured analysis on the video content of offline videos, and outputs a text description of the video content and a structured description of the video content, respectively. The second module is used to convert the output text description into vectors using the Embedding model, forming the first vector data; The third module is used to convert the output structured description into a structured format to form the first key information list; The fourth module is used to convert each element in the first key information list into a vector using the Embedding model, forming the second vector data; The fifth module is used to store the first vector data and the second vector data into the vector database and establish the association between the first vector data and the second vector data; The sixth module is used to obtain the search description content input by the user and call the large model to perform a structured description of the search description content; The seventh module is used to convert the retrieved description content into vectors using the Embedding model, forming the third vector data; The eighth module is used to query the vector database using third vector data, retrieve all first vector data that meet the similarity requirements with the third vector data, and form the first result set; The ninth module is used to retrieve the second vector data associated with each first vector data in the first result set from the vector database, forming the second result set; The tenth module is used to convert the structured description of the search results into a structured format, forming a list of second key information with the same format as the first key information. The eleventh module is used to convert each element in the second key information list into a vector using the Embedding model, forming the fourth vector data; The twelfth module is used to compare the similarity of each element's vector in the fourth vector data with the similarity of each element's vector in the second vector data in the second result set, obtain the second vector data in which the similarity of each element meets the requirements, and output the video content corresponding to the obtained second vector data as the search result.

6. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video content retrieval method according to any one of claims 1-4.

Images