Off-line video pre-analysis method, retrieval method, system and device based on size model cooperation and storage medium

Abstract

The application discloses an offline video pre-analysis method, a retrieval method, a system, equipment and a storage medium based on size model cooperation, and the following steps are included: taking the union of an input algorithm list set and a recommended algorithm list set to obtain a total algorithm list set; obtaining a small model algorithm list set, taking the intersection of the small model algorithm list set and the total algorithm list set to obtain a first algorithm list set; and obtaining a second algorithm list set according to the total algorithm list set and the first algorithm list set; the first algorithm list set is sent to the small model to analyze the offline video, and after the corresponding algorithm is analyzed, an alarm is generated and stored in a small model alarm library; the offline video is analyzed by a large model according to the second algorithm list set, a text description is made on the analysis result, the text description is converted into a vector, and the vector is stored in a vector database. The application adopts a large model and a small model cooperative analysis mode, fully utilizes the advantages of the small model and the large model, and optimizes the performance of video analysis and retrieval.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an offline video pre-analysis method, retrieval method, system, device, and storage medium based on big-small model collaboration. Background Technology

[0002] With the explosive growth of video data, effectively utilizing this data has become a major technical challenge. Currently, searches for related video content are typically conducted by searching video titles or by searching for pre-entered text descriptions by users. However, these traditional methods are insufficient for retrieving and finding video content within massive amounts of video data, resulting in low effective video utilization.

[0003] With the development of artificial intelligence technology, large VLM models, CV models, and small models are being applied to video retrieval. Small models are specifically designed for data annotation, training, and optimization of relevant algorithms, and they generally outperform large VLM models in terms of accuracy and recall. However, because the list of algorithms that small models can analyze is limited, they often cannot meet the needs of users' retrieval. Large models have significant advantages over small models in semantic understanding and processing capabilities, but they suffer from high computational resource requirements and a high dependence on training data. Summary of the Invention

[0004] The purpose of this invention is to provide an offline video pre-analysis method, retrieval method, system, device, and storage medium based on large and small model collaboration, so as to solve the above-mentioned problems existing in the current video retrieval and analysis using large and small models.

[0005] This invention is achieved through the following technical solution:

[0006] The offline video pre-analysis method based on size-model collaboration includes the following steps:

[0007] Semantic understanding is performed on the structured information recorded from offline videos to obtain a list of recording algorithms {A1};

[0008] A large model is used to analyze offline videos to obtain a list of recommendation algorithms {A2};

[0009] The union of the input algorithm list set {A1} and the recommendation algorithm list set {A2} is used to obtain the total algorithm list set {A}.

[0010] Obtain the small model algorithm list set {B}. Take the intersection of the small model algorithm list set {B} and the total algorithm list set {A} to obtain the first algorithm list set, denoted as {A}∩{B}. And based on the total algorithm list set {A} and the first algorithm list set {A}∩{B}, obtain the second algorithm list set, denoted as {A} - {A}∩{B}.

[0011] The first algorithm list set is sent to the small model to analyze the offline video. After the corresponding algorithm is analyzed, an alarm is generated and stored in the small model alarm library.

[0012] The offline video is analyzed using a second algorithm list set analysis based on a large model. The analysis results are described in text, converted into vectors, and stored in a vector database.

[0013] In some embodiments of the present invention, the step of analyzing offline videos using a large model to obtain a list set of recommendation algorithms {A2} includes:

[0014] Extract n images at equal intervals from an offline video, and extract one image before and after each image at a first set time interval;

[0015] The large model interface is called n times using the prompt method. The large model analyzes the three images extracted at intervals to obtain the recommendation algorithm list set {A2}.

[0016] In some embodiments of the present invention, the step of performing a second algorithmic list set analysis on offline videos using a large model includes:

[0017] Starting from the first frame of the offline video, three images are continuously extracted at second set time intervals, and the extracted images are analyzed by the second algorithm list set analysis using a large model.

[0018] Starting from the moment of the third image, slide the third set time and repeatedly call the large model to perform the second algorithm list set analysis on the extracted images; repeat this process until the analysis of the entire offline video is completed.

[0019] In some embodiments of the present invention, the retrieval of offline videos processed using the aforementioned size-model-based collaborative offline video pre-analysis method includes the following steps:

[0020] By analyzing the search results of a single sentence input by a user using a large model, the search results are broken down and standardized into a new search format, represented as: [First Algorithm List] or\and [Other Algorithm Lists].

[0021] The first search result was obtained by searching the small model alarm database using the [first algorithm list set].

[0022] Convert the "List of Other Algorithms" into a vector, use the "List of Other Algorithms" as the search criteria, and search the vector database to obtain the second search result;

[0023] Summarize the first and second search results and output them.

[0024] On the other hand, the present invention also provides an offline video pre-analysis system based on size-model collaboration, for executing the offline video pre-analysis method based on size-model collaboration, comprising:

[0025] The first module is used to call the large model to perform semantic understanding on the structured information recorded from offline videos, and obtain the list of recording algorithms {A1}.

[0026] The second module is used to call a large model to analyze offline videos and obtain a list of recommendation algorithms {A2}.

[0027] The third module is used to take the union of the input algorithm list set {A1} and the recommendation algorithm list set {A2} to obtain the total algorithm list set {A}.

[0028] The fourth module is used to obtain the small model algorithm list set {B}, take the intersection of the small model algorithm list set {B} and the total algorithm list set {A} to obtain the first algorithm list set, denoted as {A}∩{B}; and obtain the second algorithm list set based on the total algorithm list set {A} and the first algorithm list set {A}∩{B}, denoted as {A} - {A}∩{B}.

[0029] The fifth module is used to send the first algorithm list set to the small model to analyze the offline video. After the corresponding algorithm is analyzed, an alarm is generated and stored in the small model alarm library.

[0030] The sixth module is used to call the large model to perform a second algorithm list set analysis on the offline video, describe the analysis results in text, convert the text description into vectors, and store them in a vector database.

[0031] On the other hand, the present invention also provides an offline video retrieval system based on size-model collaboration, for executing the offline video retrieval method based on size-model collaboration, comprising:

[0032] The seventh module is used to call the large model to analyze the search content of a user's input sentence, break down the search content and standardize it into a new search format;

[0033] The eighth module is used to retrieve the first search result from the small model alarm database using the [first algorithm list set].

[0034] The ninth module is used to convert the "List of Other Algorithms" into a vector, and then use the "List of Other Algorithms" as the search criteria to search the vector database to obtain the second search result.

[0035] The tenth module is used to summarize and output the first and second search results.

[0036] On the other hand, the present invention also provides an electronic device, comprising:

[0037] Processor; and,

[0038] Memory for storing the executable instructions of the processor;

[0039] The processor is configured to execute the offline video pre-analysis method based on size model collaboration by executing the executable instructions.

[0040] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned offline video pre-analysis method based on size model collaboration.

[0041] On the other hand, the present invention also provides an electronic device, comprising:

[0042] Processor; and,

[0043] Memory for storing the executable instructions of the processor;

[0044] The processor is configured to execute the offline video retrieval method based on size model collaboration by executing the executable instructions.

[0045] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the offline video retrieval method based on size model collaboration.

[0046] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0047] This invention employs a collaborative analysis approach using large and small models. Algorithms that the small model cannot analyze are analyzed by the large model, fully leveraging the advantages of both models and optimizing the performance of video analysis and retrieval. By preprocessing offline videos using both large and small models, subsequent user searches can quickly return results for the algorithms the user is interested in, eliminating the need to analyze the entire video from beginning to end each time based on the user's question, thus significantly improving retrieval speed. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of the offline video pre-analysis method in an embodiment of the present invention.

[0050] Figure 2 This is a flowchart of the offline video retrieval method in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0052] The ability to quickly and accurately retrieve the video data that users need from massive amounts of video data is currently a hot topic in the field of video processing technology. With the development of artificial intelligence technology, it has been applied to the processing and retrieval of video data. In video retrieval and processing, the model needs to be able to quickly analyze the algorithms that users are interested in (including targets, events, and scenes) and then retrieve relevant content.

[0053] When users upload offline videos, they can input the algorithms they are interested in (including targets, events, and scenarios) in a single sentence. This allows the large-scale model to focus on the corresponding algorithms during analysis. Otherwise, because the large-scale model cannot output algorithms it does not focus on, users may not be able to find relevant videos during their search. For example, if a user inputs "search for videos showing people riding bicycles without helmets," and the large-scale model is not pre-selected with the algorithm for "riding bicycles without helmets," the model may only analyze the presence of bicycles and often fail to retrieve the specific content of "riding bicycles without helmets" within the video footage.

[0054] Because small models have undergone data labeling, training, and optimization for relevant algorithms, the accuracy and recall of the algorithms are better, but the list of algorithms that can be analyzed by small models is limited.

[0055] Therefore, this invention employs a collaborative analysis approach using both large and small models. The large model handles tasks that the small model cannot analyze, thus covering more algorithms that users wish to analyze. This invention analyzes algorithms known to the user, using the large model for rapid video analysis and recommendation, combined with algorithms from the small model. All resulting algorithms are then categorized, allowing both the small and large models to pre-analyze offline videos based on their respective categorization algorithms, facilitating subsequent retrieval.

[0056] For example, the list of algorithms that can be analyzed by the small model includes: fighting, crowd gathering, and vehicles. These three algorithms correspond to the algorithm set {B}.

[0057] When a user uploads an offline video, the algorithm list they follow is: traffic congestion, fighting, and urban flooding. These three algorithms correspond to the algorithm set {A1}.

[0058] The large model recommends the following algorithms for frame extraction analysis of the video: Congestion and Garbage Overflow, which correspond to the algorithm set {A2}.

[0059] Merging the algorithm set {A1} with the algorithm set {A2} yields the algorithm set {A}: congestion, fighting, urban flooding, and overflowing garbage.

[0060] The smaller model handles the algorithms corresponding to {A}∩{B}, meaning it only handles algorithms that "compete" with each other; the larger model handles the rest, meaning it handles algorithms corresponding to {A} - {A}∩{B}, which represent "congestion, urban flooding, and overflowing garbage," and performs pre-analysis on offline videos.

[0061] By employing the aforementioned method of collaborative analysis using both large and small models, the algorithms (targets, events, scenarios) that users wish to analyze can be covered.

[0062] Reference Figure 1 In some embodiments of the present invention, the offline video pre-analysis method based on size model collaborative analysis includes the following steps:

[0063] S11. Call the large model interface via prompt to perform semantic understanding and analysis on the structured information entered by the user during offline video upload, such as a one-sentence description of the algorithm content they are interested in, and obtain the list of entered algorithms {A1}.

[0064] S12. Based on the duration of the offline video, extract n images (e.g., 10 images) at equal intervals from the offline video, and extract one image 1 second before and after each image.

[0065] Using the prompt method, the large model interface is called n times. The large model analyzes the three images extracted at intervals to obtain the recommendation algorithm list set {A2}.

[0066] S13. Take the union of the input algorithm list set {A1} and the recommendation algorithm list set {A2} to obtain the total algorithm list set {A};

[0067] S14. Obtain the small model algorithm list set {B}. Take the intersection of the small model algorithm list set {B} and the total algorithm list set {A} to obtain the first algorithm list set, which is represented as {A}∩{B}.

[0068] Based on the total algorithm list set {A} and the first algorithm list set {A}∩{B}, obtain the second algorithm list set, which is represented as {A} - {A}∩{B};

[0069] S15. The first algorithm list set is sent to the small model to analyze the offline video. After the algorithm corresponding to the first algorithm list set is analyzed, an alarm is generated and stored in the small model alarm library.

[0070] S16. Starting from the first frame of the offline video, extract three images every first set time interval (e.g., 2 seconds). Use the prompt method to call the large model to analyze the second algorithm list set of the extracted images.

[0071] Starting from the moment of the 3rd image (the last image extracted last time), slide the second set time (e.g., 10 seconds) and repeat the step of calling the large model to analyze the second algorithm list set of the extracted images using the prompt method; then repeat this operation in turn until the analysis of the entire offline video is completed;

[0072] The results of each large model analysis are described in text. Embedding is then used to convert the obtained text descriptions into vectors and store them in a vector database.

[0073] Thus, by utilizing the collaborative operation of small and large models, the pre-analysis of offline videos is completed.

[0074] Based on the above pre-analysis of offline videos, a small model alarm library and a vector database are obtained. On this basis, some embodiments of the present invention provide an offline video retrieval method, referring to... Figure 2 This includes the following steps:

[0075] S21. The user inputs a single sentence of search content. The large model is called via prompt to identify the user's search intent, and the key content of the search is broken down and standardized into a new search format, represented as: [First Algorithm List] or\and [Other Algorithm List]; where [Other Algorithm List] is the list of all algorithms identified except for [First Algorithm List].

[0076] S22. Retrieve the first algorithm list set from the small model alarm database to obtain the first search result;

[0077] S23. Convert the "List of Other Algorithms" into a vector using embedding. Use the vector converted from the "List of Other Algorithms" as the search criteria to search the vector database and obtain the second search result.

[0078] The search can be performed using a similarity calculation method, with a similarity threshold of 85% to obtain the second search result;

[0079] S24. Summarize the above search results and return the retrieved results to the user to complete the search.

[0080] After preprocessing offline videos using large and small models, users can quickly retrieve results by asking relevant questions during subsequent searches, eliminating the need to analyze the entire video from beginning to end each time based on the user's question, thus greatly improving search speed.

[0081] On the other hand, the present invention also provides an offline video pre-analysis system based on size-model collaboration, for executing the offline video pre-analysis method based on size-model collaboration, comprising:

[0082] The first module is used to call the large model to perform semantic understanding on the structured information recorded from offline videos, and obtain the list of recording algorithms {A1}.

[0083] The second module is used to call a large model to analyze offline videos and obtain a list of recommendation algorithms {A2}.

[0084] The third module is used to take the union of the input algorithm list set {A1} and the recommendation algorithm list set {A2} to obtain the total algorithm list set {A}.

[0085] The fourth module is used to obtain the small model algorithm list set {B}, take the intersection of the small model algorithm list set {B} and the total algorithm list set {A} to obtain the first algorithm list set, denoted as {A}∩{B}; and obtain the second algorithm list set based on the total algorithm list set {A} and the first algorithm list set {A}∩{B}, denoted as {A} - {A}∩{B}.

[0086] The fifth module is used to send the first algorithm list set to the small model to analyze the offline video. After the corresponding algorithm is analyzed, an alarm is generated and stored in the small model alarm library.

[0087] The sixth module is used to call the large model to perform a second algorithm list set analysis on the offline video, describe the analysis results in text, convert the text description into vectors, and store them in a vector database.

[0088] On the other hand, the present invention also provides an offline video retrieval system based on size-model collaboration, for executing the offline video retrieval method based on size-model collaboration, comprising:

[0089] The seventh module is used to call the large model to analyze the search content of a user's input sentence, break down the search content and standardize it into a new search format;

[0090] The eighth module is used to retrieve the first search result from the small model alarm database using the [first algorithm list set].

[0091] The ninth module is used to convert the "List of Other Algorithms" into a vector, and then use the "List of Other Algorithms" as the search criteria to search the vector database to obtain the second search result.

[0092] The tenth module is used to summarize and output the first and second search results.

[0093] On the other hand, the present invention also provides an electronic device, comprising:

[0094] Processor; and,

[0095] Memory for storing the executable instructions of the processor;

[0096] The processor is configured to execute the offline video pre-analysis method based on size model collaboration by executing the executable instructions.

[0097] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned offline video pre-analysis method based on size model collaboration.

[0098] On the other hand, the present invention also provides an electronic device, comprising:

[0099] Processor; and,

[0100] Memory for storing the executable instructions of the processor;

[0101] The processor is configured to execute the offline video retrieval method based on size model collaboration by executing the executable instructions.

[0102] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the offline video retrieval method based on size model collaboration.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. An offline video pre-analysis method based on size-model collaboration, characterized in that, Includes the following steps: Semantic understanding is performed on the structured information recorded from offline videos to obtain a list of recording algorithms {A1}; A large model is used to analyze offline videos to obtain a list of recommendation algorithms {A2}; The union of the input algorithm list set {A1} and the recommendation algorithm list set {A2} is used to obtain the total algorithm list set {A}. Obtain the small model algorithm list set {B}. Take the intersection of the small model algorithm list set {B} and the total algorithm list set {A} to obtain the first algorithm list set, denoted as {A}∩{B}. And based on the total algorithm list set {A} and the first algorithm list set {A}∩{B}, obtain the second algorithm list set, denoted as {A} - {A}∩{B}. The first algorithm list set is sent to the small model to analyze the offline video. After the corresponding algorithm is analyzed, an alarm is generated and stored in the small model alarm library. The offline video is analyzed using a second algorithm list set analysis based on a large model. The analysis results are described in text, converted into vectors, and stored in a vector database.

2. The offline video pre-analysis method based on size model collaboration according to claim 1, characterized in that, The steps of analyzing offline videos using a large model to obtain the recommendation algorithm list set {A2} include: Extract n images at equal intervals from an offline video, and extract one image before and after each image at a first set time interval; The large model interface is called n times using the prompt method. The large model analyzes the three images extracted at intervals to obtain the recommendation algorithm list set {A2}.

3. The offline video pre-analysis method based on size model collaboration according to claim 1, characterized in that, The steps of performing a second algorithmic list set analysis on offline videos using a large model include: Starting from the first frame of the offline video, three images are continuously extracted at second set time intervals, and the extracted images are analyzed by the second algorithm list set analysis using a large model. Starting from the moment of the third image, slide the third set time and repeatedly call the large model to perform the second algorithm list set analysis on the extracted images; repeat this process until the analysis of the entire offline video is completed.

4. An offline video retrieval method based on size model collaboration, characterized in that, The method for retrieving offline videos processed using the offline video pre-analysis method based on size model collaboration as described in any one of claims 1-3 includes the following steps: By analyzing the search results of a single sentence input by a user using a large model, the search results are broken down and standardized into a new search format, represented as: [First Algorithm List] or\and [Other Algorithm Lists]. The first search result was obtained by searching the small model alarm database using the [first algorithm list set]. Convert the "List of Other Algorithms" into a vector, use the "List of Other Algorithms" as the search criteria, and search the vector database to obtain the second search result; Summarize the first and second search results and output them.

5. An offline video pre-analysis system based on large and small model collaboration, characterized in that, An offline video pre-analysis method based on size model collaboration as described in any one of claims 1-3, comprising: The first module is used to call the large model to perform semantic understanding on the structured information recorded from offline videos, and obtain the list of recording algorithms {A1}. The second module is used to call a large model to analyze offline videos and obtain a list of recommendation algorithms {A2}. The third module is used to take the union of the input algorithm list set {A1} and the recommendation algorithm list set {A2} to obtain the total algorithm list set {A}. The fourth module is used to obtain the small model algorithm list set {B}, take the intersection of the small model algorithm list set {B} and the total algorithm list set {A} to obtain the first algorithm list set, denoted as {A}∩{B}; and obtain the second algorithm list set based on the total algorithm list set {A} and the first algorithm list set {A}∩{B}, denoted as {A} - {A}∩{B}. The fifth module is used to send the first algorithm list set to the small model to analyze the offline video. After the corresponding algorithm is analyzed, an alarm is generated and stored in the small model alarm library. The sixth module is used to call the large model to perform a second algorithm list set analysis on the offline video, describe the analysis results in text, convert the text description into vectors, and store them in a vector database.

6. An offline video retrieval system based on size-model collaboration, characterized in that, The method for performing the offline video retrieval method based on size model collaboration as described in claim 4 includes: The seventh module is used to call the large model to analyze the search content of a user's input sentence, break down the search content and standardize it into a new search format; The eighth module is used to retrieve the first search result from the small model alarm database using the [first algorithm list set]. The ninth module is used to convert the "List of Other Algorithms" into a vector, and then use the "List of Other Algorithms" as the search criteria to search the vector database to obtain the second search result. The tenth module is used to summarize and output the first and second search results.

7. An electronic device, characterized in that, include: processor; as well as, Memory for storing the executable instructions of the processor; The processor is configured to execute the offline video pre-analysis method based on size model collaboration as described in claims 1-3 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the offline video pre-analysis method based on size model collaboration as described in claims 1-3.

9. An electronic device, characterized in that, include: processor; as well as, Memory for storing the executable instructions of the processor; The processor is configured to execute the offline video retrieval method based on size model collaboration as described in claim 4 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the offline video retrieval method based on size model collaboration as described in claim 4.

Images