Video retrieval method based on Agent

Abstract

The invention discloses an Agent-based video retrieval method, which comprises two major steps of video storage and retrieval: during video storage, an original video is subjected to time sequence segmentation to obtain video clips, a comprehensive description containing a picture description and an audio description is generated through a video description module, and then the overall description of the original video is obtained through summarization of a large language model; respectively storing the two types of descriptions into a database and generating a vector index; during retrieval, the Query processing Agent analyzes user query to clarify a retrieval intention, and the database retrieval Agent optimizes the query and obtains a retrieval result based on the vector index. In the video description module, a cue word Agent generates a targeted cue word according to picture description, an audio understanding large model is guided to extract audio information associated with a picture, and tight combination of the audio information and the picture information is achieved. The problems that in the prior art, sound and picture information association is insufficient, and user intentions are difficult to distinguish are solved, the accuracy of video retrieval is effectively improved, interaction obstacles between the user and the system are reduced, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the fields of large model prompt word engineering, AIAgent and image processing technology, specifically an agent-based video retrieval method. Background Technology

[0002] In existing video retrieval methods, when a video is added to the database, it is usually segmented and then the screen description, text information, speech text and audio modal information are extracted and stored separately. During retrieval, the user query is extracted with keywords and vectors are transformed before the retrieval results are matched.

[0003] However, existing technologies have the following drawbacks:

[0004] First, it is difficult to distinguish whether a user wants the complete video or a video segment, resulting in a poor user interaction experience;

[0005] Secondly, audio information extraction is limited to extracting speech or other audio components alone, and is not closely related to the image information, which affects the accuracy of retrieval. Summary of the Invention

[0006] The purpose of this invention is to provide an agent-based video retrieval method, which aims to solve the problems of weak correlation between audio and video information and unclear user search intent, thereby improving the accuracy of video retrieval and user experience.

[0007] To address the aforementioned technical problems, this invention provides an agent-based video retrieval method, including a video storage step and a retrieval step, as detailed below:

[0008] (1) Video import steps:

[0009] a. The original video is divided into video segments through temporal segmentation. Temporal segmentation can be achieved through a video segmentation model, which divides the original video into video segments of appropriate length to facilitate subsequent processing and retrieval.

[0010] b. The video clip is input into the video description module, which generates a comprehensive description that includes both visual and audio descriptions. The specific process is as follows: the video understanding big model analyzes the video clip and outputs a visual description; the prompt agent receives the visual description and generates targeted prompts to guide the audio understanding big model to extract audio information related to the visuals; the audio understanding big model outputs an audio description based on the prompts; the big language model merges the visual and audio descriptions to obtain a comprehensive description.

[0011] c. The large language model summarizes the comprehensive description of all video segments of the same original video to generate an overall description of the original video;

[0012] d. Store the comprehensive description of the video segment and the overall description of the original video in the database respectively, and generate corresponding vector indexes for the two types of descriptions to facilitate fast retrieval later.

[0013] (2) Search steps:

[0014] a. Receive the query input from the user and pass the query to the Query Processing Agent;

[0015] b. Query Processing Agent analyzes the query to determine if the user has a video retrieval need. If so, it further determines whether the user needs to retrieve the complete video or a video segment.

[0016] c. The database retrieval agent receives the judgment result from the query processing agent, optimizes the user query, including semantic expansion and vector transformation, and then calls the retrieval tool to perform a similarity search in the database based on the vector index to obtain the search result with the highest matching degree; d. The search results are returned to the user, which may be video clips, complete video links, or related answers depending on the judgment result.

[0017] In summary, due to the adoption of the above-mentioned technologies, the beneficial effects of this invention are:

[0018] 1. This invention generates prompts for the audio understanding model based on the image description using a prompt word agent, which closely links audio information extraction with the image content, avoids the extraction of irrelevant audio information, and improves the accuracy and efficiency of retrieval.

[0019] 2. By processing the Query Agent, the user's search intent is clarified, and the search needs for complete videos and video clips are distinguished, reducing the interaction barriers between users and the system and improving the user experience;

[0020] 3. The comprehensive description of video segments and the overall description of the original video are stored separately and a vector index is generated. Combined with the query optimization of the database retrieval agent, the accuracy and efficiency of the retrieval results are further improved. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention, making other features, objects, and advantages of the invention more apparent. The illustrative embodiments of the invention illustrated in the drawings and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0022] Figure 1 This is a schematic diagram of the overall process of video data entry into the database;

[0023] Figure 2 This is a schematic diagram of the internal processing flow of the video description module;

[0024] Figure 3 A schematic diagram of the user query processing flow;

[0025] Figure 4 This is a diagram illustrating a hardware scenario where the technology is used. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] In the description of this invention, it should be understood that the terms indicating orientation or positional relationship are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing the invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.

[0028] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific context of the specification.

[0029] This invention provides an agent-based video retrieval method, including the following embodiments:

[0030] Example 1: Video Ingestion and Retrieval in a Multimedia Content Distribution Platform

[0031] (1) Video import:

[0032] a. The multimedia content distribution platform acquires various types of raw videos, and uses a video segmentation model to perform temporal segmentation on the raw videos to obtain multiple video segments of uniform duration;

[0033] b. Input each video segment into the video understanding model. This model analyzes the content of the video segments and outputs a description of the scene, including information such as the actions of people, the environment, and the features of objects in the scene.

[0034] c. The screen description is passed to the prompt word Agent. The prompt word Agent generates prompt words based on the key information in the screen description. The prompt words focus on audio features related to the screen content, guiding the audio understanding model to extract audio information in a targeted manner.

[0035] d. After receiving the prompt words, the audio understanding model analyzes the audio part of the video segment, extracts audio information related to the picture, and generates audio descriptions, including speech content, background sound effects, music style, etc.

[0036] e. Input the visual description and audio description into the large language model. The large language model integrates the two types of descriptions to generate a logically coherent comprehensive description of the video clip.

[0037] f. The large language model collects comprehensive descriptions of all video segments from the same original video, summarizes and generalizes them, and generates an overall description that reflects the core content of the original video;

[0038] g. Store the comprehensive description of the video segment and the overall description of the original video into the platform database respectively. At the same time, perform semantic vector conversion on the two types of descriptions to generate corresponding vector indexes and store them.

[0039] (2) Search process:

[0040] a. A user enters a query on a multimedia content distribution platform, specifying a video content within a particular scene; b. The query is passed to a Query Processing Agent, which performs semantic analysis to determine that the user needs to retrieve video clips; c. The Database Retrieval Agent receives the results, semantically expands the user's query by adding relevant synonyms and scene-related words, and then converts the optimized query into a semantic vector; d. The Database Retrieval Agent invokes a retrieval tool to perform similarity matching based on the semantic vector in the database's vector index, filtering out the video clips with the highest matching degree; e. The filtered video clips are sorted by matching degree and returned to the user, who can directly view the video clips or obtain links to the complete videos.

[0041] Example 2: Multimedia content referencing in an LLM chat system

[0042] (1) Video import:

[0043] a. The LLM chat system's backend server collects various original videos in science popularization, entertainment, and education categories. It then segments the original videos temporally using a video segmentation model, dividing the video into segments based on logical breakpoints to ensure the integrity of each segment's theme; b. The video segment input video understanding model generates a scene description, covering the core elements and dynamic processes within the scene; c. The prompting agent generates prompts based on the scene description. For scenes containing experimental operations, the prompts focus on sounds during the experiment, such as instrument operation sounds.

[0044] c. The large-scale audio understanding model extracts corresponding audio information based on prompt words and generates audio descriptions;

[0045] d. The large language model merges the visual description and audio description to obtain a comprehensive description of the video segment;

[0046] e. The large language model summarizes all segments of the same original video to generate a comprehensive description;

[0047] g. Store the fragment summary description, the overall description, and the corresponding vector index into the backend database.

[0048] (2) Search process:

[0049] a. Users chat with the LLM system to inquire about video content related to a specific knowledge point;

[0050] b. The user query is passed to the Query Processing Agent. After analysis, the Agent determines that the user needs to retrieve videos, and the requirement is for complete videos.

[0051] c. The database retrieval agent optimizes user queries, identifies the core keywords corresponding to knowledge points, and converts them into semantic vectors; d. Based on the semantic vectors, the agent retrieves the corresponding original video description from the database and obtains the complete video link with the highest matching degree; e. The LLM chat system returns the complete video link and related descriptions to the user, enabling accurate referencing of multimedia content.

[0052] Example 3: Precise Retrieval of Long Video Clips

[0053] (1) Video import:

[0054] a. A video platform obtains the original video of a long documentary and uses a video segmentation model to divide it into multiple short video segments, each segment corresponding to a specific plot in the documentary;

[0055] b. The video segment input video understanding model generates a scene description, including information such as characters, scenes, and shot transitions in the plot; c. The prompting agent generates prompts based on the scene description. For interview scenes in documentaries, the prompts focus on the interviewer's voice content and background ambient sounds; d. The audio understanding model extracts audio information based on the prompts and generates an audio description, accurately recording the voice content and related sound effects; e. The large language model merges the scene description and audio description to generate a comprehensive description for each video segment.

[0056] f. The large language model summarizes the comprehensive descriptions of all segments to generate an overall description of the documentary, covering the documentary's theme, structure, core content, etc.; g. The comprehensive descriptions of the segments and the overall description are stored in the database respectively, and corresponding vector indexes are generated.

[0057] (2) Search process:

[0058] a. The user enters a query, hoping to find a specific interview segment from the documentary;

[0059] b. After the Query Agent analyzes the query, it determines whether the user needs to retrieve video clips;

[0060] c. Database retrieval agent optimizes queries, extracts key information from the query, including interviewees, topics, etc., and converts them into semantic vectors;

[0061] d. Based on semantic vectors, retrieve the corresponding comprehensive description of video segments from the vector index of the database, and filter out the video segments with the highest matching degree;

[0062] e. Return the video clip to the user, who can watch the target scene directly without watching the full documentary.

Claims

1. An agent-based video retrieval method, characterized in that: Includes the following steps: Video storage steps: The original video is segmented temporally to obtain video segments. A comprehensive description of the video segments is generated through the video description module. The comprehensive description includes the visual description and audio description of the video segment. The comprehensive descriptions of multiple video segments are summarized through a large language model to obtain the overall description of the original video. The comprehensive descriptions of the video segments and the overall description of the original video are stored in the database and a vector index is generated. Search steps: Receive user query, use the Query Processing Agent to determine whether the user needs to search for a video and whether the user needs to search for the entire video or a video segment; use the Database Search Agent to optimize the search query based on the determination result and call the search tool to retrieve the corresponding search results from the database and return them.

2. The agent-based video retrieval method according to claim 1, characterized in that, The steps for the video description module to generate a comprehensive description of a video segment include: Input video clips into the video understanding model to obtain a visual description of the video clips; The image description is input into the prompt word Agent, which generates prompt words for the large audio understanding model based on the image description. The prompt words are input into the audio understanding model to obtain an audio description of the video segment; The image and audio descriptions are input into a large language model and merged to obtain a comprehensive description of the video segment.

3. The agent-based video retrieval method according to claim 1, characterized in that, The temporal segmentation step involves dividing the original video into video segments of a preset length using a video segmentation model.

4. The agent-based video retrieval method according to claim 1, characterized in that, The steps for optimizing retrieval queries using the database retrieval agent include semantic expansion and vector transformation of the user query, followed by similarity retrieval based on the vector index.

5. The agent-based video retrieval method according to claim 1, characterized in that, The search results include video clips, complete video links, or corresponding answers, depending on the judgment result of the Query Processing Agent.

6. The agent-based video retrieval method according to claim 1, characterized in that, The prompts generated by the Agent are related to the content of the video clips and are used to guide the audio understanding model to extract audio information associated with the visuals.

7. The agent-based video retrieval method according to claim 1, characterized in that, The vector index is generated based on the semantic vector of the comprehensive description of the video segment and the overall description of the original video.

Images