Audio and video retrieval method, system and storage medium based on large language model

Abstract

The present invention provides an audio and video retrieval method, system and storage medium based on a large language model, relating to the field of computer information technology. The method comprises the following steps: obtaining natural language request information related to audio and video retrieval content; vectorizing the request information according to a large language model tool to extract retrieval information of the request information; searching in a vector database using a vector space search algorithm to screen out vectors matching the retrieval information; searching in a relational database by matching the index information of the vector to obtain associated information corresponding to the vector, the associated information including the video name, text content and start and end time; sending the target video or target audio corresponding to the vector and its associated information to a client, and being able to select a specified time point from the start and end time to play the target video or target audio.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and in particular to a video and audio retrieval method. Background Art

[0002] Video retrieval, which involves searching numerous video files for clips that meet specific criteria, is a challenging task. However, video retrieval currently has a wide range of applications. However, for retrieval technologies that are not yet fully mature, accurate video content recognition often requires complex multimodal models, advanced algorithms, and extensive computing resources. This results in complex data processing, long processing times, low search efficiency, and high server requirements. Summary of the Invention

[0003] The purpose of the present invention is to provide an audio and video retrieval method, system and storage medium based on a large language model to solve the current problems of slow video retrieval speed and high cost.

[0004] To solve the above problems, the present invention first provides an audio and video retrieval method based on a large language model, comprising the following steps: obtaining natural language request information related to audio and video retrieval content; vectorizing the request information according to a large language model tool to extract retrieval information of the request information; searching in a vector database using a vector space search algorithm to filter out vectors that match the retrieval information; searching in a relational database by matching the index information of the vector to obtain associated information corresponding to the vector, the associated information including name, text content and start and end time; sending the target video or target audio corresponding to the vector and its associated information to the client, and being able to select a specified time point from the start and end time to play the target video or target audio.

[0005] Furthermore, the construction process of the vector database and the relational database includes: file separation and recognition, using open source software tools to first extract audio files from video files, and then parsing the extracted audio files through a speech recognition model, and directly parsing the audio files through a speech recognition model; obtaining file association information, the speaker, the text content corresponding to the audio content, and the start and end time information corresponding to the text content in the audio file; preliminarily merging the recognition results and generating start and end time information corresponding to the merged text content; content merging and correction, performing a secondary merge on the preliminary merged content to form a merged paragraph content; using a large language model for semantic processing and deleting non-essential content; merging the generated results three times to form paragraphs of a preset text length, manually correcting erroneous content, and updating the corresponding start and end times of the paragraphs; vectorization and storage, using a large language model to vectorize the merged content, and storing the generated vectors in a vector database; and storing the finally merged association information in a relational database.

[0006] Furthermore, the secondary merging includes: selecting a text length merging method and a time length merging method according to the category of the video file or audio file, the text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to the preset time length.

[0007] Furthermore, the secondary merging further includes: adding overlapping content to the beginning of the paragraph formed after merging the segments, wherein the overlapping content is identical to the content portion of the corresponding length at the end of the previous paragraph according to a preset text length.

[0008] Furthermore, sending the target video corresponding to the vector to the client includes: reordering the results retrieved from the relational database according to relevance, and sending the results according to the reordered order to the client.

[0009] The present invention also provides an audio and video retrieval system based on a large language model, comprising: a vectorization module, which performs vectorization processing on natural language request information related to audio and video retrieval content obtained according to a large language model tool, and extracts retrieval information of the request information; a vector retrieval module, which searches in a vector database according to the retrieval information, and filters out vectors that match the retrieval information; an association module, which searches in a relational database by matching the index information of the vector, and obtains associated information corresponding to the vector, wherein the associated information includes name, text content and start and end time; a sending module, which reorders the target video or target audio retrieved from the relational database according to the relevance, and sends the target video or target audio and its associated information to the client according to the reordering result, and can select a specified time point from the start and end time to play the selected target video or target audio.

[0010] Furthermore, the construction process of the vector database and the relational database includes: file separation and recognition, for video files, using open source software tools to first extract audio files from video files; parsing the audio files through a speech recognition model, and obtaining related information, the speaker, the text content corresponding to the audio content, and the start and end time information corresponding to the text content in the audio file; preliminary merging of the recognition results, and generating start and end time information corresponding to the merged text content; content merging and correction, performing a secondary merge on the preliminary merged content to form a merged paragraph content; using a large language model for semantic processing to delete non-essential content; merging the generated results three times to form a paragraph of a preset text length, manually correcting the erroneous content, and updating the corresponding start and end time of the paragraph; vectorization and storage, using a large language model to vectorize the merged content, and storing the generated vector in a vector database; and storing the finally merged related information in a relational database.

[0011] Furthermore, the secondary merging includes: selecting a text length merging method and a time length merging method according to the category of the video file or audio file, the text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to the preset time length; adding overlapping content to the starting part of the paragraph formed after the merging and segmenting, and the overlapping content is the same as the content part of the corresponding length at the end of the previous paragraph according to the preset text length.

[0012] The present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed, they are used to implement the audio and video retrieval method based on a large language model as described in any one of the above technical solutions.

[0013] The present invention also provides a computer program product, including a computer program, which, when executed, implements the audio and video retrieval method based on a large language model as described in any one of the above technical solutions.

[0014] According to the audio and video retrieval method, system and storage medium based on the large language model provided by the present invention, audio and video searches are converted into more convenient text searches by cleverly combining speech recognition, text processing of the large language model and natural language search technology based on vector algorithms, thereby avoiding the complex processing required for deep recognition of video content. There is no need to rely on complex algorithms, the large model involved is relatively small in scale, and ordinary CPU servers can meet the operating requirements, which reduces costs, improves retrieval speed, and has stronger overall practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0016] Figure 1 Flowchart of the video retrieval method using a large language model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] Example 1

[0019] This embodiment provides a method for constructing a vector database and a relational database, which mainly includes three stages:

[0020] 1. File separation and identification stage;

[0021] This phase includes the following steps to convert a video file or audio file into preliminary speech recognition results.

[0022] S101: For video files, extract audio files from the video files using open source software tools;

[0023] S201: The audio file is directly parsed using a speech recognition model. The audio file extracted from the video file is also parsed using a speech recognition model to extract the speaker, the text content corresponding to the audio content, and the start and end time information of the text content in the audio file. Since the video and audio are synchronized, the start and end time of the video content are also obtained at the same time.

[0024] S202: Preliminary merging of the recognition results is performed, usually by sentence, while retaining and updating the information obtained in the above steps.

[0025] 2. Content merging and revision stage;

[0026] This stage includes the following steps to further merge and polish the speech recognition results to form high-quality text content.

[0027] S301: Perform secondary merging on the recognized content in the previous stage.

[0028] This embodiment is mainly used for searching training and conference videos. These videos are characterized by a high degree of synchronization between visual and auditory content, and all videos are accompanied by audio commentary. For example, a training PPT presentation accompanied by the instructor's audio provides strong support for in-depth analysis of the video content.

[0029] Specifically, for training videos, which are usually delivered by a single trainer, with clear structure and high-quality audio, they are merged based on text length, usually merging adjacent 150-word content into one paragraph;

[0030] For conference videos, which are characterized by multiple people speaking, intertwined content, and large fluctuations in sound quality, the time period merging method is adopted to merge adjacent content within 60 seconds into one paragraph.

[0031] When merging, consider the issue of sentence segmentation. This means you can't merge strictly based on length or time, to avoid splitting a whole sentence into two adjacent segments. Furthermore, when merging, overlapping content is added to the beginning of the resulting paragraph. This overlapping content, based on a preset text length, matches the content at the end of the previous paragraph to improve the accuracy of subsequent vector searches. The length of the overlapping text can be adjusted as needed. For example, for training videos, an overlap of approximately 30 words is acceptable, while for conference videos, a 12-second overlap is acceptable.

[0032] S401: If the merged content contains a large number of non-essential contents such as modal particles, direct reading will affect the understanding of its meaning. The large language model's ability to understand, polish, and summarize is used to process the content and form a text with relatively clear meaning.

[0033] During this process, the large model is required to retain details as much as possible to avoid information loss due to over-generalization.

[0034] S501: The results generated by the large model are further merged to avoid the content being too trivial and to form paragraphs within 300 words in length in preparation for vectorization.

[0035] At the same time, a small amount of manual inspection is carried out to correct obvious identification errors.

[0036] In each of the above merging steps, the start and end time of the content also needs to be calculated to facilitate the implementation of the subsequent search process.

[0037] 3. Vectorization and storage phase;

[0038] In this stage, the merged results are vectorized and stored.

[0039] S601: Input the merged result, including the speaker, content, start and end time information; use the large language model to vectorize the content, and store the vector in a vector database.

[0040] At the same time, other related information, including video or audio name, speaker, start and end time, and vector database index, is stored in a relational database for subsequent retrieval and analysis.

[0041] Example 2

[0042] This embodiment provides an audio and video retrieval method based on a large language model, such as Figure 1 As shown, the following steps are included:

[0043] S701: vectorizing the acquired natural language request information related to the audio and video retrieval content using a large language model tool, using the same large language model tool as step S601 in the data generation process;

[0044] S801: Searching the vector database constructed using the method of Example 1 using a vector space search algorithm to retrieve N vectors closest to the above information;

[0045] S802: Using the index of the retrieved vector match, search in the relational database to obtain the associated information corresponding to the vector;

[0046] S803: sorting the search results by the rearrangement model, sorting them by relevance, and arranging them in descending order of relevance;

[0047] S804: Returning the reordered result to the client, the result including the text, file information, and the starting point of the text in the video or audio;

[0048] S805: You can select the target video or target audio and view its associated information, and support playing the target video or target audio from a specified time point.

[0049] The construction of the above vector database and the relational database adopts the construction method described in Example 1.

[0050] This embodiment implements video or audio search using natural language. Natural language search based on vector algorithms is an advanced search technology that leverages the power of large language models to understand and process human language. The core of this search method is to convert natural language questions into numerical forms that computers can understand, namely vectors. These vectors mathematically represent the semantic content of the language, allowing computers to find the answer that best matches the question at the semantic level. Natural language search based on vector algorithms provides a more advanced and flexible search method that can better understand the user's intentions and provide more accurate and rich search results rather than relying solely on keyword matching.

[0051] In addition, the clever combination of speech recognition, text processing of large language models, and natural language search technology based on vector algorithms converts video searches into more convenient text searches, thus avoiding the complex processing required for in-depth recognition of video content. There is no need to rely on complex algorithms, and the large models involved are relatively small in scale. Ordinary CPU servers can meet the operating requirements, reducing costs, improving retrieval speed, and improving overall practicality.

[0052] Example 3

[0053] This embodiment provides a video retrieval system based on a large language model, which can implement the method of embodiment 2, including:

[0054] The vectorization module vectorizes the natural language request information related to the video retrieval content obtained using the large language model tool and extracts the retrieval information of the request information;

[0055] A vector retrieval module searches the vector database based on the retrieval information and selects vectors that match the retrieval information;

[0056] The association module searches the relational database by matching the index information of the vector to obtain the associated information corresponding to the vector, including the video name, text content, and start and end time;

[0057] The sending module sends the target video corresponding to the vector and its related information to the client, and can select a specified time point from the start and end time to play the target video.

[0058] The vector database and the relational database are constructed using the construction method described in Example 1.

[0059] Example 4

[0060] This embodiment provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed, they are used to implement the audio and video retrieval method based on a large language model as provided in Example 2.

[0061] Example 5

[0062] This embodiment provides a computer program product, including a computer program. When the computer program is executed, it implements the audio and video retrieval method based on the large language model provided in Example 2.

[0063] In the description of this embodiment, it should be noted that those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the control device through a computer program, and the program can be stored in a computer-readable storage medium. When executed, the program may include the processes of the above-mentioned method embodiments, wherein the storage medium may be a memory, a disk, an optical disk, etc.

[0064] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope defined by the claims.

[0065] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0066] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0067] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio and video retrieval method based on a large language model, characterized in that: The following steps are involved: Obtain natural language request information related to audio and video retrieval content; Performing vectorization processing on the request information according to a large language model tool to extract search information of the request information; Using a vector space search algorithm to search in a vector database, and screening out vectors therein that match the search information; By matching the index information of the vector, searching in a relational database, obtaining the associated information corresponding to the vector, the associated information including the name, text content, and start and end time; The target video or target audio corresponding to the vector and the associated information thereof are sent to the client, and the target video or target audio can be played at a specified time point from the start and end time; The construction process of the vector database and the relational database includes: File separation and recognition: For video files, use open source software tools to first extract audio files from them, and then parse the extracted audio files through a speech recognition model. For audio files, directly parse them through a speech recognition model; obtain file-related information, including the speaker, the text content corresponding to the audio content, and the corresponding start and end time information of the text content in the audio file; preliminarily merge the recognition results and generate the start and end time information corresponding to the merged text content; Content merging and correction: perform a secondary merge on the initially merged content to form merged paragraph content; use a large language model for semantic processing and delete non-essential content; merge the generated results three times to form paragraphs of a preset text length, manually correct erroneous content, and update the corresponding start and end times of the paragraphs; Vectorization and storage: using a large language model to vectorize the merged content, and storing the generated vectors in a vector database; and storing the final merged related information in a relational database.

2. The audio and video retrieval method based on a large language model according to claim 1, characterized in that The secondary merger includes: A text length merging method and a time length merging method are selected according to the category of the video file or audio file. The text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to the preset time length.

3. The audio and video retrieval method based on a large language model according to claim 2, characterized in that The secondary merger also includes: Overlapping content is added to the beginning of the paragraph formed after merging the segments, and the overlapping content is the same as the content of the corresponding length at the end of the previous paragraph according to a preset text length.

4. The audio and video retrieval method based on a large language model according to any one of claims 1 to 3, characterized in that: Sending the target video or target audio corresponding to the vector to the client includes: The results retrieved from the relational database are reordered according to relevance and sent to the client in the reordered order.

5. An audio and video retrieval system based on a large language model, characterized in that include: A vectorization module, which vectorizes the natural language request information related to the audio and video retrieval content obtained using a large language model tool, and extracts retrieval information from the request information; A vector retrieval module searches a vector database according to the retrieval information and selects vectors that match the retrieval information; An association module searches a relational database by matching the index information of the vector to obtain association information corresponding to the vector, the association information including name, text content, and start and end time; a sending module, which reorders the target video or target audio retrieved from the relational database according to their relevance, and sends the target video or target audio and their associated information to the client according to the reordering result, and is capable of selecting a specified time point from the start and end time to play the selected target video or target audio; The construction process of the vector database and the relational database includes: File separation and recognition: For video files, open source software tools are used to extract audio files from the video files. The audio files are parsed using a speech recognition model to obtain file-related information, including the speaker, the text content corresponding to the audio content, and the start and end time information of the text content in the audio file. The recognition results are preliminarily merged to generate the start and end time information corresponding to the merged text content. Content merging and correction: perform a secondary merge on the initially merged content to form merged paragraph content; use a large language model for semantic processing and delete non-essential content; merge the generated results three times to form paragraphs of a preset text length, manually correct erroneous content, and update the corresponding start and end times of the paragraphs; Vectorization and storage: using a large language model to vectorize the merged content, and storing the generated vectors in a vector database; and storing the final merged related information in a relational database.

6. The audio and video retrieval system based on a large language model according to claim 5, characterized in that The secondary merger includes: Select a text length merging method and a time length merging method based on the category of the video file or audio file. The text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to the preset time length; Overlapping content is added to the beginning of the paragraph formed after merging the segments, and the overlapping content is the same as the content of the corresponding length at the end of the previous paragraph according to a preset text length.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the audio and video retrieval method based on a large language model according to any one of claims 1 to 4.

8. A computer program product, characterized in that The invention comprises a computer program, which implements the audio and video retrieval method based on a large language model as claimed in any one of claims 1 to 4 when the computer program is executed.

Images