Audio and video retrieval method and system based on large language model and storage medium

Abstract

The invention provides an audio and video retrieval method and system based on a large language model and a storage medium, and relates to the technical field of computer information, and the method comprises the following steps: obtaining natural language request information related to audio and video retrieval content; performing vectorization processing on the request information according to a large language model tool, and extracting retrieval information of the request information; retrieving in a vector database by adopting a vector space search algorithm, and screening vectors matched with the retrieval information; searching in a relational database by matching index information of the vectors to obtain associated information corresponding to the vectors, the associated information including video names, text contents and starting and ending time; and sending the target video or the target audio corresponding to the vector and the associated information thereof to the client, and selecting a specified time point from the starting time to the ending time to play the target video or the target audio.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and in particular to a video and audio retrieval method. Background Art

[0002] Video retrieval refers to searching for clips that meet specific conditions in a large number of video files, which is a difficult task. However, the current application scenarios of video retrieval are extremely wide. For the retrieval technology that is not yet fully mature, to achieve accurate video content recognition, it often needs to rely on complex multimodal models, advanced algorithms and huge computing resources. The data processing process is complicated, the processing time is long, the search efficiency is low, and the server requirements are high. Summary of the invention

[0003] The purpose of the present invention is to provide an audio and video retrieval method, system and storage medium based on a large language model to solve the current problems of slow video retrieval speed and high cost.

[0004] To solve the above problems, the present invention first provides an audio and video retrieval method based on a large language model, comprising the following steps: obtaining natural language request information related to audio and video retrieval content; vectorizing the request information according to a large language model tool to extract retrieval information of the request information; searching in a vector database using a vector space search algorithm to filter out vectors that match the retrieval information; searching in a relational database by matching the index information of the vector to obtain associated information corresponding to the vector, the associated information including name, text content and start and end time; sending the target video or target audio corresponding to the vector and its associated information to a client, and being able to select a specified time point from the start and end time to play the target video or target audio.

[0005] Furthermore, the construction process of the vector database and the relational database includes: file separation and recognition, for video files, using open source software tools to first extract audio files from them, and then parsing the extracted audio files through a speech recognition model, and parsing the audio files directly through a speech recognition model; obtaining file association information, the speaker, the text content corresponding to the audio content, and the start and end time information corresponding to the text content in the audio file; preliminarily merging the recognition results, and generating start and end time information corresponding to the merged text content; content merging and correction, performing secondary merging of the preliminary merged content to form merged paragraph content; using a large language model for semantic processing and deleting non-essential content; merging the generated results three times to form paragraphs of a preset text length, manually correcting erroneous content, and updating the corresponding start and end time of the paragraphs; vectorization and storage, using a large language model to vectorize the merged content, and storing the generated vectors in a vector database; and storing the finally merged association information in a relational database.

[0006] Furthermore, the secondary merging includes: selecting a text length merging method and a time length merging method according to the category of the video file or audio file, the text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to the preset time length.

[0007] Furthermore, the secondary merging further comprises: adding overlapping content to the beginning of the paragraph formed after merging the segments, wherein the overlapping content is identical to the content portion of the corresponding length at the end of the previous paragraph according to a preset text length.

[0008] Furthermore, sending the target video corresponding to the vector to the client includes: reordering the results retrieved from the relational database according to relevance, and sending the results to the client according to the reordered order.

[0009] The present invention also provides an audio and video retrieval system based on a large language model, comprising: a vectorization module, which performs vectorization processing on natural language request information related to audio and video retrieval content obtained according to a large language model tool, and extracts retrieval information of the request information; a vector retrieval module, which searches in a vector database according to the retrieval information, and screens out vectors that match the retrieval information; an association module, which searches in a relational database by matching the index information of the vector, and obtains the associated information corresponding to the vector, wherein the associated information includes a name, text content, and a start and end time; a sending module, which reorders the target video or target audio retrieved from the relational database according to the relevance, and sends the target video or target audio and its associated information to the client according to the reordering result, and can select a specified time point from the start and end time to play the selected target video or target audio.

[0010] Furthermore, the construction process of the vector database and the relational database includes: file separation and recognition, for video files, using open source software tools to first extract audio files from video files; parsing the audio files through a speech recognition model, and obtaining related information, the speaker, the text content corresponding to the audio content, and the start and end time information corresponding to the text content in the audio file; preliminary merging of the recognition results, and generating start and end time information corresponding to the merged text content; content merging and correction, secondary merging of the preliminary merged content to form merged paragraph content; semantic processing using a large language model to delete non-essential content; tertiary merging of the generated results to form paragraphs of a preset text length, manual correction of erroneous content, and updating the corresponding start and end time of the paragraphs; vectorization and storage, using a large language model to vectorize the merged content, and storing the generated vectors in a vector database; and storing the finally merged related information in a relational database.

[0011] Furthermore, the secondary merging includes: selecting a text length merging method and a time length merging method according to the category of the video file or the audio file, the text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to a preset time length; adding overlapping content to the starting part of the paragraph formed after the merging and segmenting, and the overlapping content is the same as the content part of the corresponding length at the end of the previous paragraph according to the preset text length.

[0012] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed, they are used to implement the audio and video retrieval method based on a large language model as described in any one of the above technical solutions.

[0013] The present invention also provides a computer program product, including a computer program, which, when executed, implements the audio and video retrieval method based on a large language model as described in any one of the above technical solutions.

[0014] According to the audio and video retrieval method, system and storage medium based on a large language model provided by the present invention, audio and video searches are converted into more convenient text searches by cleverly combining speech recognition, text processing of a large language model and natural language search technology based on a vector algorithm, thereby avoiding the complex processing required for deep recognition of video content. There is no need to rely on complex algorithms, the large model involved is relatively small in scale, and ordinary CPU servers can meet the operating requirements, thereby reducing costs, improving retrieval speed, and having stronger overall practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0016] Figure 1 A flow chart of a video retrieval method using a large language model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] Example 1

[0019] This embodiment provides a method for constructing a vector database and a relational database, which mainly includes three stages:

[0020] 1. File separation and identification stage;

[0021] This phase includes the following steps, which are aimed at converting the video file or audio file into preliminary speech recognition results.

[0022] S101: For video files, extract audio files from the video files using open source software tools;

[0023] S201: The audio file is directly analyzed using a speech recognition model. The audio file extracted from the video file is also analyzed using a speech recognition model to extract the speaker, the text content corresponding to the audio content, and the start and end time information of the text content in the audio file. Since the video and audio are synchronized, the start and end time of the video content is also obtained at the same time.

[0024] S202: Preliminary merging of the recognition results is performed, usually by sentence, while retaining and updating the information obtained in the above steps.

[0025] 2. Content merging and revision stage;

[0026] This stage includes the following steps to further merge and polish the speech recognition results to form high-quality text content.

[0027] S301: Perform secondary merging on the recognized contents in the previous stage.

[0028] The main application scenario of this embodiment focuses on the retrieval of training and conference videos. The characteristics of such videos are that the visual content and the auditory content are highly synchronized, and all videos are accompanied by audio explanations. For example, the PPT presentation of the training is combined with the lecturer's explanation sound, which provides strong support for the in-depth analysis of the video content.

[0029] Specifically, for training videos, which are usually delivered by a single trainer, with clear structure and high-quality sound, they are merged according to the length of the text, usually merging adjacent 150-word content into one paragraph;

[0030] For conference videos, which are characterized by multiple people speaking, intertwined content, and large fluctuations in sound quality, the method of merging by time period is adopted to merge adjacent content within 60 seconds into one paragraph.

[0031] When merging, the problem of sentence segmentation must be considered, that is, merging cannot be performed strictly according to length or time, and avoid splitting a whole sentence into two adjacent paragraphs. In addition, when merging, the front and back overlap method will be adopted. The overlapping content is added to the beginning of the paragraph formed after merging the segments, that is, the overlapping content is the same as the content part of the corresponding length at the end of the previous paragraph according to the preset text length, so as to improve the accuracy of subsequent vector searches. The length of overlapping text can be adjusted according to needs. For example, for training videos, an overlap of about 30 words can be used, and for conference videos, an overlap of 12 seconds can be used.

[0032] S401: If the merged content contains a lot of non-essential content such as modal particles, direct reading will affect the understanding of its meaning. Use the understanding, polishing and summarizing capabilities of the large language model to process the content and form a text with relatively clear meaning.

[0033] In this process, the large model is required to retain details as much as possible to avoid information loss due to over-generalization.

[0034] S501: The results generated by the large model are further merged to avoid the content being too trivial and to form paragraphs within 300 words in length in preparation for vectorization.

[0035] At the same time, a small amount of manual inspection is performed to correct obvious identification errors.

[0036] In each of the above merging steps, the start and end time of the content also needs to be calculated to facilitate the implementation of the subsequent search process.

[0037] 3. Vectorization and storage phase;

[0038] In this stage, the merged results are vectorized and stored.

[0039] S601: Input the merged result, including the speaker, content, start and end time information; use the large language model to vectorize the content, and store the vector in a vector database.

[0040] At the same time, other related information, including video or audio name, speaker, start and end time, and vector database index, is stored in a relational database for subsequent retrieval and analysis.

[0041] Example 2

[0042] This embodiment provides an audio and video retrieval method based on a large language model, such as Figure 1 As shown, the following steps are included:

[0043] S701: vectorizing the acquired natural language request information related to the audio and video retrieval content using a large language model tool, and using the same large language model tool as step S601 in the data generation process;

[0044] S801: Searching in the vector database constructed by the method of Example 1 by using a vector space search algorithm, and retrieving N vectors that are closest to the above information;

[0045] S802: Using the index matching the retrieved vector, search in the relational database to obtain the associated information corresponding to the vector;

[0046] S803: sorting the search results by the rearrangement large model, and sorting them according to relevance, and arranging them in descending order of relevance;

[0047] S804: Returning the reordered result to the client, the result including the text, file information, and the starting point of the text in the video or audio;

[0048] S805: The target video or target audio may be selected to view its associated information, and the target video or target audio may be played from a specified time point.

[0049] The construction of the above vector database and the relational database adopts the construction method described in Example 1.

[0050] This embodiment implements video or audio search using natural language. Natural language search based on vector algorithm is an advanced search technology that uses the ability of large language models to understand and process human language. The core of this search method is to convert natural language questions into numerical forms that can be understood by computers, namely vectors. These vectors mathematically represent the semantic content of the language, allowing the computer to find the answer that best matches the question at the semantic level. Natural language search based on vector algorithm provides a more advanced and flexible search method that can better understand the user's intentions and provide more accurate and rich search results instead of relying solely on keyword matching.

[0051] In addition, it cleverly combines speech recognition, text processing of large language models, and natural language search technology based on vector algorithms to transform video searches into more convenient text searches, thereby avoiding the complex processing required for deep recognition of video content. There is no need to rely on complex algorithms, and the large models involved are relatively small in scale. Ordinary CPU servers can meet operating requirements, reducing costs, improving retrieval speed, and making overall practicality stronger.

[0052] Example 3

[0053] This embodiment provides a video retrieval system based on a large language model, which can implement the method of embodiment 2, including:

[0054] A vectorization module, which performs vectorization processing on the natural language request information related to the video retrieval content obtained according to the large language model tool, and extracts the retrieval information of the request information;

[0055] A vector retrieval module searches the vector database according to the retrieval information and selects the vectors matching the retrieval information;

[0056] The association module searches the relational database by matching the index information of the vector to obtain the associated information corresponding to the vector, including the video name, text content and start and end time;

[0057] The sending module sends the target video corresponding to the vector and its related information to the client, and can select a specified time point from the start and end time to play the target video.

[0058] The construction of the vector database and the relational database adopts the construction method described in Example 1.

[0059] Example 4

[0060] This embodiment provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed, they are used to implement the audio and video retrieval method based on a large language model as provided in Embodiment 2.

[0061] Example 5

[0062] This embodiment provides a computer program product, including a computer program, which implements the audio and video retrieval method based on a large language model as provided in Embodiment 2 when the computer program is executed.

[0063] In the description of this embodiment, it should be noted that those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing the control device through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above-mentioned method embodiments, wherein the storage medium may be a memory, a disk, an optical disk, etc.

[0064] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

[0065] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0066] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0067] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio and video retrieval method based on a large language model, characterized in that: The following steps are involved: Obtain natural language request information related to audio and video retrieval content; Performing vectorization processing on the request information according to a large language model tool to extract search information of the request information; Using a vector space search algorithm to search in a vector database, and screening out vectors therein that match the search information; By matching the index information of the vector, searching in a relational database, obtaining the associated information corresponding to the vector, the associated information including the name, text content and start and end time; The target video or target audio corresponding to the vector and the associated information thereof are sent to the client, and a designated time point can be selected from the start and end time to play the target video or target audio.

2. The audio and video retrieval method based on a large language model according to claim 1, characterized in that: The construction process of the vector database and the relational database includes: File separation and recognition: For video files, use open source software tools to first extract audio files from them, and then parse the extracted audio files through the speech recognition model. For audio files, directly parse them through the speech recognition model; obtain file-related information, the speaker, the text content corresponding to the audio content, and the start and end time information corresponding to the text content in the audio file; preliminarily merge the recognition results and generate the start and end time information corresponding to the merged text content; Content merging and correction: perform secondary merging of the initial merged content to form merged paragraph content; use a large language model for semantic processing and delete non-substantive content; merge the generated results three times to form paragraphs of preset text length, manually correct erroneous content, and update the corresponding start and end times of the paragraphs; Vectorization and storage: using a large language model to vectorize the merged content, and storing the generated vectors in a vector database; and storing the finally merged associated information in a relational database.

3. The audio and video retrieval method based on a large language model according to claim 2, characterized in that: The secondary merger includes: A text length merging method and a time length merging method are selected according to the category of the video file or audio file. The text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to the preset time length.

4. The audio and video retrieval method based on a large language model according to claim 3, characterized in that: The secondary merger also includes: Overlapping content is added to the beginning of the paragraph formed after merging the segments, and the overlapping content is the same as the content portion of the corresponding length at the end of the previous paragraph according to a preset text length.

5. The audio and video retrieval method based on a large language model according to any one of claims 1 to 4, characterized in that: Sending the target video or target audio corresponding to the vector to the client includes: The results retrieved from the relational database are reordered according to relevance, and the results are sent to the client in the reordered order.

6. An audio and video retrieval system based on a large language model, characterized in that: include: A vectorization module, which performs vectorization processing on the natural language request information related to the audio and video retrieval content acquired according to the large language model tool, and extracts the retrieval information of the request information; A vector retrieval module searches a vector database according to the retrieval information and selects vectors matching the retrieval information; An association module, which searches in a relational database by matching the index information of the vector to obtain association information corresponding to the vector, wherein the association information includes a name, text content, and a start and end time; The sending module reorders the target video or target audio retrieved from the relational database according to the relevance, and sends the target video or target audio and its related information to the client according to the reordering result, and can select a specified time point from the start and end time to play the selected target video or target audio.

7. The audio and video retrieval system based on a large language model according to claim 6, characterized in that: The construction process of the vector database and the relational database includes: File separation and recognition: For video files, use open source software tools to first extract audio files from video files; use speech recognition models to parse audio files and obtain file-related information, including the speaker, text content corresponding to the audio content, and the start and end time information corresponding to the text content in the audio file; perform preliminary merging of recognition results and generate start and end time information corresponding to the merged text content; Content merging and correction: perform secondary merging of the initial merged content to form merged paragraph content; use a large language model for semantic processing and delete non-substantive content; merge the generated results three times to form paragraphs of preset text length, manually correct erroneous content, and update the corresponding start and end times of the paragraphs; Vectorization and storage: using a large language model to vectorize the merged content, and storing the generated vectors in a vector database; and storing the finally merged associated information in a relational database.

8. The audio and video retrieval system based on a large language model according to claim 7, characterized in that: The secondary merger includes: Selecting a text length merging method and a time length merging method according to the category of the video file or the audio file, wherein the text length merging method merges and segments the content according to the set text length, and the time length merging method merges and segments the content according to the preset time length; Overlapping content is added to the beginning of the paragraph formed after merging the segments, and the overlapping content is the same as the content portion of the corresponding length at the end of the previous paragraph according to a preset text length.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed, are used to implement the audio and video retrieval method based on a large language model according to any one of claims 1 to 5.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed, implements the audio and video retrieval method based on a large language model as claimed in any one of claims 1 to 5.

Images