A video question-answering method, system and medium based on multimodal information fusion

Abstract

The present invention discloses a video question-answering method, system and medium based on multimodal information fusion, which belongs to the field of computer vision technology and is used to solve the technical problems that the video question-answering technology relies on single modal information, cannot make full use of the multimodal information contained in the video, and the efficiency and accuracy of video retrieval are insufficient. It includes: extracting basic information of video files to obtain video summaries and text subtitles; segmenting text subtitle sentences to obtain subtitle segments, and obtaining corresponding video segments and audio segments; clustering each audio segment to obtain several voiceprint type clusters; obtaining the video lens set and subtitle set corresponding to each voiceprint type cluster, and obtaining the character label corresponding to each type of voiceprint; obtaining key frame vectors and subtitle segment vectors based on video segments and subtitle segments; storing all information in a vector database; performing intent analysis on questions input by users based on video summaries, and calling the vector database for video retrieval and question answering.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a video question-answering method, system and medium based on multimodal information fusion. Background Art

[0002] In today's information age, the widespread application and rapid growth of video content make the management and retrieval of video data particularly important. Traditional video retrieval methods mostly rely on single-modal information, such as retrieval based on the image content or audio content of the video. This single-modal information is difficult to fully and accurately describe the diversity and complexity of video content.

[0003] When processing complex video information, existing video question-answering systems usually only rely on single-modal information analyzed from the video content, and cannot fully utilize the multi-modal information contained in the video for comprehensive analysis, which limits the ability to understand and retrieve video content, resulting in insufficient retrieval efficiency and accuracy. In particular, in application scenarios such as conference videos and teaching videos that require detailed analysis of the content of a person's speech, current technologies cannot meet users' needs for accurate video retrieval and video question-answering. Summary of the invention

[0004] The embodiments of the present invention provide a video question-answering method, system and medium based on multimodal information fusion, which are used to solve the following technical problems: the current video question-answering technology relies on single modal information and cannot fully utilize the multimodal information contained in the video, resulting in insufficient efficiency and accuracy of video retrieval.

[0005] The embodiment of the present invention adopts the following technical solutions:

[0006] On the one hand, an embodiment of the present invention provides a video question-answering method based on multimodal information fusion, the method comprising: extracting basic information from an input video file to obtain a video summary and text subtitles;

[0007] Sentence segmentation is performed on the text subtitles to obtain subtitle segments, and corresponding video segments and audio segments are obtained according to the subtitle segments;

[0008] Map each audio clip into a voiceprint vector and cluster them to obtain several voiceprint type clusters;

[0009] Obtain the video shot set and subtitle set corresponding to each voiceprint type cluster, and input them into the multimodal model to obtain the character label corresponding to each type of voiceprint;

[0010] According to the video segment and the subtitle segment, a key frame vector and a subtitle segment vector are obtained;

[0011] The key frame vector, subtitle segment vector, character label, video segment, subtitle segment, audio segment and corresponding timestamp are stored in a vector database in units of segments;

[0012] Based on the video summary, the intent of the question input by the user is analyzed, and the vector database is called to perform video retrieval and question answering according to the analysis result.

[0013] In a feasible implementation, basic information is extracted from the input video file to obtain a video summary and text subtitles, specifically including:

[0014] Split the input video file into channels to obtain video channel information and audio channel information;

[0015] Summarizing the video channel information through a multimodal large model to obtain the video summary;

[0016] The audio channel information is feature extracted using a paraformer speech recognition model, and the feature is output as a text subtitle and a timestamp of each word in the text subtitle is output.

[0017] In a feasible implementation manner, the text subtitles are segmented into sentences to obtain subtitle segments, and corresponding video segments and audio segments are obtained according to the subtitle segments, which specifically includes:

[0018] Using a segmentation tool, the text subtitles are segmented by sentences using punctuation marks as segmentation points to obtain a plurality of subtitle segments;

[0019] Determine the time interval corresponding to each subtitle segment according to the timestamp of the beginning character and the timestamp of the ending character of each subtitle segment;

[0020] The video channel information and the audio channel information are segmented according to the time interval to obtain the video segments and the audio segments.

[0021] In a feasible implementation, each audio clip is mapped to a voiceprint vector and clustered to obtain several voiceprint type clusters, specifically including:

[0022] The voiceprint recognition model is used to extract features from each audio clip and map them to a high-dimensional vector space to obtain the voiceprint vector corresponding to each audio clip.

[0023] All voiceprint vectors are clustered by a clustering algorithm to obtain several voiceprint type clusters; wherein the voiceprint vectors in each voiceprint type cluster are by default the voiceprint vectors of the same speaker.

[0024] In a feasible implementation, the video shot set and subtitle set corresponding to each voiceprint type cluster are obtained and input into the multimodal model to obtain the character label corresponding to each type of voiceprint, specifically including:

[0025] The video clips corresponding to each voiceprint vector in each voiceprint type cluster are spliced ​​together to obtain a video shot set for each type of voiceprint;

[0026] The subtitle segments corresponding to each voiceprint vector in each voiceprint type cluster are spliced ​​to obtain a subtitle set for each type of voiceprint;

[0027] Through OCR recognition of the cover frames of each type of video shot set, and through the multimodal large model, the voiceprint source of each type of video shot set and subtitle set is determined;

[0028] According to the OCR recognition results and the voiceprint source judgment results, determine the person information corresponding to each type of voiceprint; wherein the person information at least includes: the person's name, the person's clothing characteristics and the voice source;

[0029] According to the character information, a character label corresponding to each type of voiceprint is generated.

[0030] In a feasible implementation manner, obtaining a key frame vector and a subtitle segment vector according to the video segment and the subtitle segment specifically includes:

[0031] Extract key frames of each video clip;

[0032] The key frame and the subtitle segment are vectorized by using the multimodal model to obtain the key frame vector and the subtitle segment vector.

[0033] In a feasible implementation manner, performing intent analysis on the question input by the user based on the video summary specifically includes:

[0034] Obtaining a question input by a user, and performing vectorization processing on the question to obtain a question vector;

[0035] Analyzing the correlation between the question vector and each stored video summary, and analyzing the character features implied in the question vector;

[0036] Perform user intention analysis based on the correlation and character characteristics to obtain intention analysis results;

[0037] The intention analysis result includes at least: pointing to a specific speaker, not pointing to a specific speaker but related to a certain video summary, and not related to all video summaries.

[0038] In a feasible implementation, calling the vector database for video retrieval and question answering according to the analysis results specifically includes:

[0039] If the intent analysis result points to a specific speaker, the character tag of the speaker is obtained, and the corresponding video clip and subtitle clip are retrieved in the vector database according to the character tag, and input into the multimodal large model for question answering;

[0040] If the intention analysis result does not specifically point to a specific speaker but is related to a certain video summary, the question vector is mixed with the key frame vector and the subtitle segment vector in the vector database and sorted, and the sorting result is input into the multimodal large model for question answering;

[0041] If the intent analysis result is irrelevant to all video summaries, the question vector is directly input into the large language model for question answering.

[0042] On the other hand, an embodiment of the present invention further provides a video question answering system based on multimodal information fusion, the system comprising:

[0043] The basic information extraction module is used to extract basic information from the input video file to obtain the video summary and text subtitles;

[0044] A video segmentation module is used to segment the text subtitles into sentences to obtain subtitle segments, and obtain corresponding video segments and audio segments according to the subtitle segments;

[0045] The voiceprint recognition module is used to map each audio clip into a voiceprint vector and cluster it to obtain several voiceprint type clusters; obtain the video shot set and subtitle set corresponding to each voiceprint type cluster, and input them into the multimodal model to obtain the character label corresponding to each type of voiceprint;

[0046] A vector storage module is used to obtain a key frame vector and a subtitle segment vector according to the video segment and the subtitle segment; and store the key frame vector, subtitle segment vector, character label, video segment, subtitle segment, audio segment and corresponding timestamp into a vector database in units of segments;

[0047] The intent recognition and question-answering module is used to perform intent analysis on questions input by users based on the video summary, and call the vector database for video retrieval and question answering according to the analysis results.

[0048] Finally, an embodiment of the present invention also provides a storage medium, which is a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores at least one program, each of which includes instructions, and when the instructions are executed by a terminal, the terminal executes the video question-and-answer method based on multimodal information fusion.

[0049] Compared with the prior art, the video question-answering method, system and medium based on multimodal information fusion provided by the embodiments of the present invention have the following beneficial effects:

[0050] The present invention realizes accurate understanding and efficient retrieval of video content by splitting audio and video information and combining multimodal models to comprehensively analyze voice, video, text and other information. The audio and video channels are split to obtain video information and audio information; a video summary is generated through a multimodal large model; the audio is converted into text subtitles using speech recognition technology; the video and audio are segmented by sentence according to the subtitle text; the voiceprint recognition model is used to process audio clips to obtain voiceprint vectors and cluster them to distinguish different speakers; the multimodal model is combined to obtain relevant information of the speaker for character tagging; video key frames are extracted and vectorized; the voice vectors, video vectors and other metadata are packaged and stored in a vector database; finally, the intention is recognized according to the questions entered by the user, and questions and answers are conducted through the multimodal large model.

[0051] This method can not only improve the accuracy of video retrieval, but also provide more detailed and accurate answers based on the content of a specific speaker when a user asks a question, effectively improving the accuracy of video retrieval and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0053] Figure 1 A flow chart of a video question-answering method based on multimodal information fusion provided by an embodiment of the present invention;

[0054] Figure 2 A flowchart of a specific implementation of a video question-answering method based on multimodal information fusion provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the structure of a video question-answering system based on multimodal information fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] The embodiment of the present invention provides a video question answering method based on multimodal information fusion, such as Figure 1 As shown, the video question answering method based on multimodal information fusion specifically includes steps S101-S106:

[0058] S101, extracting basic information from an input video file to obtain a video summary and text subtitles.

[0059] Specifically, the input video file is split into channels to obtain video channel information and audio channel information. Then the video channel information is summarized by a multimodal large model to obtain a video summary. The paraformer speech recognition model is used to extract features from the audio channel information, output as text subtitles, and output the timestamp of each word in the text subtitles.

[0060] As a feasible implementation method, Figure 2 A flowchart of a specific implementation of a video question-answering method based on multimodal information fusion provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, firstly, the built-in tool is used to split the video files of various formats into channels to obtain the video information. and audio information Then, in the summary generation module, the video information is summarized through a multimodal large model to obtain a video summary. In the audio processing module, the ASR algorithm is used to process the audio information. Perform text conversion to obtain text subtitles And the audio information is analyzed through the paraformer speech recognition model Perform feature extraction and output text subtitles The timestamp of each word in .

[0061] S102: Segment the text subtitles into sentences to obtain subtitle segments, and obtain corresponding video segments and audio segments according to the subtitle segments.

[0062] Specifically, a segmentation tool is used to segment text subtitles by sentence using punctuation marks as segmentation points to obtain a number of subtitle segments.

[0063] Furthermore, the time interval corresponding to each subtitle segment is determined according to the timestamp of the beginning character and the timestamp of the ending character of each subtitle segment. According to the time interval, the video channel information and the audio channel information are segmented respectively to obtain the video segment and the audio segment.

[0064] As a feasible implementation method, Figure 2 As shown, when getting audio information After that, the video segmentation module is based on the text subtitles , split by sentence to get subtitle segments , get the timestamp corresponding to each sentence, and use the timestamp to compare the video information and audio information Split to get video clips and audio clips The specific operation method is: use the segmentation tool to segment the subtitle text based on punctuation marks, and give the time interval corresponding to the text according to the timestamps of the beginning and end characters of each text segment, and then and audio information Split to get video clips and audio clips .

[0065] S103: Map each audio clip into a voiceprint vector and perform clustering to obtain a number of voiceprint type clusters.

[0066] Specifically, the voiceprint recognition model is used to extract features from each audio clip and map them to a high-dimensional vector space to obtain a voiceprint vector corresponding to each audio clip.

[0067] Furthermore, all voiceprint vectors are clustered by a clustering algorithm to obtain several voiceprint type clusters; wherein the voiceprint vectors in each voiceprint type cluster are by default the voiceprint vectors of the same speaker.

[0068] As a feasible implementation method, Figure 2 As shown, the ERes2Net model is used for each audio clip Perform vectorization processing to obtain the voiceprint vector set Then, the voiceprint vector set is analyzed by DBSCAN algorithm. Perform cluster analysis. Each cluster represents a type of voiceprint set. The voiceprint vectors in a type of voiceprint set are the voiceprints of the same person. The process is as follows:

[0069] according to , determine the voiceprint vector The eps neighborhood ;in, For distance Less than or equal to The set of all points of is the preset threshold. If the voiceprint vector The number of vectors in the eps neighborhood satisfies ,but is a core point; is the minimum threshold of the number of cluster vectors. Each core point determined and all points in its eps neighborhood are clustered into a voiceprint type cluster, and several voiceprint type clusters are obtained. .

[0070] S104, obtaining a video shot set and a subtitle set corresponding to each voiceprint type cluster, and inputting them into a multimodal model to obtain a character label corresponding to each type of voiceprint.

[0071] Specifically, the video clips corresponding to each voiceprint vector in each voiceprint type cluster are spliced ​​to obtain a video shot set of each type of voiceprint. The subtitle clips corresponding to each voiceprint vector in each voiceprint type cluster are spliced ​​to obtain a subtitle set of each type of voiceprint.

[0072] Furthermore, the cover frames of each type of video shot set are subjected to OCR recognition, and the voiceprint source of each type of video shot set and subtitle set is determined through a multimodal large model. Then, based on the OCR recognition results and the voiceprint source determination results, the character information corresponding to each type of voiceprint is determined; wherein the character information at least includes: the character's name, the character's clothing characteristics, and the source of the sound. Finally, based on the character information, the character label corresponding to each type of voiceprint is generated.

[0073] As a feasible implementation method, Figure 2 As shown, the sound tagging module clusters according to the voiceprint type Each voiceprint vector Corresponding video clips and subtitle clips , respectively, to obtain the video lens set and subtitle set of each type of sound, by editing each type of video The cover frame is OCR recognized and combined with the video lens set and subtitles , the source of the voiceprint is judged through a multimodal large model, and relevant information of the source (including the person’s name, clothing characteristics, etc.) is output, and a person label S is formed based on this information.

[0074] In one embodiment, the voiceprints of different teachers are clustered in the teaching video, so that the voiceprints of different teachers are separated, and then the video clips corresponding to each teacher's voiceprint are spliced, and the subtitles are also spliced, so that all the video shots and all the speaking subtitles of each teacher in the entire teaching video can be obtained. Then, the cover frame in each video shot set is OCR recognized to identify the character features of each teacher, such as facial features, clothing features, etc. The source of the sound in the video shot set is identified through a large model, such as which teacher it comes from, what the teacher's name is, and other information. Then, a character label for each teacher is formed based on this information. The same is true when the scene is changed to a conference video, and the character label of each speaker is formed according to the above process.

[0075] S105, obtaining key frame vectors and subtitle segment vectors according to the video segment and the subtitle segment, and storing the key frame vectors, subtitle segment vectors, character labels, video segments, subtitle segments, audio segments and corresponding timestamps in a vector database in units of segments.

[0076] Specifically, the key frames of each video clip are extracted, and the key frames and subtitle clips are vectorized through the multimodal model to obtain key frame vectors and subtitle clip vectors.

[0077] Furthermore, key frame vectors, subtitle segment vectors, character labels, video segments, subtitle segments, audio segments and corresponding timestamps are stored in a vector database in segments.

[0078] As a feasible implementation method, Figure 2 As shown, the vectorization module extracts each video segment using the key frame extraction model. Keyframes , use the multimodal embedding model for video key frames And subtitles Perform vectorization to obtain the key frame vector and subtitle segment vector The vector storage module then stores the key frame vectors in units of fragments. and subtitle segment vector Combine character label S and video clip , subtitle clips , Audio Clip The corresponding timestamp and other source information are stored in the vector database.

[0079] S106: Perform intent analysis on the question input by the user based on the video summary, and call the vector database for video retrieval and question answering according to the analysis result.

[0080] Specifically, the question input by the user is obtained and vectorized to obtain the question vector. Then, the correlation between the question vector and each stored video summary is analyzed, as well as the character features implied in the question vector. The user intention is analyzed based on the correlation and character features to obtain the intention analysis result.

[0081] The intent analysis result includes at least: pointing to a specific speaker, not pointing to a specific speaker but related to a certain video summary, not pointing to a specific speaker and not related to all video summaries.

[0082] Furthermore, if the intent analysis result points to a specific speaker, the character label of the speaker is obtained, and the corresponding video clips and subtitle clips are retrieved in the vector database according to the character label, and then input into the multimodal large model for question answering.

[0083] If the intent analysis result does not specifically point to a specific speaker but is related to a video summary, the question vector is mixed with the key frame vectors and subtitle segment vectors in the vector database and sorted, and the sorting result is input into the multimodal large model for question answering.

[0084] If the intent analysis result is irrelevant to all video summaries, the question vector is directly input into the large language model for question answering.

[0085] As a feasible implementation method, based on the user's questions, the multimodal big model is used to analyze the intent and divide the questions into three categories. If the question only involves the opinion of a certain person, the video clip and subtitles corresponding to the person are extracted and input into the multimodal big model for question and answer; if the question is not related to the summary, it is directly input into the big model for question and answer; if none of the above situations occurs, the question is vectorized and the key frame vector in the vector database is used. and subtitle segment vector Calculate the mixed similarity and then sort. Return the sorted results, and input the highest-ranked video clips and subtitles into the multimodal large model for question answering.

[0086] In addition, the embodiment of the present invention also provides a video question-answering system based on multimodal information fusion, such as Figure 3 As shown, the video question answering system 300 based on multimodal information fusion specifically includes:

[0087] A basic information extraction module 310 is used to extract basic information from an input video file to obtain a video summary and text subtitles;

[0088] The video segmentation module 320 is used to segment the text subtitles into sentences to obtain subtitle segments, and obtain corresponding video segments and audio segments according to the subtitle segments;

[0089] The voiceprint recognition module 330 is used to map each audio clip into a voiceprint vector and cluster it to obtain a number of voiceprint type clusters; obtain the video shot set and subtitle set corresponding to each voiceprint type cluster, and input them into the multimodal model to obtain the character label corresponding to each type of voiceprint;

[0090] The vector storage module 340 is used to obtain a key frame vector and a subtitle segment vector according to the video segment and the subtitle segment; and store the key frame vector, subtitle segment vector, character label, video segment, subtitle segment, audio segment and corresponding timestamp into a vector database in units of segments;

[0091] The intention recognition and question-answering module 350 is used to perform intention analysis on the questions input by the user based on the video summary, and call the vector database for video retrieval and question answering according to the analysis results.

[0092] Finally, an embodiment of the present invention further provides a storage medium, which is a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores at least one program, each of which includes instructions, and when the instructions are executed by a terminal, the terminal executes:

[0093] Extract basic information from the input video file to obtain video summary and text subtitles;

[0094] Sentence segmentation is performed on the text subtitles to obtain subtitle segments, and corresponding video segments and audio segments are obtained according to the subtitle segments;

[0095] Map each audio clip into a voiceprint vector and cluster them to obtain several voiceprint type clusters;

[0096] Obtain the video shot set and subtitle set corresponding to each voiceprint type cluster, and input them into the multimodal model to obtain the character label corresponding to each type of voiceprint;

[0097] According to the video segment and the subtitle segment, a key frame vector and a subtitle segment vector are obtained;

[0098] The key frame vector, subtitle segment vector, character label, video segment, subtitle segment, audio segment and corresponding timestamp are stored in a vector database in units of segments;

[0099] Based on the video summary, the intent of the question input by the user is analyzed, and the vector database is called to perform video retrieval and question answering according to the analysis result.

[0100] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0101] The above describes specific embodiments of the present invention. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. A video question answering method based on multimodal information fusion, characterized in that: The method comprises: Extract basic information from the input video file to obtain video summary and text subtitles; Sentence segmentation is performed on the text subtitles to obtain subtitle segments, and corresponding video segments and audio segments are obtained according to the subtitle segments; Map each audio clip into a voiceprint vector and cluster them to obtain several voiceprint type clusters; Obtain the video shot set and subtitle set corresponding to each voiceprint type cluster, and input them into the multimodal model to obtain the character label corresponding to each type of voiceprint; According to the video segment and the subtitle segment, a key frame vector and a subtitle segment vector are obtained; The key frame vector, subtitle segment vector, character label, video segment, subtitle segment, audio segment and corresponding timestamp are stored in a vector database in units of segments; Based on the video summary, the intent of the question input by the user is analyzed, and according to the analysis result, the vector database is called to perform video retrieval and question answering, specifically including: Obtaining a question input by a user, and performing vectorization processing on the question to obtain a question vector; Analyzing the correlation between the question vector and each stored video summary, and analyzing the character features implied in the question vector; Perform user intention analysis based on the correlation and character characteristics to obtain intention analysis results; The intention analysis result at least includes: pointing to a specific speaker, not pointing to a specific speaker but related to a certain video summary, and not related to all video summaries; If the intent analysis result points to a specific speaker, the speaker's character tag is obtained, and the corresponding video clip and subtitle clip are retrieved in the vector database according to the character tag, and input into the multimodal large model for question answering; If the intent analysis result does not specifically point to a specific speaker but is related to a video summary, the question vector is mixed with the key frame vector and the subtitle segment vector in the vector database and sorted, and the sorting result is input into the multimodal large model for question answering; If the intent analysis result is irrelevant to all video summaries, the question vector is directly input into the large language model for question answering.

2. The video question answering method based on multimodal information fusion according to claim 1, characterized in that: Extract basic information from the input video file to obtain video summary and text subtitles, including: Split the input video file into channels to obtain video channel information and audio channel information; Summarizing the video channel information through a multimodal large model to obtain the video summary; The audio channel information is subjected to feature extraction using a paraformer speech recognition model, and the feature is output as a text subtitle and a timestamp of each word in the text subtitle is output.

3. The video question answering method based on multimodal information fusion according to claim 2, characterized in that: Sentence segmentation is performed on the text subtitles to obtain subtitle segments, and corresponding video segments and audio segments are obtained according to the subtitle segments, specifically including: Using a segmentation tool, the text subtitles are segmented by sentences using punctuation marks as segmentation points to obtain a plurality of subtitle segments; Determine the time interval corresponding to each subtitle segment according to the timestamp of the beginning character and the timestamp of the ending character of each subtitle segment; The video channel information and the audio channel information are segmented according to the time interval to obtain the video segments and the audio segments.

4. The video question answering method based on multimodal information fusion according to claim 1, characterized in that: Each audio clip is mapped to a voiceprint vector and clustered to obtain several voiceprint type clusters, including: The voiceprint recognition model is used to extract features from each audio clip and map them to a high-dimensional vector space to obtain the voiceprint vector corresponding to each audio clip. All voiceprint vectors are clustered by a clustering algorithm to obtain several voiceprint type clusters; wherein the voiceprint vectors in each voiceprint type cluster are by default the voiceprint vectors of the same speaker.

5. The video question answering method based on multimodal information fusion according to claim 1, characterized in that: Get the video shot set and subtitle set corresponding to each voiceprint type cluster, and input them into the multimodal model to get the character label corresponding to each type of voiceprint, including: The video clips corresponding to each voiceprint vector in each voiceprint type cluster are spliced ​​together to obtain a video shot set for each type of voiceprint; The subtitle segments corresponding to each voiceprint vector in each voiceprint type cluster are spliced ​​to obtain a subtitle set for each type of voiceprint; Through OCR recognition of the cover frames of each type of video shot set, and through the multimodal large model, the voiceprint source of each type of video shot set and subtitle set is determined; According to the OCR recognition results and the voiceprint source judgment results, determine the person information corresponding to each type of voiceprint; wherein the person information at least includes: the person's name, the person's clothing characteristics and the voice source; According to the character information, a character label corresponding to each type of voiceprint is generated.

6. The video question answering method based on multimodal information fusion according to claim 1, characterized in that: According to the video segment and the subtitle segment, a key frame vector and a subtitle segment vector are obtained, specifically including: Extract key frames of each video clip; The key frame and the subtitle segment are vectorized by using the multimodal model to obtain the key frame vector and the subtitle segment vector.

7. A video question-answering system based on multimodal information fusion, characterized in that: The system comprises: The basic information extraction module is used to extract basic information from the input video file to obtain the video summary and text subtitles; A video segmentation module is used to segment the text subtitles into sentences to obtain subtitle segments, and obtain corresponding video segments and audio segments according to the subtitle segments; The voiceprint recognition module is used to map each audio clip into a voiceprint vector and cluster it to obtain several voiceprint type clusters; obtain the video shot set and subtitle set corresponding to each voiceprint type cluster, and input them into the multimodal model to obtain the character label corresponding to each type of voiceprint; A vector storage module is used to obtain a key frame vector and a subtitle segment vector according to the video segment and the subtitle segment; and store the key frame vector, subtitle segment vector, character label, video segment, subtitle segment, audio segment and corresponding timestamp into a vector database in units of segments; The intent recognition and question-answering module is used to perform intent analysis on the questions input by the user based on the video summary, and call the vector database for video retrieval and question answering according to the analysis results, specifically including: Obtaining a question input by a user, and performing vectorization processing on the question to obtain a question vector; Analyzing the correlation between the question vector and each stored video summary, and analyzing the character features implied in the question vector; Perform user intention analysis based on the correlation and character characteristics to obtain intention analysis results; The intention analysis result at least includes: pointing to a specific speaker, not pointing to a specific speaker but related to a certain video summary, and not related to all video summaries; If the intent analysis result points to a specific speaker, the character tag of the speaker is obtained, and the corresponding video clip and subtitle clip are retrieved in the vector database according to the character tag, and input into the multimodal large model for question answering; If the intention analysis result does not specifically point to a specific speaker but is related to a certain video summary, the question vector is mixed with the key frame vector and the subtitle segment vector in the vector database and sorted, and the sorting result is input into the multimodal large model for question answering; If the intent analysis result is irrelevant to all video summaries, the question vector is directly input into the large language model for question answering.

8. A storage medium, characterized in that: The storage medium is a non-volatile computer-readable storage medium, and the non-volatile computer-readable storage medium stores at least one program, each of which includes instructions, and when the instructions are executed by the terminal, the terminal executes a video question-answering method based on multimodal information fusion according to any one of claims 1-6.

Images