Video question answering system and method based on voice recognition and video picture selection

Abstract

The invention discloses a video question answering system and method based on voice recognition and video picture selection. The system comprises an audio data acquisition unit, a voice recognition unit, an intelligent video picture selection unit, an information fusion unit and a question answering unit. The voice recognition unit and the intelligent video picture selection unit are arranged to recognize and analyze the voice and video data acquired by the audio data acquisition unit in real time, so that high-quality text data and video picture data can be obtained, and redundant pictures can be effectively reduced; the information fusion unit carries out information fusion on the text data and the video picture data, the synchronism and consistency of text content and video content can be effectively improved, finally, the question and answer unit is set to combine the fused text and visual information, questions or prompts sent by a user are reasonably analyzed, and a final question and answer result is generated. The voice and video picture information integration capability of the system is effectively improved, and the overall accuracy and efficiency of the question-answering system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing and natural language processing, and in particular to a video question-answering system and method based on speech recognition and video picture selection. Background Art

[0002] There are multiple modalities of information in a video, such as scenes, objects, audio, etc. These key elements can largely reflect the visual information content of the video. In addition, the natural language description of the video can also reflect the main content of the video to a certain extent. The existing pre-trained models can extract these elements from different perspectives.

[0003] In response to the object-level elements that appear in the video, Redmon et al. proposed the YOLO object detection and framework. This framework can identify the objects that appear in the image and locate them, thereby effectively extracting the target elements.

[0004] The question-and-answer system is one of many language or video processing systems. In enterprises and institutions, the question-and-answer system can be used as an intelligent customer service to provide users with fast and accurate answers. The question-and-answer system can also be used for knowledge management in enterprises and institutions to help employees quickly acquire the knowledge they need. The question-and-answer system can also be integrated into intelligent assistants (such as smartphone assistants, smart speakers, etc.) to provide users with convenient services. Users can interact with intelligent assistants through voice or text, ask various questions, and obtain the information and services they need.

[0005] At present, there are existing question-answering systems based on speech recognition: this system only relies on the speech in the video for recognition, and converts the speech content into text for processing. However, this system cannot use the visual information in the video to answer the user's questions, so it is not effective when dealing with questions containing rich visual information, or even cannot be used at all; in addition, words with similar pronunciations are also common problems of misrecognition by the speech recognition system, which will cause serious deviations in the understanding of the content of the video; in addition, video content that only contains video images and no audio cannot be interpreted. The prior art also proposes a question-answering system based on video images: this system mainly analyzes the video images to answer questions; the commonly used methods in the current technology include directly extracting image information from the video, extracting 12 frames or 24 frames at equal intervals, and limiting the input length of the video. However, long videos cannot be interpreted, and they are usually cut into 1-minute videos before being input into the algorithm for interpretation, which directly leads to the loss of picture information. Moreover, without considering the content of the picture, the video frames are directly extracted at equal intervals, which will produce a large number of redundant pictures, resulting in low processing efficiency and less useful information. In addition, there is a lack of voice information, which makes it impossible to interpret the core information of videos such as interviews whose main information is in the voice.

[0006] In summary, the existing question-answering system still has the technical problem of being unable to effectively integrate voice and video information, and thus unable to efficiently process long video content, resulting in low overall accuracy and efficiency of the question-answering system. Summary of the invention

[0007] In order to overcome the technical problem that the existing question-answering system cannot effectively integrate voice and video picture information, and thus cannot efficiently process long video content, resulting in low overall accuracy and efficiency of the question-answering system, the present invention proposes a video question-answering system and method based on voice recognition and video picture selection, which can effectively integrate voice and video picture information, thereby efficiently processing long video content and improving the overall accuracy and efficiency of the question-answering system.

[0008] The purpose of the present invention is achieved by the following technical solutions:

[0009] A video question-answering system based on speech recognition and video picture selection, the system comprising: an audio data acquisition unit, a speech recognition unit, an intelligent video picture selection unit, an information fusion unit and a question-answering unit;

[0010] The audio data acquisition unit is used to acquire voice and video data;

[0011] The speech recognition unit is used to perform real-time speech recognition on the speech and video data and generate corresponding text data;

[0012] The intelligent video picture selection unit is used to analyze the voice and video data, extract key frames in the video content, and generate video picture data based on the extracted key frames;

[0013] The information fusion unit is used to fuse the generated text data with the video picture data to obtain combined text and visual information;

[0014] The question-answering unit is used to obtain the user's question information, and analyze the question information in combination with the information fusion result to generate the final question-answering result.

[0015] In the above technical scheme, by setting up a voice recognition unit and an intelligent video screen selection unit to perform real-time recognition and analysis of voice and video data, high-quality text data and video screen data can be obtained, and the generation of redundant screens can be effectively reduced; an information fusion unit is set to fuse text data and video screen data to obtain combined text and visual information, which can effectively improve the synchronization and consistency of text content and video content; finally, a question-and-answer unit is set to obtain questions or prompts issued by users, and the questions or prompts issued by users are reasonably analyzed in combination with the fused text and visual information, and the final question-and-answer results are generated, which effectively improves the system's ability to integrate voice and video screen information, thereby efficiently processing long video content and improving the overall accuracy and efficiency of the question-and-answer system.

[0016] Preferably, the speech recognition unit includes an audio encoding module;

[0017] The audio encoding module performs speech-to-text processing on the audio data in the voice and video data, generates text data with a timestamp, and performs text segmentation on the text data.

[0018] Preferably, the audio encoding module sets a deep learning model to learn the audio data and time features after speech-to-text processing, and obtains a speech recognition model that masters speech-to-text processing and time alignment;

[0019] The speech recognition model aligns text with time and generates text data with timestamps.

[0020] In the above technical solution, a deep learning model is set up to learn the audio data and time features after speech-to-text processing, so as to obtain a speech recognition model that can master speech-to-text processing and time alignment, and then use this model to align the text with time to generate text data with timestamps, thereby achieving synchronization of video and audio content, and effectively improving the accuracy and reliability of speech recognition.

[0021] Preferably, the intelligent video picture selection unit includes a split-screen detection module and a first video encoding module;

[0022] The split-screen detection module extracts the video frame by frame from the voice and video data to extract pixels and deep learning features, and predicts split-screen boundaries based on the extracted pixels and deep learning features;

[0023] The video encoding module allocates video key frames according to the split-screen detection result, and extracts semantic information of the video picture content according to the allocated video key frames.

[0024] Preferably, the split-screen detection module includes a classification model, and the split-screen detection module uses a sliding window technology to extract and scale a plurality of frames of video from the voice and video data each time, so as to extract pixels and deep learning features of the video, and splice the pixels and deep learning features;

[0025] The classification model predicts the frame boundaries of the video based on the spliced ​​pixels and deep learning features;

[0026] Wherein, the classification model is a neural network model.

[0027] Preferably, the video encoding module uses a picture encoder to extract features of the video key frames according to the assigned video key frames, and uses a multi-layer perceptron MLP to align the extracted video key frame features with text data to generate a picture with semantic information.

[0028] Preferably, the process of extracting pixel features of the video by the storyboard detection module includes extracting histogram statistical information of the video image and the grayscale value of the video image after grayscale conversion, and flattening the video image into a long vector form to have a holistic and local representation of the underlying image information.

[0029] Preferably, the process of extracting the histogram statistical information of the video image includes counting the total frequency of pixel values ​​appearing in a specified interval, expressed as:

[0030] H(r k )=n k ,k=0,1,2,...,N;

[0031] The grayscale value obtained after the video image is grayed is normalized and expanded into a one-dimensional vector. The expression is:

[0032]

[0033] Among them, r k Represents the interval of pixel values, n k represents the frequency, and k represents the kth interval.

[0034] In the above technical solution, the storyboard detection module can effectively detect the storyboard switching and provide the storyboard timestamp and duration in the process of extracting the voice and video data frame by frame, so as to ensure that each scene is fully represented; the video encoding module can reasonably allocate key frames in the process of allocating video key frames according to the storyboard detection results, so as to improve the video encoding efficiency and capture representative pictures; the set classification model can effectively improve the accuracy and reliability of the system in predicting the storyboard boundaries of the video, so as to accurately extract and analyze the video content, especially in complex video processing, to ensure that each scene is fully understood.

[0035] Preferably, the question-answering unit includes a text encoding module and a large language model;

[0036] The text editing module performs text segmentation on the instructions input by the user and ensures alignment with the video and audio content;

[0037] The large language model performs multimodal information processing on the pictures with semantic information generated by the video encoding module, the text data with timestamps generated by the audio encoding module, and the user instructions after the text segmentation by the text editing module to generate the final question-answering result;

[0038] The large language model is a Llama 3 model obtained after training.

[0039] In the above technical solution, the set text editing module can make corresponding operations according to the user's instructions, ensure alignment with the video and audio content, and generate the required output; the set large language model can effectively combine the three modal inputs of video, audio and text, so as to fully and efficiently understand the video content; it is convenient for the system to understand the key scenes and instruction interactions in the video, and generate a detailed summary with a timestamp, thereby improving the performance of the system in scenarios where it is necessary to combine the video screen and voice content at the same time to understand the video content.

[0040] A video question-answering method based on speech recognition and video picture selection, the method comprising the following steps:

[0041] Get audio and video data;

[0042] Performing real-time speech recognition on the speech and video data and generating corresponding text data;

[0043] Analyzing the voice and video data, extracting key frames in the video content, and generating video picture data based on the extracted key frames;

[0044] Fusing the generated text data with the video image data;

[0045] Obtain the user's question information, analyze the question information in combination with the information fusion results, and generate the final question-answering results.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention proposes a video question-and-answer system and method based on speech recognition and video frame selection. First, by setting a speech recognition unit and an intelligent video frame selection unit to perform real-time recognition and analysis on the acquired speech and video data, high-quality text data and video frame data can be obtained, and the generation of redundant frames can be effectively reduced; then an information fusion unit is set to fuse the text data and the video frame data to obtain combined text and visual information, which can effectively improve the synchronization and consistency of the text content and the video content; finally, a question-and-answer unit is set to obtain questions or prompts issued by the user, and the questions or prompts issued by the user are reasonably analyzed in combination with the fused text and visual information, and a final question-and-answer result is generated, which effectively improves the system's ability to integrate speech and video frame information, thereby efficiently processing long video content and improving the overall accuracy and efficiency of the question-and-answer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the structure of a video question-answering system based on speech recognition and video picture selection provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the video storyboard technology provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the system function implementation provided by the embodiment of the present application;

[0051] Figure 4 A flowchart of the steps of a video question-answering method based on speech recognition and video frame selection provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0054] Embodiment 1:

[0055] This embodiment provides a video question-answering system based on speech recognition and video frame selection. Figure 1 , the system includes: an audio data acquisition unit, a speech recognition unit, an intelligent video picture selection unit, an information fusion unit and a question-answering unit;

[0056] The audio data acquisition unit is used to acquire voice and video data;

[0057] The speech recognition unit is used to perform real-time speech recognition on the speech and video data and generate corresponding text data;

[0058] The intelligent video picture selection unit is used to analyze the voice and video data, extract key frames in the video content, and generate video picture data based on the extracted key frames;

[0059] The information fusion unit is used to fuse the generated text data with the video picture data to obtain combined text and visual information;

[0060] The question-answering unit is used to obtain the user's question information, and analyze the question information in combination with the information fusion result to generate the final question-answering result.

[0061] In this embodiment, by setting up a voice recognition unit and an intelligent video screen selection unit to perform real-time recognition and analysis of voice and video data, high-quality text data and video screen data can be obtained, and the generation of redundant screens can be effectively reduced; an information fusion unit is set to fuse text data and video screen data to obtain combined text and visual information, which can effectively improve the synchronization and consistency of text content and video content; finally, a question-and-answer unit is set to obtain questions or prompts issued by users, and the questions or prompts issued by users are reasonably analyzed in combination with the fused text and visual information, and the final question-and-answer results are generated, which effectively improves the system's ability to integrate voice and video screen information, thereby efficiently processing long video content and improving the overall accuracy and efficiency of the question-and-answer system.

[0062] Embodiment 2:

[0063] This embodiment is based on the first embodiment, and further describes the components of each unit in the first embodiment, as follows:

[0064] As a preferred embodiment, the speech recognition unit includes an audio encoding module;

[0065] The audio encoding module performs speech-to-text processing on the audio data in the voice and video data, generates text data with a timestamp, and performs text segmentation on the text data.

[0066] As a preferred embodiment, the audio encoding module sets a deep learning model to learn the audio data and time features after speech-to-text processing, and obtains a speech recognition model that masters speech-to-text processing and time alignment;

[0067] The speech recognition model Paraformer (ASR model) aligns text with time and generates text data with timestamps.

[0068] More specifically, the audio encoding module performs ASR (speech-to-text) processing on the audio in the video, uses the speech recognition model Paraformer (ASR model) to generate text with timestamps, and then inputs the text into the large language model after text segmentation through Text Tokenizer (text segmentation).

[0069] It can be understood that a deep learning model is set up to learn the audio data and time features after speech-to-text processing, so as to obtain a speech recognition model that can master speech-to-text processing and time alignment, and then use this model to align the text with time, generate text data with timestamps, and achieve synchronization of video and audio content, effectively improving the accuracy and reliability of speech recognition.

[0070] As a preferred embodiment, the intelligent video picture selection unit includes a split-screen detection module and a first video encoding module;

[0071] The split-screen detection module extracts the video frame by frame from the voice and video data to extract pixels and deep learning features, and predicts split-screen boundaries based on the extracted pixels and deep learning features;

[0072] The video encoding module allocates video key frames according to the split-screen detection result, and extracts semantic information of the video picture content according to the allocated video key frames.

[0073] As a preferred embodiment, the split-screen detection module includes a classification model, and the split-screen detection module uses a sliding window technology to extract and scale a plurality of frames of video from the voice and video data each time to extract pixels and deep learning features of the video, and splice the pixels and deep learning features;

[0074] The classification model predicts the frame boundaries of the video based on the spliced ​​pixels and deep learning features;

[0075] Wherein, the classification model is a neural network model.

[0076] As a preferred embodiment, the video encoding module uses a picture encoder to extract features of the video key frames according to the assigned video key frames, and uses a multi-layer perceptron MLP to align the extracted video key frame features with text data to generate a picture with semantic information.

[0077] More specifically,

[0078] For example, see Figure 2 ,Frame extraction and feature processing: The storyboard detection module extracts frame by frame from the video, and slides 50 frames of video from all video frames along the time axis each time, and scales the 50-frame video frame sequence to a screen size of 48x27, and extracts features from the scaled image. The feature extraction scheme uses a scheme that simultaneously extracts pixel features (RGB) and deep learning features, and splices the two features together and sends them to the classification model to predict the boundaries of the storyboard.

[0079] Frame detection and keyframe extraction: Based on the results of frame detection, the timestamp information of the frames is output. Keyframes are allocated reasonably according to the number and duration of frames. For example, in a sample video, the duration of frame 1 is 0-10 seconds, and the duration of frame 2 is 10-57 seconds. Therefore, among the 9 keyframes, 2 frames are allocated to frame 1 and 7 frames are allocated to frame 2, and the frames of each frame are evenly extracted.

[0080] As a preferred embodiment, the process of extracting pixel features of the video by the storyboard detection module includes extracting histogram statistical information of the video image and the grayscale value of the video image after grayscale conversion, and flattening the video image into a long vector form to have a holistic and local representation method for the underlying image information.

[0081] Specifically, the process of extracting the histogram statistics of the video image includes counting the total frequency of pixel values ​​appearing in a specified interval, and the expression is:

[0082] H(r k )=n k ,k=0,1,2,...,N;

[0083] In order to make the result more general, normalization is performed here, dividing the histogram statistics by the total number of pixels. The expression is:

[0084]

[0085] The grayscale processing of the video image is expressed as:

[0086] Gray=R×0.299+G×0.587+B×0.114;

[0087] Similarly, the grayscale values ​​obtained after grayscale conversion need to be normalized and expanded into a one-dimensional vector. The expression is:

[0088]

[0089] Expand into a one-dimensional vector operation, reshape the scaled 48x27 image into a one-dimensional vector of 1x1296;

[0090] Among them, r k Represents the interval of pixel values, n k represents the frequency, and k represents the kth interval.

[0091] It can be understood that the storyboard detection module can effectively detect the storyboard switching, provide the storyboard timestamp and duration, ensure that each scene is fully represented, reduce redundant information, and improve the efficiency of information processing during the process of extracting the voice and video data frame by frame; the video encoding module can reasonably allocate key frames, improve the video encoding efficiency, and capture representative pictures during the process of allocating video key frames according to the storyboard detection results; the set classification model can effectively improve the accuracy and reliability of the system in predicting the storyboard boundaries of the video, so as to accurately extract and analyze the video content, especially in complex video processing, to ensure that each scene is fully understood.

[0092] As a preferred embodiment, the question-answering unit includes a text encoding module and a large language model;

[0093] The text editing module performs text segmentation on the instructions input by the user and ensures alignment with the video and audio content;

[0094] The large language model performs multimodal information processing on the pictures with semantic information generated by the video encoding module, the text data with timestamps generated by the audio encoding module, and the user instructions after the text segmentation by the text editing module to generate the final question-answering result;

[0095] The large language model is a Llama 3 model obtained after training.

[0096] For example, see Figure 3 , in the video encoding module:

[0097] The video encoding module receives video input, extracts key frames of the video (9 frames are extracted by default), and inputs them into the picture encoder to extract the features of the picture sequence. Then, the picture features are aligned with the text through the MLP (multi-layer perceptron) and then input into the large model. At this time, the video screen has been converted into an input containing a large amount of semantic information. Figure 3 As shown in the figure, taking the sample input as an example, the video encoding module extracted that the video contains a yellow Shiba Inu, and the owner is interacting with the Shiba Inu. Under some instructions from the owner, the Shiba Inu responds. At first, the Shiba Inu did not understand the meaning of the actions. After a series of teaching and food rewards, the Shiba Inu slowly understood the owner's instructions.

[0098] In the audio encoding module:

[0099] The audio encoding module receives audio input, performs ASR processing (speech-to-text recognition) on the audio input to convert it into text output with timestamp information, and then passes through the Text Tokenizer (text segmentation) and inputs it into the large model. Figure 3 As shown, taking the sample input as an example, the output of ASR is as follows:

[0100] [00:00:01.00-00:00:03.00]Sit down, sit down, um, good boy, claw the ground, claw the ground, claw the floor tiles.

[0101] [00:00:03.00-00:00:06.00]Hurry up and sit down. You sit down first. You sit down first. Hey, you sit down first. Sit down first.

[0102] [00:00:06.00-00:00:10.00]Hey, baby, baby, please sit down first. Come here, sit down.

[0103] [00:00:40.00-00:00:43.00]Good job, come on, sit down, sit down, sit down, sit down, and I'll give it to you.

[0104] [00:00:43.00-00:00:46.00]Sit down, sit down, sit down, sit down, sit down. Sit down nicely.

[0105] [00:00:46.00-00:00:48.00]Let’s sit in another corner.

[0106] [00:00:48.00-00:00:51.00]Okay, sit down nicely.

[0107] [00:00:51.00-00:00:54.00]Okay, good boy, you are so good, so good.

[0108] [00:00:54.00-00:00:57.00]Sit down, oh good dog, do it, oh, good dog, be good.

[0109] In the text encoding module:

[0110] The text encoding module receives text input and inputs it into the large model after passing through the Text Tokenizer. Figure 3 As shown, taking the sample input as an example, the input question prompt (instruction): summarize the video content according to the timeline.

[0111] In the large language model:

[0112] The large language model combines the outputs of the video encoding module, the audio encoding module, and the text encoding module. The specific operation is to add a specific token identifier to the output of each module. Used to distinguish different frames, added before and after the output of the audio encoding module <audio> and< / audio> The mark indicates the start and end position of the audio, and the text encoding module output adds <query>< / query> The markers indicate the starting and ending positions of the question prompt and are input into the large language model.

[0113] Description of other modules: Image Encoder uses ViT; MLP uses two layers for image and text alignment; ASR uses paraformer for speech recognition; large language model uses Llama 3. Fine-tuned modules: MLP and Llama 3.

[0114] In this embodiment, the set text editing module can make corresponding operations according to the user's instructions, ensure alignment with the video and audio content, and generate the required output; the set large language model can effectively combine the three modal inputs of video, audio and text, so as to fully and efficiently understand the video content; it is convenient for the system to understand the key scenes and instruction interactions in the video, and generate a detailed summary with a timestamp, so as to improve the performance of the system in scenarios where it is necessary to combine the video screen and voice content at the same time to understand the video content, such as dog training scenarios.

[0115] Embodiment three:

[0116] This embodiment provides a video question-answering method based on speech recognition and video frame selection. Figure 4 , the method comprises the following steps:

[0117] Step S1: Acquire voice and video data;

[0118] Step S2: performing real-time speech recognition on the speech and video data, and generating corresponding text data;

[0119] Step S3: Analyze the voice and video data, extract key frames in the video content, and generate video picture data based on the extracted key frames;

[0120] Step S4: fusing the generated text data with the video image data;

[0121] Step S5: Obtain the user's question information, and analyze the question information in combination with the information fusion result to generate the final question-answering result.

[0122] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video question-answering system based on speech recognition and video screen selection, characterized in that: The system includes: an audio data acquisition unit, a speech recognition unit, an intelligent video picture selection unit, an information fusion unit and a question-answering unit; The audio data acquisition unit is used to acquire voice and video data; The speech recognition unit is used to perform real-time speech recognition on the speech and video data and generate corresponding text data; The intelligent video picture selection unit is used to analyze the voice and video data, extract key frames in the video content, and generate video picture data based on the extracted key frames; The information fusion unit is used to fuse the generated text data with the video picture data to obtain combined text and visual information; The question-answering unit is used to obtain the user's question information, and analyze the question information in combination with the information fusion result to generate the final question-answering result.

2. The video question-answering system based on speech recognition and video picture selection according to claim 1, characterized in that: The speech recognition unit includes an audio encoding module; The audio encoding module performs speech-to-text processing on the audio data in the voice and video data, generates text data with a timestamp, and performs text segmentation on the text data.

3. The video question-answering system based on speech recognition and video picture selection according to claim 2 is characterized in that: The audio encoding module sets a deep learning model to learn the audio data and time features after speech-to-text processing, and obtains a speech recognition model that masters speech-to-text processing and time alignment; The speech recognition model aligns text with time and generates text data with timestamps.

4. The video question-answering system based on speech recognition and video picture selection according to claim 1, characterized in that: The intelligent video picture selection unit includes a split-screen detection module and a first video encoding module; The split-screen detection module extracts the video frame by frame from the voice and video data to extract pixels and deep learning features, and predicts split-screen boundaries based on the extracted pixels and deep learning features; The video encoding module allocates video key frames according to the split-screen detection result, and extracts semantic information of the video picture content according to the allocated video key frames.

5. The video question-answering system based on speech recognition and video picture selection according to claim 4 is characterized in that: The split-screen detection module includes a classification model, and the split-screen detection module uses a sliding window technology to extract a plurality of frames of video from the voice and video data each time and scale them to extract pixels and deep learning features of the video, and splice the pixels and deep learning features; The classification model predicts the frame boundaries of the video based on the spliced ​​pixels and deep learning features; Wherein, the classification model is a neural network model.

6. The video question-answering system based on speech recognition and video picture selection according to claim 4, characterized in that: The video encoding module uses a picture encoder to extract features of the video key frames according to the assigned video key frames, and uses a multi-layer perceptron MLP to align the extracted video key frame features with text data to generate a picture with semantic information.

7. The video question-answering system based on speech recognition and video picture selection according to claim 5, characterized in that: The process of extracting pixel features of the video by the split-screen detection module includes extracting the histogram statistical information of the video image and the grayscale value of the video image after grayscale conversion, and flattening the video image into a long vector form to have a holistic and local representation method for the underlying image information.

8. The video question-answering system based on speech recognition and video frame selection according to claim 7, characterized in that: The process of extracting the histogram statistics of a video image includes counting the total frequency of pixel values ​​appearing in a specified interval, expressed as: H(r k )=n k ,k=0,1,2,...,N; The grayscale value obtained after the video image is grayed is normalized and expanded into a one-dimensional vector. The expression is: Among them, r k Represents the interval of pixel values, n k represents the frequency, and k represents the kth interval.

9. The video question-answering system based on speech recognition and video picture selection according to any one of claims 1 to 8, characterized in that: The question-answering unit includes a text encoding module and a large language model; The text editing module performs text segmentation on the instructions input by the user and ensures alignment with the video and audio content; The large language model performs multimodal information processing on the pictures with semantic information generated by the video encoding module, the text data with timestamps generated by the audio encoding module, and the user instructions after the text segmentation by the text editing module to generate the final question-answering result; The large language model is a Llama 3 model obtained after training.

10. A video question-answering method based on speech recognition and video picture selection, characterized in that: The method comprises the following steps: Get audio and video data; Performing real-time speech recognition on the speech and video data and generating corresponding text data; Analyzing the voice and video data, extracting key frames in the video content, and generating video picture data based on the extracted key frames; The generated text data is fused with the video image data to obtain combined text and visual information; Obtain the user's question information, analyze the question information in combination with the information fusion results, and generate the final question-answering results.

Images