Video intelligent question-answering method and system based on incremental video retrieval enhancement generation

Abstract

The invention discloses a video intelligent question-answering method and system based on incremental video retrieval enhancement generation, and the method comprises the steps: segmenting a video, generating text description, storing the text description in a text vector database, and combining target detection with a multi-scale attention mechanism to extract video features; when a user queries, preferentially generating an answer based on text retrieval and by utilizing a large language model; if the answer does not meet the condition, an incremental retrieval process is started, related video features are retrieved from the video vector database, a description text is generated to update the database, and the question and answer ability is dynamically enhanced. The system comprises a video preprocessing module, a target attention enhancement video coding module, a query processing and question and answer module and an incremental video retrieval enhancement generation module. The method improves the response speed and accuracy of video questions and answers, and is suitable for instant analysis scenes such as intelligent security and protection, traffic monitoring and the like.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a video intelligent question answering method and system based on incremental video retrieval enhancement generation. Background Technology

[0002] With economic development, surveillance cameras are being deployed extensively in various indoor and outdoor locations, resulting in a massive increase in the amount of surveillance video. Consequently, the demand for video retrieval and intelligent querying is growing daily. Surveillance video retrieval and querying typically focus on the behavior recognition and detection of target objects such as pedestrians and vehicles appearing in the video.

[0003] Traditional video retrieval question answering systems segment videos and encode them into vectors stored in a vector database. For user-input queries, a video-text retrieval model matches the most semantically relevant video segment in the vector database. However, this process involves video feature extraction and the matching of video and text features, both of which utilize video understanding models with large parameter sets, limiting the system's application in real-time analysis scenarios. Therefore, improving the efficiency and reducing the retrieval cost of video retrieval question answering systems is one of the current challenges.

[0004] For video understanding models, traditional methods apply Transformer self-attention to the video domain. Videos contain both temporal and spatial information; as video resolution increases, the time complexity of traditional Transformer encoders increases threefold, severely impacting the encoding and retrieval of high-resolution videos. Furthermore, Transformers cannot directly extract the locations of pedestrians, vehicles, and other targets in the video. This leads to the influence of background image features on target behavior recognition, thus affecting the accuracy of behavior recognition. Therefore, reducing the computational cost of video understanding models and improving the ability to understand target region features is one of the current challenges. Summary of the Invention

[0005] The purpose of this invention is to provide a video intelligent question answering method and system based on incremental video retrieval enhancement generation, so as to solve the problems of low video retrieval efficiency and unsuitability for real-time analysis scenarios.

[0006] To achieve the above objectives, the present invention provides a video intelligent question answering method based on incremental video retrieval enhancement generation, comprising the following steps: S1: Divide the video to be retrieved into time segments to generate sub-videos, and preprocess each sub-video; S2: Input the sub-video into the target attention-enhanced video coding module, use the detection results of the target detection model in it to determine the target region in the video frame, and then generate a target region saliency map. Then, extract the spatiotemporal dependency of the sub-video segments through a multi-scale multi-head self-attention mechanism, extract features from the target region, generate video coding features and save them to the video vector database. S3: Input the query text, vectorize it, perform semantic retrieval in the text vector database to obtain a set of text blocks with semantic similarity scores higher than a preset threshold; concatenate the query text and the set of text blocks according to a preset template, input them into the Large Language Model (LLM), and obtain the initial answer from the Large Language Model (LLM); S4: Process the initial response content based on the Large Language Model (LLM): If the response contains a video link, output the search result; if the response is unanswerable and does not exceed the preset threshold, start the incremental video retrieval enhancement generation process; if the response is unanswerable and the number of times exceeds the preset threshold, output "unanswerable" directly. The incremental video retrieval enhancement generation process includes: performing semantic retrieval in the video vector database to obtain several video segment features with the highest semantic similarity to the query text; inputting the video segment features into the GPT-based video description generation module to generate video description text related to the query text; updating the generated video description text to the text vector database, and re-executing step S3.

[0007] To optimize the above technical solution, the specific measures also include: In step S1, the preprocessing of each sub-video specifically includes: For each sub-video, keyframes are extracted at fixed frame intervals to obtain a set of keyframes. The keyframes are input into the object detection model to obtain the object detection results, including bounding box coordinates and object category; Count the number of occurrences of different target categories in keyframes and generate a target list; Furthermore, based on the target detection results, a text description for each keyframe is generated. The text descriptions of all keyframes are concatenated into a document, which is then segmented, vectorized, and stored in a text vector database.

[0008] In step S2, the sub-video is input to the target attention-enhanced video coding module, and the target region in the video frame is determined using the detection results of the target detection model therein. A target region saliency map is then generated, and the spatiotemporal dependencies of the sub-video segments are extracted using a multi-scale multi-head self-attention mechanism to extract features from the target region. Specifically, this includes: Represent the video as a four-dimensional tensor, with dimension . Where T represents time, H represents height, W represents width, and 3 represents RGB channels; the video tensor is subdivided into multiple sub-regions, mapped to vectors, and recombined into new tensors; Furthermore, based on the target detection results, the target region in the video frame is determined, and a three-dimensional target region saliency map is generated to provide prior knowledge guidance for the target region; A multi-scale, multi-head self-attention mechanism is used to extract the long-range spatiotemporal dependencies of sub-video segments, where the attention calculation introduces a three-dimensional target region saliency map as an additive bias; Multi-scale feature aggregation is used to sort, group, and aggregate vectors to extract the encoded features of the target region.

[0009] Furthermore, the generation of a 3D target region saliency map based on the target detection results specifically includes: A two-dimensional saliency map is constructed based on the bounding box set in the object detection results to quantify the proximity of pixels to the object detection boxes. The calculation formula is as follows:

[0010] in, The standard significance value is... Update the values ​​for the difference matrix. These represent saliency plots. The element value at that position; The two-dimensional saliency map is expanded into a three-dimensional saliency map by time-dilation, and the saliency maps of all frames are stitched together to obtain the final saliency map.

[0011] Furthermore, the process of sorting, grouping, and aggregating vectors through multi-scale feature aggregation to extract the encoded features of the target region specifically involves: Based on the significance value assigned to each vector in the 3D target region significance map, all vectors within the tensor are sorted in descending order and divided into multiple tensor groups. Assign an aggregation factor to each tensor group and perform the aggregation operation, using the following formula:

[0012] in, To output a tensor, It is a linear mapping function. For input tensors, The aggregation factor is used to concatenate tensors according to their spatial positions before grouping, thus obtaining the encoded features of the video.

[0013] Furthermore, in step S4, the GPT-based video description generation module is a deep learning model with the Transformer architecture at its core, handling the video-to-text generation task, and the generation formula is:

[0014] in, This indicates the total number of words in the video description; This indicates the single character or word to be generated at position J; This represents all word sequences that have been generated before J.

[0015] As another important technical solution, the present invention also provides a video intelligent question-answering system based on incremental video retrieval enhancement generation, comprising: The video input and preprocessing module is used to segment the video to be retrieved into time segments, generate sub-videos, and preprocess each sub-video; The target attention enhancement module is used to input sub-videos into the target attention enhancement video coding module, use the detection results of the target detection model in it to determine the target region in the video frame, and then generate a target region saliency map. Then, the spatiotemporal dependency of the sub-video segments is extracted through a multi-scale multi-head self-attention mechanism, the target region is feature extracted, video coding features are generated and saved to the video vector database. The semantic retrieval module is used to input query text, vectorize it, and perform semantic retrieval in the text vector database to obtain a set of text blocks with semantic similarity scores higher than a preset threshold. The question-and-answer generation module is used to concatenate query text and text block sets according to preset templates, input them into the large language model LLM, and obtain the initial answer from the large language model LLM; The answer decision and incremental retrieval control module is used to process the initial answer content of the large language model LLM: if the answer contains a video link, the retrieval result is output; if the answer is unanswerable and does not exceed the preset threshold, the incremental video retrieval enhancement generation process is started; if the answer is unanswerable and the number of times exceeds the preset threshold, the unanswerable result is output directly.

[0016] Furthermore, the target detection model in the target attention enhancement module is a YOLOX model, which includes a backbone network, a neck network, and a detection head. The model output filters redundant bounding boxes using a non-maximum suppression algorithm. The vectorization model used in text description generation is the bge-reranker-large model, and the vectorization process uses cosine similarity for calculation.

[0017] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a video intelligent question answering method based on incremental video retrieval enhancement generation as described above.

[0018] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute a video intelligent question-answering method based on incremental video retrieval enhancement generation as described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: The target attention-enhanced video coding module proposed in this invention effectively reduces the computational complexity of spatiotemporal feature extraction in high-resolution videos through multi-scale feature aggregation and saliency-based guidance. It solves the problem that the computational complexity of traditional video Transformer models increases cubically with resolution, making it more feasible to process high-definition video streams.

[0020] The target attention enhancement mechanism of this invention integrates prior knowledge of target detection into the video encoding process, making the model feature extraction more focused on key target areas such as pedestrians and vehicles, effectively suppressing background interference, thereby generating higher quality video features and descriptive text that better match the user's query intent.

[0021] This invention features a unique incremental video retrieval enhancement generation mechanism that enables the system to learn from failed queries. By dynamically generating video descriptions strongly relevant to the query and updating the database, the system's knowledge base is continuously enriched and refined, achieving a gradual improvement in question-answering accuracy and overcoming the limitation of traditional static database systems where performance is fixed once initialized.

[0022] This invention utilizes a lightweight video preprocessing workflow and lightweight object detection models such as YoloX to generate preliminary text descriptions, avoiding the direct invocation of large video understanding models in the initial question-and-answer stage. This enables the system to respond quickly to most routine queries, making it particularly suitable for real-time analysis scenarios requiring rapid response, such as surveillance and security.

[0023] The method and system of this invention can be widely applied to video analysis and question-answering tasks in various fields such as intelligent security, video content review, smart transportation, and smart homes, including vehicle tracking, pedestrian behavior analysis, and abnormal event detection, and have high commercial value and practical prospects. Attached Figure Description

[0024] Figure 1 : Flowchart of video retrieval and question answering in this embodiment of the invention.

[0025] Figure 2 : Flowchart of video preprocessing in this embodiment of the invention.

[0026] Figure 3 : Flowchart of incremental video retrieval enhancement generation in this embodiment of the invention.

[0027] Figure 4: Flowchart of target attention enhancement video coding in this embodiment of the invention. Detailed Implementation

[0028] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0029] The following is an explanation of some of the terms used in this invention: Transformer is a deep learning model architecture used for natural language processing (NLP) tasks. It is based on the self-attention mechanism, which captures long-distance dependencies by computing attention weights at all positions in a sequence in parallel, and performs well when processing long sequences.

[0030] Large Language Models (LLMs) are deep learning-based models designed to understand and generate human language. A key characteristic of LLMs is their massive parameter scale, typically containing billions or even trillions of parameters, enabling them to process and generate highly complex and diverse text.

[0031] Retrieval Augmented Generation (RAG) provides information sources for large language models by retrieving data from multiple modalities, including video, text, and images, thus assisting in the generation of answers.

[0032] In some implementations, such as Figure 1 As shown, this invention provides a video intelligent question answering method based on incremental video retrieval enhancement generation, comprising the following steps: S1: Divide the video to be retrieved into time segments to generate sub-videos, preprocess each sub-video, generate text descriptions, and store them in a text vector database; In some implementations, the user inputs the video to be searched. and specific queries .

[0033] For video Perform time segmentation to generate a set of sub-videos. .

[0034] For sub-videos, Keyframes are extracted at intervals of 10 frames to obtain a set. For each sub-video Generate text description The vectorized text is then saved to the TextDB vector database. The text vectorization is implemented using the bge-reranker-large model, as shown in the following formula:

[0035] S2: Input the sub-video into the target attention-enhanced video coding module, use the detection results of the target detection model in the module to generate a saliency map, then extract the spatiotemporal dependencies of the sub-video segments through a multi-scale multi-head self-attention mechanism, extract features from the target region, generate video coding features and save them to the video vector database.

[0036] In some implementations, sub-videos The input is fed into the target attention-enhanced video coding module to generate video coding features. Save it to the vector database VideoDB.

[0037] keyframes The input is given to the YoloX model to obtain the object detection results, as shown in the following formula:

[0038] in, The model consists of three parts: the backbone, the neck, and the detection head. Convert the model output into bounding box coordinates. This is a non-maximum suppression algorithm used to filter redundant bounding boxes.

[0039] Based on the target detection results, generate The text description is in the following format: Detected Quantity is , Quantity is ,…, Quantity is ; in, Keyframe The time period in the original video.

[0040] Will All keyframes By concatenating the text descriptions, we obtain the document. .Will After segmentation and vectorization, the data is stored in TextDB to support the data generated for enhanced retrieval.

[0041] In some implementations, the specific process of video preprocessing is as follows: Figure 2 As shown, it includes keyframe extraction, object detection, text description generation, and vectorized storage.

[0042] Based on SG-Former, it is extended from image classification to video understanding, and combined with object detection results, the feature extraction capability of target regions is improved.

[0043] In some implementations, such as Figure 4 As shown, the video Represented as a four-dimensional tensor , dimension ,in It is the time dimension. and These represent the height and width of the image, respectively, and 3 indicates the RGB color channel.

[0044] Preferably, the target detection results are input, and the target regions in the video frames are determined using the detection results of the target detection model. A three-dimensional target region saliency map is generated, providing prior knowledge guidance for the target regions and increasing attention to the target regions. The specific process is as follows: For each frame of sampled image in V Obtain the set of bounding boxes calculated during the video preprocessing stage. Each detection box consists of a four-dimensional vector. It means that among them and These represent the coordinates of the top-left and bottom-right corners of the bounding box, respectively.

[0045] based on Construct a saliency map The saliency map is a two-dimensional matrix used to quantify the proximity of each pixel in a video frame to the target detection bounding box. Any position within the matrix... The element values ​​all represent the distance from the corresponding pixel in the original video to the detection box: the closer the distance, the larger the value. The calculation formula is as follows:

[0046] in, The standard significance value is... Update the values ​​for the difference matrix. These represent saliency plots. The element value at position.

[0047] Preferably, by dilating the time dimension, Expanded into a three-dimensional saliency map And by stitching together the saliency maps of all frames, we obtain the final saliency map:

[0048] Will Subdivided into multiple subregion Mapped to a length of The vector is given by the following formula:

[0049] Reorganize the mapped vectors according to their spatial arrangement to form a vector with dimension [ ]. tensor .

[0050] The spatiotemporal long-range dependencies of sub-video segments are extracted using a multi-scale, multi-head self-attention mechanism, as shown in the following formula:

[0051]

[0052]

[0053]

[0054] in, These represent the input features used to... Transformed into a learnable weight matrix of query, key, and value vectors. These represent the query vector, key vector, and value vector, respectively, and are core components of the self-attention mechanism. They are implemented using tensors. With the corresponding weight matrix ( The product is obtained by multiplying the two products. Represents the key vector matrix transpose, Represents the key vector Dimensions.

[0055] This represents a normalized exponential function used to convert the original attention scores into probability distribution weights that sum to 1.

[0056] Indicates the scaling factor. By using stride and kernel size The 3D convolution operation divides the tensor Y into... Each subregion is assigned a vector, and the set of vectors within that subregion is mapped to a single vector.

[0057] For the 3D window partitioning function, the tensor Divided into sizes The subtensors, all numbered according to their spatial location, return the sequence number as... A three-dimensional window.

[0058] As a preferred option, Indicates the sequence number is The attention weight vector is calculated from the subtensor, and the values ​​of the elements reflect the degree of attention the model pays to different parts of the input when generating the output. Input features The corresponding saliency plot has the following dimensions: Recorded The saliency value corresponding to all the marked indices.

[0059] This indicates a concatenation operation, used to join multiple vectors or tensors along a specified dimension into a longer vector or tensor.

[0060] Multi-scale feature aggregation reduces computational costs while ensuring the granularity of feature extraction from the target region. First, based on the saliency map of the 3D target region... Assigning significance values ​​to each vector, for tensors Sort all vectors within the range in descending order and divide them into... A set of tensors is formed from a group of tensors. .

[0061] Preferably, for each tensor group Specify an aggregation factor sci and perform the following aggregation operation:

[0062] in, This is a linear mapping function used to transform the feature dimensions. After this operation, the output tensor is... The number of vectors will be reduced to the input tensor set. Vector count .

[0063] After aggregation, the data is pieced together according to the spatial positions before grouping. To achieve saliency-guided multi-scale feature extraction. That is, video The coding features.

[0064] S3: Input the query text, vectorize it, perform semantic retrieval in the text vector database, and obtain a set of text blocks with semantic similarity scores higher than a preset threshold; concatenate the query text and the retrieved text block set according to a preset template, and input them into the Large Language Model (LLM) to obtain the initial answer of the Large Language Model (LLM); In some implementations, such as Figure 3 As shown, for user queries Perform vectorized encoding, and use it in VideoDB. Semantic retrieval is performed on the video encoding vectors of the sub-videos to obtain the features corresponding to the k video segments with the highest semantic similarity scores. .

[0065] Will The input is fed into the GPT-based video description generation module to generate video description text relevant to the user's query. The video description generation module based on GPT is a deep learning model with a Transformer architecture at its core, specifically designed for video-to-text generation tasks. It receives encoded video feature vectors as input and, based on the video content, ultimately generates video description text that conforms to natural language grammar and is relevant to the user's query. The video description generation formula for the GPT-based video description generation module is as follows:

[0066] in, This indicates the total number of words in the video description. Indicates the first The position will generate a single character or word. Indicates in All previously generated word sequences.

[0067] User queries and the generated description text The content is added to the description document D of V, and the content of TextDB is updated synchronously.

[0068] S4: Process the initial response content based on the Large Language Model (LLM): If the response contains a video link, output the search result; if the response is unanswerable and does not exceed the preset threshold, start the incremental video retrieval enhancement generation process; if the response is unanswerable and the number of times exceeds the preset threshold, output "unanswerable" directly. In another embodiment of the present invention, a video intelligent question-answering system based on incremental video retrieval enhancement generation is proposed, comprising: The video input and preprocessing module is used to segment the video to be retrieved into time segments, generate sub-videos, and preprocess each sub-video; The target attention enhancement module is used to take sub-videos as input, generate a saliency map using the detection results of the target detection model in the module, and then extract the spatiotemporal dependencies of sub-video segments through a multi-scale multi-head self-attention mechanism, extract features of the target region, generate video coding features, and save them to the video vector database. The semantic retrieval module is used to input query text, vectorize it, and perform semantic retrieval in the text vector database to obtain a set of text blocks with semantic similarity scores higher than a preset threshold. The question-and-answer generation module is used to concatenate the query text and the retrieved text block set according to the preset template, and input them into the large language model LLM to obtain the initial answer of the large language model LLM; The answer decision and incremental retrieval control module is used to process the initial answer content of the large language model LLM: if the answer contains a video link, the retrieval result is output; if the answer is unanswerable and does not exceed the preset threshold, the incremental video retrieval enhancement generation process is started; if the answer is unanswerable and the number of times exceeds the preset threshold, the unanswerable result is output directly. Preferably, the target detection model in the target attention enhancement module is the YOLOX model, which includes a backbone network, a neck network, and a detection head. The model output filters redundant bounding boxes using a non-maximum suppression algorithm. The vectorization model used in text description generation is the bge-reranker-large model, and the vectorization process uses cosine similarity for calculation.

[0069] In another embodiment of the present invention, an electronic device is proposed, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a video intelligent question answering method based on incremental video retrieval enhancement generation as described above.

[0070] In another embodiment of the present invention, a computer-readable storage medium is provided storing a computer program that causes a computer to execute a video intelligent question-answering method based on incremental video retrieval enhancement generation as described above.

[0071] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A video intelligent question answering method based on incremental video retrieval enhancement generation, characterized in that, Includes the following steps: S1: Divide the video to be retrieved into time segments to generate sub-videos, and preprocess each sub-video; S2: Input the sub-video into the target attention-enhanced video coding module, use the detection results of the target detection model in it to determine the target region in the video frame, and then generate a target region saliency map. Then, extract the spatiotemporal dependency of the sub-video segments through a multi-scale multi-head self-attention mechanism, extract features from the target region, generate video coding features and save them to the video vector database. S3: Input the query text, vectorize it, perform semantic retrieval in the text vector database to obtain a set of text blocks with semantic similarity scores higher than a preset threshold; concatenate the query text and the set of text blocks according to a preset template, input them into the Large Language Model (LLM), and obtain the initial answer from the Large Language Model (LLM); S4: Process the initial response from the Large Language Model (LLM): If the response contains a video link, output the search result; if the response is unanswerable and does not exceed a preset threshold, initiate the incremental video retrieval enhancement generation process. If the answer is "cannot be answered" and the number of attempts exceeds the preset threshold, then output "cannot be answered" directly. The incremental video retrieval enhancement generation process includes: performing semantic retrieval in the video vector database to obtain several video segment features with the highest semantic similarity to the query text; inputting the video segment features into the GPT-based video description generation module to generate video description text related to the query text; updating the generated video description text to the text vector database, and re-executing step S3.

2. The video intelligent question answering method based on incremental video retrieval enhancement generation according to claim 1, characterized in that: In step S1, the preprocessing of each sub-video specifically includes: For each sub-video, keyframes are extracted at fixed frame intervals to obtain a set of keyframes. The keyframes are input into the object detection model to obtain the object detection results, including bounding box coordinates and object category; Count the number of occurrences of different target categories in keyframes and generate a target list; Based on the target detection results, a text description for each keyframe is generated. The text descriptions of all keyframes are concatenated into a document, which is then segmented, vectorized, and stored in a text vector database.

3. The video intelligent question answering method based on incremental video retrieval enhancement generation according to claim 1, characterized in that: In step S2, the sub-video is input to the target attention-enhanced video coding module, and the target region in the video frame is determined using the detection results of the target detection model therein. A target region saliency map is then generated, and the spatiotemporal dependencies of the sub-video segments are extracted using a multi-scale multi-head self-attention mechanism to extract features from the target region. Specifically, this includes: Represent the video as a four-dimensional tensor, with dimension . Where T represents time, H represents height, W represents width, and 3 represents RGB channels; the video tensor is subdivided into multiple sub-regions, mapped to vectors, and recombined into new tensors; Based on the target detection results, the target region in the video frame is determined, a three-dimensional target region saliency map is generated, and prior knowledge of the target region is provided for guidance. A multi-scale, multi-head self-attention mechanism is used to extract the long-range spatiotemporal dependencies of sub-video segments, where the attention calculation introduces a three-dimensional target region saliency map as an additive bias; Multi-scale feature aggregation is used to sort, group, and aggregate vectors to extract the encoded features of the target region.

4. The video intelligent question answering method based on incremental video retrieval enhancement generation according to claim 3, characterized in that: The generation of a 3D target region saliency map based on the target detection results is specifically as follows: A two-dimensional saliency map is constructed based on the bounding box set in the object detection results to quantify the proximity of pixels to the object detection boxes. The calculation formula is as follows: in, The standard significance value is... Update the values ​​for the difference matrix. These represent saliency plots. The element value at that position; The two-dimensional saliency map is expanded into a three-dimensional saliency map by time-dilation, and the saliency maps of all frames are stitched together to obtain the final saliency map.

5. The video intelligent question answering method based on incremental video retrieval enhancement generation according to claim 3, characterized in that: The process of sorting, grouping, and aggregating vectors through multi-scale feature aggregation to extract the encoded features of the target region specifically involves: Based on the significance value assigned to each vector in the 3D target region significance map, all vectors within the tensor are sorted in descending order and divided into multiple tensor groups. Assign an aggregation factor to each tensor group and perform the aggregation operation, using the following formula: in, To output a tensor, It is a linear mapping function. For input tensors, The aggregation factor is used to concatenate tensors according to their spatial positions before grouping, thus obtaining the encoded features of the video.

6. The video intelligent question answering method based on incremental video retrieval enhancement generation according to claim 1, characterized in that: In step S4, the GPT-based video description generation module is a deep learning model with the Transformer architecture at its core, which processes the video-to-text generation task. The generation formula is as follows: in, This indicates the total number of words in the video description; This indicates the single character or word to be generated at position J; This represents all word sequences that have been generated before J.

7. The system used in the video intelligent question answering method based on incremental video retrieval enhancement as described in claim 1, characterized in that, include: The video input and preprocessing module is used to segment the video to be retrieved into time segments, generate sub-videos, and preprocess each sub-video; The target attention enhancement module is used to take sub-videos as input, generate a saliency map using the detection results of the target detection model in the module, and then extract the spatiotemporal dependencies of sub-video segments through a multi-scale multi-head self-attention mechanism, extract features of the target region, generate video coding features, and save them to the video vector database. The semantic retrieval module is used to input query text, vectorize it, and perform semantic retrieval in the text vector database to obtain a set of text blocks with semantic similarity scores higher than a preset threshold. The question-and-answer generation module is used to concatenate the query text and the retrieved text block set according to the preset template, and input them into the large language model LLM to obtain the initial answer of the large language model LLM; The answer decision and incremental retrieval control module is used to process the initial answer content of the large language model LLM: if the answer contains a video link, the retrieval result is output; if the answer is unanswerable and does not exceed the preset threshold, the incremental video retrieval enhancement generation process is started; if the answer is unanswerable and the number of times exceeds the preset threshold, the unanswerable result is output directly.

8. The system according to claim 7, characterized in that: The target attention enhancement module uses a YOLOX model for target detection, which consists of a backbone network, a neck network, and a detection head. The model output filters redundant bounding boxes using a non-maximum suppression algorithm. The vectorization model used in text description generation is the bge-reranker-large model, and the vectorization process uses cosine similarity for calculation.

9. An electronic device, characterized in that, include: The present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a video intelligent question answering method based on incremental video retrieval enhancement generation as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute a video intelligent question answering method based on incremental video retrieval enhancement as described in any one of claims 1 to 6.

Images