Scene text video question answering method and system based on selection and focus mechanism

Abstract

The application proposes a scene text video question answering method and system based on a selection and focusing mechanism, which first extracts scene text and corresponding video frames in the video; decomposes the original question into coarse-grained sub-questions and fine-grained sub-questions; screens out candidate scene text related to the coarse-grained question according to the coarse-grained sub-questions, and filters to obtain reserved scene text; screens out key frames according to the original question and the reserved scene text; captures the spatial dynamics and time existence of the scene text in the cross frame by using the attention mechanism to obtain local features; inputs the key frames and the local features into the visual language model together with the original question to generate the final output result. The application adopts the "coarse-fine" granularity question guiding selection strategy, discards the traditional uniform frame sampling method, and automatically filters the redundant frames and locks the key frames containing the answers according to the question semantics, so that the question related frames are not missed.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimedia processing technology, and in particular to a scene text video question answering method and system based on selection and focusing mechanisms. Background Technology

[0002] Understanding scene text within videos plays a crucial role in interpreting the physical world and is a key component in numerous practical applications, such as providing assistance to the visually impaired and facilitating autonomous driving. As a representative task in the field of scene text understanding, Video Text-based Visual Question Answering (VQA) has received widespread attention in recent years. Compared to traditional Video Question Answering (VQA), Video Text-based VQA is more complex because it explicitly requires models to read and understand the scene text throughout the video, establishing a connection between visual context and textual semantics to answer a given question. In recent years, numerous methods have been proposed for the Video Text-based VQA task, achieving significant progress. For example, methods extract video features based on the Transformer architecture and fuse them with text for inference, or aggregate key clues by reconstructing the spatiotemporal relationships of visual entities in the video. However, most existing technologies face two serious unresolved technical shortcomings.

[0003] First, it relies on a uniform temporal sampling strategy: most existing methods employ a static, query-independent uniform temporal sampling strategy to extract frames from the video, treating all frames in the video equally. This approach is highly prone to missing key frames that are highly relevant to the question. When asking about a brief appearance of text information in the video, uniform sampling often fails to capture the precise frame containing the answer.

[0004] Second, it ignores the spatio-temporal evolution of scene text across different video frames: most current mainstream methods only establish static positional relationships between different instances within a single frame, ignoring the spatial position changes and temporal existence of the same scene text across consecutive video frames. This mechanism hinders the model from understanding the real spatio-temporal connections between different texts in the video, and makes it impossible to accurately model the movement trajectory and duration of the same text across multiple frames.

[0005] Furthermore, with the application of Large Language Models (LLMs) in video understanding, directly inputting complete video frame sequences containing a large amount of redundant background into the model leads to extremely high visual token overhead and computational costs. Moreover, existing large video models primarily focus on action recognition and event understanding, lacking specific mechanisms for capturing the spatiotemporal evolution of fine-grained scene text (such as text movement, occlusion, disappearance, and reappearance). Therefore, how to accurately select video frames containing key text while reducing computational overhead, and effectively model the spatiotemporal consistency of text, has become a bottleneck that urgently needs to be addressed in this field. Summary of the Invention

[0006] In view of the above, the main objective of this invention is to propose a scene-based text video question-answering method and system based on selection and focusing mechanisms to solve the aforementioned technical problems.

[0007] This invention proposes a scene-based text-video question-answering method based on a selection and focusing mechanism, the method comprising the following steps: Step 1: Based on the input video and the corresponding original question, extract the scene text (OCR text) and the corresponding video frames from the video; The original problem is broken down into a coarse-grained subproblem and a fine-grained subproblem. Step 2: Perform coarse filtering using a vision-language model. Input the coarse-grained sub-problems, video frames, and scene text into the vision-language model to initially filter out candidate scene texts related to the coarse-grained problems. Step 3: Perform fine-grained selection and video frame localization, and use fine-grained sub-problems to verify and further filter the candidate scene text to obtain the retained scene text; The CLIP model is used to calculate the relevance score between the original question and all video frames of the retained scene text, and the frame with the highest relevance score is selected as the key frame. Step 4: Construct the spatiotemporal evolution features of the scene text, assign a unique feature embedding to each scene text in the keyframe, and add it to the image patch at the corresponding position of the keyframe to obtain the fused feature representation; The fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames and obtain local features; Step 5: Infer the final answer by combining global and local information: Input the keyframes (global information) selected by the question-guided selection module and the local features (local information) extracted by consistency processing, along with the original question, into the visual language model to obtain global candidate answers and local candidate answers; calculate the confidence score of each candidate answer and select the answer with the highest confidence score as the final output result.

[0008] This invention also proposes a scene-based text-video question-answering system based on a selection and focus mechanism, wherein the system applies the scene-based text-video question-answering method based on the selection and focus mechanism described above, and the system includes: The question-guided selection module is used for: Based on the input video and the corresponding original question, extract the scene text and the corresponding video frames from the video; The original problem is decomposed into coarse-grained subproblems and fine-grained subproblems; The coarse-grained sub-problem, video frames, and scene text are input into the visual language model to initially filter out candidate scene texts related to the coarse-grained problem; Fine-grained sub-problems are used to verify and further filter candidate scene texts, resulting in retained scene texts; The CLIP model is used to calculate the relevance score between the original question and all video frames of the retained scene text, and the frame with the highest relevance score is selected as the key frame. The text-aware region consistency module is used for: A unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding position of the keyframe to obtain the fused feature representation; The fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames and obtain local features; The global timing-aware reordering module is used for: The keyframes and local features, along with the original question, are input into the visual language model to obtain global and local candidate answers. The confidence score of each candidate answer is calculated, and the answer with the highest confidence score is selected as the final output.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Overcoming information loss and achieving accurate frame extraction: This invention adopts a coarse-fine granular question-guided selection strategy, abandoning the traditional uniform frame sampling method. It can automatically filter redundant frames and lock the key frames of scene text containing the answer based on the semantics of the question. This significantly reduces information noise during reasoning from the source, ensuring that no question-related frames are missed.

[0010] 2. Enhanced Spatiotemporal Dynamic Association Capability of Multimodal Features: This invention innovatively assigns independent color mapping embedding features to each scene text and establishes cross-frame connections between scene texts through a self-attention mechanism. The model not only grasps the static relative position within a single frame but also dynamically tracks the frequency of occurrence and spatial displacement (i.e., spatiotemporal evolution) of the target text on the time axis. This greatly improves the recognition accuracy and tracking stability when highly similar or overlapping text exists in the video.

[0011] 3. Complementary Advantages of Local and Global Perspectives: This invention utilizes a Global Temporal Aware Reordering (GTR) mechanism, simultaneously considering the global and local spatiotemporal evolution of scene text, ensuring robustness and high accuracy of reasoning results for complex problems. Extensive experiments on mainstream public datasets (such as RoadTextVQA and M4-ViteVQA) demonstrate that this invention surpasses existing state-of-the-art methods in both accuracy and Average Normalized Levenshtein Similarity (ANLS).

[0012] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by means of embodiments of the invention. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the steps of the scene-based text video question-answering method proposed in this invention, which is based on a selection and focusing mechanism. Figure 2 This is a diagram illustrating the overall architecture of the scene-based text video question-answering system proposed in this invention, which is based on a selection and focusing mechanism. Figure 3 This is a structural diagram of the Problem-Guided Selection (QGS) module proposed in this invention; Figure 4 This is a structural diagram of the Text Aware Region Consistency (TRC) module proposed in this invention; Figure 5 This is a structural diagram of the Global Time-Aware Rearrangement (GTR) module proposed in this invention. Detailed Implementation

[0014] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0015] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0016] Please see Figure 1 This embodiment provides a scene-based text video question-answering method based on a selection and focusing mechanism. The method includes the following steps: Step 1: Based on the input video and the corresponding original question, extract the scene text and corresponding video frames from the video; The original problem is decomposed into coarse-grained subproblems and fine-grained subproblems; In a preferred embodiment of the present invention, scene text and corresponding video frames are extracted from the video, and the original problem is decomposed into coarse-grained sub-problems and fine-grained sub-problems, specifically including the following steps: A pre-trained video text extraction model (VTS) is used to uniformly sample and scan the input video frame by frame to extract the video frame sequence; each video frame in the video frame sequence contains a set of scene text and the corresponding spatial bounding box coordinates; By using a large language model, the original input question is semantically parsed and decoupled. The semantic part containing the core query entity is extracted into a coarse-grained sub-question, and the semantic part containing specific attributes or additional constraints of the core query entity is extracted into a fine-grained sub-question.

[0017] In this embodiment, when using a large language model to decompose the problem, prompt words containing instruction templates are constructed to guide the model in identifying the main localization targets and additional constraints in the problem. For example, when the original problem is "Which exit has to be taken for Linden Blvd?", the large language model decouples it into: a coarse-grained sub-problem "Which exit should Itake?" used to locate the approximate position of the scene text, and a fine-grained sub-problem "Is this exit for Linden Blvd?" used to verify whether the scene text meets specific attributes. This coarse-to-fine decoupling method can effectively reduce the retrieval space of the subsequent visual language model in complex video frames.

[0018] Step 2: Input the coarse-grained sub-problem, video frames, and scene text into the visual language model to initially filter out candidate scene texts related to the coarse-grained problem; In a preferred embodiment of the present invention, candidate scene texts are verified and further filtered using fine-grained sub-problems to obtain retained scene texts, specifically including the following steps: The coarse-grained sub-problems, video frames, and corresponding scene text sets are combined to construct a multimodal input prompt, which is then input into the visual language model. We use a visual language model to understand the core entities mentioned in coarse-grained sub-problems and analyze the semantic relationships between the scene text and the core entities in the corresponding keyframes. Based on semantic relevance, a visual language model is used to identify and filter out scene texts that are semantically related to the core entities from all scene texts, and retain them as candidate scene texts. This filters out redundant scene texts that are irrelevant to coarse-grained sub-problems, resulting in the retained scene texts.

[0019] In step 2, the CLIP model is used to calculate the correlation score between the original question and all video frames containing retained scene text, specifically including the following steps: The CLIP model is used to encode the original question and all video frames containing preserved scene text into feature vectors; The cosine similarity between the feature vectors of the original question and all video frames containing preserved scene text is calculated to obtain a relevance score. The corresponding process follows the following formula: ; in, The relevance score between the video frame representation containing the i-th scene text and the problem features is represented by this value. The visual feature representation of the video frame representing the i-th scene text. This indicates taking the L2 norm. Indicates the characteristics of the original problem. This represents the time index of the video frame containing the i-th scene text. This represents the dot product of vectors.

[0020] Step 3: Use fine-grained sub-problems to verify and further filter the candidate scene texts to obtain the retained scene texts; The CLIP model is used to calculate the relevance score between the original question and all video frames of the retained scene text, and the frame with the highest relevance score is selected as the key frame. Step 4: Assign a unique feature embedding to each scene text in the keyframe and add it to the image patch at the corresponding position of the keyframe to obtain the fused feature representation; The fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames and obtain local features; In a preferred embodiment of the present invention, a unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding position of the keyframe to obtain the fused feature representation. Specifically, the steps include: For each keyframe, extract the content of each scene text and its corresponding spatial bounding box coordinates to determine the scene text instances that need to be modeled; The keyframes are input into the visual encoder, the entire keyframe image is divided into blocks and mapped into high-dimensional vectors to obtain the image block visual embedding. An embedding table is constructed in the feature space of the visual encoder. The embedding table contains a set of pre-defined, learnable high-dimensional feature vectors. These vectors are orthogonal to each other or have high discriminative power, and are logically equivalent to different "color labels".

[0021] For each different scene text instance, a unique feature vector is selected from the embedding table as an identity identifier to obtain the identity identifier vector; this ensures that text instances with different content in the same video obtain different feature vectors, thereby marking their uniqueness in the feature space.

[0022] Based on the spatial bounding box coordinates, the corresponding image patch is located, and the identity vector is added to the visual embedding of the corresponding image patch to form a fused feature representation. This operation is equivalent to "coloring" each text instance in the visual feature stream.

[0023] In this embodiment, an embedding table is constructed in the feature space of the visual encoder. This embedding table employs a "color mapping" mechanism: a series of unique high-dimensional feature vectors are pre-defined for different content-based scene text in the visual feature space. These feature vectors are logically equivalent to different "color labels," used to prominently identify different text instances in the visual feature stream. Subsequently, the image patch feature sequence carrying the identification is input into the text-aware spatiotemporal self-attention module. When calculating attention across frames, the module can track the spatial position changes and temporal occurrence states of the same text instance through the same identification, thereby explicitly modeling its spatiotemporal evolution.

[0024] In this embodiment, a unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding position of the keyframe to obtain the fused feature representation. The corresponding process has the following relationship: ; ; in, This indicates the embedded table being constructed. This represents the text of the i-th scene. The feature embeddings representing the assignment, This represents the visual embedding of the image patch at the i-th scene text location. This represents the feature representation after fusion.

[0025] In this embodiment, the fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames, thereby obtaining local features. The corresponding process has the following relationship: ; ; ; in, The attention score represents the relationship between the i-th scene text input and the j-th scene text input, which is used to measure the correlation strength between the two scene texts in the feature space. The term represents the spatiotemporal relationship bias between the i-th scene text and the j-th scene text. It is used to encode the spatiotemporal relationship between the two scene texts, enabling the attention calculation process to perceive the relative position information and temporal evolution information of the scene texts. This indicates the transpose operation. This represents the different learnable parameters of the self-attention layer. Indicates the scaling factor. This represents the final output feature representation of the image block corresponding to the i-th scene text after incorporating global context information; This represents the attention weight of the i-th scene text to the j-th scene text, which is used to determine how much attention to allocate to the j-th scene text when updating the features of the i-th scene text; Represents the normalized exponential function, Let represent the fused feature representation of the j-th scene text. This represents matrix multiplication.

[0026] Step 5: Input the keyframes and local features along with the original question into the visual language model to obtain global and local candidate answers; calculate the confidence score of each candidate answer and select the answer with the highest confidence score as the final output.

[0027] In a preferred embodiment of the present invention, the confidence score of each candidate answer is calculated, and the answer with the highest confidence score is selected as the final output result. This specifically includes the following steps: The candidate answers to be evaluated, the original question, and the corresponding video frames are concatenated to form a verification query prompt, which is constructed in the form of 'Based on the provided video content, is the answer [candidate answer] correct for the question [original question]? Please only answer Y or N'. Input the validation query prompts into the visual language model. For each candidate answer, the visual language model outputs a positive label score and a negative label score. The confidence score is calculated using the following formula: ; in, This represents the confidence score. Indicates candidate answers, To indicate a positive score, Indicates a negative score; Comparing the confidence scores of global and local candidate answers, and selecting the one with the higher score as the final answer, follows the relationship: ; in, This indicates the final answer. Indicates local candidate answers, Represents the global candidate answers. The confidence score represents the local candidate answer. This represents the confidence score of all candidate answers.

[0028] Please refer to Figure 2 This embodiment also provides a scene-based text-video question-answering system based on a selection and focus mechanism, wherein the system applies the scene-based text-video question-answering method based on a selection and focus mechanism as described above, and the system includes: The question-guided selection module is used for: Based on the input video and the corresponding original question, extract the scene text and the corresponding video frames from the video; The original problem is decomposed into coarse-grained subproblems and fine-grained subproblems; The coarse-grained sub-problem, video frames, and scene text are input into the visual language model to initially filter out candidate scene texts related to the coarse-grained problem; Fine-grained sub-problems are used to verify and further filter candidate scene texts, resulting in retained scene texts; The CLIP model is used to calculate the relevance score between the original question and all video frames of the retained scene text, and the frame with the highest relevance score is selected as the key frame. The text-aware region consistency module is used for: A unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding position of the keyframe to obtain the fused feature representation; The fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames and obtain local features; The global timing-aware reordering module is used for: The keyframes and local features, along with the original question, are input into the visual language model to obtain global and local candidate answers. The confidence score of each candidate answer is calculated, and the answer with the highest confidence score is selected as the final output.

[0029] Please refer to Figure 3 As a preferred embodiment of the present invention, the working process of the question-guided selection module is as follows: A pre-trained video text extraction model (VTS) is used to uniformly sample and scan the input video frame by frame to extract the video frame sequence; each video frame in the video frame sequence contains a set of scene text and the corresponding spatial bounding box coordinates; By using a large language model, the original input question is semantically parsed and decoupled. The semantic part containing the core query entity is extracted into a coarse-grained sub-question, and the semantic part containing specific attributes or additional constraints of the core query entity is extracted into a fine-grained sub-question.

[0030] In this embodiment, when using a large language model to decompose the problem, prompt words containing instruction templates are constructed to guide the model in identifying the main localization targets and additional constraints in the problem. For example, when the original problem is "Which exit has to be taken for Linden Blvd?", the large language model decouples it into: a coarse-grained sub-problem "Which exit should Itake?" used to locate the approximate position of the scene text, and a fine-grained sub-problem "Is this exit for Linden Blvd?" used to verify whether the scene text meets specific attributes. This coarse-to-fine decoupling method can effectively reduce the retrieval space of the subsequent visual language model in complex video frames.

[0031] Please refer to Figure 4 As a preferred embodiment of the present invention, the working process of the text-aware region consistency module is as follows: The coarse-grained sub-problems, video frames, and corresponding scene text sets are combined to construct a multimodal input prompt, which is then input into the visual language model. We use a visual language model to understand the core entities mentioned in coarse-grained sub-problems and analyze the semantic relationships between the scene text and the core entities in the corresponding keyframes. Based on semantic relevance, a visual language model is used to identify and filter out scene texts that are semantically related to the core entities from all scene texts, and retain them as candidate scene texts. This filters out redundant scene texts that are irrelevant to coarse-grained sub-problems, resulting in the retained scene texts.

[0032] The CLIP model is used to encode the original question and all video frames containing preserved scene text into feature vectors; The cosine similarity between the feature vectors of the original question and all video frames containing preserved scene text is calculated to obtain a relevance score. The corresponding process follows the following formula: ; in, The relevance score between the video frame representation containing the i-th scene text and the problem features is represented by this value. The visual feature representation of the video frame representing the i-th scene text. This indicates taking the L2 norm. Indicates the characteristics of the original problem. This represents the time index of the video frame containing the i-th scene text. Represents the dot product of vectors; After obtaining the relevance score, the frame with the highest relevance score is selected as the key frame.

[0033] For each keyframe, extract the content of each scene text and its corresponding spatial bounding box coordinates to determine the scene text instances that need to be modeled; The keyframes are input into the visual encoder, the entire keyframe image is divided into blocks and mapped into high-dimensional vectors to obtain the image block visual embedding. An embedding table is constructed in the feature space of the visual encoder. The embedding table contains a set of pre-defined, learnable high-dimensional feature vectors. These vectors are orthogonal to each other or have high discriminative power, and are logically equivalent to different "color labels".

[0034] For each different scene text instance, a unique feature vector is selected from the embedding table as an identity identifier to obtain the identity identifier vector; this ensures that text instances with different content in the same video obtain different feature vectors, thereby marking their uniqueness in the feature space.

[0035] Based on the spatial bounding box coordinates, the corresponding image patch is located, and the identity vector is added to the visual embedding of the corresponding image patch to form a fused feature representation. This operation is equivalent to "coloring" each text instance in the visual feature stream.

[0036] In this embodiment, an embedding table is constructed in the feature space of the visual encoder. This embedding table employs a "color mapping" mechanism: a series of unique high-dimensional feature vectors are pre-defined for different content-based scene text in the visual feature space. These feature vectors are logically equivalent to different "color labels," used to prominently identify different text instances in the visual feature stream. Subsequently, the image patch feature sequence carrying the identification is input into the text-aware spatiotemporal self-attention module. When calculating attention across frames, the module can track the spatial position changes and temporal occurrence states of the same text instance through the same identification, thereby explicitly modeling its spatiotemporal evolution.

[0037] In this embodiment, a unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding position of the keyframe to obtain the fused feature representation. The corresponding process has the following relationship: ; ; in, This indicates the embedded table being constructed. This represents the text of the i-th scene. The feature embeddings representing the assignment, This represents the visual embedding of the image patch at the i-th scene text location. This represents the feature representation after fusion.

[0038] In this embodiment, the fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames, thereby obtaining local features. The corresponding process has the following relationship: ; ; ; in, The attention score represents the relationship between the i-th scene text input and the j-th scene text input, which is used to measure the correlation strength between the two scene texts in the feature space. The term represents the spatiotemporal relationship bias between the i-th scene text and the j-th scene text. It is used to encode the spatiotemporal relationship between the two scene texts, enabling the attention calculation process to perceive the relative position information and temporal evolution information of the scene texts. This indicates the transpose operation. This represents the different learnable parameters of the self-attention layer. Indicates the scaling factor. This represents the final output feature representation of the image block corresponding to the i-th scene text after incorporating global context information; This represents the attention weight of the i-th scene text to the j-th scene text, which is used to determine how much attention to allocate to the j-th scene text when updating the features of the i-th scene text; Represents the normalized exponential function, Let represent the fused feature representation of the j-th scene text. This represents matrix multiplication.

[0039] Please refer to Figure 5 As a preferred embodiment of the present invention, the working process of the global timing-aware reordering module is as follows: The candidate answers to be evaluated, the original question, and the corresponding video frames are concatenated to form a verification query prompt, which is constructed in the form of 'Based on the provided video content, is the answer [candidate answer] correct for the question [original question]? Please only answer Y or N'. Input the validation query prompts into the visual language model. For each candidate answer, the visual language model outputs a positive label score and a negative label score. The confidence score is calculated using the following formula: ; in, This represents the confidence score. Indicates candidate answers, To indicate a positive score, Indicates a negative score; Comparing the confidence scores of global and local candidate answers, and selecting the one with the higher score as the final answer, follows the relationship: ; in, This indicates the final answer. Indicates local candidate answers, Represents the global candidate answers. The confidence score represents the local candidate answer. This represents the confidence score of all candidate answers.

[0040] To verify the effectiveness of this invention, this embodiment was extensively validated on mainstream public scene text video question answering datasets (RoadTextVQA and M4-ViteVQA), achieving significant performance improvements compared to existing technologies. On the RoadTextVQA dataset: the method of this invention achieved an accuracy of 69.13% and an average normalized Levenshtein similarity (ANLS) of 74.66%. Compared to the existing state-of-the-art method GAT (accuracy 50.23%, ANLS 58.12%), this invention improves accuracy by nearly 19%, significantly surpassing existing technologies.

[0041] On the M4-ViteVQA dataset (Task1Split1): the proposed method achieved 65.55% accuracy and 74.09% ANLS on the validation set, and 62.14% accuracy and 70.75% ANLS on the test set, representing improvements of 2.24% and 1.88% respectively compared to the baseline model. Ablation experiments were conducted on the validation set of the M4-ViteVQA dataset (Task1Split1), demonstrating that introducing either the question-guided selection module or the text-aware region consistency module alone can effectively improve the model's accuracy and ANLS. When the global temporal-aware reordering module is used in conjunction, the model achieves a total accuracy improvement of 2.24% compared to the baseline, proving the necessity of integrating global and local spatiotemporal change information to improve the accuracy of complex question-answering reasoning.

[0042] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A scene-based text-video question-answering method based on selection and focusing mechanisms, characterized in that, The method includes the following steps: Step 1: Based on the input video and the corresponding original question, extract the scene text and corresponding video frames from the video; The original problem is decomposed into coarse-grained subproblems and fine-grained subproblems; Step 2: Input the coarse-grained sub-problem, video frames, and scene text into the visual language model to initially filter out candidate scene texts related to the coarse-grained problem; Step 3: Use fine-grained sub-problems to verify and further filter the candidate scene texts to obtain the retained scene texts; The CLIP model is used to calculate the relevance score between the original question and all video frames of the retained scene text, and the frame with the highest relevance score is selected as the key frame. Step 4: Assign a unique feature embedding to each scene text in the keyframe and add it to the image patch at the corresponding position of the keyframe to obtain the fused feature representation; The fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames and obtain local features; Step 5: Input the keyframes and local features along with the original question into the visual language model to obtain global and local candidate answers; calculate the confidence score of each candidate answer and select the answer with the highest confidence score as the final output.

2. The scene text video question answering method based on selection and focusing mechanism according to claim 1, characterized in that, In step 1, scene text and corresponding video frames are extracted from the video, and the original problem is decomposed into coarse-grained sub-problems and fine-grained sub-problems, specifically including the following steps: A pre-trained video text extraction model is used to uniformly sample and scan the input video frame by frame to extract the video frame sequence; each video frame in the video frame sequence contains a set of scene text and the corresponding spatial bounding box coordinates. By using a large language model, the original input question is semantically parsed and decoupled. The semantic part containing the core query entity is extracted into a coarse-grained sub-question, and the semantic part containing specific attributes or additional constraints of the core query entity is extracted into a fine-grained sub-question.

3. The scene text video question answering method based on selection and focusing mechanism according to claim 2, characterized in that, In step 2, the candidate scene text is validated and further filtered using fine-grained sub-problems to obtain the retained scene text. This process includes the following steps: The coarse-grained sub-problems, video frames, and corresponding scene text sets are combined to construct a multimodal input prompt, which is then input into the visual language model. We use a visual language model to understand the core entities mentioned in coarse-grained sub-problems and analyze the semantic relationships between the scene text and the core entities in the corresponding keyframes. Based on semantic relevance, a visual language model is used to identify and filter out scene texts that are semantically related to the core entities from all scene texts, and retain them as candidate scene texts. This filters out redundant scene texts that are irrelevant to coarse-grained sub-problems, resulting in the retained scene texts.

4. The scene text video question answering method based on selection and focusing mechanism according to claim 2, characterized in that, In step 2, the CLIP model is used to calculate the correlation score between the original question and all video frames containing retained scene text, specifically including the following steps: The CLIP model is used to encode the original question and all video frames containing preserved scene text into feature vectors; The cosine similarity between the feature vectors of the original question and all video frames containing preserved scene text is calculated to obtain a relevance score. The corresponding process follows the following formula: ； in, Indicates containing the first i The correlation score between the video frame representation of the scene text and the problem features. Indicates the first i Visual feature representation of video frames for a scene text This indicates taking the L2 norm. Indicates the characteristics of the original problem. This represents the time index of the video frame containing the i-th scene text. This represents the dot product of vectors.

5. The scene text video question answering method based on selection and focusing mechanism according to claim 2, characterized in that, In step 4, a unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding location of the keyframe to obtain the fused feature representation. This specifically includes the following steps: For each keyframe, extract the content of each scene text and its corresponding spatial bounding box coordinates to determine the scene text instances that need to be modeled; The keyframes are input into the visual encoder, the entire keyframe image is divided into blocks and mapped into high-dimensional vectors to obtain the image block visual embedding. An embedding table is constructed in the feature space of the visual encoder, which contains a set of pre-defined, learnable high-dimensional feature vectors. For each different scene text instance, a unique feature vector is selected from the embedding table as an identity identifier to obtain the identity identifier vector; Based on the spatial bounding box coordinates, the corresponding image blocks are located, and the identity vector is added to the image block visual embedding of the corresponding image block to form a fused feature representation.

6. The scene text video question answering method based on selection and focusing mechanism according to claim 5, characterized in that, In step 4, a unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding location of the keyframe to obtain the fused feature representation. The corresponding process has the following relationship: ； ； in, This indicates the embedded table being constructed. This represents the text of the i-th scene. The feature embeddings representing the assignment, This represents the visual embedding of the image patch at the i-th scene text location. This represents the feature representation after fusion.

7. The scene text video question answering method based on selection and focusing mechanism according to claim 6, characterized in that, In step 4, the fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal presence of scene text across frames, thereby obtaining local features. The corresponding process has the following relationship: ； ； ； in, This represents the attention score between the i-th scene text input and the j-th scene text input. This represents the spatiotemporal relationship bias term between the i-th scene text and the j-th scene text. This indicates the transpose operation. This represents the different learnable parameters of the self-attention layer. Indicates the scaling factor. This represents the final output feature representation of the image block corresponding to the i-th scene text after incorporating global context information; This represents the attention weight of the i-th scene text to the j-th scene text, which is used to determine how much attention to allocate to the j-th scene text when updating the features of the i-th scene text; Represents the normalized exponential function, Let represent the fused feature representation of the j-th scene text. This represents matrix multiplication.

8. The scene text video question answering method based on selection and focusing mechanism according to claim 7, characterized in that, In step 5, the confidence score of each candidate answer is calculated, and the answer with the highest confidence score is selected as the final output. This includes the following steps: Verification query prompts are created by stitching together the candidate answers to be evaluated, the original question, and the corresponding video frames. Input the validation query prompts into the visual language model. For each candidate answer, the visual language model outputs a positive label score and a negative label score. The confidence score is calculated using the following formula: ； in, This represents the confidence score. Indicates candidate answers, To indicate a positive score, Indicates a negative score; Comparing the confidence scores of global and local candidate answers, and selecting the one with the higher score as the final answer, follows the relationship: ； in, This indicates the final answer. Indicates local candidate answers, Represents the global candidate answers. The confidence score represents the local candidate answer. This represents the confidence score of all candidate answers.

9. A scene-based text-video question-answering system based on a selection and focusing mechanism, characterized in that, The system applies the scene-based text-video question-answering method based on selection and focusing mechanisms as described in any one of claims 1 to 8, and the system comprises: The question-guided selection module is used for: Based on the input video and the corresponding original question, extract the scene text and the corresponding video frames from the video; The original problem is decomposed into coarse-grained subproblems and fine-grained subproblems; The coarse-grained sub-problem, video frames, and scene text are input into the visual language model to initially filter out candidate scene texts related to the coarse-grained problem; Fine-grained sub-problems are used to verify and further filter candidate scene texts, resulting in retained scene texts; The CLIP model is used to calculate the relevance score between the original question and all video frames of the retained scene text, and the frame with the highest relevance score is selected as the key frame. The text-aware region consistency module is used for: A unique feature embedding is assigned to each scene text in the keyframe and added to the image patch at the corresponding position of the keyframe to obtain the fused feature representation; The fused feature representation is input into the text-aware spatiotemporal self-attention module to capture the spatial dynamics and temporal existence of scene text across frames and obtain local features; The global timing-aware reordering module is used for: The keyframes and local features, along with the original question, are input into the visual language model to obtain global and local candidate answers. The confidence score of each candidate answer is calculated, and the answer with the highest confidence score is selected as the final output.

Images