Video question answering method and device

Abstract

The application relates to the technical field of video question answering, and discloses a video question answering method and device, which comprises the following steps: acquiring a video to be analyzed and a user question raised by a user based on the video to be analyzed; analyzing the correlation between each video frame in the video to be analyzed and the user question by using a pre-trained correlation scoring model to obtain first correlation scores of the user question and each video frame; sorting the video frames from high to low based on the first correlation scores, and screening first target video frames with a first preset number of high ranks; generating first visual features and text description features according to the first correlation scores of the first target video frames, a preset score threshold and a preset feature extraction path; and generating an answer to the user question based on the first visual features, the text description features and the user question by using a pre-trained video question answering prediction model. The application improves the accuracy of video question answering.

Description

Technical Field

[0001] This application relates to the field of video question answering technology, specifically to a video question answering method and apparatus. Background Technology

[0002] Video Question Answering (VideoQA) is an important research direction at the intersection of artificial intelligence, computer vision, and natural language processing. Its goal is to enable models to understand video content and answer questions related to the video.

[0003] Traditional video question answering methods sample video frames uniformly or treat all video frames equally, resulting in a large number of background frames that are irrelevant to the user's question being included in the model's calculation. This makes it difficult for the model to accurately locate the key video frames that actually contain clues to the answer from the massive amount of redundant information, thereby reducing the accuracy of video question answering. Summary of the Invention

[0004] The purpose of this application is to provide a video question-and-answer method and apparatus to address the technical deficiency of low accuracy in existing video question-and-answer technologies.

[0005] To achieve the above objectives, the first aspect of this application provides a video question-answering method, the method comprising: Acquire the video to be analyzed and the user questions raised by users based on the video; By using a pre-trained relevance scoring model, the relevance between each video frame in the video to be analyzed and the user's question is analyzed, and the first relevance score between the user's question and each video frame is obtained. Based on the first relevance score, the video frames are sorted from high to low, and the first target video frames with the first preset number of top-ranked frames are selected. Based on the first relevance score of the first target video frame, the preset score threshold, and the preset feature extraction path, generate the first visual feature and text description feature; Based on a pre-trained video question answering prediction model, answers to user questions are generated according to first visual features, text description features, and user questions.

[0006] In this embodiment of the application, a first visual feature and a text description feature are generated based on a first relevance score of the first target video frame, a preset score threshold, and a preset feature extraction path, including: If the first relevance scores of the first target video frames are all less than the first preset score threshold, the event clustering process is performed on each video frame in the video to be analyzed by a clustering algorithm to obtain a semantically coherent first event set. Generate a first-event-level description for each event in the first event set; Determine the second relevance score of each first-event-level description to the user's question; The target event set is determined based on multiple second relevance scores and a third preset scoring threshold; Determine the second target video frame based on the center frame of each event in the target event set; Extract the video features of each second target video frame as the first visual feature; Extract the event-level descriptions of the target event set and use them as text descriptions.

[0007] In this embodiment of the application, a first visual feature and a text description feature are generated based on a first relevance score of the first target video frame, a preset score threshold, and a preset feature extraction path, including: If any first relevance score is greater than or equal to the first preset score threshold and less than the second preset score threshold, the event clustering process is performed on each video frame in the video to be analyzed by a clustering algorithm to obtain a semantically coherent second event set. Select a second preset number of video frames from the first target video frame to obtain the second target video frame; Match the second target video frame with the events in the second event set to determine the event to which the second target video frame belongs, wherein the event to which the video frame belongs includes one or more video frames; In each attribution event, determine the third relevance score of each second target video frame to the user question; Based on the third relevance score, each second target video frame is sorted from high to low, and the top-ranked third preset number of second target videos are selected as the third target video frames. Extract the text description of each third target video frame as the second event-level description; Generate a first frame-level description based on the second target video frame; Extract the video features of each third target video frame as the first visual feature; Text description features are obtained based on the second event-level description and the first frame-level description.

[0008] In this embodiment of the application, a first visual feature and a text description feature are generated based on a first relevance score of the first target video frame, a preset score threshold, and a preset feature extraction path, including: If any first correlation score is greater than or equal to a first preset score threshold and greater than or equal to a second preset score threshold, the video features of the first target video frame are extracted as the first visual features. Generate a second frame-level description based on the first target video frame; Based on the second-frame level description, the text description features are obtained.

[0009] In this embodiment of the application, the method further includes: If any first correlation score is greater than or equal to the first preset score threshold and less than the second preset score threshold, and if any first correlation score is greater than or equal to the first preset score threshold and greater than or equal to the second preset score threshold, the first visual feature obtained by extracting the video features of each third target video frame and the first visual feature obtained by extracting the video features of the first target video frame are fused to obtain the fused first visual feature. The text description features obtained from the second event-level description and the first frame-level description, as well as the text description features obtained from the second frame-level description, are fused to obtain the fused text description features.

[0010] In this embodiment of the application, a pre-trained relevance scoring model is used to analyze the relevance between each video frame in the video to be analyzed and the user's question, and to obtain the first relevance score between the user's question and each video frame, including: Based on the video to be analyzed, the second visual features of the video to be analyzed are extracted; Based on the user's question, we extract the textual features of the user's question; The second visual feature is converted into a query representation of the second visual feature through a pre-trained visual feature conversion model. By using a pre-trained relevance scoring model, the relevance between each video frame in the video to be analyzed and the user's question is analyzed based on the query representation and text features, and the first relevance score between the user's question and each video frame is obtained.

[0011] In this embodiment of the application, a pre-trained video question-answering prediction model generates answers to user questions based on first visual features, text description features, and the user question, including: Based on the characteristics of user questions and text descriptions, determine the fourth relevance score between user questions and each text description feature; Based on the fourth relevance score and a preset threshold, text description features with a fourth relevance score greater than or equal to the preset threshold are selected as target text description features. Based on a pre-trained video question answering prediction model, answers to user questions are generated according to first visual features, target text description features, and user questions.

[0012] In this embodiment of the application, text description features with a fourth relevance score greater than or equal to a preset threshold are selected as target text description features based on a fourth relevance score and a preset threshold, including: Based on the user's question, determine the question type, which includes how the question is categorized, what the question is, and why the question is categorized. Based on the type of user problem, determine the preset threshold corresponding to the problem type; Based on the fourth relevance score and the preset threshold corresponding to the question type, text description features with a fourth relevance score greater than or equal to the preset threshold corresponding to the question type are selected as target text description features.

[0013] In this embodiment of the application, a preset threshold corresponding to the problem type is determined based on the problem type of the user's problem, including: When the question type is "How Question Type", the preset threshold is determined to be the first preset threshold. Given the type of the user's question, the second preset threshold is determined. When the user's question type is "Why Question Type", the preset threshold is determined to be the third preset threshold, wherein the first preset threshold is greater than the second preset threshold, and the second preset threshold is greater than or equal to the third preset threshold.

[0014] A second aspect of this application provides a video question-and-answer device, the device comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing the instructions, to implement the video question-and-answer method provided in the first aspect above.

[0015] The above technical solution utilizes a pre-trained relevance scoring model to perform coarse-grained filtering of each frame of the video to be analyzed and the user's question, obtaining a first relevance score and selecting the first target video frame with a high score, thus significantly reducing irrelevant visual redundancy. Based on the first relevance score and a preset scoring threshold, different preset feature extraction paths are dynamically selected, thereby adaptively generating accurate first visual features and textual description features. The pre-trained video question-answering prediction model integrates the first visual features and textual description features with the user's question to generate an answer, thereby significantly improving fine-grained semantic alignment capability and question-answering accuracy.

[0016] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings: Figure 1 The illustration shows a flowchart of a video question-and-answer method according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0019] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0020] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0021] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0022] Figure 1 The illustration schematically shows a flowchart of a video question-and-answer method according to an embodiment of this application. For example... Figure 1 As shown in the figure, this application provides a video question-and-answer method, which may include the following steps: Step S110: Obtain the video to be analyzed and the user questions raised by the user based on the video to be analyzed; Step S120: Analyze the correlation between each video frame and the user question in the video to be analyzed using a pre-trained correlation scoring model, and obtain the first correlation score between the user question and each video frame. Step S130: Sort the video frames from high to low based on the first relevance score, and select the first target video frames with the first preset number of frames at the top of the sort; Step S140: Generate first visual features and text description features based on the first relevance score of the first target video frame, the preset score threshold, and the preset feature extraction path; Step S150: Based on the pre-trained video question answering prediction model, generate the answer to the user's question according to the first visual features, text description features and the user's question.

[0023] In step S110, the video to be analyzed is first acquired. The video to be analyzed can be a dynamic scene recording of any length. At the same time, natural language questions raised by the user regarding the video to be analyzed are also acquired. The questions may involve inquiries about objects, actions, events, or causal relationships in the video to be analyzed.

[0024] In step S120, the video to be analyzed comprises multiple video frames, and each video frame in the video to be analyzed undergoes preliminary screening. Specifically, a pre-trained relevance scoring model is used to extract the visual features of each video frame and the textual features of the question, and a first relevance score is calculated between each video frame and the question. The first relevance score is a semantic relevance score. The first relevance score is used to measure whether a video frame contains key information related to the question, thereby providing a basis for subsequent keyframe screening and avoiding redundant calculations caused by global uniform sampling. Illustratively, the relevance scoring model can be a combination of a Q-Former (Querying Transformer) model and a frozen large language model.

[0025] In step S130, after obtaining the first relevance score between each video frame and the problem, all video frames are sorted from high to low according to the first relevance score, and a first preset number of video frames with the highest scores are selected as the initial keyframes (i.e., the first target video frames). This process achieves coarse-grained screening of video content, effectively narrowing the visual range of subsequent processing and reducing interference from background frames unrelated to the problem. Indicatively, the first preset number can be set according to actual needs, and this embodiment does not impose a specific limitation.

[0026] In step S140, based on the relevance scores of the initially selected original keyframes (i.e., the first target video frame), the first target video frame is assigned to a preset feature extraction path using a preset scoring threshold. The preset feature extraction path generates corresponding first visual features and textual description features, achieving multi-granularity and adaptive extraction of visual information. Indicatively, the preset scoring threshold can be set according to actual needs, and this embodiment does not impose specific limitations.

[0027] In step S150, the generated first visual features are fused with the text description features and input into a pre-trained video question-answering prediction model. Combining the user's question with preset prompts, an answer to the user's question is generated. As an example, the video question-answering prediction model can be a Flan-T5 (Fine-tuned Language Net - Text-to-TextTransfer Transformer) model.

[0028] In this embodiment, a pre-trained relevance scoring model is used to perform coarse-grained filtering of each frame of the video to be analyzed and the user's question to obtain a first relevance score and select the first target video frame with a high score, which greatly reduces irrelevant visual redundancy. Different preset feature extraction paths are dynamically selected based on the first relevance score and a preset scoring threshold, thereby adaptively generating accurate first visual features and text description features. The pre-trained video question answering prediction model integrates the first visual features and text description features with the user's question to generate an answer, thereby significantly improving fine-grained semantic alignment capability and question answering accuracy.

[0029] Furthermore, step S140 may include the following steps: Step S141a: When the first relevance scores of all first target video frames are less than the first preset score threshold, the event clustering process is performed on each video frame in the video to be analyzed by a clustering algorithm to obtain a semantically coherent first event set. Step S142a: Generate a first event-level description for each event in the first event set; Step S143a: Determine the second relevance score of each first event-level description to the user's question; Step S144a: Determine the target event set based on multiple second relevance scores and a third preset scoring threshold; Step S145a: Determine the second target video frame based on the center frame of each event in the target event set; Step S146a: Extract the video features of each second target video frame as the first visual feature; Step S147a: Extract the event-level descriptions of the target event set as text descriptions.

[0030] In step S141a, for the extreme case where the first target video frame screening fails—that is, all the first target video frames have a first relevance score to the question that is less than the preset lower limit score threshold—indicating that the coarse-grained screening has failed to locate effective clues, the original keyframes are no longer relied upon. Instead, global video analysis is employed, using a clustering algorithm to perform unsupervised event segmentation on the entire video frame sequence. Visually similar and temporally coherent video frames are aggregated into semantically complete event segments, forming the first event set. As an example, the clustering algorithm can be the DPC-KNN clustering algorithm.

[0031] In step S142a, a first event-level description is generated for each event in the first event set. After obtaining a semantically coherent event set, a corresponding text description, i.e., a first event-level description, is generated for each event. The first event-level description typically uses a visual language model to understand and summarize the continuous frame sequence within the event, transforming dynamic visual content into natural language expression, covering the actions, objects, and their temporal relationships within the event.

[0032] In step S143a, each first event-level description is input into the pre-trained language model along with the user's question, and the cosine similarity between the two is calculated, which is the second relevance score. The second relevance score measures whether the first event-level description matches the question's intent in terms of content, reflecting whether the event is likely to contain the key information needed to answer the question. Through this evaluation, candidate events highly relevant to the question can be screened at the event level, compensating for the lack of frame-level localization.

[0033] In step S144a, events with second relevance scores higher than a third preset score threshold are selected based on the second relevance score to form a target event set. Events in the target event set are considered semantically highly relevant to the question and are an important source of information for generating subsequent answers. For example, events with second relevance scores higher than the third preset score threshold from among multiple events can be identified as the target event set; if the second relevance scores of all events in the first event set do not exceed the third preset score threshold, then all events in the first event set are identified as the target event set.

[0034] In step S145a, after determining the target event set, a center frame is selected from each event in the target event set as the second target video frame. The center frame is usually located at the midpoint of the event's time, which can better represent the core visual content of the event and avoid the semantic incompleteness problem that may be caused by selecting event boundary frames.

[0035] In step S146a, visual features are extracted from the selected second target video frames (i.e., the center frames of each target event). A pre-trained visual encoder can be used to convert the frame images into high-dimensional feature vectors, which constitute the first visual features. The visual features obtained through event-level relocalization effectively compensate for the failure of the initial frame-level retrieval. As an example, the visual encoder can be a ViT (Vision Transformer) model.

[0036] In step S147a, the event-level description of each event in the target event set is used as a text description feature. Corresponding to the visual features, these text descriptions provide event-level semantic information, which can more completely reflect the action sequence and contextual relationships related to the problem in the video.

[0037] The embodiments of this application effectively solve the problem of visual cues being lost when coarse-grained screening fails. Through event-level semantic analysis and relocation, the previously unusable fragmented video frame information is transformed into an event-level representation with a complete temporal context, realizing the leap from frame-level localization failure to event-level relocation. This significantly enhances the robustness and fine-grained semantic capture capability when facing complex, ambiguous, or long-temporal problems.

[0038] Furthermore, step S140 may include the following steps: Step S141b: If any first relevance score is greater than or equal to the first preset score threshold and less than the second preset score threshold, perform event clustering processing on each video frame in the video to be analyzed using a clustering algorithm to obtain a semantically coherent second event set. Step S142b: Select a second preset number of video frames from the first target video frame to obtain the second target video frame; Step S143b: Match the second target video frame with the events in the second event set to determine the event to which the second target video frame belongs. The event to which the video frame belongs includes one or more video frames. Step S144b: In each attribution event, determine the third relevance score of each second target video frame to the user question; Step S145b: Sort each second target video frame from high to low based on the third relevance score, and select the top-ranked second target videos as the third target video frames; Step S146b: Extract the text description of each third target video frame as the second event-level description; Step S147b: Generate a first frame-level description based on the second target video frame; Step S148b: Extract the video features of each third target video frame as the first visual feature; Step S149b: Obtain text description features based on the second event-level description and the first frame-level description.

[0039] In step S141b, for scenarios where the keyframe score is in the middle range, i.e., there exists a first relevance score for any first target video frame that is greater than or equal to a first preset score threshold and less than a second preset score threshold, it is determined that although the first target video frame has some relevance, its semantic information may be incomplete, requiring supplementation of temporal context. Therefore, a clustering algorithm is used to perform event segmentation on all video frames of the video to be analyzed, aggregating temporally coherent and visually similar frames into semantically complete event segments, forming a second event set.

[0040] In step S142b, a second preset number of video frames are selected from the first target video frames to obtain the second target video frames. In one example, the semantic relevance of each first target video frame to the user's question is calculated, and a second preset number of video frames with higher semantic relevance are selected from multiple semantic relevance pairs as the second target video frames.

[0041] In step S143b, after obtaining the second event set, each second target video frame is matched with the events in the second event set to determine the event to which the second target video frame belongs.

[0042] In step S144b, the semantic relevance between each second target video frame and the user question in each attribution event is recalculated to obtain a third relevance score.

[0043] In step S145b, within each attribution event, a second preset number of video frames (i.e., context windows) with the highest relevance to the user's question are selected as the third target video frames. This mechanism directly filters the most valuable context frames based on semantic relevance within each attribution event, rather than sampling from a fixed window based on temporal position, thereby ensuring that the supplementary context frames are both temporally relevant and semantically accurate.

[0044] In step S146b, a text description is generated for the selected third target video frame, forming a second event-level description. Since the third target video frame constitutes a contextualized frame set surrounding the original keyframe, the second event-level description can reflect the dynamic process, action sequence, and temporal relationship in the event segment, thereby providing rich temporal contextual semantic support for the original keyframe, enabling a previously isolated frame to obtain a complete event-level understanding.

[0045] In step S147b, a single-frame-level text description, i.e., a first-frame-level description, is directly generated for each second target video frame.

[0046] In step S148b, visual features are extracted for each third target video frame, and the spatial structure, object information and scene layout within the third target video frame are encoded into high-dimensional feature vectors to form the first visual features.

[0047] In step S149b, the second event-level description and the first frame-level description are integrated to form the final text description features. The event-level description provides temporal context and dynamic semantics, while the frame-level description retains static, fine-grained information. The combination of the two enables the text description features to possess both the ability to understand complex actions and processes and the ability to accurately depict local details, thus providing semantically rich and hierarchically distinct textual support for subsequent question-answering prediction.

[0048] This application's embodiments break through the rigid mode of traditional fixed window sampling. By using event-level semantic boundary localization and frame-level filtering guided by relevance, it achieves accurate supplementation of the temporal context of the original key frames. This avoids visual noise introduced by redundant frames and ensures the temporal integrity required to understand dynamic processes, significantly improving the model's ability to capture complex actions and fine-grained semantics.

[0049] Furthermore, step S140 may include the following steps: Step S141c: If any first correlation score is greater than or equal to the first preset score threshold and greater than or equal to the second preset score threshold, extract the video features of the first target video frame as the first visual features; Step S142c: Generate a second frame-level description based on the first target video frame; Step S143c: Obtain text description features based on the second frame-level description.

[0050] In step S141c, if any first relevance score is greater than or equal to a first preset score threshold and not lower than a second preset score threshold, it is determined that the first target video frame contains sufficient and accurate visual evidence, without the need for additional contextual supplementation or event relocation. Therefore, the deep visual representation of each first target video frame is directly extracted through the visual encoder and used as the first visual feature to provide high-confidence visual input for the subsequent question-answering prediction module, ensuring the efficiency and simplicity of the reasoning process. As an example, the visual encoder can be a ViT (Vision Transformer) model.

[0051] In step S142c, a corresponding single-frame text description, i.e., a second-frame-level description, is generated for each first target video frame. The second-frame-level description focuses on static visual content within the frame, such as key objects, scene attributes, or local details, converting visual information into natural language expressions. By preserving frame-level semantic information, it provides fine-grained textual clues that correspond to visual features for the question-answering prediction module, which is particularly suitable for question types that do not require temporal reasoning, such as asking about object attributes, colors, and locations.

[0052] In step S143c, the second frame-level description is integrated and encoded to form the final text description feature. The text description feature carries the visual semantic information in the highly relevant keyframe in the form of a text modality, forming a multimodal complement with the first visual feature.

[0053] This application's embodiments skip the complex process of context supplementation and event relocation, retaining the most essential visual evidence through the simplest path, avoiding redundant calculations and the introduction of irrelevant information, significantly improving processing efficiency while ensuring the accuracy of question answering, and achieving effective avoidance of visual information overload and precise alignment of fine-grained semantics.

[0054] In steps S141a-S147a, S141b-S149b, and S141c-S143c, different visual localization and description generation paths are employed for different paths: for low-matching frames, clustering algorithms are used for event-level relocalization; for medium-matching frames, contextual descriptions are supplemented through question-guided visual contextualization; and for high-matching frames, frame-level descriptions are directly generated. Through the organic integration of these three feature extraction paths, the optimal processing strategy can be dynamically selected based on the degree of matching between visual evidence and the question. This avoids visual information overload while ensuring the flexibility of temporal modeling, significantly improving the overall performance of video question answering in terms of fine-grained semantic capture and overall accuracy.

[0055] Furthermore, embodiments of this application may also include the following steps: Step S210: If any first correlation score is greater than or equal to the first preset score threshold and less than the second preset score threshold, and any first correlation score is greater than or equal to the first preset score threshold and greater than or equal to the second preset score threshold, the first visual feature obtained by extracting the video features of each third target video frame and the first visual feature obtained by extracting the video features of the first target video frame are fused to obtain the fused first visual feature. Step S220: The text description features obtained based on the second event-level description and the first frame-level description are fused together to obtain the fused text description features.

[0056] In step S210, if the first relevance score of any first target video frame is between the first preset score threshold and the second preset score threshold (i.e., the middle matching path), and at the same time there is another first target video frame whose first relevance score is greater than or equal to the second preset score threshold (i.e., the high matching path), the first visual feature obtained by extracting the video features of each third target video frame from the middle matching path is fused with the first visual feature obtained by directly extracting the video features of the first target video frame from the high matching path to obtain the fused first visual feature.

[0057] In step S220, when the middle matching path and the high matching path coexist, the text description features obtained by fusing the middle matching path according to the second event-level description and the first frame-level description are fused with the text description features obtained from the high matching path according to the second frame-level description to obtain the fused text description features.

[0058] In steps S210-S220, multi-source fusion of visual and textual features is achieved when medium-matching paths and high-matching paths coexist. This allows the video question-answering prediction model to be pre-trained with the fused first visual features and the fused textual description features to generate answers to user questions. This avoids information loss caused by a single path selection, provides the question-answering prediction module with a richer and more robust multimodal representation, and significantly improves the model's semantic understanding and reasoning ability in complex question-answering scenarios.

[0059] Furthermore, step S120 may include the following steps: Step S121: Extract the second visual features of the video to be analyzed; Step S122: Extract the text features of the user's question based on the user's question; Step S123: Convert the second visual feature into a query representation of the second visual feature using a pre-trained visual feature conversion model; Step S124: Using a pre-trained relevance scoring model, the relevance between each video frame in the video to be analyzed and the user's question is analyzed based on the query representation and text features, and the first relevance score between the user's question and each video frame is obtained.

[0060] In step S121, the pre-trained visual encoder extracts features from each frame of the video to be analyzed, mapping the visual content in the original pixel space into a high-dimensional semantic feature vector to form a second visual feature. For example, the visual encoder can be a ViT (Vision Transformer) model.

[0061] In step S122, the user question is semantically encoded using a pre-trained text encoder, converting the natural language question into a high-dimensional text feature vector. The text feature representation of the user question captures the semantic intent, keyword information, and syntactic structure of the question. For example, the text encoder can be a Flan-T5 model.

[0062] In step S123, the second visual features are converted into a compact representation relevant to the question query using a pre-trained visual feature transformation model. As an example, the visual feature transformation model can be a Q-Former model. Q-Former interacts with visual features through learnable query vectors, compressing the original high-dimensional visual features into a task-oriented query representation. This allows the visual information to be better aligned with the question semantics in the subsequent language model, providing semantically adapted visual input for relevance scoring calculation.

[0063] In step S124, the visual query representation and the textual features of the user's question are input into a pre-trained relevance scoring model. Combined with preset scoring prompts, the semantic matching degree between each video frame and the question is calculated, outputting a first relevance score. This first relevance score quantifies whether each video frame contains the key information required for the answer, providing a reliable ranking basis for subsequent keyframe selection and achieving preliminary semantic alignment between visual information and question intent. As an example, the relevance scoring model can be a frozen Flan-T5 model.

[0064] In this embodiment, a question-oriented compression of visual features is achieved by introducing a learnable query vector. This transforms high-dimensional redundant visual information into a compact representation that is semantically aligned with the user's question. At the same time, relevance evaluation is performed, providing accurate and reliable semantic anchors for subsequent keyframe screening. This effectively solves the problem of inaccurate positioning caused by visual information overload in traditional methods.

[0065] Furthermore, step S150 may include the following steps: Step S151: Based on the user question and text description features, determine the fourth relevance score between the user question and each text description feature; Step S152: Based on the fourth relevance score and the preset threshold, select text description features with a fourth relevance score greater than or equal to the preset threshold as target text description features; Step S153: Based on the pre-trained video question answering prediction model, generate the answer to the user's question according to the first visual features, the target text description features, and the user's question.

[0066] In step S151, the user question and each text description feature are input into a pre-trained language model, and the semantic matching degree between them is calculated to obtain a fourth relevance score. The fourth relevance score is used to measure the strength of the content-related association between each text description feature and the question intent.

[0067] In step S152, the text description features are filtered by a preset threshold, and only the text description features with a fourth relevance score higher than the preset threshold are retained to form the target text description features.

[0068] In step S153, the filtered target text description features and the first visual features are fused in a multimodal manner, and the answer to the user's question is output by combining the user's question input with the pre-trained video question answering prediction model.

[0069] In this embodiment, the problem of information redundancy and semantic misalignment caused by non-question-oriented description generation is effectively solved by filtering with a preset threshold. It achieves accurate multimodal fusion of visual evidence and textual clues, significantly improves the model's question-answering accuracy in fine-grained reasoning tasks, and especially enhances its ability to understand complex dynamic processes.

[0070] Furthermore, step S152 may include the following steps: Step S1521: Based on the user's question, determine the question type of the user's question, where the question type includes how to ask the question, what the question type is, and why the question type is asked. Step S1522: Determine the preset threshold corresponding to the problem type based on the problem type of the user's problem; Step S1523: Based on the fourth relevance score and the preset threshold corresponding to the question type, filter the text description features whose fourth relevance score is greater than or equal to the preset threshold corresponding to the question type as target text description features.

[0071] In step S1521, the user's questions are semantically classified to identify the question type. Specifically, question types can be divided into "How," "What," and "Why" types. Different types of questions have fundamentally different requirements for the granularity of visual information: "How" questions emphasize the temporal integrity of action sequences and dynamic processes; "What" questions focus on the static recognition of entities, objects, or scenes; and "Why" questions involve causal reasoning and logical connections.

[0072] In step S1522, after determining the question type, corresponding preset thresholds are assigned to different types according to preset mapping rules. Specifically, a stricter filtering threshold is set for "how" type questions to retain high-value descriptions rich in action details and eliminate redundant background information; a relatively lenient threshold is set for "what" and "why" type questions to ensure that necessary entity information and causal clues are not missed. This differentiated threshold setting mechanism allows description filtering to adaptively match the dynamic needs of the question for information granularity.

[0073] In step S1523, the fourth relevance score is compared with a predetermined threshold corresponding to the question type. Only text description features with a fourth relevance score not lower than the predetermined threshold are retained to form target text description features. Through this type-adaptive filtering mechanism, for "how" type questions, only the core description highly related to the action process is retained; for "what" type questions and "why" type questions, a wider range of semantic cues are retained, thereby achieving precise alignment between the description filtering granularity and the question intent.

[0074] This application embodiment achieves adaptive alignment between description filtering granularity and problem intent by identifying the problem type to which the user's question belongs, dynamically determining the corresponding preset threshold, and filtering text description features based on the fourth relevance score, thus accurately eliminating irrelevant and redundant information.

[0075] Furthermore, step S1522 may include the following steps: Step S1522a: If the question type is "how question type", determine the preset threshold as the first preset threshold; Step S1522b: Given the type of the user's question, determine the second preset threshold as the preset threshold. Step S1522c: When the user's question type is "why question type", determine the preset threshold as the third preset threshold, wherein the first preset threshold is greater than the second preset threshold, and the second preset threshold is greater than or equal to the third preset threshold.

[0076] In steps S1522a-S1522c of this embodiment, a differentiated threshold setting mechanism dynamically matches the corresponding preset thresholds based on the question type of the user's question. Since "how" type questions typically focus on the execution method, process details, and temporal logic of actions, they require the highest granularity of information description and need to retain high-value descriptions rich in action sequences and dynamic changes. Therefore, a first preset threshold is set for "how" type questions to achieve the strictest description filtering and retain high-value action details. A second preset threshold is set for "what" type questions to balance entity information integrity and background removal. A third preset threshold is set for "why" type questions to leniently retain the semantic context required for causal reasoning. Through this hierarchical setting where the first preset threshold is greater than the second preset threshold, and the second preset threshold is greater than or equal to the third preset threshold, precise alignment between description filtering granularity and question intent is achieved, effectively solving the problems of information redundancy and semantic misalignment caused by non-question-oriented description generation. Illustratively, the specific values ​​of the first, second, and third preset thresholds can be set according to actual needs, and this embodiment does not impose specific limitations on them.

[0077] In this embodiment, the first preset quantity can be set to 4, the first preset scoring threshold can be set to 0.2, the second preset scoring threshold can be set to 0.35, the third preset scoring threshold can be set to 0.35, the second preset quantity can be set to 6, the third preset quantity can be dynamically determined based on the value of the first preset quantity and the scoring distribution, and the first preset threshold can be set to 0.1. The aforementioned parameter values ​​are preferred values ​​verified by experiments, and this application does not limit the specific parameter values.

[0078] This application embodiment also provides a video question-and-answer device, the device comprising: The memory is configured to store instructions; The processor is configured to retrieve instructions from memory and, when executing instructions, to implement the video question-and-answer method described above.

[0079] It is understood that the video question-and-answer device provided in this application embodiment can implement each process of the video question-and-answer method in the above embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0085] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0086] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0087] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0088] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A video question-answering method, characterized in that, The method includes: Acquire the video to be analyzed and user questions raised by users based on the video to be analyzed; By using a pre-trained relevance scoring model, the relevance between each video frame in the video to be analyzed and the user question is analyzed, and the first relevance score between the user question and each video frame is obtained. Based on the first correlation score, the video frames are sorted from high to low, and a first preset number of first target video frames that are ranked first are selected. Based on the first relevance score of the first target video frame, the preset score threshold, and the preset feature extraction path, generate the first visual feature and the text description feature. Based on a pre-trained video question answering prediction model, an answer to the user question is generated according to the first visual feature, the text description feature, and the user question.

2. The method according to claim 1, characterized in that, The step of generating first visual features and textual description features based on the first relevance score of the first target video frame, a preset score threshold, and a preset feature extraction path includes: If the first relevance scores of the first target video frame are all less than the first preset score threshold, the video frames in the video to be analyzed are subjected to event clustering processing by a clustering algorithm to obtain a semantically coherent first event set. Generate a first event-level description for each event in the first event set; Determine a second relevance score for each of the first event-level descriptions to the user's question; The target event set is determined based on multiple second relevance scores and a third preset scoring threshold; Based on the center frame of each event in the target event set, determine the second target video frame; Extract the video features of each second target video frame as the first visual feature; Extract the event-level descriptions from the target event set and use them as the text descriptions.

3. The method according to claim 1, characterized in that, The step of generating first visual features and textual description features based on the first relevance score of the first target video frame, a preset score threshold, and a preset feature extraction path includes: If any of the first relevance scores is greater than or equal to the first preset score threshold and less than the second preset score threshold, an event clustering process is performed on each video frame in the video to be analyzed by a clustering algorithm to obtain a semantically coherent second event set. A second preset number of video frames are selected from the first target video frame to obtain a second target video frame; The second target video frame is matched with events in the second event set to determine the event to which the second target video frame belongs, wherein the event to which the video frame belongs includes one or more video frames; In each attribution event, a third relevance score is determined for each of the second target video frames and the user question; Based on the third correlation score, each second target video frame is sorted from high to low, and the top-ranked second target videos are selected as the third target video frames. Extract the text description of each of the third target video frames as a second event-level description; Generate a first frame-level description based on the second target video frame; Extract the video features of each of the third target video frames as the first visual features; The text description features are obtained based on the second event-level description and the first frame-level description.

4. The method according to claim 1, characterized in that, The step of generating first visual features and textual description features based on the first relevance score of the first target video frame, a preset score threshold, and a preset feature extraction path includes: If any of the first correlation scores is greater than or equal to the first preset score threshold and greater than or equal to the second preset score threshold, the video features of the first target video frame are extracted as the first visual features. Generate a second frame-level description based on the first target video frame; The text description features are obtained based on the second frame-level description.

5. The method according to claim 1, characterized in that, The method further includes: If any of the first correlation scores is greater than or equal to the first preset score threshold and less than the second preset score threshold, and if any of the first correlation scores is greater than or equal to the first preset score threshold and greater than or equal to the second preset score threshold, the first visual features obtained by extracting the video features of each third target video frame and the first visual features obtained by extracting the video features of the first target video frame are fused to obtain the fused first visual features. The text description features obtained from the second event-level description and the first frame-level description, as well as the text description features obtained from the second frame-level description, are fused to obtain the fused text description features.

6. The method according to claim 1, characterized in that, The process involves using a pre-trained relevance scoring model to analyze the relevance between each video frame in the video to be analyzed and the user question, obtaining a first relevance score between the user question and each video frame, including: Based on the video to be analyzed, the second visual features of the video to be analyzed are extracted; Based on the user's question, the text features of the user's question are extracted; The second visual feature is converted into a query representation of the second visual feature using a pre-trained visual feature conversion model; Using a pre-trained relevance scoring model, the relevance of each video frame in the video to be analyzed to the user question is analyzed based on the query representation and the text features, and the first relevance score of the user question to each video frame is obtained.

7. The method according to claim 1, characterized in that, The pre-trained video question-answering prediction model generates an answer to the user question based on the first visual feature, the text description feature, and the user question, including: Based on the user question and the text description features, a fourth relevance score is determined between the user question and each of the text description features; Based on the fourth relevance score and the preset threshold, text description features with a fourth relevance score greater than or equal to the preset threshold are selected as target text description features. Based on a pre-trained video question-answering prediction model, an answer to the user question is generated according to the first visual feature, the target text description feature, and the user question.

8. The method according to claim 7, characterized in that, The step of selecting text description features whose fourth relevance score is greater than or equal to the preset threshold as target text description features based on the fourth relevance score and the preset threshold includes: Based on the user's question, determine the question type of the user's question, wherein the question type includes how to ask the question, what the question type is, and why the question type is. Based on the type of the user's question, determine a preset threshold corresponding to the question type; Based on the fourth relevance score and the preset threshold corresponding to the question type, text description features with a fourth relevance score greater than or equal to the preset threshold corresponding to the question type are selected as target text description features.

9. The method according to claim 8, characterized in that, The step of determining a preset threshold corresponding to the problem type based on the problem type of the user problem includes: When the question type is "how question type", the preset threshold is determined to be the first preset threshold; Given the type of the user's question, the preset threshold is determined to be the second preset threshold. When the user's question type is "Why Question Type", the preset threshold is determined to be the third preset threshold, wherein the first preset threshold is greater than the second preset threshold, and the second preset threshold is greater than or equal to the third preset threshold.

10. A video question-and-answer device, characterized in that, The device includes: The memory is configured to store instructions; A processor is configured to retrieve the instructions from the memory and, when executing the instructions, to implement the video question-answering method according to any one of claims 1 to 9.

Images