Video question answering method and device

Abstract

The present disclosure provides a video question answering method and device, electronic equipment, storage medium and computer program product, relates to the technical field of artificial intelligence, in particular to the technical field of large model, agent, natural language understanding, computer vision, multi-modal information processing and the like, and can be applied to a video question answering scene. The specific implementation scheme is as follows: determining an organization mode of response evidence of a question in a video; determining candidate response evidence of the question from a scene representation library including representation data of scenes in the video by using a search strategy corresponding to the organization mode; and generating response data of the question according to the candidate response evidence. The present disclosure analyzes the organization mode of the response evidence required by the question in the video, selects a corresponding evidence search strategy to obtain candidate response evidence for response, improves the accuracy and comprehensiveness of the candidate response evidence, and the adaptation degree between the candidate response evidence and the question, and further improves the accuracy and comprehensiveness of the response data.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the technical fields of large models, intelligent agents, natural language understanding, computer vision, and multimodal information processing, and in particular to a video question answering method, device, electronic device, storage medium, and computer program product, which can be applied to video question answering scenarios. Background Technology

[0002] With the rapid growth of long-form video data in scenarios such as teaching, surveillance, film and television, sports events, live replays, and vehicle recording, long-form video question answering has become an important application requirement for multimodal intelligence. Since long videos typically contain a large number of visual frames and continuous audio information, and answering specific questions often relies on only a small amount of key evidence, existing methods usually employ a uniform retrieval strategy to obtain candidate segments from the video. However, the required evidence structure varies significantly across different questions, and a uniform retrieval strategy can easily lead to problems such as insufficient evidence coverage, overly concentrated temporal distribution of candidate segments, or an imbalance of evidence among candidate answers. Summary of the Invention

[0003] This disclosure provides a video question-answering method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to the first aspect, a video question answering method is provided, comprising: determining the organization pattern of the response evidence for a question in the video; using a retrieval strategy corresponding to the organization pattern to determine candidate response evidence for the question from a scene representation library including representation data of the scene in the video; and generating response data for the question based on the candidate response evidence.

[0005] According to a second aspect, a video question-answering device is provided, comprising: a pattern determination unit configured to determine the organization pattern of response evidence for a question in the video; an evidence determination unit configured to determine candidate response evidence for the question from a scene representation library including representation data of the scene in the video using a retrieval strategy corresponding to the organization pattern; and a response unit configured to generate response data for the question based on the candidate response evidence.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0009] According to the technology disclosed herein, a video question-answering method and apparatus are provided. The method determines the organization pattern of the response evidence for a question in the video; employs a retrieval strategy corresponding to the organization pattern to identify candidate response evidence for the question from a scene representation library including scene representation data in the video; and generates response data for the question based on the candidate response evidence. This method specifically analyzes the organization pattern of the required response evidence for the question in the video and selects an appropriate evidence retrieval strategy to obtain candidate response evidence for answering, thereby improving the accuracy and comprehensiveness of the candidate response evidence, as well as the fit between the candidate response evidence and the question, and ultimately improving the accuracy and comprehensiveness of the response data.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure; Figure 2 This is a flowchart of an embodiment of the video question-and-answer method according to the present disclosure; Figure 3 This is a schematic diagram of the framework of the video question-answering system according to this embodiment; Figure 4 This is a schematic diagram illustrating an application scenario of the video question-and-answer method according to this embodiment; Figure 5 This is a flowchart of yet another embodiment of the video question-and-answer method according to the present disclosure; Figure 6 This is a structural diagram of one embodiment of the video question-and-answer device according to the present disclosure; Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] The technical solutions disclosed herein involve the collection, storage, use, processing, transmission, provision, and disclosure of various types of information, such as user personal information, in accordance with relevant laws and regulations and do not violate public order and good morals.

[0014] Figure 1 An exemplary architecture 100 is shown that can be applied to the video question-answering method and apparatus disclosed herein.

[0015] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0016] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data acquisition, interaction, and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0017] Server 105 can be a server that provides various services, such as analyzing the organization pattern of the required response evidence in the video for videos and questions uploaded by terminal devices 101, 102, and 103, and selecting appropriate evidence retrieval strategies to obtain candidate response evidence for response. As an example, server 105 can be a cloud server.

[0018] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0019] It should also be noted that the video question-answering method provided in the embodiments of this disclosure is generally executed by a server, but the possibility of it being executed by a terminal device, or by a combination of the server and the terminal device, is not excluded. Accordingly, all the components (e.g., units) of the video question-answering device can be located entirely in the server, entirely in the terminal device, or separately in the server and the terminal device.

[0020] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. When the electronic devices on which the video question-answering method runs do not require data transmission with other electronic devices, the system architecture may only include the electronic devices on which the video question-answering method runs (e.g., servers or terminal devices).

[0021] Please refer to Figure 2 , Figure 2 A flowchart illustrating a video question-and-answer method provided in this embodiment of the disclosure. (Continue to refer to...) Figure 3 The diagram illustrates the framework of a video question-answering system. Process 200 includes the following steps: Step 201: Determine the organization pattern of the response evidence to the video questions in the video.

[0022] In this embodiment, the execution entity of the video question-answering method (e.g., Figure 1 The server in the video acquires video remotely or locally via a wired or wireless network connection and determines the organization pattern of the response evidence in the video to the questions raised.

[0023] Video generally refers to digital multimedia data that uses a continuous sequence of image frames as its carrier and may contain synchronous or asynchronous audio streams. In terms of type, it encompasses surveillance footage, films and television dramas, recorded lessons, meeting minutes, sports broadcasts, UGC (User-Generated Content) short videos, and industrial inspection videos, among others. In terms of content, videos typically contain multimodal information elements such as people, objects, scenes, actions, events, text labels, voice dialogues, and environmental sound effects. These elements dynamically evolve and semantically relate to each other over time. For example, a 30-minute recorded lesson video includes visual content such as the teacher's blackboard writing and PowerPoint slide transitions, as well as auditory content such as explanations and interactive questions, and also implicitly contains the logical order and temporal rhythm of the knowledge points being presented.

[0024] A question related to a video refers to a query request made by a user or system in natural language or other structured form, aiming to extract specific information from the video. This question can revolve around dimensions such as people, objects, events, time, place, cause, and effect in the video. It can be a factual question (e.g., "When did the person in the red coat enter the room?"), an inference question (e.g., "What was the key turning point in the score reversal between the two players?"), or a summary question (e.g., "What resolutions were reached in this meeting?"). The form of the question is not limited to interrogative sentences; it can also be a combination of keywords, an instruction phrase, or a semantic fragment. Its core is to express a clear intention to extract, integrate, or infer information from the video content.

[0025] Response evidence refers to the smallest semantic unit or set of semantic units in a video that can directly support, partially support, or reasonably infer an answer to a question. This evidence is not the original pixels or audio waveform itself, but rather a fragment of content that can be recognized and referenced after semantic analysis. Its forms include, but are not limited to: a visual scene within a certain time period, a speech-to-text transcript, a subtitle entry, a sequence of actions, a text label, or a composite information body composed of the above elements under a specific temporal and logical relationship. Response evidence must possess the following characteristics: locatability (can be mapped to a specific spatiotemporal range in the video), interpretability (can establish a semantic connection with the question), and verifiability (its existence and meaning can be confirmed through the video content itself).

[0026] Organizational patterns refer to the structural distribution of response evidence within the video's timeline and semantic space, reflecting the temporal arrangement, semantic aggregation, and logical connection among evidentiary elements. This pattern is an abstract characterization of the inherent organizational characteristics of evidence, rather than a specification of external retrieval behavior or processing procedures. For example, if the response evidence is concentrated within a short segment of the video and semantically highly focused, its organizational pattern can be characterized as "focused"; if the response evidence is scattered across multiple discontinuous time periods but collectively constitutes a complete answer, its organizational pattern can be characterized as "covered"; if the response evidence reflects a comparative relationship between two or more sets of content in terms of attributes, states, or behaviors, its organizational pattern can be characterized as "contrastive". Determining the organizational pattern essentially involves identifying and classifying the structural features of information in the video, providing a basis for subsequently adapting differentiated evidence extraction paths.

[0027] As an example, by analyzing the matching relationship between the semantic features of the questions and the statistical characteristics of the video content, the organization pattern of the response evidence in the video can be determined.

[0028] First, semantic analysis is performed on the question to extract semantic slots such as key entities, action predicates, time constraints, quantity limits, and logical connectors, thereby determining the type of information requirement. For example, if the question contains words such as "separately," "each," or "different," or involves a comparison of attributes of two or more subjects, it is preliminarily determined that the corresponding response evidence may have a comparative organizational tendency; if the question contains words such as "whole process," "each stage," or "from beginning to end," or requires listing multiple discrete events, it is preliminarily determined that it may have a comprehensive organizational tendency; if the question points to a single event, a specific moment, or a single object, it is preliminarily determined that it may have a focused organizational tendency.

[0029] Secondly, the video content is preprocessed and analyzed to obtain statistical features such as time segmentation information, scene switching points, audio activity intervals, subtitle density distribution, and visual saliency change curves. The aforementioned semantic analysis results are cross-compared with the video statistical features: when the semantics of the question point to a single target and there are short-term, high-density information intervals strongly related to it in the video, the organization mode is determined to be a focused mode; when the semantics of the question point to multi-time period coverage and the relevant semantic content in the video is discretely distributed on the time axis but the total amount is sufficient, the organization mode is determined to be a coverage mode; when the semantics of the question clearly contain a comparative intent and two or more sets of semantically symmetrical or opposing content blocks can be identified in the video, the organization mode is determined to be a contrastive mode.

[0030] Finally, a confidence verification mechanism is introduced. Based on the matching strength between the question's semantics and video features, the prior distribution of patterns from similar historical questions, and the metadata tags of the current video (such as video category, duration, and shooting scene), the confidence level of the determined organizational pattern is comprehensively calculated. If the confidence level is lower than a preset threshold, it reverts to the default organizational pattern or is marked as requiring further confirmation, ensuring the robustness of the pattern determination results.

[0031] As another example, by training a dedicated pattern recognition neural network model, the organization pattern of the response evidence to a question in a video can be determined. This model takes the question text and the multimodal representation of the video as joint inputs and directly outputs the organization pattern category and its confidence level.

[0032] The neural network model employs a two-stream architecture: one stream is a text encoding branch, used to transform the question into a dense semantic vector; the other stream is a video representation encoding branch, used to compress the temporal segmentation representation of the video, scene semantic labels, and global contextual information into structured video vectors. The two vectors interact through a cross-modal attention fusion layer before being fed into a classification head network, outputting probability distributions corresponding to predefined organization pattern categories (including focused patterns, covering patterns, contrasting patterns, and other scalable patterns). During model training, a manually labeled or weakly supervised "question-video-organization pattern" triplet dataset is used. The loss function employs labeled smoothed cross-entropy, supplemented by a semantic distance regularization term between patterns, enabling the model to not only learn category boundaries but also capture the semantic continuity between patterns.

[0033] In actual reasoning, the problem to be addressed and the multimodal representation of the corresponding video are input into a trained neural network model. The model outputs the predicted probability of each organizational pattern category. The one with the highest probability is selected as the candidate organizational pattern, and its probability value is used as the confidence level for determining that pattern. If the confidence level is greater than or equal to a preset confidence threshold, the candidate organizational pattern is determined as the final organizational pattern; if the confidence level is lower than the threshold, the model output is not directly adopted, and a backup determination mechanism is triggered instead.

[0034] Step 202: Using the retrieval strategy corresponding to the organizational pattern, candidate response evidence for the question is determined from the scene representation library, which includes the representation data of the scene in the video.

[0035] In this embodiment, the aforementioned execution entity adopts a retrieval strategy corresponding to the organizational pattern to determine candidate response evidence for the problem from a scene representation library that includes representation data of scenes in the video.

[0036] A retrieval strategy refers to a set of retrieval rules and execution logic, formulated based on a defined organizational model, used to locate candidate response evidence semantically related to the question within a scene representation library. A retrieval strategy is not a simple matching operation, but a systematic specification of elements such as the retrieval scope, retrieval granularity, matching criteria, evidence aggregation methods, and termination conditions. Its core purpose is to adapt the retrieval behavior to the actual distribution structure of response evidence in the video. Specifically, a retrieval strategy specifies the spatiotemporal scale at which the search is conducted, the semantic relevance used as the criterion for including candidates, whether cross-time period aggregation or cross-scene association is necessary, and under what conditions the retrieval is considered complete. Different organizational models correspond to different retrieval strategies, with differences reflected in the retrieval deployment method, coverage logic, and evidence organization assumptions.

[0037] A scene refers to the smallest unit of information in a video that is continuous in time, semantically consistent, and relatively complete in content. Its division is not based on fixed duration or frame count, but rather on the inherent structural boundaries of the video content: it is uninterrupted in time or with jumps in camera angles; semantically, it revolves around the same theme or event and can be understood independently; functionally, it can serve as the basic operational unit for representation generation and evidence retrieval. For example, in a video representing a product launch, the speaker continuously explains the chip architecture for 1 minute and 20 seconds, using visuals, audio, and text to collaboratively express key technical points, thus constituting a scene.

[0038] Representational data refers to the data format formed after semantic encoding and structured abstraction of scenes in a video. This data can be stored in a scene representation library and supports computational retrieval. Representational data is not the original pixel frames or original audio waveforms of the video, but rather a digital expression that reflects the essential attributes of the scene content, obtained through feature extraction and semantic mapping. Its content includes, but is not limited to: the semantic tags of the scene, the categories and spatial relationships of objects contained in the scene, the descriptions of actions or events carried by the scene, the timestamp interval corresponding to the scene, the location identifier of the scene in the overall narrative of the video, and the vectorized semantic embedding of the scene.

[0039] The representation data is organized by scene as the basic unit and stored in the scene representation library according to the video timeline or semantic hierarchy, which enables the retrieval operation to efficiently complete semantic matching and localization without repeatedly parsing the original video.

[0040] As an example, firstly, during the system initialization phase, a corresponding retrieval strategy template is developed for each defined organizational pattern. Each retrieval strategy template, based on its corresponding organizational pattern's assumptions regarding the distribution structure of response evidence in the video, specifies the retrieval deployment method, the spatiotemporal range covered by the retrieval, the semantic association criteria used for matching, the granularity requirements for evidence aggregation, and the termination conditions for determining the completion of candidate response evidence. The retrieval strategy templates corresponding to different organizational patterns differ in the specific values ​​and combinations of the above elements. This difference reflects the differences in the distribution characteristics of response evidence in the video under different organizational patterns. However, all templates follow a unified parameterized description framework to ensure the standardization and reusability of the retrieval execution process.

[0041] During the actual execution phase, the system receives the identified organizational pattern identifier, searches for the retrieval strategy template corresponding to the organizational pattern in the mapping table, fills each parameter slot in the template with the semantic parsing results of the current question and the scene representation library meta-information of the current video, and generates an executable retrieval instruction sequence.

[0042] Finally, according to the retrieval instruction sequence, a matching operation is performed in the scene representation library. Scene units or sets that meet the matching criteria are marked as candidate response evidence, and the scene identifier, timestamp interval and matching score corresponding to each candidate response evidence are recorded as the output of this step.

[0043] As another example, a neural network model is generated by training a dedicated retrieval strategy, which takes organizational pattern identifiers and question semantic representations as joint inputs, adaptively outputs retrieval strategy parameters, and determines candidate response evidence in a scene representation library based on these parameters.

[0044] The retrieval strategy generation neural network model employs a conditional generation architecture. The model's input consists of three parts: the first is a category encoding vector of the organizational pattern, used to inform the model of the structural distribution type of the current response evidence; the second is a semantic embedding vector of the question, obtained by encoding the question using a text encoder, used to provide semantic anchors for retrieval; and the third is a global statistical description vector of the scene representation library, including the total number of scenes, time span, and semantic category distribution histogram, used to provide prior constraints on the retrieval space. The model's output is a retrieval strategy parameter vector, which encodes retrieval execution elements such as retrieval window width, matching semantic threshold, evidence aggregation granularity, time axis search direction, and termination criteria.

[0045] During model training, a "organizational pattern-question-correct candidate response evidence" triple, either manually labeled or semi-automatically constructed, is used as a supervision signal. The training objective is to maximize the overlap between the candidate response evidence set obtained after the model outputs the retrieval policy parameters and performs a retrieval on the scene representation library, and the labeled correct candidate response evidence set. The loss function employs set matching loss at the candidate set level, supplemented by a policy parameter smoothing regularization term to prevent the model from outputting overly extreme retrieval parameters.

[0046] In actual reasoning, the currently determined organizational pattern category code, the semantic embedding vector of the current question, and the global statistical description vector of the current video scene representation library are input into the trained retrieval strategy to generate a neural network model. The model outputs a retrieval strategy parameter vector. Subsequently, based on this parameter vector, a parameterized retrieval operation is performed in the scene representation library: using the question semantic embedding as the query vector, using the representation data of each scene in the scene representation library as the retrieval object, similarity calculation and filtering are performed according to the retrieval window width and matching semantic threshold output by the model, matching scenes are grouped or merged according to the evidence aggregation granularity output by the model, the traversal order is determined according to the search direction of the time axis output by the model, and the termination criterion output by the model determines when to stop the retrieval. Finally, scene units or sets that satisfy all parameter constraints are identified as candidate response evidence for the question, and the corresponding scene identifier, timestamp interval, and matching score are recorded.

[0047] In some optional implementations of this embodiment, the execution entity performs step 202 as follows to determine candidate response evidence: in response to the organization mode being a focused mode, a first preset number of candidate response evidence are determined from the scenario representation library based on the correlation between the problem and the representation data in the scenario representation library.

[0048] In response to the focused organizational model, a single query is constructed based on the question. Specifically, the system semantically encodes the question, encoding the core inquiry object, inquiry intent, and local directional information of the required evidence carried by the question into a unified query vector. This query vector serves as a single query representation of the question's retrieval request for representational data in the scenario representation library.

[0049] After obtaining a single query, the relevance between the question and the representation data in the scenario representation library is calculated. Specifically, each representation data stored in the scenario representation library is traversed, and a relevance score is calculated for each single query and each representation data. The relevance score measures the degree of semantic matching between the question and the scenario corresponding to each representation data; a higher score indicates that the scenario corresponding to the representation data is closer to the response requirements of the question.

[0050] After calculating all relevance scores, the representative data are sorted in descending order based on their relevance scores. A first preset number of representative data with the highest scores are selected, and the scenarios corresponding to these first preset number of representative data are determined as candidate response evidence. Here, the first preset number is a pre-set positive integer used to limit the upper limit of the number of candidate response evidence returned in the focused mode.

[0051] In this implementation, in the focused mode, a single query is used to directly match the most relevant representation data in the scenario representation library, so that the acquisition of candidate response evidence is focused on the local evidence pointed to by the question, avoiding the redundant introduction of irrelevant scenarios, and improving the targeting of evidence location and retrieval efficiency.

[0052] In some optional implementations of this embodiment, the execution entity performs step 202 as follows to determine candidate response evidence: First, in response to the organization mode being a coverage mode, a second preset number of initial response evidence is determined from the scenario representation library based on the correlation between the problem and the representation data in the scenario representation library; then, a first preset number of candidate response evidence that meets the time separation condition is determined from the initial response evidence, wherein the first preset number is less than the second preset number.

[0053] When the organization mode is determined to be a coverage mode, the following input data is obtained: the semantic expression of the question, the preset budget parameter (i.e., the first preset quantity, denoted as K), the expansion factor α (α>1, for example, 4), the time dispersion threshold (e.g., the number of frames corresponding to 20 seconds), and the scene index (i.e., the joint vector and time metadata carried by each representation data in the scene representation library). Here, the first preset quantity K is a pre-set positive integer used to limit the number of candidate response evidence in the final output; the expansion factor α is a positive number greater than 1 used to expand the initial candidate range based on the first preset quantity; and the time dispersion threshold is a pre-set minimum time interval used to constrain the time distance between the final candidate response evidence.

[0054] First, a query encoder is used to encode the question into a query vector, which carries the core semantic information of the question and serves as a unified expression for subsequent retrieval. Then, a vector nearest neighbor search is performed on the scene representation database, calculating the similarity score between the query vector and the joint vector of each representation data in the scene representation database. The similarity score is used to measure the degree of semantic matching between the question and the scene corresponding to each representation data.

[0055] All representation data are sorted in descending order based on similarity scores. The second preset number of representation data with the highest scores are selected, and the scenarios corresponding to the second preset number of representation data are determined as initial response evidence to form an expanded candidate pool.

[0056] The initial response evidence is associated and recorded with its corresponding time position (i.e., the starting frame) and similarity score, and then sorted in descending order of similarity score. Subsequently, the system employs a greedy selection strategy, examining each initial response evidence starting with the one with the highest similarity score: For the currently examined initial response evidence, its starting frame is compared with the starting frames of each response evidence already selected in the candidate set, and the time interval is calculated. If the time interval between the initial response evidence and all response evidence already selected in the candidate set is greater than or equal to the time dispersion threshold, then the initial response evidence is deemed to meet the time separation condition and is selected into the candidate set; if the time interval between the initial response evidence and any response evidence already selected in the candidate set is less than the time dispersion threshold, then the initial response evidence is deemed not to meet the time separation condition, and is temporarily stored in the pending backfill list, continuing to examine the next initial response evidence. This greedy selection process is repeated until the number of response evidences contained in the candidate set reaches a first preset number, or all initial response evidences have been traversed.

[0057] If, after the aforementioned temporal separation screening, the number of response evidences in the candidate set that meet the temporal separation criteria is still insufficient to reach the first preset number, then a backfilling process is performed. Specifically, initial response evidence is selected sequentially from the list to be backfilled, in descending order of similarity score, and added to the candidate set one by one until the number of response evidences in the candidate set reaches the first preset number. This backfilling mechanism ensures that the final output number of candidate response evidences is always equal to the first preset number, avoiding insufficient candidate evidence due to overly strict temporal separation criteria.

[0058] In this implementation, the initial candidate range is first expanded in the coverage mode and then filtered according to the temporal separation condition. This allows the candidate response evidence to achieve distributed coverage in terms of time location while maintaining semantic relevance, avoiding the concentration of traditional search results in the same temporal neighborhood and improving the coverage of repeated and scattered events in long videos.

[0059] In some optional implementations of this embodiment, the execution entity performs step 202 as follows to determine candidate response evidence: First, in response to the organization mode being a comparison mode, each candidate answer in the question and the candidate answer set corresponding to the question is formed into a query pair; then, based on the first preset quantity and the quantity of query pairs, the retrieval quantity corresponding to the query pair is determined; then, based on the correlation between the query pair and the representation data in the scenario representation library, the initial response evidence of the retrieval quantity is determined from the scenario representation library; finally, candidate response evidence is obtained based on the initial response evidence corresponding to each query pair.

[0060] Retrieves the question and its corresponding set of candidate answers. The candidate answer set contains one or more candidate answers, each representing a possible competing hypothesis about the question. The system iterates through each candidate answer in the set, combining the question with that candidate answer to form a query pair.

[0061] The first preset quantity is allocated based on the number of query pairs, such that the sum of the retrieval quantities corresponding to each query pair is approximately equal to the first preset quantity. The allocation method can be an equal distribution strategy, where when the number of query pairs is J, the retrieval quantity corresponding to each query pair is the first preset quantity divided by J and then rounded down; or a non-equal distribution strategy can be used, where the retrieval quantity for each query pair is allocated differently based on the prior confidence or semantic complexity of each candidate answer, but the sum of the retrieval quantities corresponding to each query pair is still constrained by the overall first preset quantity.

[0062] For each query pair, based on the correlation between the query pair and the representation data in the scenario representation library, the number of initial response evidence corresponding to the query pair is determined from the scenario representation library.

[0063] Specifically, for each query pair, the question and corresponding candidate answer in the query pair are jointly encoded into a query vector. This query vector simultaneously carries the inquiry intent of the question and the hypothetical semantics of the candidate answer. Subsequently, vector retrieval is performed on a scene representation database, calculating the relevance score between each query vector and each representation data in the database. The representation data are then sorted in descending order based on their relevance scores. The highest-scoring representation data corresponding to the query pair is selected, and the scene corresponding to this number of retrieved representation data is determined as the initial response evidence for the query pair.

[0064] All query pairs are merged to form a merged candidate set. Since different query pairs may recall the same or highly overlapping time positions, time-based deduplication is performed on the merged candidate set: for multiple initial response evidences in the merged candidate set that have overlapping time positions or time intervals less than a preset threshold, the system retains the one with the highest relevance score and removes the remaining duplicates.

[0065] In this implementation, in the comparative mode, query pairs are independently constructed and retrieved for each candidate answer. After unified deduplication and budget control, each competing hypothesis receives relatively fair evidence support, avoiding a single query biased towards a certain hypothesis and resulting in an unbalanced distribution of evidence.

[0066] In some optional implementations of this embodiment, before executing step 202, the execution entity may also perform the following operations: First, divide the video into scenes to obtain a scene set; then, encode the visual and audio data in the scenes in the scene set to obtain scene vectors; then, combine the scene vectors and scene metadata to generate scene representation data; finally, combine the representation data of each scene in the scene set to obtain a scene representation library.

[0067] First, scene segmentation is performed on the video, dividing the continuous video stream into multiple semantically coherent scene units, resulting in a scene set. Each scene set contains multiple scenes, each corresponding to a continuous time segment within the video. For each scene in the scene set, its corresponding global frame interval and time interval are recorded, serving as the temporal metadata portion of the scene's metadata. Simultaneously, representative frames are selected within each scene to represent its visual content; and speech-recognized text overlapping with the scene's time interval is collected to represent its speech content. These representative frames and speech-recognized text maintain alignment along the temporal dimension.

[0068] Then, a multimodal scene encoder is used to jointly encode the visual and speech data in each scene. Specifically, for each scene in the scene set, the selected representative frame is used as the visual input, and the speech recognition text aligned with the time interval of that scene is used as the speech input, both of which are fed into the multimodal scene encoder. The multimodal scene encoder performs cross-modal fusion encoding of the visual and speech information, outputting a unified scene vector.

[0069] Then, for each scene in the scene set, the scene vector of that scene is associated and combined with the scene metadata of that scene to form a complete representation data entry. The scene metadata includes the start frame, end frame, time interval, concatenated text of overlapping speech sentences within that interval, and duration.

[0070] Finally, the representation data of each scene are uniformly written into a shared vector index to obtain a scene representation library. The scene representation library stores all scene representation data entries in a structured form, supporting similarity retrieval based on scene vectors and time-range queries based on scene metadata.

[0071] Since the construction process of the scene representation library is completely independent of any specific problem and depends only on the content and structure of the input video itself, the scene representation library can be shared and reused for different problems corresponding to the same video, retrieval strategies under different organizational modes, and re-retrieval in multiple rounds of interaction, without having to repeatedly perform video encoding operations for each query.

[0072] In this implementation, by dividing the video into scenes and jointly encoding visual and speech information to generate a scene representation library independent of the problem, multiple searches of the same video can share and reuse the index, avoiding redundant encoding overhead and significantly improving the overall processing efficiency of the system.

[0073] Step 203: Generate response data for the question based on the candidate response evidence.

[0074] In this embodiment, the aforementioned executing entity generates response data for the question based on the candidate response evidence.

[0075] As an example, the system pre-configures multiple response structure templates. Each template corresponds to a type of response data organization format. The template contains several semantic slots, and each slot is labeled with the semantic type, source constraints, and format requirements of the content to be filled in.

[0076] During the generation phase, firstly, a suitable response structure template is selected from the template library based on the semantic category of the question and the attribute characteristics of the candidate response evidence. Then, the meta-information such as scene identifiers, timestamp intervals, semantic tags, and matching scores in the candidate response evidence are mapped to the corresponding semantic slots of the selected template according to the content filling rules. For derived information required in the template but not directly provided by the candidate response evidence, it is calculated or obtained from the existing slot content according to the preset derivation rules, rather than being re-searched or obtained externally. Finally, the format validation and text rendering are performed on the filled template, and response data that conforms to the preset specifications is output.

[0077] As another example, candidate response evidence is used as the sole source of content input, and together with the original question text and pre-set generation instructions, it is constructed into a structured prompt. The generation instructions explicitly limit the model to only reorganize the expression and refine the language based on the provided candidate response evidence, and prohibit the introduction of training knowledge, external information or speculative content.

[0078] After receiving the structured prompt, the large language model is subject to content anchoring constraints during decoding, ensuring that its output token sequence always revolves around the factual elements in the candidate response evidence and cannot deviate or expand upon them. The generated response data is verified by the post-processing module to confirm that all its statements can be traced back to the input candidate response evidence. If any content fragments that cannot be traced back are found, they are automatically removed and local regeneration is triggered until the output is fully compliant.

[0079] In some optional implementations of this embodiment, the execution entity performs step 203 as follows: The first step involves identifying and executing target evidence processing operations from the operation set based on the assessment results of the candidate response evidence regarding its comprehensiveness and credibility. The operation set includes evidence supplementation operations, evidence verification operations, and response generation operations.

[0080] The second step involves performing the target evidence processing operation to update the candidate response evidence, in response to whether the target evidence processing operation is an evidence supplementation operation or an evidence verification operation.

[0081] Evidence supplementation refers to the process of expanding and enriching candidate response evidence when the evaluation results of the comprehensiveness of the candidate response evidence do not meet the preset requirements, in order to address the coverage gaps in the existing candidate response evidence. Its purpose is to ensure that the updated candidate response evidence meets the information completeness requirements of the response in terms of semantic coverage.

[0082] Evidence verification refers to the process of verifying and confirming questionable items in existing candidate response evidence when the evaluation results of the candidate response evidence in terms of credibility do not meet the preset requirements. Its purpose is to ensure that the updated candidate response evidence meets the information credibility conditions required for the response in terms of factual accuracy and source reliability.

[0083] The response generation operation refers to the processing operation that directly generates response data for a question based on the current candidate response evidence when the evaluation results of the candidate response evidence in terms of comprehensiveness and credibility meet the preset requirements. Its purpose is to transform the candidate response evidence that has met the quality conditions into the final response output oriented towards the question.

[0084] The above three operations together constitute an operation set. They are mutually exclusive in terms of execution conditions and form a closed loop in terms of logical relationship: the evidence supplementation operation and the evidence verification operation take updating candidate response evidence as output, and the updated candidate response evidence needs to re-enter the evaluation stage; the response generation operation takes generating response data as output, marking the end of the response process.

[0085] The evidence supplementation operation is implemented through semantic scene retrieval. Its purpose is to address coverage gaps in the comprehensiveness of candidate response evidence by retrieving scene fragments semantically relevant to the question from the shared scene vector index, thereby expanding the semantic coverage of the candidate response evidence. The input data for this operation consists of the query text (typically generated during inference based on inference data) and the top_k recall counts. During processing, a query encoder encodes the query text into a query vector, performs nearest neighbor retrieval on the shared scene vector index, and sorts the results by similarity score. In coverage mode, strategies such as expanding the candidate pool, temporal dispersion, and backfilling are implemented; in other modes, the Top K results are directly used. No video frames are loaded during the entire retrieval process. The output data of this operation is a list of candidate scenes. Each record in the list includes the start time, end time, start frame, end frame, similarity score, and the corresponding audio preview for that scene. It should be emphasized that all candidates in the above candidate scenario list are marked as "tentative reference" and cannot be used directly as evidence. This is because the operation only performs retrieval and candidate recall, and does not perform visual content verification or voice content confirmation of the recall results. Its role is limited to promoting the comprehensiveness of candidate response evidence.

[0086] The evidence verification process includes three specific operations: temporal visual inspection, time-aligned speech transcription, and problem-conditional segment analysis. These three operations work together to verify and confirm the questionable content in the candidate response evidence, elevating "provisional reference evidence" to verified evidence, thereby enhancing the credibility of the candidate response evidence.

[0087] Temporal visual inspection is used to verify the true visual content of candidate scenes. This operation takes one or more time intervals (each containing a start and end frame) and a frame extraction budget as input data. Within each time interval, the number of frames to be extracted is allocated proportionally to the interval length. Video frames are extracted, and the shorter side of each extracted frame is scaled. The scaled frames are then stitched together to form a mesh image, which is fed into a visual description model. This model is required to describe the main scene or action within that time interval using a concise text. The output data includes the actual number of extracted frames, the time interval, the total duration, and a scene description text. The candidate scene list recalled through semantic scene retrieval is based solely on vector similarity; the actual video content has not yet been confirmed. Temporal visual inspection transforms candidate scenes from a "provisional reference" state to a "verified observation" state through actual frame extraction and visual description.

[0088] Time-aligned speech-to-text transcription aligns spoken words within a specific time interval in candidate response evidence with corresponding visual images within that time interval. This operation takes one or more time intervals as input data, extracts audio segments corresponding to those time intervals from the video, calls a speech recognition model to transcribe the extracted audio segments with silence filtering enabled during the transcription process, and then maps the relative timestamps output by the speech recognition model back to the video's global frame index. The output data consists of multiple transcribed text segments (each containing a start frame, end frame, and text content) and a concatenated full text obtained by splicing the multiple transcribed text segments. This operation establishes a correlation between speech and visual content within a unified global frame index coordinate system, providing a multimodal consistency verification basis for subsequent evaluation.

[0089] Conditionalized fragment analysis is used for fine-grained verification or conflict resolution of specific content in candidate response evidence. This operation takes a time interval and a sub-question text for that time interval as input data. A specified number of frames (up to 50 frames) are uniformly extracted within the time interval. Each frame is labeled with a global frame index, and the labeled frame sequence is fed into a visual question-answering model. The model then provides a judgment answer and corresponding confidence score for the sub-question text. The output data includes the time interval, the actual number of frames extracted, the judgment answer for the sub-question, the confidence score (ranging from 0 to 1), and the sub-question text. When there are inconsistencies between the outputs of time-series visual inspection or time-aligned speech-to-text, or when targeted confirmation of a specific sub-question is required, this operation provides a judgment result with a clear confidence score, further enhancing the credibility of candidate response evidence.

[0090] As an example, firstly, a comprehensive assessment and a credibility assessment are performed on the candidate response evidence. The comprehensive assessment examines the extent to which the current candidate response evidence covers all semantic components involved in the question, and outputs the comprehensive assessment result; the credibility assessment examines the sufficiency of factual basis and the reliability of sources for each item in the current candidate response evidence, and outputs the credibility assessment result. The two assessment results together constitute the complete output of this assessment.

[0091] Subsequently, based on the comprehensiveness assessment results and the credibility assessment results, the target evidence processing operation is determined from the operation set. Specifically, if the comprehensiveness assessment results indicate that there are coverage gaps in the candidate response evidence, the evidence supplementation operation is determined as the target evidence processing operation; if the comprehensiveness assessment results meet the requirements but the credibility assessment results indicate that there is questionable content in the candidate response evidence, the evidence verification operation is determined as the target evidence processing operation; if both the comprehensiveness assessment results and the credibility assessment results meet the preset requirements, the response generation operation is determined as the target evidence processing operation.

[0092] If the target evidence processing operation is an evidence supplementation operation, it is executed to expand the candidate response evidence to address the coverage gaps indicated by the comprehensiveness assessment results, generating updated candidate response evidence. If the target evidence processing operation is an evidence verification operation, it is executed to verify and confirm the candidate response evidence to address the questionable content indicated by the credibility assessment results, generating updated candidate response evidence. The updated candidate response evidence replaces the original candidate response evidence and re-enters the comprehensiveness assessment and credibility assessment stages until the assessment results meet the preset requirements, at which point the target evidence processing operation is determined to be a response generation operation.

[0093] In response to the target evidence processing operation being a response generation operation, response data for the question is generated based on the current candidate response evidence. This step is now complete.

[0094] In this implementation, the determination and execution of target evidence processing operations in the operation set are driven by the evaluation results. This allows candidate response evidence to be updated in a targeted manner when its comprehensiveness and credibility are insufficient, avoiding direct response generation when the evidence quality does not meet the standards, thereby improving the integrity and reliability of the response data.

[0095] In some optional implementations of this embodiment, the execution entity performs step 203 as follows: in response to the target evidence processing operation being a response generation operation, it performs the target evidence processing operation and generates response data based on the current candidate response evidence.

[0096] When the comprehensiveness and credibility assessments of the candidate response evidence both meet preset requirements, the target evidence processing operation is determined to be the response generation operation. In response to this determination, the current candidate response evidence is used as the sole factual basis for response generation, and is input into the response generation stage along with the original question text (e.g., implemented based on a large artificial intelligence model).

[0097] In the response generation stage, based on the semantic content of the current candidate response evidence, and focusing on the key information requested by the question, the relevant content in the candidate response evidence is organized into a question-oriented response statement. During the generation process, all factual statements in the response data are confined to the current candidate response evidence, and no information outside of the current candidate response evidence is introduced. After generation, the response data is returned as the final output of this step.

[0098] In this implementation, the response generation operation is triggered only after both comprehensiveness and credibility assessments have met the standards, ensuring that the response data is strictly generated based on verified candidate response evidence, thus avoiding the production of unreliable response results when the quality of evidence is insufficient.

[0099] In some optional implementations of this embodiment, the execution entity performs the first step described above in the following manner, determining and executing the target evidence processing operation from the operation set: First, based on the operation results of historical evidence processing operations during the generation of question and response data, the evaluation results of the candidate response evidence regarding comprehensiveness and credibility are determined; then, based on the evaluation results, the target evidence processing operation that can improve the comprehensiveness and credibility of the candidate response evidence is determined from the operation set.

[0100] Historical evidence processing operations refer to the collective evidence supplementation and / or evidence verification operations performed before the current round of evaluation during the generation of response data for the current issue. Each historical evidence processing operation corresponds to an operation result, which records the status of the candidate response evidence before the operation, the processing content taken during the operation, and the updated status of the candidate response evidence after the operation. The operation results of historical evidence processing operations constitute one of the input bases for the evaluation process. Their role is to provide the evolution trajectory and existing processing information of the candidate response evidence for the comprehensive evaluation and credibility evaluation of the current round, so that the evaluation conclusion not only reflects the current static state of the candidate response evidence, but also reflects its dynamic changes in previous rounds of operations.

[0101] As an example, firstly, the input information required for the evaluation is gathered. The original text of the current question is extracted, and the operation results of all historical evidence processing operations are retrieved from the operation records of the response data generation process. If this is the first time entering the evaluation stage, the operation results of the historical evidence processing operations are an empty set, and the evaluation is based solely on the original question text and the initial candidate response evidence. If this is not the first time entering the evaluation stage, the operation results of the historical evidence processing operations record the specific processing content of the evidence supplementation operations and / or evidence verification operations performed in previous rounds, as well as the updates to the candidate response evidence.

[0102] Subsequently, based on the operational results of the problem and historical evidence processing, the comprehensiveness assessment results of the candidate response evidence are determined. The comprehensiveness assessment uses each semantic component involved in the problem as a reference benchmark to examine the coverage of each semantic component by the current candidate response evidence. At the same time, it refers to the processing records of coverage gaps in previous rounds in the operational results of historical evidence processing to determine whether there are still unfilled gaps in the semantic coverage of the current candidate response evidence, and outputs the comprehensiveness assessment result.

[0103] Subsequently, based on the operational results of the problem and historical evidence processing, the credibility assessment results of the candidate response evidence are determined. The credibility assessment examines the sufficiency of factual basis and the reliability of sources for each item in the current candidate response evidence. At the same time, it refers to the processing records of questionable content in previous rounds in the operational results of historical evidence processing to determine whether there are still unresolved doubts about the factual accuracy and source reliability of the current candidate response evidence, and outputs the credibility assessment results.

[0104] Finally, based on the comprehensiveness assessment results and the credibility assessment results, target evidence processing operations that can improve the comprehensiveness and credibility of candidate response evidence are determined from the operation set. Specifically, if the comprehensiveness assessment results indicate that there are still gaps in the semantic coverage of candidate response evidence, then the evidence supplementation operation is determined as the target evidence processing operation to improve the comprehensiveness of candidate response evidence; if the comprehensiveness assessment results meet the requirements but the credibility assessment results indicate that there are still questionable contents in candidate response evidence, then the evidence verification operation is determined as the target evidence processing operation to improve the credibility of candidate response evidence; if both the comprehensiveness assessment results and the credibility assessment results meet the preset requirements, then the response generation operation is determined as the target evidence processing operation, indicating that the comprehensiveness and credibility of candidate response evidence no longer need to be further improved, and the response data generation stage can begin.

[0105] In this implementation, the results of the problem and historical evidence processing operations are comprehensively evaluated to ensure that the determination of comprehensiveness and credibility takes into account the current state and evolution of the candidate response evidence. This allows for the accurate identification of target evidence processing operations that can improve evidence quality, avoiding blind duplication or omission of processing.

[0106] In some optional implementations of this embodiment, the execution entity determines the evaluation results of the candidate response evidence regarding its comprehensiveness and credibility in the following manner: First, the operation results of the historical evidence processing operation are structurally organized in terms of temporal, semantic, and logical relationships to obtain a temporal evidence summary; based on the question, the operation results of the historical evidence processing operation, and the temporal evidence summary, the evaluation results of the candidate response evidence regarding its comprehensiveness and credibility are determined.

[0107] A temporal evidence summary is a compressed view of evidence obtained by structuring the results of historical evidence processing operations in terms of temporal, semantic, and logical relationships. Essentially, it merges and structures the discrete and heterogeneous results generated from each round of evidence processing according to a unified organizational framework. This provides a reference basis for subsequent evaluation stages that considers both overall coverage and local details, rather than simply listing the original outputs.

[0108] The temporal evidence summary records the chronological order and coverage of the time intervals involved in each operation result in terms of temporal relationships. For example, the coverage view indicates that one interval has been checked while another interval is still unchecked. In the relational view, it records the correspondence between each operation result and the semantic components involved in the question in terms of semantic relationships. For example, one operation result corresponds to the question about the subject's behavior while another operation result corresponds to the question about the time sequence. In terms of logical relationships, it records the consistency or contradiction between each operation result. For example, two observations have conflicting descriptions of the same event, or a certain counting result has not yet been cross-confirmed.

[0109] In the structured processing phase, the results of all historical evidence processing operations during the response data generation process are retrieved as the processing objects. In the temporal relationship processing, the time intervals involved in each operation result are arranged chronologically, all checked intervals are merged to obtain the current time coverage, and time areas not yet covered by any operation result are marked; these uncovered areas constitute the next constraint in the time dimension and are explicitly recorded in the temporal evidence summary. In the semantic relationship processing, each operation result is correlated with the semantic components involved in the question, identifying which semantic components are supported by operation results and which semantic components do not yet have corresponding information in the current operation result; these semantic components without corresponding information constitute the next constraint in the semantic dimension and are also explicitly recorded in the temporal evidence summary. In the logical relationship processing, the existence of contradictory descriptions, unconfirmed values ​​or states, and conflicting observations requiring further verification among the operation results is examined, and these contradictions and matters to be verified are explicitly recorded; these contradictions and matters to be verified constitute the next constraint in the logical dimension and are also included in the temporal evidence summary. After being organized in the above three aspects, the results of each operation are organized into a structured temporal evidence summary. This temporal evidence summary not only presents the current state of the candidate response evidence, but also explicitly exposes the next constraints such as areas with insufficient coverage, hypotheses with missing evidence, conflicting observations, and unconfirmed values ​​or states, serving as a compressed reference for subsequent evaluation stages.

[0110] During the assessment and determination phase, the original question text, the results of historical evidence processing, and the chronological evidence summary serve as assessment inputs. For a comprehensive assessment, the semantic components involved in the question are used as references. Combined with records of semantic coverage, time coverage annotations, and explicit semantic and temporal constraints in the chronological evidence summary, the assessment determines whether the current candidate response evidence adequately covers each semantic component and whether any gaps remain unfilled. Simultaneously, the results of previous rounds of historical evidence processing addressing these gaps are used to determine whether existing processing has substantially resolved the corresponding gaps. For credibility assessment, the sufficiency and reliability of factual evidence in the current candidate response evidence are examined. Combined with records of logical contradictions and conflicts observed in the chronological evidence summary and explicit logical constraints, the assessment determines whether contradictory descriptions still exist in the candidate response evidence and whether any key information remains unverified. Simultaneously, the results of previous rounds of historical evidence processing verifying questionable content are used to determine whether existing verification has substantially eliminated the corresponding doubts. Based on the above assessments, the results of the comprehensiveness and credibility assessments of the candidate response evidence are output.

[0111] In some examples, a time-series evidence ledger (ledger) is used to uniformly manage the multimodal observation results (operation results) generated during the exploration process, and dynamically generate compressed structured evidence views based on different questions. Specifically, the ledger not only stores the source, time position, content, and confidence level of each observation, but also maintains information such as time coverage, temporal relationships between events, repeating events, state changes, conflicting evidence, and supporting and rebuttal evidence for different candidate hypotheses. Formally, the ledger first normalizes the output of each evidence operation into a set of entries. Each entry includes the source and interaction sequence of evidence, the global frame interval and time interval, the query or sub-question that triggered the observation, the observation content, and confidence level information. The aforementioned executing entity uses the above examples to structure the entries in the time-series evidence ledger in terms of temporal, semantic, and logical relationships, resulting in structured data.

[0112] In this implementation, the operation results are structured in three dimensions—time, semantics, and logic—to form a time-series evidence summary. This enables the evaluation process to efficiently locate gaps and contradictions in the compressed view, thereby improving the accuracy and efficiency of the comprehensiveness and credibility assessment.

[0113] In some optional implementations of this embodiment, the execution entity can perform step 201 as follows to determine the organization pattern of the response evidence in the video: determine the organization pattern of the response evidence in the video and the evidence arrangement pattern of the temporal evidence summary.

[0114] Evidence arrangement mode refers to the structured strategy adopted by temporal evidence summaries when organizing verified observations. Its core lies in selecting appropriate evidence merging, arrangement, and association methods based on the organization pattern of the response evidence in the video, enabling the temporal evidence summary to present the evidence status in a form that best fits the needs of the question. Observation refers to the structured operation result returned by the inference model after performing an evidence processing operation, including but not limited to the following data: source operation type (semantic scene retrieval / temporal visual inspection / time-aligned speech-to-text / question-conditional fragment analysis), time interval (start time, end time), frame range (start frame, end frame), observation content (visual description text, or speech-to-text text, or sub-question judgment answer), and confidence level.

[0115] When the question pertains to local identification or description, the response evidence in the video is typically concentrated within a specific time period or area of ​​the frame. In this case, the evidence arrangement pattern becomes focused, with the temporal evidence summary emphasizing the compact organization of verified observations directly related to the local object, while downplaying information covering irrelevant time intervals. For example, if the question is "What is the person in the red coat on the left side of the frame holding?", the temporal evidence summary would be compactly arranged only around the time period in which the person appears and the visual inspection results of the object they are holding.

[0116] When the question pertains to recurring events, counts, or sequences, the response evidence is distributed across multiple discrete time points in the video and exhibits either cumulative or sequential characteristics. In this case, the evidence arrangement mode is mapped to a coverage type, with the temporal evidence summary focusing on recording the coverage status of each time interval and the relationships between events, such as sequence and frequency. For example, if the question is "How many times did the host raise his right hand throughout the entire event?", the temporal evidence summary would arrange the hand-raising event records within each checked interval in chronological order and explicitly mark the time areas that are not yet covered.

[0117] When a question involves competing answers or competing explanations, the evidence presented in the video may simultaneously support or refute multiple mutually exclusive hypotheses. In this case, the evidence arrangement pattern is mapped to a contrastive type, with the temporal evidence summary focusing on separately compiling supporting and conflicting evidence for each candidate hypothesis and presenting them in a comparative structure. For example, if the question is "Did the vehicle cross the stop line before or after the red light turned on?", the temporal evidence summary would list verified observations supporting both the "crossing the line before the red light" and "crossing the line after the red light" hypotheses, and mark the points of contradiction or matters to be verified between them.

[0118] As an example, semantic parsing is performed on the questions to identify the inherent distribution pattern of the response evidence in the video, thereby determining the organization pattern of the response evidence in the video. Specifically, the core question object, questioning method, and spatiotemporal distribution characteristics of the required evidence in the questions are analyzed: if the question points to a local object in a specific time period or a specific area of ​​the video, and the required evidence does not involve cross-time period accumulation or the discrimination between multiple mutually exclusive candidates, then the organization pattern of the response evidence in the video is determined to be focused; if the question points to repetitive events distributed at multiple discrete time points in the video, and the required evidence has cumulative, countable, or sequential characteristics, then the organization pattern of the response evidence in the video is determined to be covering; if the question points to multiple mutually exclusive candidate answers or candidate explanations, and the required evidence must simultaneously support or refute each of the above candidates, then the organization pattern of the response evidence in the video is determined to be contrastive.

[0119] Based on the organizational model, the evidence arrangement pattern of the time-series evidence summary is determined. The evidence arrangement pattern is a structured strategy adopted by the time-series evidence summary when organizing verified observations. It is adjusted according to different organizational models to ensure that the presentation format of the time-series evidence summary matches the evidence requirements of the issue. Specifically: when the organization model of the response evidence in the video is focused, the evidence arrangement pattern of the time-series evidence summary is determined to be an organizational strategy that emphasizes the compact collection of local evidence directly related to the issue, while weakening the coverage of time intervals unrelated to the issue; when the organization model of the response evidence in the video is comprehensive, the evidence arrangement pattern of the time-series evidence summary is determined to be an organizational strategy that emphasizes the time coverage and the relationship between events, arranging the event records of each examined interval in chronological order and explicitly marking the uncovered time areas; when the organization model of the response evidence in the video is comparative, the evidence arrangement pattern of the time-series evidence summary is determined to be an organizational strategy that emphasizes the supporting evidence and conflicting evidence for each hypothesis, using a comparative structure to separately collect verified observations corresponding to each candidate hypothesis and their contradictions. The output of this determination process is the evidence arrangement pattern of the time-series evidence summary, which serves as the organizational basis for the merging, arrangement, and association of verified observations in the subsequent structured processing.

[0120] In this implementation, the aforementioned executing entity obtains the temporal evidence summary in the following manner: First, the results of historical evidence processing operations are structurally organized in terms of temporal, semantic, and logical relationships to obtain structured data; then, the structured data is arranged using an evidence arrangement pattern to generate a temporal evidence summary that includes a verified evidence set, evidence comprehensiveness, and evidence correlation graph.

[0121] First, the results of all historical evidence processing operations during the response data generation process are retrieved as the objects of organization, and then the results are organized in a structured manner from the aspects of time relationship, semantic relationship and logical relationship.

[0122] In the structured organization of temporal relationships, the results of each historical evidence processing operation are arranged sequentially according to their corresponding time position or time interval, determining the distribution order of each operation result on the video timeline; the time intervals involved in each operation result are merged to obtain the currently covered time range; the currently covered time range is compared with the overall video duration to mark the time areas not yet covered by any operation results. Through the above organization, the operation results form an ordered arrangement structure with coverage markings in the time dimension.

[0123] In the structured organization of semantic relationships, the problem is parsed into several semantic components. The results of each historical evidence processing operation are then correlated with these semantic components one by one, indicating whether each semantic component currently has an operational result to support it. For semantic components that do not yet have an operational result, they are marked as evidence gaps. Through the above organization, each operational result forms a mapping structure corresponding to the semantic components of the problem in the semantic dimension.

[0124] In the structured organization of logical relationships, the process examines whether there are contradictory descriptions among the results of each historical evidence processing operation, whether there are any values ​​or states that have not yet been cross-confirmed, and whether there are any conflicting observations that require further verification. These contradictory matters, matters requiring confirmation, and conflicting observations are then identified and recorded. Through this organization, the results of each operation form a logically related structure with consistency and conflict annotations.

[0125] After the above three aspects of structuring are completed, the results of each historical evidence processing operation are organized into a structured data set. In this structured data set, each operation result simultaneously carries its temporal location information, the semantic component attribution information, and the logical relationship information between it and other operation results.

[0126] When the evidence arrangement mode adopts a focused organizational strategy, the arrangement emphasizes the compact collection of operational results directly related to the local object addressed by the problem, while downplaying time interval coverage information irrelevant to the problem, thus focusing the temporal evidence summary on the presentation of local evidence. When the evidence arrangement mode adopts a coverage-oriented organizational strategy, the arrangement emphasizes the chronological arrangement of event records for each examined interval, explicitly retaining the annotations of uncovered time areas and the relationship information such as the sequence and frequency between events, thus focusing the temporal evidence summary on the presentation of time coverage and the relationships between events. When the evidence arrangement mode adopts a comparative organizational strategy, the arrangement emphasizes the separate collection of supporting and conflicting evidence for each candidate hypothesis, presenting the evidence distribution of each hypothesis and the contradictions between them in a comparative structure, thus focusing the temporal evidence summary on the presentation of supporting and conflicting evidence for each hypothesis.

[0127] The generated temporal evidence summary comprises three components: first, a set of verified evidence, which is an ordered collection of verified observations after structured organization and arrangement, with each verified observation carrying its temporal location, semantic attribution, and logical association annotations; second, evidence comprehensiveness, which is a record of coverage status based on temporal and semantic component coverage, used to reflect the degree to which the current verified evidence covers the evidence required for the issue; and third, an evidence association graph, which is a graphical representation of the associations and contradictions between the operational results formed by the structured organization based on logical relationships, used to reflect the consistency, dependence, or conflict among the verified evidence.

[0128] In this implementation, the dispersed operation results are organized into a three-dimensional structure based on time, semantics, and logic, and then arranged according to an adaptation mode. This allows the temporal evidence summary to present the evidence status in a compact and complete form, providing a structured reference for subsequent evaluation and helping to improve the accuracy and reliability of the reasoning process.

[0129] In some optional implementations of this embodiment, the execution entity performs the process of determining the organization pattern and evidence arrangement pattern in the following manner: First, it determines the model prediction organization pattern of the response evidence in the video and the model prediction arrangement pattern of the temporal evidence summary by using the pattern determination model; then, in response to the confidence level of the model prediction organization pattern and the model prediction arrangement pattern being greater than or equal to a preset confidence threshold, it determines the model prediction organization pattern and the model prediction arrangement pattern as the organization pattern and the evidence arrangement pattern, respectively.

[0130] First, the semantic features of the question and relevant information of the currently acquired candidate response evidence are fed into a pre-configured pattern determination model. The pattern determination model extracts features and performs comprehensive judgment on the input, outputting two prediction results: one is the model's predicted organization pattern, i.e., the pattern determination model's prediction of the organization pattern of the question's response evidence in the video; the other is the model's predicted arrangement pattern, i.e., the pattern determination model's prediction of the arrangement pattern of the temporal evidence summary. Simultaneously with outputting these two prediction results, the pattern determination model also outputs the confidence scores corresponding to the model's predicted organization pattern and the model's predicted arrangement pattern, respectively, to characterize the degree of certainty of the pattern determination model for each prediction result.

[0131] Then, the confidence level of the model's predicted organizational pattern is compared with a preset confidence threshold, and the confidence level of the model's predicted orchestration pattern is also compared with the same preset confidence threshold. The model is deemed to have sufficient confidence in the prediction result if and only if the confidence level of the model's predicted organizational pattern is greater than or equal to the preset confidence threshold, and the confidence level of the model's predicted orchestration pattern is also greater than or equal to the preset confidence threshold.

[0132] Under the aforementioned confidence level conditions, the model-predicted organization pattern is directly determined as the organization pattern of the response evidence in the video, and the model-predicted arrangement pattern is directly determined as the evidence arrangement pattern of the time-series evidence summary.

[0133] In this implementation, the pattern determination model automatically predicts organizational and arrangement patterns, and a pre-set confidence threshold is used as the adoption threshold to ensure that the determined patterns have sufficient judgment reliability and reduce the adverse impact of misjudgments on subsequent evidence arrangement.

[0134] In some optional implementations of this embodiment, the execution entity determines the organization pattern of the response evidence in the video and the evidence arrangement pattern of the temporal evidence summary in the following manner: in response to the confidence of the model prediction organization pattern and the model prediction arrangement pattern being less than a preset confidence threshold, the organization pattern of the response evidence in the video and the evidence arrangement pattern of the temporal evidence summary are determined according to preset prior rules.

[0135] In response to the model's predicted organizational and orchestration patterns having confidence levels below a pre-set confidence threshold, semantic features of the question are extracted. These features include the type of object the question targets, the method of inquiry, the spatiotemporal distribution characteristics of the required evidence, and whether multiple mutually exclusive candidates are involved. Based on pre-set prior rules, these semantic features are matched item by item with pre-set organizational pattern determination conditions: if the question's semantic features meet the determination conditions for a focused pattern, the organizational pattern of the response evidence in the video is determined to be focused; if the question's semantic features meet the determination conditions for a covered pattern, the organizational pattern of the response evidence in the video is determined to be covered; if the question's semantic features meet the determination conditions for a contrastive pattern, the organizational pattern of the response evidence in the video is determined to be contrastive.

[0136] After determining the organization pattern of the response evidence in the video, the evidence arrangement pattern of the chronological evidence summary is determined based on the correspondence between the organization pattern and the evidence arrangement pattern in the preset a priori rules. When the organization pattern is focused, the evidence arrangement pattern of the chronological evidence summary determined according to the preset a priori rules is an organization strategy that emphasizes the compact collection of local evidence; when the organization pattern is comprehensive, the evidence arrangement pattern of the chronological evidence summary determined according to the preset a priori rules is an organization strategy that emphasizes the temporal coverage and the relationship between events; when the organization pattern is comparative, the evidence arrangement pattern of the chronological evidence summary determined according to the preset a priori rules is an organization strategy that emphasizes the comparison between supporting evidence and conflicting evidence.

[0137] It should be noted that when the question includes candidate answers, the candidate answers in the question text can be parsed to obtain a set of candidate answers. The question and the set of candidate answers can then be used as a reference for the pattern recognition model and preset prior rules for data processing.

[0138] In this implementation, when the confidence level of the model prediction is insufficient, a preset prior rule is used to determine the organization mode and arrangement mode, ensuring that there is always a clear basis for the mode determination process, and avoiding the loss of organizational foundation for subsequent evidence arrangement due to prediction uncertainty.

[0139] In this embodiment, the above-described implementation methods can be executed by a video question-and-answer agent to generate response data.

[0140] See also Figure 4 , Figure 4This is a schematic diagram of an application scenario 400 of the video question-answering method according to this embodiment. User 401 sends video 404 and question 405 to server 403 through terminal device 402. The server pre-divides the video into scenes and determines the representation data of each scene, thereby obtaining a scene representation library. For the user's question about the video, firstly, the organization pattern of the answer evidence for the video question is determined in the video; then, a retrieval strategy corresponding to the organization pattern is used to determine the candidate answer evidence for the question from the scene representation library, which includes the representation data of the scenes in the video; finally, the answer data for the question is generated based on the candidate answer evidence.

[0141] In this embodiment, by determining the organization pattern of the response evidence for the video question in the video; using the retrieval strategy corresponding to the organization pattern, candidate response evidence for the question is determined from the scene representation library including the representation data of the scene in the video; and based on the candidate response evidence, response data for the question is generated. This approach specifically analyzes the organization pattern of the response evidence required for the question in the video and selects the appropriate evidence retrieval strategy to obtain candidate response evidence for response, thereby improving the accuracy and comprehensiveness of the candidate response evidence, as well as the fit between the candidate response evidence and the question, and thus improving the accuracy and comprehensiveness of the response data.

[0142] Continue to refer to Figure 5 The illustration shows a schematic flow 500 of yet another embodiment of the video question-answering method according to the present disclosure. Flow 500 includes the following steps: Step 501: Determine the organization pattern of the response evidence to the video questions and the evidence arrangement pattern of the temporal evidence summary in the video.

[0143] Step 502: Using the retrieval strategy corresponding to the organizational pattern, candidate response evidence for the question is determined from the scene representation library, which includes the representation data of the scene in the video.

[0144] Step 503: The results of the historical evidence processing operation are structured in terms of temporal, semantic, and logical relationships to obtain structured data.

[0145] The set of operations includes evidence supplementation operations, evidence verification operations, and response generation operations.

[0146] Step 504: Arrange structured data using the evidence arrangement pattern to generate a time-series evidence summary that includes the verified evidence set, evidence comprehensiveness, and evidence correlation graph.

[0147] Step 505: Based on the problem, the results of the historical evidence processing operations, and the time-series evidence summary, determine the evaluation results of the candidate response evidence regarding its comprehensiveness and credibility.

[0148] Step 506: Based on the evaluation results, identify the target evidence processing operations from the set of operations that can improve the comprehensiveness and credibility of candidate response evidence.

[0149] Step 507: In response to the target evidence processing operation being an evidence supplementation operation or an evidence verification operation, perform the target evidence processing operation to update the candidate response evidence.

[0150] After completing step 507, return to step 503 to continue execution.

[0151] Step 508: In response to the target evidence processing operation being a response generation operation, execute the target evidence processing operation and generate response data based on the current candidate response evidence.

[0152] The video question-answering method in this embodiment, in process 500, compared to process 200 above, specifically describes the process of generating response data. It analyzes the organization pattern of the response evidence required for the question in the video, selects the appropriate evidence retrieval strategy to obtain candidate response evidence for response, and generates response data after passing the comprehensiveness and credibility evaluation, thereby improving the accuracy and comprehensiveness of the response data.

[0153] Continue to refer to Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a video question-and-answer device, which is similar to... Figure 2 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0154] like Figure 6 As shown, the video question answering device 600 includes: a pattern determination unit 601, configured to determine the organization pattern of the response evidence for a question in the video; an evidence determination unit 602, configured to use a retrieval strategy corresponding to the organization pattern to determine candidate response evidence for the question from a scene representation library including representation data of the scene in the video; and a response unit 603, configured to generate response data for the question based on the candidate response evidence.

[0155] In some optional implementations of this embodiment, the response unit 603 is further configured to: determine and execute a target evidence processing operation from an operation set based on the evaluation results of the candidate response evidence regarding its comprehensiveness and credibility, wherein the operation set includes an evidence supplementation operation, an evidence verification operation, and a response generation operation; and, in response to the target evidence processing operation being an evidence supplementation operation or an evidence verification operation, execute the target evidence processing operation to update the candidate response evidence.

[0156] In some optional implementations of this embodiment, the response unit 603 is further configured to: in response to the target evidence processing operation being a response generation operation, perform the target evidence processing operation and generate response data based on the current candidate response evidence.

[0157] In some optional implementations of this embodiment, the response unit 603 is further configured to: determine the evaluation results of the comprehensiveness and credibility of the candidate response evidence based on the operation results of historical evidence processing operations during the generation of question and response data; and determine the target evidence processing operation from the operation set that can improve the comprehensiveness and credibility of the candidate response evidence based on the evaluation results.

[0158] In some optional implementations of this embodiment, the response unit 603 is further configured to: perform structured organization of the operation results of the historical evidence processing operation in terms of time relationship, semantic relationship and logical relationship to obtain a time-series evidence summary; and determine the evaluation results of the candidate response evidence in terms of comprehensiveness and credibility based on the question, the operation results of the historical evidence processing operation and the time-series evidence summary.

[0159] In some optional implementations of this embodiment, the pattern determination unit 601 is further configured to: determine the organization pattern of the response evidence in the video and the evidence arrangement pattern of the time-series evidence summary; and the response unit 603 is further configured to: perform structured organization of the operation results of historical evidence processing operations in terms of time relationship, semantic relationship and logical relationship to obtain structured data; arrange the structured data using the evidence arrangement pattern to generate a time-series evidence summary including the verified evidence set, evidence comprehensiveness and evidence correlation graph.

[0160] In some optional implementations of this embodiment, the pattern determination unit 601 is further configured to: determine the model prediction organization pattern of the response evidence in the video and the model prediction arrangement pattern of the temporal evidence summary through the pattern determination model; and determine the model prediction organization pattern and the model prediction arrangement pattern as the organization pattern and the evidence arrangement pattern, respectively, in response to the confidence levels of the model prediction organization pattern and the model prediction arrangement pattern being greater than or equal to a preset confidence threshold.

[0161] In some optional implementations of this embodiment, the pattern determination unit 601 is further configured to: in response to the confidence of the model prediction organization pattern and the model prediction orchestration pattern being less than a preset confidence threshold, determine the organization pattern of the response evidence in the video and the evidence orchestration pattern of the temporal evidence summary according to preset prior rules.

[0162] In some optional implementations of this embodiment, the evidence determination unit 602 is further configured to: in response to the organization mode being a focused mode, determine a first preset number of candidate response evidences from the scenario representation library based on the correlation between the problem and the representation data in the scenario representation library.

[0163] In some optional implementations of this embodiment, the evidence determination unit 602 is further configured to: in response to the organization mode being a coverage mode, determine a second preset number of initial response evidence from the scenario representation library based on the correlation between the problem and the representation data in the scenario representation library; and determine a first preset number of candidate response evidence that satisfy the time separation condition from the initial response evidence, wherein the first preset number is less than the second preset number.

[0164] In some optional implementations of this embodiment, the evidence determination unit 602 is further configured to: in response to the organization mode being a comparison mode, form a query pair by combining the question and each candidate answer in the candidate answer set corresponding to the question; determine the retrieval quantity corresponding to the query pair based on a first preset quantity and the quantity of query pairs; determine the initial response evidence of the retrieval quantity from the scene representation library based on the correlation between the query pair and the representation data in the scene representation library; and obtain candidate response evidence based on the initial response evidence corresponding to each query pair.

[0165] In some optional implementations of this embodiment, the above apparatus further includes: a scene representation unit (not shown in the figure), configured to: divide the video into scenes to obtain a scene set; encode the visual data and audio data in the scenes in the scene set to obtain scene vectors; combine the scene vectors and scene metadata of the scenes to generate scene representation data; and combine the representation data of each scene in the scene set to obtain a scene representation library.

[0166] In this embodiment, a video question-answering device is provided. A pattern determination unit determines the organization pattern of the response evidence for a question in the video. The evidence determination unit uses a retrieval strategy corresponding to the organization pattern to determine candidate response evidence for the question from a scene representation library that includes representation data of the scene in the video. The response unit generates response data for the question based on the candidate response evidence. This approach specifically analyzes the organization pattern of the response evidence required for the question in the video and selects an appropriate evidence retrieval strategy to obtain candidate response evidence for response, thereby improving the accuracy and comprehensiveness of the candidate response evidence and the fit between the candidate response evidence and the question, and thus improving the accuracy and comprehensiveness of the response data.

[0167] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the video question-and-answer method described in any of the above embodiments.

[0168] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the video question-and-answer method described in any of the above embodiments when executed.

[0169] This disclosure provides a computer program product that, when executed by a processor, can implement the video question-and-answer method described in any of the above embodiments.

[0170] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0171] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0172] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0173] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as video question-answering methods. For example, in some embodiments, the video question-answering method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video question-answering method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform video question-answering methods by any other suitable means (e.g., by means of firmware).

[0174] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0175] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable video question-and-answer device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0176] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers incorporating blockchain technology.

[0180] According to the technical solution of this disclosure, a video question-answering method and apparatus are provided. The method determines the organization pattern of the response evidence for a question in the video; employs a retrieval strategy corresponding to the organization pattern to determine candidate response evidence for the question from a scene representation library including scene representation data in the video; and generates response data for the question based on the candidate response evidence. This method specifically analyzes the organization pattern of the required response evidence for the question in the video and selects an appropriate evidence retrieval strategy to obtain candidate response evidence for answering, thereby improving the accuracy and comprehensiveness of the candidate response evidence, as well as the fit between the candidate response evidence and the question, and ultimately improving the accuracy and comprehensiveness of the response data.

[0181] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0182] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video question-answering method, comprising: Determine the organizational pattern of the response evidence to the video in the video; Using the retrieval strategy corresponding to the organizational pattern, candidate response evidence for the question is determined from a scene representation library that includes representation data of the scene in the video; Based on the candidate response evidence, generate response data for the question.

2. The method according to claim 1, wherein, The step of generating response data for the question based on the candidate response evidence includes: Based on the evaluation results of the candidate response evidence regarding its comprehensiveness and credibility, target evidence processing operations are determined and executed from the operation set, wherein the operation set includes evidence supplementation operations, evidence verification operations, and response generation operations; In response to the target evidence processing operation being either the evidence supplementation operation or the evidence verification operation, the target evidence processing operation is performed to update the candidate response evidence.

3. The method according to claim 2, wherein, The step of generating response data for the question based on the candidate response evidence further includes: In response to the target evidence processing operation being the response generation operation, the target evidence processing operation is executed to generate the response data based on the current candidate response evidence.

4. The method according to claim 2, wherein, The step of determining and executing target evidence processing operations from the operation set based on the evaluation results of the candidate response evidence regarding comprehensiveness and credibility includes: Based on the results of historical evidence processing operations during the generation of the question and the response data, the evaluation results of the candidate response evidence regarding comprehensiveness and credibility are determined; Based on the evaluation results, target evidence processing operations that can improve the comprehensiveness and credibility of the candidate response evidence are identified from the set of operations.

5. The method according to claim 4, wherein, The step of determining the evaluation results of the candidate response evidence regarding comprehensiveness and credibility based on the operation results of historical evidence processing during the generation of the question and the response data includes: The results of the historical evidence processing operations are structured in terms of temporal, semantic, and logical relationships to obtain a temporal evidence summary; Based on the question, the results of the historical evidence processing operation, and the temporal evidence summary, the evaluation results of the candidate response evidence regarding comprehensiveness and credibility are determined.

6. The method according to claim 5, wherein, The determination of the organization pattern of the response evidence in the video includes: Determine the organization pattern of the response evidence in the video and the evidence arrangement pattern of the temporal evidence summary; and The results of the historical evidence processing operations are structured in terms of temporal, semantic, and logical relationships to obtain a temporal evidence summary, including: The results of the historical evidence processing operations are structured in terms of temporal, semantic, and logical relationships to obtain structured data; The structured data is arranged using the aforementioned evidence arrangement pattern to generate a time-series evidence summary that includes a verified evidence set, evidence comprehensiveness, and evidence correlation graph.

7. The method according to claim 6, wherein, The method of determining the organization pattern of the response evidence in the video and the evidence arrangement pattern of the temporal evidence summary includes: The model is determined by pattern identification, which identifies the model prediction organization pattern of the response evidence in the video and the model prediction arrangement pattern of the temporal evidence summary. In response to the confidence levels of the model prediction organization pattern and the model prediction orchestration pattern being greater than or equal to a preset confidence threshold, the model prediction organization pattern and the model prediction orchestration pattern are respectively determined as the organization pattern and the evidence orchestration pattern.

8. The method according to claim 7, wherein, The method of determining the organization pattern of the response evidence in the video and the evidence arrangement pattern of the temporal evidence summary further includes: In response to the fact that the confidence levels of the model-predicted organization pattern and the model-predicted orchestration pattern are less than the preset confidence threshold, the organization pattern of the response evidence in the video and the orchestration pattern of the temporal evidence summary are determined according to preset prior rules.

9. The method according to claim 1, wherein, The step of using the retrieval strategy corresponding to the organizational pattern to determine candidate response evidence for the question from a scene representation library including scene representation data in the video includes: In response to the organization mode being a focused mode, a first preset number of candidate response evidences are determined from the scenario representation library based on the correlation between the question and the representation data in the scenario representation library.

10. The method according to claim 1, wherein, The step of using the retrieval strategy corresponding to the organizational pattern to determine candidate response evidence for the question from a scene representation library including scene representation data in the video includes: In response to the organization mode being a coverage mode, a second preset number of initial response evidences are determined from the scenario representation library based on the correlation between the question and the representation data in the scenario representation library; From the initial response evidence, a first preset number of candidate response evidences that satisfy the time sequence separation condition are determined, wherein the first preset number is less than the second preset number.

11. The method according to claim 1, wherein, The step of using the retrieval strategy corresponding to the organizational pattern to determine candidate response evidence for the question from a scene representation library including scene representation data in the video includes: In response to the organization mode being a comparison mode, a query pair is formed by combining the question and each candidate answer in the set of candidate answers corresponding to the question; The number of searches corresponding to each query pair is determined based on the first preset number and the number of query pairs. Based on the correlation between the query pair and the representation data in the scene representation library, the number of initial response evidences retrieved are determined from the scene representation library. The candidate response evidence is obtained based on the initial response evidence corresponding to each query pair.

12. The method according to claim 1, wherein, Before determining candidate response evidence for the question from a scene representation library including scene representation data in the video using the retrieval strategy corresponding to the organizational pattern, the method further includes: The video is divided into scenes to obtain a scene set; The visual and audio data in the scenes of the scene set are encoded to obtain scene vectors; By combining the scene vector and the scene metadata, representation data of the scene is generated; The scene representation library is obtained by combining the representation data of each scene in the scene set.

13. A video question-and-answer device, comprising: The pattern determination unit is configured to determine the organization pattern of response evidence to a question in the video within the video; The evidence determination unit is configured to use a retrieval strategy corresponding to the organization mode to determine candidate response evidence for the question from a scene representation library that includes representation data of the scene in the video. The response unit is configured to generate response data for the question based on the candidate response evidence.

14. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images