Question and answer task processing, visual question and answer model training method and device

Abstract

This specification provides embodiments of question-answering task processing, a method for training a visual question-answering model, and an apparatus. The question-answering task processing method includes: acquiring question-answering task data and target visual content associated with it; using the visual perception unit of the visual question-answering model, calling visual tools to locate target entities within the target visual content, and cropping the image region corresponding to the target entity based on the location result to obtain visual context information; using the network search unit of the visual question-answering model, performing a network search based on the visual context information to obtain search evidence information; and generating question-answering results based on the visual context information, search evidence information, and question-answering task data. This "see first, then search, search driven by seeing" process design ensures that the model's answers must be based on both real visual observation and external evidence, fundamentally suppressing reliance on memorized internal parameters.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to a question-answering task processing, visual question-answering model training method and apparatus. Background Technology

[0002] In recent years, with the rapid development of multimodal large-scale models and intelligent agent technologies, deep question answering and reasoning capabilities based on video content have become a key direction for artificial intelligence to move towards general cognition. Massive amounts of unstructured video data contain rich visual semantic information, providing an important foundation for building intelligent systems with perception, retrieval, and reasoning capabilities.

[0003] However, existing methods generally suffer from modal bias when dealing with complex video question answering tasks: the models tend to rely directly on text priors or internal parameter memory to answer, skipping fine-grained visual analysis of video content, which makes it impossible to reliably perform multi-hop reasoning based on real visual evidence and difficult to generate high-quality question answering results with information gaps. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a question-answering task processing method. One or more embodiments of this specification also relate to a visual question-answering model training method, a question-answering task processing apparatus, a visual question-answering model training apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a question-answering task processing method is provided, including: Obtain the target visual content associated with the question-and-answer task data; The visual perception unit of the visual question answering model calls visual tools to locate target entities in the target visual content, and crop the image region corresponding to the target entity based on the location result to obtain visual context information. The network search unit of the visual question answering model performs network retrieval based on visual context information to obtain retrieval evidence information; Question-answering results are generated based on visual contextual information, retrieval evidence information, and question-answering task data.

[0006] According to a second aspect of the embodiments of this specification, a visual question answering model training method is provided, comprising: Acquire sample question-and-answer task data, sample visual content associated with the sample question-and-answer task data, and acquire question-and-answer decision trajectory. The question-and-answer decision trajectory includes an action sequence of first using visual tools to locate entities and perform visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate answers. The initial model is trained under supervision using sample question-answering task data, sample visual content, and question-answering decision trajectories to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and is used to call the visual perception unit and the network exploration unit according to the action sequence. The intermediate model is trained using reinforcement learning based on sample question-answering task data and sample visual content to obtain a trained visual question-answering model.

[0007] According to a third aspect of the embodiments of this specification, a question-answering task processing apparatus is provided, comprising: The first acquisition module is configured to acquire the question-and-answer task data and the target visual content associated with the question-and-answer task data; The visual processing module is configured to use the visual perception unit of the visual question answering model to call visual tools to locate target entities in the target visual content, and crop the image region corresponding to the target entity based on the location result to obtain visual context information. The search module is configured to perform network retrieval based on visual context information through the network search unit of the visual question answering model to obtain retrieval evidence information. The generation module is configured to generate question-and-answer results based on visual context information, retrieved evidence information, and question-and-answer task data.

[0008] According to a fourth aspect of the embodiments of this specification, a visual question-answering model training apparatus is provided, comprising: The second acquisition module is configured to acquire sample question-and-answer task data, sample visual content associated with the sample question-and-answer task data, and acquire question-and-answer decision trajectory. The question-and-answer decision trajectory includes an action sequence of first using visual tools to perform entity localization and visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate an answer. The first training module is configured to supervise the training of the initial model using sample question-and-answer task data, sample visual content, and question-and-answer decision trajectories to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and is used to call the visual perception unit and the network exploration unit according to the action sequence. The second training module is configured to use sample question-answering task data and sample visual content to perform reinforcement learning training on the intermediate model, so as to obtain a trained visual question-answering model.

[0009] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-described question-answering task processing method or visual question-answering model training method.

[0010] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described question-answering task processing method or visual question-answering model training method.

[0011] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described question-answering task processing method or visual question-answering model training method.

[0012] One embodiment of this specification implements the acquisition of target visual content associated with question-and-answer task data. Through the visual perception unit of the visual question-and-answer model, visual tools are invoked to locate target entities in the target visual content, and the image region corresponding to the target entity is cropped based on the location result to obtain visual context information. Through the network search unit of the visual question-and-answer model, network retrieval is performed based on the visual context information to obtain retrieval evidence information. Based on the visual context information, retrieval evidence information, and question-and-answer task data, question-and-answer results are generated. The model is required to locate and crop entities in the target visual content through the visual perception unit, thereby explicitly extracting task-related visual context information. Only on this basis is it allowed to trigger network retrieval based on the visual context to obtain external evidence. This "see first, then search, search driven by seeing" process design ensures that the model's answer must be based on both real visual observation and external evidence, fundamentally suppressing reliance on internal parameter memorization. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a question-and-answer task processing method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a visual question-answering model training method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing steps of a visual question-answering model training method provided in one embodiment of this specification. Figure 4 This is a flowchart illustrating the processing steps of another visual question-answering model training method provided in one embodiment of this specification. Figure 5 This is a schematic diagram of the structure of a question-and-answer task processing device provided in one embodiment of this specification; Figure 6This is a schematic diagram of the structure of a visual question-answering model training device provided in one embodiment of this specification; Figure 7 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] LLM: Large Language Model.

[0020] VLM: Vision-Language Model.

[0021] DR: Deep Research.

[0022] KL divergence: also known as Kullback-Leibler divergence.

[0023] In recent years, the development of Artificial General Intelligence (AGI) has been accelerating towards a new stage with autonomous perception, reasoning, and action capabilities, among which the rise of Deep Research Agents (DRAs) is particularly noteworthy. These agents, relying on the powerful generalization capabilities of large language models and multimodal foundational models, are already able to proactively invoke external tools to perform multi-hop information retrieval, evidence integration, and complex problem-solving in open network environments. With the maturity of Visual-Language Models (VLMs), the capabilities of these agents have further expanded from the pure text domain to hybrid environments including rich media content such as images and web page layouts, initially demonstrating their potential for end-to-end cognitive exploration in the real digital world.

[0024] Against this backdrop, users have raised higher-level demands on intelligent systems: they not only require them to understand static text and image content, but also expect them to delve into dynamic, continuous, and information-dense video streams to complete complex question-and-answer tasks with real information gaps. For example, in an unannotated news video or educational recording, a user might ask, "Which component of this experimental setup deformed in the third minute? What are its material properties?"—such questions require the system to accurately locate specific entities appearing across frames, extract their visual features, and proactively initiate external knowledge retrieval based on these entities, ultimately fusing multi-source evidence to generate a reliable answer. This demand essentially points to a new type of deep video research capability, where the intelligent agent needs to achieve an organic unity of fine-grained perception, tool-based exploration, and multi-hop reasoning across spatiotemporal dimensions.

[0025] To address these challenges, existing technologies primarily follow the paradigm of static multimodal agents, simplifying videos into sets of keyframes and directly applying the Vision-DR framework. Typical approaches include: extracting representative frames using a visual model, independently performing object detection or image-text matching on each frame, and then combining this with a larger model for question understanding and answer generation; some methods introduce simple tool-calling mechanisms, allowing the model to selectively trigger image cropping or text search during inference. These solutions have achieved some success in closed-domain or short-time video tasks, initially realizing the association between visual content and external knowledge.

[0026] However, when faced with long-term, highly complex video deep search tasks in real-world scenarios, existing technologies reveal fundamental flaws. First, their architecture fails to structurally decouple visual perception from network exploration, making the model highly susceptible to "modal bias"—that is, prioritizing internal parameter memory or directly initiating broad text searches, completely skipping the active localization and verification of specific entities in the video, thus losing the mandatory reliance on visual evidence. Second, the lack of phased constraints on tool invocation order and permissions prevents the agent from forming a reliable inference chain of "visual anchoring first, then network evidence gathering." Furthermore, existing training data largely originates from manually labeled short question-and-answer pairs, lacking both real-world multi-step tool interaction trajectories and the dynamic characteristics of visual entity evolution over time, preventing the model from learning long-term decision-making logic for videos. These problems collectively lead to the current system's poor performance and unreliability when facing complex tasks requiring cross-frame tracking, entity cropping, and multi-hop retrieval.

[0027] To address the aforementioned issues, this specification provides a question-answering task processing method, a question-answering task processing device, a visual question-answering model training device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0028] See Figure 1 , Figure 1 This is a flowchart of a question-and-answer task processing method provided in one embodiment of this specification, specifically including the following steps 102-108.

[0029] Step 102: Obtain the target visual content associated with the question-and-answer task data.

[0030] Question-and-answer task data refers to the text of questions posed by users that require answers in conjunction with visual information, such as "What action did the person in the red clothes in the video take?".

[0031] Target visual content refers to images or video clips that are semantically related to the question-answering task data. These can be provided by the user or automatically retrieved by the system based on the question text. For example, a surveillance video or short video clip showing a person in red clothing moving around.

[0032] In practical applications, the system first receives question-and-answer task data input by the user, and then determines whether the task includes target visual content. If the user has provided target visual content, it is directly bound to the question-and-answer task data; if the user only inputs a text question, the system retrieves the most relevant visual material from a local repository or external resources based on the question's semantics as the target visual content. The retrieval process can be implemented using methods such as keyword matching, cross-modal embedding alignment, or similarity calculation using a pre-trained visual language model. Finally, the system pairs the question-and-answer task data with the determined target visual content for subsequent module processing.

[0033] For example, in a user video question-and-answer assistance system, a user submits the question, "What device is the person in the gray coat operating in this video?", along with a 15-second short video clip as the target visual content. Upon receiving the request, the system binds the question text and the video clip together as a complete question-and-answer task input, ensuring a direct correspondence between the question and the visual evidence.

[0034] Furthermore, the process of acquiring the question-and-answer task data and the target visual content associated with it includes: acquiring the question-and-answer task data and acquiring multiple original videos associated with it; performing a duration-based threshold filter on each original video to obtain multiple candidate videos; calling a video filtering model to evaluate the content complexity of each candidate video, obtaining an information content evaluation result, and removing candidate videos with information content below a preset condition based on the information content evaluation result to obtain retained videos; and calling an image-text matching model to extract at least one keyframe from the retained videos based on the question-and-answer task data, and generating target visual content based on at least one keyframe.

[0035] Raw video refers to unprocessed video obtained from external data sources that is semantically or thematically related to the question-and-answer task data, such as video clips containing demonstrations of experimental devices retrieved from live streams of technology conferences or open-source educational video libraries.

[0036] The candidate videos are the set of videos retained after the original videos are filtered by a duration threshold, and their duration is within a preset reasonable range, such as between 3 seconds and 20 minutes.

[0037] Selected videos refer to candidate videos that meet the preset conditions after content complexity assessment. They usually include dynamic objects, text labels, or multi-element interactive scenes.

[0038] The target visual content, which is the final visual input used to support question-answering reasoning, consists of one or more keyframe images extracted from the retained video.

[0039] In practical applications, the system first receives question-answering task data and retrieves multiple original videos semantically related to the data from streaming media platforms and publicly available multi-domain video datasets. Then, the system performs duration-based filtering on each original video, eliminating videos shorter than 3 seconds or longer than 20 minutes, resulting in multiple candidate videos. Next, the system calls a video filtering model to evaluate the content complexity of each candidate video. This model analyzes the color distribution, motion changes, and semantic richness of the video frame sequence, outputting an information content evaluation result. Based on this result, the system removes candidate videos with information content scores below a preset threshold, obtaining the retained videos. Finally, the system calls an image-text matching model to encode the question-answering task data into text embeddings and calculates similarity between these embeddings and the image embeddings extracted frame-by-frame from the retained videos. At least one frame with the highest similarity is selected as a keyframe, and the target visual content is generated based on these keyframes.

[0040] For the step "calling the video screening model to evaluate the content complexity of each candidate video", one option is to use a pre-trained convolutional neural network combined with optical flow analysis to calculate the dynamic entropy value of the image. Another option is to call a large language model as the screening agent, input the sampled frame and its metadata into the model and prompt it to output a structured information score. No limitation is made here.

[0041] In the embodiments of this specification, a two-stage filtering mechanism and a semantically aligned keyframe extraction strategy are used to effectively eliminate low information density and irrelevant video interference, ensuring that the target visual content focuses on visual segments that are highly relevant to the question-answering task and rich in information.

[0042] For example, the system obtains the question-and-answer task data "What is the name of the chemical reaction vessel demonstrated in this teaching video?" and retrieves 50 original videos associated with this question-and-answer task data from publicly available datasets such as educational streaming media platforms. The system first performs filtering based on a duration threshold, removing videos shorter than 5 seconds or longer than 15 minutes, leaving 38 videos as candidates. Next, the system calls a video filtering model built with a large language model, sampling each candidate video at 1 frame per second. The image and title are input into the model for content complexity evaluation, obtaining information content evaluation results. The model identifies 12 videos as invalid segments that are purely explanatory without experimental footage or have a single background, their information content being lower than the preset condition, and these are removed, retaining 26 videos containing valid experimental operation footage. Subsequently, the system calls the CLIP image-text matching model, extracting the keyframes with the highest matching degree from the 26 retained videos based on the question-and-answer task data "name of chemical reaction vessel". Finally, the three keyframes with the highest global scores are selected and spliced ​​to generate the target visual content for subsequent modules to analyze the container's morphological features.

[0043] Furthermore, using an image-text matching model, at least one keyframe is extracted from the retained video based on the question-and-answer task data. This includes: calling the image-text matching model to extract the visual features of each video frame in the retained video based on the question-and-answer task data, and calculating the similarity of visual features between consecutive video frames in the retained video; calculating the pixel complexity of each video frame; and removing consecutive video frames with similarity higher than a similarity threshold and / or video frames with pixel complexity lower than a pixel complexity threshold from each video frame to obtain at least one keyframe.

[0044] Image-text matching models are multimodal models that can map images and text to a unified semantic vector space. A typical example is CLIP-ViT-L / 14, which achieves cross-modal alignment through contrastive learning.

[0045] Visual features refer to the high-dimensional vector representations extracted from video frames by the image encoder, which carry the semantic and structural information of the image.

[0046] Similarity is used to measure the degree of visual similarity between consecutive video frames, and is usually calculated using cosine similarity.

[0047] Pixel complexity reflects the texture richness and information content of a single frame image, and can be quantified by indicators such as standard deviation, gradient magnitude, or edge density.

[0048] Keyframes are representative video frames that have been retained after multiple filtering processes and possess both semantic relevance and high information content.

[0049] In practical applications, the system invokes an image-text matching model to extract visual features from each frame of the retained video based on question-and-answer task data. Simultaneously, the model's image encoder generates standardized high-dimensional embedding vectors. The system then calculates the cosine similarity of visual features between adjacent frames to identify redundant segments with subtle content changes. At the same time, the system independently calculates the pixel complexity of each frame to exclude solid-color backgrounds, blurry images, or static scenes. Based on this, the system executes a joint filtering strategy: removing consecutive frames with similarity higher than a preset threshold (e.g., 0.8) and simultaneously removing frames with pixel complexity lower than a set lower limit, ultimately retaining at least one keyframe that meets both conditions as the basic unit of the target visual content.

[0050] For the step "Calculate the pixel complexity of each video frame", one option is to evaluate the global texture change based on the standard deviation of the grayscale image, and another option is to use the Sobel operator to calculate the sum of gradient magnitudes to measure the amount of edge information. This is not limited here.

[0051] In the embodiments of this specification, by fusing the dual criteria of semantic similarity and pixel complexity, the interference of visual redundancy and low-information frames is effectively suppressed, ensuring that the extracted keyframes are both highly relevant to the question-answering task and have sufficient detail expression capabilities.

[0052] For example, the system receives the question-and-answer task data "What is the model of the end effector of the robotic arm shown in the picture?" For a preserved video, the system samples 60 frames at a frequency of 2 frames per second. First, it calls the CLIP-ViT-L / 14 model to extract the visual feature vector of each frame and performs L2 normalization on the vector. Next, it calculates the cosine similarity between the i-th frame and the (i+1)-th frame, and finds that the similarity of frames 10-15 and 30-35 both exceed 0.82, and is judged as static display images, which are then merged and deduplicated. At the same time, the system calculates the pixel complexity of each frame, and uses the grayscale standard deviation method to remove 5 frames with a standard deviation of less than 15 and pure white background images. Finally, in the remaining 42 frames, the system combines the question-and-answer text with the CLIP image-text matching score of each frame, and selects the 3 frames with the highest matching degree as keyframes. These frames clearly show the different angles of the robotic arm end effector and have no duplicate or low-quality images.

[0053] Step 104: Using the visual perception unit of the visual question answering model, the visual tools are invoked to locate the target entity in the target visual content, and the image region corresponding to the target entity is cropped based on the location result to obtain visual context information.

[0054] The visual perceptual unit (VPU) is a submodule in a visual question-answering model responsible for parsing input images or video frames. Its structure may include components such as convolutional neural networks, detection heads, or Transformer encoders to generate spatially perceptual features. For example, a VPU based on the DETR architecture can output bounding boxes and class predictions.

[0055] Visual tools refer to callable components with specific visual operation capabilities, such as keyframe extraction and image cropping, used to locate entities such as people and vehicles.

[0056] The target entity refers to the key visual object that is semantically related to the question-answering task data, such as the "person in blue clothes", "the sign in the upper left corner" or "the hand operating the instrument" mentioned in the question. It needs to be identified and located in the target visual content by visual tools.

[0057] Visual context information refers to the local image regions of the target entity and their derived features that are retained after cropping. It is usually represented in the form of fixed-size image blocks, feature vectors, or patch sequences with positional encoding, and is used for subsequent fusion and inference.

[0058] In practical applications, the system inputs the target visual content into the visual perception unit, which triggers a pre-defined visual tool to perform entity localization. The visual tool receives the entire image or video frame and outputs one or more candidate bounding boxes and their confidence scores. The system filters the most matching target entity bounding boxes based on semantic cues (such as noun phrases and spatial relation words) in the question-answering task data, for example, by ranking them by calculating the semantic similarity between the entity name in the question and the detection category label. After selecting the bounding box, the system expands the region by a preset ratio (such as 10%) and crops the corresponding sub-image from the original image to form a high-resolution visual context image. This image can be further fed into a lightweight encoder to generate a compact feature vector, or directly used as the visual input for multimodal fusion. The cropping process supports dynamic adjustment: when the target entity is located at the edge of the image, the system automatically pads with zeros to maintain the integrity of the cropped region; when there are multiple candidate entities, the system can crop multiple regions in parallel to form a multi-instance visual context set.

[0059] For example, when analyzing a food review video, a user asks, "What's the name of the creative fusion dish the blogger made in the video? Which restaurant serves it?" The system first sends the video frames to the visual perception unit, calls the pre-trained dish detection and scene recognition tools, and outputs bounding boxes and labels such as "[x1=200,y1=150,w=180,h=120,label=exquisitely plated main course]". The system combines the question keywords to filter out the target dish area, expands and crops it to 300×200 pixels at a 1.2x ratio as visual context. Since the specific name of the dish and the corresponding restaurant information cannot be obtained visually alone, the system passes the cropped image along with the original question to the next module. If the question is changed to "What brand is the signature sauce used in the video? Where can I buy it?", the system similarly first locates and crops the sauce bottle area to ensure the label details are visible, and then must rely on online search to find brand information and e-commerce purchase links.

[0060] Furthermore, the visual tool includes a keyframe selection tool; by using the visual perception unit of the visual question-answering model, the visual tool is invoked to locate target entities in the target visual content, and the image region corresponding to the target entity is cropped based on the location result to obtain visual context information, including: invoking the keyframe selection tool through the visual perception unit of the visual question-answering model to select target keyframes containing target entities from the target visual content; predicting bounding box coordinates for the target entities in the target keyframes; cropping the image region corresponding to the target entities based on the bounding box coordinates to obtain a cropped image, and generating visual context information based on the cropped image.

[0061] The keyframe selection tool is a submodule integrated into the visual question answering model. It is used to identify and select representative frames containing task-related target entities from the target visual content.

[0062] The target entity refers to the specific visual object that the question-answering task focuses on, such as "experimental apparatus", "traffic sign" or "product packaging".

[0063] The bounding box coordinates are rectangular region parameters represented in the form of (x_min, y_min, x_max, y_max), used to accurately define the position of the target entity in the image.

[0064] A cropped image refers to a local area of ​​a target entity extracted from the original keyframe based on the bounding box coordinates.

[0065] Visual context information is a structured visual representation of the target entity generated from a cropped image, which is used to support subsequent reasoning.

[0066] In practical applications, the system uses the visual perception unit of the visual question-answering model to invoke a keyframe selection tool to filter out target keyframes containing the target entity from the generated target visual content. This tool combines image-text matching scores with entity existence prediction to ensure that the selected frames are both highly relevant to the question-answering semantics and clearly present the target entity. Subsequently, the system predicts bounding box coordinates for the target entity in the target keyframe. This process can be completed by a pre-trained object detection model or the localization head within a large-parameter vision-language model. After obtaining the bounding box coordinates, the system performs an image cropping operation on the target keyframe based on these coordinates, extracting a cropped image containing only the target entity and its nearest neighbor environment. This cropped image is then input into the subsequent encoding module to generate focused visual context information.

[0067] For the step "predict bounding box coordinates for target entities in target keyframes", one option is to use a dedicated detection model such as YOLO or Faster R-CNN for coordinate regression. Another option is to utilize the prompting localization capability of a multimodal large model to generate bounding boxes through text guidance. This is not limited here.

[0068] In the embodiments of this specification, the coordinated processing of keyframe selection, entity localization and region clipping effectively isolates irrelevant background interference, enhances the visual representation of the target entity, and improves the question-answering model's ability to capture fine-grained visual information.

[0069] For example, when the question-and-answer task data is "What is the official price and sales channel of this limited edition figure in the picture?", the system first identifies the only frame that clearly displays the figure and the base nameplate from multiple keyframes of the target visual content using a keyframe selection tool. Then, the visual perception unit calls the built-in positioning module to predict the bounding box coordinates of the area where the figure and base markings are located, for example (150, 90, 280, 220). Based on these coordinates, the system crops a 130×130 pixel local image from the target keyframe, which fully contains the figure's detailed features and the base text information. Finally, this cropped image is fed into the visual encoder to generate visual context information specifically for external retrieval, avoiding interference from background clutter or other irrelevant labels in the entire frame.

[0070] Step 106: Using the network search unit of the visual question answering model, perform network retrieval based on visual context information to obtain retrieval evidence information.

[0071] The web search unit in a visual question-answering model is a submodule specifically designed to access external web resources for supplementary information. It can integrate search engine interfaces, knowledge graph query tools, or API clients, and is responsible for translating visual context into retrieval requests and parsing the returned results. For example, a module that encapsulates a general-purpose search engine API constitutes a web search unit.

[0072] Retrieval evidence refers to text, structured data, or multimedia snippets retrieved from the internet or designated knowledge sources and associated with a visual context, used to support answer generation. For example, encyclopedia entries, news reports, or official information pages retrieved for an image of a landmark building.

[0073] In practical applications, the system sends visual context information to the web search unit, which first performs semantic parsing of the visual context: if the input is an image, it calls OCR, image description generation models, or attribute recognition tools to convert it into a natural language query; if the input is already a text description, it directly optimizes keyword weights. Subsequently, the web search unit constructs a structured retrieval request, calls external search engine or knowledge base interfaces, and submits a query containing elements such as entity names, scene types, and timelines. The system receives the returned search results list, sorts them by relevance, selects the top K items (e.g., K=3), and extracts titles, abstracts, or text fragments to form a structured set of retrieval evidence information. This set can be further processed through noise reduction, deduplication, and key sentence extraction to ensure concise and focused information. The entire retrieval process supports multiple iterations: if the initial results are not relevance enough, the system can adjust the query terms and re-initiate the request based on feedback.

[0074] For example, when analyzing a user-uploaded street video frame, the system has cropped out an area of ​​a bus with a unique paint scheme as visual context. The web search unit first uses an image description model to generate text: "A blue and white bus with 'City Transit Line 7' printed on its body is parked next to a bus stop with the number '15'." This description is then broken down into the keyword combination "City Transit Line 7 bus route" and submitted to the city's transportation information search engine. The system returns three results: a bus route announcement on a city's official website, a local news report about the Line 7 route change, and the route details page on a public transportation app. The web search unit extracts the operating hours, stops, and vehicle model information from each result and integrates them into structured retrieval evidence information for use by the subsequent answer generation module.

[0075] Furthermore, the network search unit of the visual question-answering model performs network retrieval based on visual context information to obtain retrieval evidence information, including: generating text query information based on visual context information using the network search unit of the visual question-answering model; calling a text search tool to perform a text query and obtain a candidate webpage list; calling a webpage access tool to extract content from the webpages in the candidate webpage list to obtain candidate content, and performing semantic matching between the candidate content and the cropped image; and filtering out retrieval evidence information related to the visual context information from the candidate content based on the semantic matching results.

[0076] A candidate webpage list refers to a set of webpage links returned by a text search tool that are relevant to the query information.

[0077] Candidate content consists of text, image descriptions, or structured data extracted from candidate web pages using web access tools.

[0078] Semantic matching is used to measure the semantic consistency between candidate content and cropped image, and similarity is usually calculated using a multimodal embedding model.

[0079] The retrieved evidence information consists of external knowledge fragments that have undergone semantic filtering and are highly relevant to the target entity, used to support the generation of the final answer.

[0080] The video metadata dictionary is a structured output, containing fields such as keyframe index, bounding box coordinates, entity name, and corresponding search summary.

[0081] In practical applications, the system first uses the network search unit of the visual question-answering model to generate accurate text query information based on visual context information (especially the cropped image and its implicit semantics), for example, transforming "pressure gauge pointer points to the green area" into "meaning of the green area on a fire extinguisher pressure gauge". Then, the system calls a text search tool to execute the query and obtain a list of candidate web pages. Next, it calls a web page access tool to crawl the content of each web page in the list, extracting readable text as candidate content. Based on this, the system performs semantic matching: on the one hand, it uses an auxiliary model (such as CLIP or a dedicated image-text alignment model) to calculate the cross-modal similarity between the cropped image and the candidate content; on the other hand, it combines the Crop_Search image search tool to verify whether the content returned by the network truly corresponds to the appearance or structure of the entity displayed in the cropped area. Finally, the system filters out highly relevant candidate content based on the semantic matching score, forming retrieval evidence information.

[0082] For the step "semantic matching based on candidate content and cropped image", one option is to encode the candidate content into a text embedding and calculate the cosine similarity with the visual embedding of the cropped image; another option is to input the key descriptions in the candidate content and the cropped image into a multimodal large model, which will determine the consistency. This is not limited here.

[0083] In the embodiments of this specification, by integrating text retrieval, content extraction and multimodal semantic verification mechanisms, a high degree of alignment between external evidence and visual entities is ensured, effectively improving the reliability and interpretability of open-domain visual question answering.

[0084] For example, in a video question-and-answer scenario, a user asks the system, "What other well-known movies has the actress in the red dress dancing in the rain in the video starred in?" The system first uses a keyframe selection tool to identify target keyframes (e.g., keyframe index 78) from the target video that contain a clear frontal shot of the actress, and then uses a visual perception unit to predict its bounding box coordinates (e.g., [210, 150, 340, 400]). Subsequently, the keyframe is cropped based on the bounding box to obtain a cropped image focusing on the actress's face and clothing, and visual context information is generated based on this. Next, the web search unit automatically generates text query information based on this visual context information, such as: "Who is the actress dancing in the rain in a red dress?" or "The female lead in the movie dancing in the rain in a red dress". The system calls the text search tool to execute the query and obtain a list of candidate web pages. Then, the web page access tool extracts candidate content such as actress names, representative works, and role descriptions from each web page.

[0085] To further ensure accuracy, the system simultaneously invokes the Crop_Search image search tool, using the cropped image as input to perform reverse image retrieval and verify whether the returned results point to the same person. Meanwhile, auxiliary models (such as a multimodal alignment model) evaluate the semantic and identity consistency between the webpage text (e.g., "Emma Stone, star of La La Land," "Oscar-winning actress") and the cropped image. After semantic matching filtering, the system retains high-confidence retrieval evidence.

[0086] Step 108: Generate question-and-answer results based on visual contextual information, retrieved evidence information, and question-and-answer task data.

[0087] In practical applications, the system inputs question-answering task data, visual context information, and external evidence features into the answer generation unit of the visual question-answering model. This unit first encodes the three types of input: question-answering task data is converted into semantic vectors by a text encoder; visual context information is used to extract features through a lightweight visual encoder (such as ResNet-18 or ViT-Tiny); and external evidence features are processed by a text encoder or knowledge graph embedding module into vector representations in a unified semantic space. Subsequently, the three features undergo cross-modal alignment and interaction at the fusion layer. One option is to use a cross-attention mechanism to guide the weighted aggregation of visual and evidence features; another option is to dynamically adjust the contribution weights of each modality through a gated fusion network. The fused joint representation is then fed into the decoder to generate natural language answers word-by-word in an autoregressive manner. During decoding, the system can introduce constraint mechanisms, such as prohibiting the generation of entity names not supported in the visual or evidence context, or forcing the citation of key numerical values ​​from the retrieved evidence, to improve answer reliability.

[0088] For example, when processing the user question "When was the building in the picture completed?", the system has already obtained the cropped image of the building (visual context) and text fragments retrieved from encyclopedias and municipal archives (external evidence features: "The building was started in 1828 and completed in 1831"). The answer generation unit inputs the question text, the evidence summary of "started in 1828 and completed in 1831", and the building's exterior feature vector into the fusion module. During the generation process, the decoder prioritizes sampling "1831" as the completion time and combines it with the question focus word "completed", finally outputting "The building was completed in 1931". If the question is "What is the top speed of this car?", and the visual context shows the car model as "Tesla Model SPlaid", the external evidence returns the official parameter "top speed 322 km / h", and the system generates "The car's top speed is 322 km / h", strictly based on the evidence data, without introducing potentially outdated parameter memories within the model.

[0089] Furthermore, after generating the question-and-answer results based on visual contextual information, retrieved evidence information, and question-and-answer task data, the process includes: receiving feedback information from the target object regarding the question-and-answer results; and adjusting the visual perception unit and / or web search unit based on the feedback information.

[0090] Feedback information refers to the target audience's evaluation signals regarding the correctness, relevance, or completeness of the question-and-answer results. This includes explicit feedback such as textual instructions like "entity recognition error" or "work omission," as well as implicit feedback such as follow-up questions or negative actions.

[0091] The visual perception unit is responsible for extracting keyframes from the video, locating target objects, and generating cropped images. Its output directly affects the accuracy of subsequent query generation and image retrieval.

[0092] The web search unit is responsible for generating text queries based on visual context, performing external knowledge retrieval, and integrating multi-source evidence. Its performance determines the coverage and accuracy of the auxiliary information obtained by the system.

[0093] In practical applications, after generating question-and-answer results, the system actively receives feedback information from the target object. If the feedback indicates a deviation in the question-and-answer results, the system will analyze the feedback content to determine the source of the error. When the error originates from the visual perception unit, such as bounding box offset or improper keyframe selection, the system can adjust the confidence threshold of the object detection model, optimize the non-maximum suppression parameters, or introduce temporal consistency constraints to improve cross-frame localization stability. When the error originates from the web search unit, such as the retrieval returning irrelevant entities or missing key works, the system can reconstruct the text query template, adding scene semantics, temporal cues, or appearance details to the original visual description, and enhancing the collaborative weights between Crop_Search and the auxiliary model during the verification phase. Regarding the processing logic of feedback information, one option is to build a lightweight feedback classifier to automatically identify the error type and trigger the corresponding module's correction process; another option is to cache the feedback samples in an offline queue and perform incremental fine-tuning or rule updates in the background to avoid affecting online service latency.

[0094] In the embodiments described in this specification, by receiving feedback information and making targeted adjustments to the visual perception unit and / or web search unit, the system can continuously optimize multimodal understanding and retrieval strategies without relying on manual annotation, thereby improving its ability to analyze complex video content.

[0095] For example, in a video question-answering task, the system identifies a female actor in a dance scene and returns a list of her representative works. The target user reports, "This actress is not the person in the video." After analyzing this feedback, the system determines that the problem stems from the visual perception unit misidentifying a supporting character with a similar appearance in the background as the main character. Subsequently, the system calls the temporal alignment module to compare the consistency of the target's motion trajectory and posture in adjacent keyframes, re-selects the most representative main-view frames, and reduces the bounding box to eliminate clothing interference. Simultaneously, the web search unit generates an enhanced query containing contextual keywords such as "solo dance," "center position," and "spotlight" based on the newly cropped image, and performs cross-validation by combining it with reverse image search results. Finally, the system updates the question-answering results based on the corrected perception and retrieval output to ensure that the entity identity is consistent with the video content.

[0096] Furthermore, the visual question answering model is trained through the following steps: acquiring sample question answering task data, sample visual content associated with the sample question answering task data, and acquiring the question answering decision trajectory, wherein the question answering decision trajectory includes an action sequence of first using visual tools to perform entity localization and visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate the answer; supervising training the initial model using the sample question answering task data, sample visual content, and question answering decision trajectory to obtain an intermediate model, the intermediate model including a visual perception unit and a network exploration unit, and the intermediate model is used to call the visual perception unit and the network exploration unit according to the action sequence; reinforcement learning training of the intermediate model using the sample question answering task data and sample visual content to obtain the trained visual question answering model.

[0097] Sample question-answering task data refers to question-answer pairs used for training. The questions involve specific entities or events in the video, and the answers need to be generated based on external knowledge or visual analysis.

[0098] Sample visual content refers to the video clips or keyframe set corresponding to the sample question-and-answer task data, which serves as the input source for the visual perception unit.

[0099] The question-answering decision trajectory refers to the sequence of actions performed by the system during the completion of the question-answering task. This includes calling visual tools for entity localization and visual search to obtain visual context, as well as calling network tools for text retrieval to obtain retrieval evidence and finally generating an answer.

[0100] The visual perception unit is responsible for processing visual input and performing operations such as object detection, keyframe selection, cropping, and visual feature extraction.

[0101] The network exploration unit is responsible for generating queries based on visual context, performing external retrievals, and fusing multi-source evidence to support answer generation.

[0102] In practical applications, the system first acquires a large amount of sample question-answering task data and its associated sample visual content, and simultaneously collects or synthesizes corresponding question-answering decision trajectories. This trajectory clearly records the sequence of actions the system should take when answering each question: first, it should call the visual perception unit to complete entity localization and context extraction; then, it should call the network exploration unit to perform text retrieval and evidence integration. Subsequently, the system uses the aforementioned triplet data to conduct supervised training on the initial model, employing an autoregressive negative log-likelihood loss function to optimize the model parameters, enabling it to accurately reproduce the tool-calling logic in the expert trajectory, thus obtaining an intermediate model. This intermediate model already possesses the ability to coordinate the invocation of the visual perception unit and the network exploration unit according to a preset action sequence. Based on this, the system further utilizes the same sample data to conduct reinforcement learning training on the intermediate model: for each question, the model generates multiple sampling trajectories in parallel. The system assigns a binary reward (1 for correct, 0 for incorrect) to each trajectory through an automated verification mechanism, and uses the Group Relative Policy Optimization (GRPO) algorithm to update the policy. The advantage function is calculated based on the relative performance of the trajectories within the group, while a KL divergence penalty is introduced to constrain the policy update magnitude, and negative gradients are downsampled to suppress format spoofing.

[0103] For the supervised training phase, one option is to use a hybrid approach of pure text deep search trajectories and multimodal expert trajectories to balance the model's visual and text processing capabilities; another option is to train using only manually labeled high-quality visual question-answering trajectories. For the reinforcement learning phase, one option is to use the GRPO algorithm for relative policy optimization of the valueless network; another option is to combine traditional reinforcement learning frameworks such as PPO or A2C and introduce an independent value estimation module.

[0104] In the embodiments of this specification, through a two-stage training mechanism, the model not only internalizes the reasonable decision-making process of "visual localization first, then network exploration", but also enhances its ability to autonomously explore and correct unknown scenarios during the reinforcement learning stage, thus avoiding the rigidity of behavior caused by simple imitation.

[0105] For example, in a video understanding task platform, the system collected 100,000 question-and-answer pairs related to film and television character recognition, each associated with a 30-second video clip. Simultaneously, the expert system generated a complete decision trajectory for each question and answer: first, the visual perception unit was invoked to locate the female character dancing in a red dress in the video, outputting her bounding box and cropped image; then, the network exploration unit was invoked to generate a text query based on the image and retrieve relevant actor information; finally, the evidence was integrated to generate the answer. During supervised training, the model learned to reproduce this trajectory, significantly improving accuracy. In the reinforcement learning phase, the model generated five different trajectories for the same question. Some trajectories attempted to skip visual localization and directly retrieve the information, but these were verified as errors and received no reward; while the trajectories following the correct process received a reward of 1. The GRPO algorithm adjusted the policy weights accordingly, gradually converging the model to a high-success-rate tool-calling pattern, ultimately forming a robust visual question-answering capability.

[0106] Furthermore, the initial model is trained under supervision using sample question-and-answer task data, sample visual content, and question-and-answer decision trajectories to obtain an intermediate model. This includes: inputting sample visual content and sample question-and-answer task data into the initial model to generate a predicted action sequence; calculating the loss based on the difference between the predicted action sequence and the question-and-answer decision trajectory; and updating the parameters of the initial model based on the loss to obtain the intermediate model.

[0107] Predicted action sequence refers to the sequence of tool invocation and inference steps that the initial model autonomously generates under the given sample visual content and sample question-and-answer task data.

[0108] Intermediate models refer to visual question answering models that have undergone supervised training and possess preliminary multi-tool collaborative capabilities. They contain visual perception units and network exploration units that can be invoked sequentially.

[0109] In practical applications, the system inputs sample visual content and sample question-answering task data into the initial model, triggering its autoregressive generation of a complete predicted action sequence. This sequence includes the tool type (such as visual localization, image cropping, text query generation, web page retrieval, etc.) and its corresponding operational parameters selected by the model at each step. Subsequently, the system aligns this predicted action sequence with the question-answering decision trajectory provided by experts token-by-token, and uses a negative log-likelihood loss function to calculate the difference between the two. The loss value reflects the degree of deviation of the model in action selection and sequence arrangement. Based on this loss, the system updates the parameters of the initial model through a backpropagation algorithm, gradually approximating the reasonable decision-making pattern reflected by the expert trajectory. After multiple rounds of iterative training, the action sequence output by the model is highly consistent with the question-answering decision trajectory. The model obtained at this point is the intermediate model, which has the ability to coordinate and call the visual perception unit and the network exploration unit according to preset logic.

[0110] For the loss calculation method, an optional way is to adopt the standard autoregressive language modeling loss, that is = ∑ log πθ(at∣a<t, x), where x is the joint input containing visual and text context, and at is the true action at the t-th step in the trajectory; another optional implementation is to introduce an action type-aware weighted loss, assigning higher weights at key decision points (such as tool switching points) to strengthen the sequential constraints.

[0111] In the embodiments of this specification, by using the question-answering decision-making trajectory as the supervision signal, the model can internalize the correct execution logic of multi-tool collaboration at the initial stage of training, avoiding the policy confusion brought by unsupervised exploration, and laying a stable behavioral foundation for the subsequent reinforcement learning stage.

[0112] Exemplarily, in a movie character question-answering task, the sample question-answering task data is "Who is the actress dancing in a red dress in the video?", the sample visual content is a 30-second video clip of a musical, and the question-answering decision-making trajectory sequentially includes the following actions: (1) Call the visual perception unit to detect all characters; (2) Filter out the target wearing a red dress and in the central position; (3) Crop the target area; (4) Generate a text query "Red dress female dancer musical" based on the cropped image; (5) Call the network exploration unit to retrieve relevant actor information; (6) Integrate the retrieval results to generate the final answer. When the initial model first inputs this sample, it may wrongly perform a network search first or omit the cropping step. The system calculates the loss and updates the parameters by comparing the predicted action sequence with the above trajectory. After multiple rounds of training, when the model faces similar inputs, it can stably reproduce the correct six-step action sequence and form a reliable intermediate model.

[0113] Furthermore, the initial model is supervised and trained using the sample question-answering task data, sample visual content, and question-answering decision-making trajectory, including: obtaining pure text deep search trajectory data, which is composed of question-answering tasks that only involve text search and reasoning; mixing the pure text deep search trajectory data and the question-answering decision-making trajectory in a preset ratio to obtain mixed training data; and using the sample question-answering task data, sample visual content, and mixed training data to supervise and train the initial model.

[0114] The pure text deep search trajectory data refers to the action sequence generated by question-answering tasks that only rely on text input and can be completed without visual content. Its typical tasks include fact-based question-answering, logical reasoning, or complex query decomposition based on a knowledge base. The action sequence usually includes multiple steps such as multi-round text query generation, web page retrieval, evidence screening, and answer integration.

[0115] The question-answering decision-making trajectory refers to the complete tool call sequence of "first visual localization, then network retrieval" executed by the system in question-answering tasks involving visual content.

[0116] Hybrid training data refers to a unified training set formed by combining the two types of trajectories in a preset ratio (such as 1:1), which is used to balance the learning of behavioral patterns of the model under different task types.

[0117] In practical applications, the system first acquires a large amount of pure text deep search trajectory data from historical logs or synthetic data to ensure coverage of diverse text reasoning scenarios. Then, this data is mixed with existing multimodal question-answering decision trajectories according to a preset ratio (e.g., 50% pure text trajectories, 50% visual-text hybrid trajectories) to construct a unified hybrid training dataset. During supervised training, the system inputs sample question-answering task data, corresponding sample visual content (if available), and the hybrid training data into the initial model: for multimodal samples, the model needs to combine visual content to generate action sequences including visual tool calls; for pure text samples, it only generates text search and reasoning action sequences based on the question text. The model predicts each token of the action sequence using an autoregressive approach and calculates a negative log-likelihood loss with the real trajectories in the hybrid training data to update the parameters. This process enables the model to continuously maintain its deep processing capabilities for complex text tasks while learning visual perception and tool collaboration.

[0118] Regarding the setting of the mixing ratio, one option is to use a fixed ratio (such as 1:1) to ensure the balanced development of the two types of capabilities; another option is to dynamically adjust the ratio according to the task difficulty or data distribution, focusing on multimodal trajectories in the early stage of training and gradually adding plain text trajectories in the later stage to enhance generalization ability.

[0119] In the embodiments of this specification, by jointly supervising the training of the model by mixing the pure text deep search trajectory and the multimodal question answering decision trajectory, the model can improve the performance of visual question answering while avoiding the degradation of text reasoning ability caused by over-focusing on the visual module, thus achieving a balance of capabilities in multi-task scenarios.

[0120] For example, in a comprehensive question-answering platform, the system collected two types of training data: one type was plain text question-answering questions like "What is the name of the black hole in *Interstellar*?", whose trajectory included generating the query "Interstellar blackhole name", searching encyclopedias, extracting "Gargantua", and verifying the source; the other type was questions like "Who is the actress dancing in the red dress in the video?", whose trajectory included multimodal steps such as visual localization, cropping, and image retrieval. The system mixed these two types of trajectories in a 1:1 ratio to form 100,000 training samples. During training, the initial model alternately received plain text questions and questions with videos, outputting corresponding action sequences respectively. After multiple rounds of training, the model could accurately execute complex visual-retrieval links and efficiently complete deep text reasoning tasks without visual input, demonstrating good multi-task adaptability.

[0121] Furthermore, obtaining sample question-and-answer task data includes: obtaining candidate question-and-answer task data; using a teacher question-and-answer model to perform static inference on the candidate question-and-answer task data to obtain a first inference result; if the first inference result is correct, removing the candidate question-and-answer task data; if the first inference result is incorrect, using the candidate question-and-answer task data as sample question-and-answer task data.

[0122] Candidate question-answering task data refers to the raw question-answer pairs initially collected from video platforms, user logs, or synthesis systems. The questions vary in format and difficulty.

[0123] Teacher question-answering models refer to closed-source or frozen-parameter models (such as large language models or pre-trained visual language models) that have strong performance but lack the ability to invoke tools. They can only perform single-step inference based on given video descriptions or static frame information and cannot perform dynamic visual localization or network retrieval.

[0124] The first inference result refers to the answer generated by the teacher model for the candidate question based solely on static input (such as video summaries, keyframe titles, or fixed visual features).

[0125] The semantic gap refers to the information gap that the entities, relationships, or events involved in a problem cannot be directly inferred from a static visual description and must be bridged by actively using tools (such as accurately locating the target object, cropping the image, and reverse searching).

[0126] In practical applications, the system first acquires a large amount of candidate question-answering task data. Then, each candidate question and its associated static visual information (such as automatic captions, scene labels, or global image embeddings) are input into the teacher question-answering model, and a static inference is performed to obtain the first inference result. The system uses automated verification mechanisms (such as comparison with the standard answer, logical consistency checks, or multi-model voting) to determine whether the result is correct. If the first inference result is correct, it indicates that the question can be solved without using any tools, belonging to a semantically simple sample, and is therefore discarded. If the first inference result is incorrect, it indicates that the question has a semantic gap and requires more refined visual perception or external knowledge retrieval to solve; therefore, this candidate question-answering task data is retained as the final sample question-answering task data.

[0127] For the design of the verification mechanism, one option is to use exact matching or fuzzy matching (such as BLEU, ROUGE) to evaluate the consistency of the answers; another option is to introduce an auxiliary discrimination model to perform secondary verification of the confidence and reasonableness of the teacher model's answers in order to reduce false screening.

[0128] In the embodiments of this specification, the static reasoning ability of the teacher model is used as a "filter" to effectively remove simple problems with low difficulty and high redundancy, ensuring that the training dataset focuses on complex tasks that require multiple tools to solve together, thereby improving the model training efficiency and reasoning depth.

[0129] For example, in a video question-and-answer data construction process, candidate questions include: "How many actors appear in the video?" (simple question) and "Who is the man in the blue suit holding a pocket watch in the video?" (complex question). The teacher model can correctly answer the former based on the video subtitle "Three people enter the room, one of whom checks the pocket watch," so this sample is discarded. However, for the latter, because the subtitle does not mention clothing details or the character's identity, the teacher model incorrectly answers "unknown character," and the verification fails. Therefore, this question is retained as sample question-and-answer task data for subsequent training of an agent with visual localization and network retrieval capabilities.

[0130] Furthermore, obtaining the question-and-answer decision trajectory includes: obtaining candidate decision trajectories; using the teacher question-and-answer model to infer a second inference result from the sample question-and-answer task data according to the candidate decision trajectories; removing the candidate decision trajectories if the second inference result is incorrect; and using the candidate decision trajectories as the question-and-answer decision trajectories if the second inference result is correct.

[0131] Candidate decision trajectories refer to the initial sequence of tool call actions generated for a specific sample of question-and-answer task data. They typically include steps such as visual localization, image cropping, query generation, web retrieval, and evidence fusion, and may originate from expert annotation, rule systems, or sampling.

[0132] Teacher question-answering models refer to a reference model with strong reasoning ability but frozen parameters (such as a high-performance closed-source large model). Its role is not to generate trajectories, but to faithfully execute the given candidate decision trajectories—that is, to strictly follow the tool calling order and parameters specified in the trajectory, gradually obtain visual context and retrieval evidence, and finally generate an answer.

[0133] The second reasoning result refers to the final answer generated by the teacher question-and-answer model through the complete reasoning process performed strictly following the candidate decision trajectory.

[0134] In practical applications, the system first acquires one or more candidate decision trajectories for each sample question-answering task data. Then, this trajectory is "replayed" to the teacher's question-answering model: the model sequentially executes each step of the trajectory (e.g., first calling visual tools to locate the target area, then generating a text query based on that area and retrieving web pages), and outputs the answer as the second inference result in the final step. The system evaluates the correctness of this result through automated verification mechanisms (such as comparison with the standard answer, fact-checking, or semantic equivalence judgment). If the second inference result is incorrect, it indicates that the candidate decision trajectory has logical flaws, tool misuse, or missing information, and cannot guide the model to obtain the correct answer; therefore, it is discarded. If the second inference result is correct, it indicates that the trajectory is a valid and reproducible reasonable behavioral path, and it is retained as the official question-answering decision trajectory for subsequent supervised training.

[0135] Regarding the verification method, one option is to use strict string matching or precise comparison after normalization of standardized answers; another option is to introduce a third-party discriminator or a soft matching mechanism based on embedding similarity to handle answers that have different expressions but the same semantics.

[0136] In the embodiments of this specification, by using the teacher model as the "execution engine" to perform closed-loop verification of candidate trajectories, a screening leap from "formally correct" to "functionally correct" is achieved, ensuring that all retained question-and-answer decision trajectories are not only structurally complete but also practically effective, thereby providing the model with high-fidelity supervision signals.

[0137] For example, in a task about "Who is the woman in the white dress playing the piano in the video?", the system generates two candidate decision trajectories: Trajectory A: First detect all people → filter those wearing white dresses → crop their upper bodies → use image reverse search → obtain "EmmaStone"; Track B: Direct full-text search for "woman in white dress playing piano" without using visual tools.

[0138] After the teacher model executes trajectory A, it successfully retrieves the actor's identity, and the answer is correct, so trajectory A is retained. However, when executing trajectory B, due to the lack of specific visual cues, the retrieval returns a noisy result of "unknown musician," and the answer is incorrect; trajectory B is discarded. Ultimately, only trajectory A, which leads to the correct answer, is included in the training set and used as the question-answering decision trajectory. Further, after obtaining the trained visual question-answering model, the process includes: acquiring verification question-answering task data and the verification visual content associated with the verification question-answering task data; inputting the verification question-answering task data and the verification visual content into the trained visual question-answering model to obtain the verification results.

[0139] Furthermore, after obtaining the trained visual question answering model, the process also includes: acquiring the verification question answering task data and the verification visual content associated with the verification question answering task data; inputting the verification question answering task data and the verification visual content into the trained visual question answering model to obtain the verification results.

[0140] Validation question-answering task data refers to question-answer pairs collected from data sources that are distributed in the same way as the training set but do not overlap with it. The question design covers different levels of difficulty, including complex queries that require combining visual details with external knowledge (such as "What movie character is the man in the red jacket riding a black motorcycle in the video?"), in order to comprehensively test the boundaries of the model's capabilities.

[0141] Visual content verification refers to video clips, keyframe sequences, or image sets that correspond one-to-one with the data from the verification question-and-answer task, serving as input for the model to perform visual perception and tool calls.

[0142] The verification result refers to the answer or action sequence that the trained visual question answering model autonomously executes after receiving verification question answering task data and verification visual content, including visual localization, image cropping, web retrieval, evidence integration, etc., and finally outputs.

[0143] In practical applications, the system first constructs a high-quality validation set, ensuring it covers diverse entity types, action scenarios, occlusion conditions, and semantic complexity, while excluding any entries that are duplicates or highly similar to the training samples. Then, each validation question-answering task data and its associated validation visual content are input into the trained visual question-answering model. The model autoregressively generates action sequences based on its internal strategies, calls appropriate tools to obtain intermediate information, and finally outputs a natural language answer as the validation result. This result will be used for subsequent automatic or manual evaluation, including dimensions such as accuracy, robustness, and the appropriateness of tool use.

[0144] Regarding the evaluation method, one option is to use automated metrics (such as Exact Match, F1 score, BLEU, or semantic equivalence judgment based on a large model) to compare the verification results with the standard answer; another option is to introduce human scoring to give fine-grained scores for the factual correctness of the answer, the completeness of the reasoning chain, and the logic of tool calls.

[0145] In the embodiments of this specification, by running the complete inference process on an independent validation set, not only can the model performance be quantified, but its weaknesses in specific scenarios (such as low light, small targets, and ambiguous expressions) can also be exposed, providing a clear direction for model iterative optimization.

[0146] For example, in a film and television character recognition verification task, the verification question-and-answer task data is "Who is the actress holding a gold trophy and standing on the left side of the stage in the video?", and the verification visual content is a keyframe from an awards ceremony video. The trained model first uses visual tools to detect the positions of all characters and trophies, locates the woman holding the trophy on the left, crops her face, generates the query "Golden Globe Awards 2025 left-side actress," retrieves "SandraOh," and outputs this answer as the verification result. This result is automatically compared and confirmed to be consistent with the standard answer, indicating that the model performed correctly on this verification sample. By summarizing a large number of such verification results, the overall accuracy can be calculated, forming an objective evaluation of the model's capabilities.

[0147] Furthermore, acquiring verification question-and-answer task data includes: acquiring seed question-and-answer task data formed by the target object's annotation of verification visual content; expanding the seed question-and-answer pairs with keywords, and conducting web searches based on the expanded keywords to obtain webpage evidence; generating multi-level reasoning questions based on the webpage evidence, and performing filtering processing on the multi-level reasoning questions through the target object; conducting quality assessment on the filtered multi-level reasoning questions, and generating verification question-and-answer task data based on the target multi-level reasoning questions that meet the quality assessment conditions.

[0148] Seed question-and-answer task data consists of initial question-and-answer pairs generated by the target audience through manual annotation based on visual content verification, reflecting directly observable facts within the visual content. For example, a staff member annotates a video with the question "What is the person in the red shirt doing in the video?" and provides the answer "Playing basketball".

[0149] Keyword expansion refers to the process of extracting core entities or attributes from seed question-and-answer pairs and supplementing them with relevant terms for the reasoning dimension from the target object. For example, keywords such as "sports brand," "professional team," and "event name" can be expanded from "red clothes" and "playing basketball."

[0150] Webpage evidence refers to external information sources obtained through web searches based on expanded keywords, used to support the generation of subsequent multi-level reasoning questions. For example, searching for "red jersey professional basketball league" returns relevant news, team websites, or social media content.

[0151] Multi-level reasoning questions are those that require two or more steps of logical deduction, combining visual content and webpage evidence, to answer. For example, "Did the team of the player wearing the red jersey in the video make the playoffs in 2025?"

[0152] The filtering process involves manually judging the automatically generated multi-level reasoning questions for the target object, retaining questions that meet the task requirements, and eliminating ambiguous, unsolvable, or visually irrelevant questions.

[0153] The quality assessment involves scoring the selected questions based on preset standards, retaining only those questions that meet the criteria of accuracy, solvability, and reasoning depth as the final verification data.

[0154] In practical applications, the system first acquires seed question-and-answer task data, generated by the target user's manual observation and annotation of the verification visual content, ensuring a strong correlation between the questions and the visual content and verifiable answers. Subsequently, the system automatically extracts key entities and actions from the seed question-and-answer pairs, and the target user evaluates or supplements these with keywords for reasoning, forming an expanded keyword set. Based on this set, the system calls a web search interface to obtain relevant web page evidence, including structured or unstructured text such as news reports, encyclopedia entries, and social media posts. Next, the system utilizes the cross-relationship between the web page evidence and the original visual information to automatically generate questions requiring multi-step reasoning. These multi-level reasoning questions are then processed by the target user, who manually filters them based on criteria such as solvability, reliance on visual input, and the presence of factual errors. For questions that pass the screening, the target user further performs a quality assessment, including checking the uniqueness of the answer, the rationality of the reasoning chain, and the clarity of the language. Only when a question meets the preset quality assessment conditions is it integrated with the corresponding verification visual content, standard answer, and reference evidence to generate the final verification question-and-answer task data.

[0155] For the keyword expansion step, one option is for the target user to manually add common-sense attribute words related to the visual entity; another option is to use a pre-trained language model to automatically recommend expansion words, which are then confirmed by the target user. For the multi-level reasoning question generation step, one option is template-based generation; another option is to use a large language model to freely generate questions based on visual descriptions and webpage evidence.

[0156] In the embodiments of this specification, the manual intervention of the target object in the keyword expansion, question screening and quality assessment stages ensures that the generated verification question-answering task data has high factual accuracy, strong visual dependence and reasonable reasoning depth, effectively improving the discrimination of the verification set on the model's ability to perform complex tasks.

[0157] For example, in the construction of a film and television character recognition verification task, the system provides an awards ceremony video as verification visual content. After watching the video, the target user labels a seed question-and-answer pair: "Who is the woman standing in the center of the stage holding a gold trophy in the video?" The answer is "Emma Stone." The system extracts keywords such as "Emma Stone," "gold trophy," and "awards ceremony," and the target user adds inference dimension words such as "2025," "film awards," and "winning work." Based on these keywords, the system initiates a web search and obtains web evidence that "Emma Stone won the 2025 Golden Globe Award for Best Actress for 'Moonlight Sonata.'" Subsequently, the system generates a multi-level inference question: "For which film did the actress holding the Golden Globe trophy in the video win the award?" This question is given to the target user for selection, confirming that they need to first identify the person's identity and then combine external knowledge to locate the winning work, which meets the requirements of multi-level inference. The target user further evaluates that their answer is clear, unambiguous, and linguistically correct, and is determined to meet the quality assessment criteria. Ultimately, this question, along with the original video, the standard answer "Moonlight Sonata," and the webpage evidence link, constituted a verification question-and-answer task dataset.

[0158] Furthermore, after conducting a quality assessment on the selected target multi-level reasoning questions and generating verification question-answering task data based on the target multi-level reasoning questions that meet the quality assessment criteria, the process also includes updating the seed question-answering task data based on the target multi-level reasoning questions.

[0159] In practical applications, after the system completes the quality assessment of the selected target multi-level reasoning questions and generates verification question-answering task data based on the questions that meet the quality assessment criteria, it further incorporates these high-quality target multi-level reasoning questions into the seed question-answering task data pool. Specifically, the system identifies target multi-level reasoning questions that have passed all evaluation stages and are determined to be high-value, and stores them in a structured manner as new seed question-answer pairs. The question portion serves as the new seed question, and its corresponding standard answer serves as the seed answer. These new seed question-answer pairs, together with the original manually labeled seed data, constitute the updated seed question-answering task dataset, which is used for the next round of keyword expansion and the generation of deeper reasoning questions. This update process can be executed periodically, forming a closed-loop mechanism of "generation—screening—evaluation—recirculation," enabling the seed pool to continuously absorb high-difficulty, high-fidelity question samples.

[0160] For the seed question-and-answer task data update step, one option is to only return target multi-level reasoning questions with scores reaching a preset threshold (such as 4.5 / 5 or above) to the seed. Another option is to have the target object confirm a second time whether to add specific high-quality questions to the seed library, so as to balance the efficiency of automation and the reliability of human judgment.

[0161] In the embodiments of this specification, target multi-level reasoning questions that meet the quality assessment conditions are fed back to update the seed question-answering task data, enabling the seed bank to have self-evolution capabilities. It can gradually accumulate high-quality basic questions covering a wider range of knowledge domains and reasoning patterns, providing a richer starting point for the subsequent generation of higher-order verification tasks.

[0162] For example, in a verification data construction task for an international conference scenario, the system initially uses seed questions labeled by staff: "Who is the man in the dark blue suit speaking in the video?", with the answer being "Speaker A". After keyword expansion, webpage evidence acquisition, and multi-level reasoning generation, the question is obtained: "At which higher education institution did the former department head introducing the climate cooperation initiative in the video attend?". This question, through target object screening and quality assessment, confirms that its reasoning chain is complete, the answer is clear, and it relies on visual identity recognition and external knowledge fusion. Subsequently, the system adds this question and its answer "Northern University" as a new seed question-answer pair to the seed question-answering task dataset. In the next data generation cycle, this new seed can be used to further expand keywords (such as "Northern University" and "educational background of public affairs figures"), thereby generating deeper reasoning questions such as "In which administrative region did the speaker who graduated from a well-known research university in the video serve as a public official?", achieving continuous enhancement of the seed bank and a step-by-step increase in task complexity.

[0163] In one embodiment of this specification, the model is required to locate and crop entities in the target visual content through the visual perception unit, thereby explicitly extracting task-related visual context information. Only on this basis is it allowed to trigger network retrieval based on the visual context to obtain external evidence. This "see first, then search, and search by seeing" process design ensures that the model's answer must be based on both real visual observation and external evidence, fundamentally suppressing the reliance on internal parameter memory.

[0164] See Figure 2 , Figure 2 This is a flowchart of a visual question answering model training method provided in one embodiment of this specification, specifically including the following steps 202-206.

[0165] Step 202: Obtain sample question-answering task data, sample visual content associated with the sample question-answering task data, and obtain the question-answering decision trajectory. The question-answering decision trajectory includes an action sequence of first using visual tools to perform entity localization and visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate an answer.

[0166] Step 204: Supervised training of the initial model is performed using sample question-answering task data, sample visual content, and question-answering decision trajectory to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and is used to call the visual perception unit and the network exploration unit according to the action sequence.

[0167] Step 206: Use sample question-answering task data and sample visual content to perform reinforcement learning training on the intermediate model to obtain the trained visual question-answering model.

[0168] It is understood that steps 202-206 above are essentially the same as the aforementioned embodiment of visual question answering model training, and will not be repeated here.

[0169] One embodiment of this specification introduces a question-answering decision trajectory containing a structured action sequence of "first visual localization, then network retrieval" to supervise the training of the initial model. This allows the intermediate model to explicitly internalize the execution logic of decoupling perception and exploration at the parameter level, effectively suppressing modality bias. Building upon this, reinforcement learning training is further implemented on the intermediate model using original sample data, enabling it to autonomously generate high-value decision paths in complex video scenarios while preserving the correct tool invocation order. This two-stage training strategy ensures both the interpretability and reliability of the model's behavior and stimulates its generalization and exploration potential on unseen tasks, thereby significantly improving the evidential dependence and inference depth of answers in video deep question-answering tasks.

[0170] The following is in conjunction with the appendix Figure 3 , Figure 4 Taking the application of the visual question-answering model training method provided in this specification in a multi-turn visual question-answering agent as an example, the visual question-answering model training method will be further explained. Figure 3 This is a flowchart of the processing procedure of a visual question answering model training method provided in one embodiment of this specification, specifically including the following steps 302-312. Figure 4 This is a flowchart illustrating the processing procedure of another visual question-answering model training method provided in one embodiment of this specification.

[0171] Step 302: Multi-domain video filtering.

[0172] Specifically, in combination Figure 4As shown in Phase 1, the server first receives raw multi-domain video streams from streaming media platforms and public datasets, and performs two-stage filtering: first, it performs coarse filtering by rules based on preset duration thresholds to remove redundant videos; then, it calls a large language model as a filtering agent to evaluate the complexity of video content and filter out invalid segments with too little information or solid backgrounds.

[0173] Step 304: In-depth research and question-answer pair generation.

[0174] Specifically, in combination Figure 4 As shown in Phase Two, the system performs crucial CLIP-based keyframe preprocessing and entity visual localization operations. Specifically, the server uses a CLIP model (such as the CLIP-ViT-L / 14 network) to extract high-dimensional visual feature vectors from video frames. After standardizing the feature vectors, the cosine similarity between consecutive frames is calculated. The system automatically discards consecutive frames with similarity scores greater than 0.8 and monochromatic frames containing no information, strictly limiting the number of candidate keyframes for each video to a preset upper limit (e.g., 20 frames). Based on this, the system inputs the preprocessed candidate frames into a large-parameter vision-language model, which ultimately determines the set of keyframes and predicts continuous bounding box coordinates for salient target entities in each keyframe. Next, the system performs visual cropping of entities based on these bounding boxes, executes an image search tool (Crop_Search), and calls an auxiliary model to verify the semantic alignment between the cropped region and the content returned by the network. Finally, it outputs a video metadata dictionary containing keyframe indices, bounding box coordinates, entity names, and search summaries. Based on this dictionary, the server synthesizes initial video question-and-answer pairs using both single-entity and multi-entity modes. To prevent the model from cheating by relying on internal memory, the system forcibly introduces a tool-free pre-test filtering mechanism to prevent parameter leakage: for each generated question, four forward static inferences are performed without enabling any search tools. If the model can answer correctly in any attempt, the sample is permanently removed, thus ensuring that the output dataset depends on the retrieval and inference of external tools.

[0175] Step 306: Trajectory generation.

[0176] Specifically, in combination Figure 4As shown in Phase 3, the core of Phase 3 lies in overcoming the agent's "modal bias" through a phased tool unlocking mechanism. The server loads the generated questions and constructs an execution pipeline that decouples perception and exploration. In the initial visual perception restriction phase, the system strictly limits the agent's action space to a subset of pure visual tools, allowing only the use of keyframe selection (Select_Keyframe) and crop search (Crop_Search) tools. The agent is forced to perform multiple cropping and visual retrievals of different entities in different keyframes until the system determines that it has obtained sufficient visual context or reached the maximum number of perception steps. This design completely blocks the agent's shortcut of directly requesting text search. After the visual localization task is completed, the system smoothly transitions to the open network exploration phase, automatically unlocking the text search (Search) and web page visit (Visit) tools, prompting the agent to perform multi-hop network information extraction based on the previously acquired visual entity attributes, and synthesizing the information to generate the final answer. Ultimately, the server uses a rejection sampling mechanism to call a discriminative model to verify the correctness of the output answer, retaining only the historical state-action execution sequence of successfully solving the problem, thus generating a high-quality multimodal expert decision trajectory dataset.

[0177] Step 308: Evaluation benchmark construction.

[0178] Specifically, in combination Figure 4 As shown in Phase 4, to scientifically measure the true capabilities of the trained visual depth research agents, the system constructed and executed a human-machine collaborative evaluation benchmark generation workflow. In this process, the client and server synchronize data through a secure API interface, ensuring real-time disk storage of labeled data and seamless integration with the multi-agent pipeline. On the client side, human annotators first select timestamps to pause the video, use a cropping search tool to select entities, verify external evidence, and compose preliminary seed question-answer pairs. Subsequently, this seed data is encrypted and uploaded to the server-side multi-agent collaborative pipeline: the Drafting Agent is responsible for expanding semantic directions to generate keywords; the search engine module crawls relevant web pages in the open network; and the QA Generation Agent synthesizes complex cross-modal multi-hop questions based on this web page evidence. The generated questions are then filtered again by the server's tool-free leak prevention mechanism and sent back to the client for secondary solvability verification by humans. Finally, the ranking agent on the server side gives a comprehensive score to the verified questions. The highest-scoring instance is retained not only as the final evaluation benchmark test data, but also as a new seed to be returned to the drafting agent in a loop. Through this continuous recursive iterative closed loop, the system can automatically synthesize long-cycle video depth search evaluation test cases with deeper levels and more complex logic.

[0179] Step 310: Model training.

[0180] Specifically, based on the expert trajectory generated in step 306, the server cluster then initiates a two-stage training process for the multimodal agent.

[0181] The first stage is supervised fine-tuning of hybrid text and image (SFT), whose input includes the aforementioned generated long-period video expert trajectory and proportionally mixed pure text depth search trajectory data. This hybrid strategy aims to compensate for the potential degradation of text processing capabilities that may result from the enhancement of visual perception. During training, the model is optimized using an autoregressive negative log-likelihood loss function, the formula of which is as follows (1): (1) in For a context that includes visual input and interaction history, This is the target action sequence. This stage enables the model to successfully complete the cold start and internalize the correct execution logic of "visual localization first, then network exploration".

[0182] The second stage is Group Relative Policy Optimization (GRPO) reinforcement learning, designed to break through the bottleneck of static imitation and stimulate endogenous exploration capabilities. For trajectory data of moderate difficulty, the SFT model generates multiple independent sampled trajectories (Rollouts) in parallel for each problem. The system automatically validates these trajectories and assigns sparse binary rewards (1 for correct, 0 for incorrect). The GRPO algorithm directly calculates the advantage function through intra-group relative rewards. This eliminates the memory overhead of a separate value network in traditional reinforcement learning. To prevent the model from exploiting format loopholes to cheat for rewards, the system downsamples negative gradients during backpropagation (e.g., applying negative gradient updates with only a 20% probability). Model Strategy The update is achieved by minimizing the objective function with KL divergence penalty, as shown in the following formula (2): (2) Step 312: Evaluate the trained model using the evaluation benchmark.

[0183] Through steps 302-312 above, when constructing data, the unlocking mechanism of forcing visual localization before text exploration is used, and samples that the model can answer directly based on internal memory are removed. This not only overcomes the inherent "modal bias" of the model, but also ensures that the training data strictly depends on the call of real external multi-step search tools.

[0184] This training method can calculate relative advantage without an independent value network and prevents training from getting stuck in meaningless format error loops by using negative gradient downsampling. While significantly saving server distributed training memory overhead, it effectively stimulates the agent's autonomous error correction and exploration potential.

[0185] Simultaneously, it utilizes multiple proxy models to automatically diverge network retrieval and combines manual verification to ensure evidence alignment. It can continuously synthesize complex test cases that require joint multi-frame visual localization and multi-hop knowledge reasoning, filling the gap in the high-fidelity evaluation system in the field of video deep search.

[0186] Corresponding to the above method embodiments, this specification also provides embodiments of a question-answering task processing device. Figure 5 This is a schematic diagram of the structure of a question-and-answer task processing device provided in one embodiment of this specification. Figure 5 As shown, the device includes: The first acquisition module 502 is configured to acquire the question-and-answer task data and the target visual content associated with the question-and-answer task data.

[0187] The visual processing module 504 is configured to use the visual perception unit of the visual question answering model to call visual tools to locate target entities in the target visual content, and to crop the image region corresponding to the target entity based on the location result to obtain visual context information.

[0188] Search module 506 is configured to perform network retrieval based on visual context information through a network search unit of a visual question answering model to obtain retrieval evidence information.

[0189] The generation module 508 is configured to generate question-and-answer results based on visual context information, retrieval evidence information, and question-and-answer task data.

[0190] Optionally, the first acquisition module 502 is further configured to acquire question-and-answer task data and acquire multiple original videos associated with the question-and-answer task data; perform filtering on each original video based on a duration threshold to obtain multiple candidate videos; call a video filtering model to evaluate the content complexity of each candidate video, obtain information content evaluation results, and remove candidate videos with information content lower than a preset condition based on the information content evaluation results to obtain retained videos; call an image-text matching model to extract at least one keyframe from the retained videos based on the question-and-answer task data, and generate target visual content based on at least one keyframe.

[0191] Optionally, the first acquisition module 502 is further configured to invoke an image-text matching model, extract visual features of each video frame in the retained video based on the question-and-answer task data, and calculate the similarity of visual features between consecutive video frames in the retained video; calculate the pixel complexity of each video frame; and in each video frame, remove consecutive video frames with similarity higher than the similarity threshold and / or video frames with pixel complexity lower than the pixel threshold to obtain at least one key frame.

[0192] Optionally, the visual tool includes a keyframe selection tool; correspondingly, the visual processing module 504 is further configured to invoke the keyframe selection tool through the visual perception unit of the visual question answering model to select a target keyframe containing the target entity from the target visual content; predict bounding box coordinates for the target entity in the target keyframe; perform image cropping on the image region corresponding to the target entity based on the bounding box coordinates to obtain a cropped image, and generate visual context information based on the cropped image.

[0193] Optionally, the search module 506 is further configured to generate text query information based on visual context information through a network search unit of a visual question answering model; call a text search tool to perform a text query and obtain a list of candidate web pages; call a web page access tool to extract content from the web pages in the candidate web page list to obtain candidate content, and perform semantic matching based on the candidate content and the cropped image; and filter out retrieval evidence information related to visual context information from the candidate content based on the semantic matching results.

[0194] Optionally, the question-answering task processing device further includes a feedback adjustment module, configured to receive feedback information from the target object regarding the question-answering results; and adjust the visual perception unit and / or the web search unit based on the feedback information.

[0195] Optionally, the question-answering task processing device further includes a training module configured to acquire sample question-answering task data, sample visual content associated with the sample question-answering task data, and acquire a question-answering decision trajectory. The question-answering decision trajectory includes an action sequence that first uses visual tools for entity localization and visual search to obtain visual context, then uses network tools for text retrieval to obtain retrieval evidence and generate an answer. The initial model is then trained under supervised conditions using the sample question-answering task data, sample visual content, and question-answering decision trajectory to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and is used to invoke the visual perception unit and network exploration unit according to the action sequence. Finally, the intermediate model is trained using reinforcement learning with the sample question-answering task data and sample visual content to obtain a trained visual question-answering model.

[0196] Optionally, the training module is further configured to input sample visual content and sample question-answering task data into the initial model to generate a predicted action sequence; calculate the loss based on the difference between the predicted action sequence and the question-answering decision trajectory; and update the parameters of the initial model based on the loss to obtain an intermediate model.

[0197] Optionally, the training module is further configured to acquire plain text deep search trajectory data, which consists of question-answering tasks involving only text search and reasoning; mix the plain text deep search trajectory data with the question-answering decision trajectory at a preset ratio to obtain mixed training data; and supervise the initial model using sample question-answering task data, sample visual content, and mixed training data.

[0198] Optionally, the training module is further configured to acquire candidate question-answering task data; perform static inference on the candidate question-answering task data using the teacher question-answering model to obtain a first inference result; remove the candidate question-answering task data if the first inference result is correct; and use the candidate question-answering task data as sample question-answering task data if the first inference result is incorrect.

[0199] Optionally, the training module is further configured to acquire candidate decision trajectories; use the teacher question-answering model to infer a second inference result from the sample question-answering task data according to the candidate decision trajectories; remove the candidate decision trajectories if the second inference result is incorrect; and use the candidate decision trajectories as question-answering decision trajectories if the second inference result is correct.

[0200] Optionally, the training module is further configured to acquire the verification question-answering task data and the verification visual content associated with the verification question-answering task data; input the verification question-answering task data and the verification visual content into the trained visual question-answering model to obtain the verification results.

[0201] Optionally, the training module is further configured to acquire seed question-and-answer task data formed by the target object's annotation of the verification visual content; expand the seed question-and-answer pairs with keywords, and perform web searches based on the expanded keywords to obtain web page evidence; generate multi-level reasoning questions based on the web page evidence, and perform filtering processing on the multi-level reasoning questions through the target object; perform quality assessment on the filtered multi-level reasoning questions, and generate verification question-and-answer task data based on the target multi-level reasoning questions that meet the quality assessment conditions.

[0202] Optionally, the training module is further configured to update the seed question-answering task data based on the target multi-level reasoning question.

[0203] In this question-and-answer task processing device, the first acquisition module 502 first acquires the question-and-answer task data and its associated target visual content, providing an input basis for subsequent processing. Subsequently, the visual processing module 504 forcibly invokes the visual tools in the visual perception unit to locate specific entities in the target visual content and crop out the corresponding image regions, generating task-related visual context information. This step ensures that the model must "see first," that is, complete fine-grained perception based on real video content, and cannot skip visual analysis to answer directly. On this basis, the search module 506 is activated, which relies on the network search unit and initiates network retrieval based solely on the visual context information output by the visual processing module 504 to obtain external evidence, realizing the constraint mechanism of "seeing to promote searching." Finally, the generation module 508 integrates the visual context from the visual processing module 504, the retrieval evidence from the search module 506, and the original question-and-answer task data to generate the answer.

[0204] The above is an illustrative scheme of a question-and-answer task processing device according to this embodiment. It should be noted that the technical solution of this question-and-answer task processing device and the technical solution of the question-and-answer task processing method described above belong to the same concept. For details not described in detail in the technical solution of the question-and-answer task processing device, please refer to the description of the technical solution of the question-and-answer task processing method described above.

[0205] Corresponding to the above method embodiments, this specification also provides embodiments of a visual question-answering model training device. Figure 6 This is a schematic diagram of the structure of a visual question-answering model training device provided in one embodiment of this specification. Figure 6 As shown, the device includes: The second acquisition module 602 is configured to acquire sample question-and-answer task data, sample visual content associated with the sample question-and-answer task data, and acquire question-and-answer decision trajectory. The question-and-answer decision trajectory includes an action sequence of first using visual tools to perform entity localization and visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate an answer.

[0206] The first training module 604 is configured to supervise the training of the initial model using sample question-and-answer task data, sample visual content, and question-and-answer decision trajectories to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and is used to call the visual perception unit and the network exploration unit according to the action sequence.

[0207] The second training module 606 is configured to use sample question-answering task data and sample visual content to perform reinforcement learning training on the intermediate model, so as to obtain a trained visual question-answering model.

[0208] In this visual question-answering model training device, the second acquisition module 602 first collects question-answering decision trajectories containing structured action sequences of "visual localization first, then network retrieval," and connects them with sample question-answering task data and sample visual content to provide a supervisory signal with sequence constraints for training. Subsequently, the first training module 604 uses these data to supervise the training of the initial model, forcing the intermediate model to learn and internalize the phased calling logic at the parameter level—that is, it must first activate the visual perception unit to complete entity localization and cropping, and then trigger the network exploration unit to initiate retrieval, thereby fundamentally suppressing the model's modal bias of relying on text shortcuts or internal memory. On this basis, the second training module 606 further uses the original sample question-answering task data and sample visual content as input to perform reinforcement learning training on the intermediate model output by the first training module 604, optimizing its ability to autonomously generate high-value decision paths in complex, open video scenes while retaining the correct tool calling order (fixed by 604). The entire process, through the cascading of modules 602→604→606, realizes the training paradigm of "standardizing behavior first, then improving intelligence," which not only ensures the interpretability and evidence dependence of the model's reasoning process, but also significantly enhances its generalization and deep reasoning capabilities on unseen tasks.

[0209] The above is an illustrative scheme of a visual question-answering model training device according to this embodiment. It should be noted that the technical solution of this visual question-answering model training device and the technical solution of the visual question-answering model training method described above belong to the same concept. For details not described in detail in the technical solution of the visual question-answering model training device, please refer to the description of the technical solution of the visual question-answering model training method described above.

[0210] Figure 7 This is a structural block diagram of a computing device according to one embodiment of this specification. The components of the computing device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and a database 750 is used to store data.

[0211] The computing device 700 also includes an access device 740, which enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 740 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0212] In one embodiment of this specification, the above-described components of the computing device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0213] The computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 700 can also be a mobile or stationary server.

[0214] The processor 720 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned question-answering task processing method or visual question-answering model training method.

[0215] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are relatively simple in description because they are fundamentally similar to the question-answering task processing method or visual question-answering model training method embodiments. Relevant details can be found in the descriptions of the question-answering task processing method or visual question-answering model training method embodiments.

[0216] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described question-answering task processing method or visual question-answering model training method.

[0217] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer-readable storage medium embodiments are relatively simple in description because they are fundamentally similar to the question-answering task processing method or visual question-answering model training method embodiments. Relevant details can be found in the descriptions of the question-answering task processing method or visual question-answering model training method embodiments.

[0218] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described question-answering task processing method or visual question-answering model training method.

[0219] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solution of the question-answering task processing method or the visual question-answering model training method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the question-answering task processing method or the visual question-answering model training method described above.

[0220] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0221] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0222] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0223] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0224] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A question-answering task processing method, characterized in that, include: Obtain the target visual content associated with the question-and-answer task data; The visual perception unit of the visual question answering model calls visual tools to locate the target entity in the target visual content, and crop the image region corresponding to the target entity based on the location result to obtain visual context information. The network search unit of the visual question-answering model performs network retrieval based on the visual context information to obtain retrieval evidence information. Based on the visual context information, the retrieval evidence information, and the question-answering task data, a question-answering result is generated.

2. The method according to claim 1, characterized in that, The acquisition of target visual content associated with the question-answering task data includes: Obtain the question-and-answer task data, and obtain multiple original videos associated with the question-and-answer task data; Filter each original video based on a duration threshold to obtain multiple candidate videos; The video filtering model is invoked to evaluate the content complexity of each candidate video, obtain the information content evaluation result, and remove candidate videos with information content lower than the preset condition based on the information content evaluation result to obtain the retained videos. The image-text matching model is invoked to extract at least one keyframe from the retained video based on the question-and-answer task data, and target visual content is generated based on the at least one keyframe.

3. The method according to claim 2, characterized in that, The step of calling the image-text matching model to extract at least one keyframe from the retained video based on the question-answering task data includes: The image-text matching model is invoked to extract the visual features of each video frame in the retained video based on the question-and-answer task data, and the similarity of visual features between consecutive video frames in the retained video is calculated. Calculate the pixel complexity of each video frame; In each video frame, consecutive video frames with similarity higher than the similarity threshold and / or video frames with pixel complexity lower than the pixel threshold are removed to obtain at least one key frame.

4. The method according to claim 1, characterized in that, The visual tools include a keyframe selection tool; the visual perception unit of the visual question-answering model invokes the visual tools to locate target entities in the target visual content, and crops the image region corresponding to the target entity based on the location result to obtain visual context information, including: The visual perception unit of the visual question answering model calls the keyframe selection tool to select target keyframes containing target entities from the target visual content. Predict bounding box coordinates for the target entity in the target keyframe; The image region corresponding to the target entity is cropped according to the bounding box coordinates to obtain a cropped image, and visual context information is generated based on the cropped image.

5. The method according to claim 4, characterized in that, The network search unit using the visual question-answering model performs network retrieval based on the visual context information to obtain retrieval evidence information, including: The network search unit of the visual question-answering model generates text query information based on the visual context information. The text search tool is invoked to perform the text query and obtain a list of candidate web pages; A web page access tool is invoked to extract content from the web pages in the candidate web page list to obtain candidate content, and semantic matching is performed between the candidate content and the cropped image. Based on the semantic matching results, retrieval evidence information related to the visual context information is filtered out from the candidate content.

6. The method according to claim 1, characterized in that, After generating the question-answering result based on the visual context information, the retrieval evidence information, and the question-answering task data, the process includes: Receive feedback information from the target object regarding the question-and-answer results; Based on the feedback information, the visual perception unit and / or the web search unit are adjusted.

7. The method according to any one of claims 1-6, characterized in that, The visual question-answering model was trained through the following steps: Acquire sample question-and-answer task data, sample visual content associated with the sample question-and-answer task data, and acquire question-and-answer decision trajectory, wherein the question-and-answer decision trajectory includes an action sequence of first using visual tools to perform entity localization and visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate an answer. The initial model is trained under supervision using the sample question-answering task data, the sample visual content, and the question-answering decision trajectory to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and the intermediate model is used to call the visual perception unit and the network exploration unit according to the action sequence. The intermediate model is trained using reinforcement learning based on the sample question-answering task data and the sample visual content to obtain a trained visual question-answering model.

8. The method according to claim 7, characterized in that, The step of supervising the training of the initial model using the sample question-answering task data, the sample visual content, and the question-answering decision trajectory to obtain an intermediate model includes: The sample visual content and the sample question-and-answer task data are input into the initial model to generate a predicted action sequence; The loss is calculated based on the difference between the predicted action sequence and the question-and-answer decision trajectory; The parameters of the initial model are updated based on the loss to obtain the intermediate model.

9. The method according to claim 8, characterized in that, The supervised training of the initial model using the sample question-answering task data, the sample visual content, and the question-answering decision trajectory includes: Obtain pure text deep search trajectory data, which consists of question-answering tasks involving only text search and reasoning; The plain text deep search trajectory data is mixed with the question-answering decision trajectory at a preset ratio to obtain mixed training data; The initial model is subjected to supervised training using the sample question-answering task data, the sample visual content, and the mixed training data.

10. The method according to claim 7, characterized in that, The acquisition of sample question-answering task data includes: Obtain candidate question-answering task data; Static reasoning is performed on the candidate question-answering task data using a teacher question-answering model to obtain a first reasoning result; If the first reasoning result is correct, the candidate question-answering task data is removed; If the first reasoning result is incorrect, the candidate question-answering task data will be used as the sample question-answering task data.

11. The method according to claim 10, characterized in that, The acquisition of the question-and-answer decision trajectory includes: Obtain candidate decision trajectories; Using the teacher question-and-answer model, a second inference result is derived from the sample question-and-answer task data according to the candidate decision trajectory; If the second reasoning result is incorrect, the candidate decision trajectory is removed; If the second reasoning result is correct, the candidate decision trajectory is taken as the question-and-answer decision trajectory.

12. The method according to claim 7, characterized in that, After obtaining the trained visual question-answering model, the following is also included: Obtain the verification question-and-answer task data and the verification visual content associated with the verification question-and-answer task data; The verification question-answering task data and the verification visual content are input into the trained visual question-answering model to obtain the verification results.

13. The method according to claim 12, characterized in that, The acquisition of verification question-and-answer task data includes: Acquire seed question-and-answer task data formed by the target object's annotation of the visual content to be verified; The seed question-and-answer pairs are expanded with keywords, and web searches are performed based on the expanded keywords to obtain webpage evidence; Multi-level reasoning questions are generated based on the webpage evidence, and the multi-level reasoning questions are filtered through the target object. The quality of the selected multi-level reasoning questions is assessed, and validation question-answering task data is generated based on the target multi-level reasoning questions that meet the quality assessment criteria.

14. The method according to claim 13, characterized in that, After performing quality assessment on the selected target multi-level reasoning questions and generating verification question-answering task data based on the target multi-level reasoning questions that meet the quality assessment criteria, the process further includes: Based on the target multi-level reasoning problem, update the seed question-answering task data.

15. A method for training a visual question-answering model, characterized in that, include: Acquire sample question-and-answer task data, sample visual content associated with the sample question-and-answer task data, and acquire question-and-answer decision trajectory, wherein the question-and-answer decision trajectory includes an action sequence of first using visual tools to perform entity localization and visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate an answer. The initial model is trained under supervision using the sample question-answering task data, the sample visual content, and the question-answering decision trajectory to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and the intermediate model is used to call the visual perception unit and the network exploration unit according to the action sequence. The intermediate model is trained using reinforcement learning based on the sample question-answering task data and the sample visual content to obtain a trained visual question-answering model.

16. A question-and-answer task processing device, characterized in that, include: The first acquisition module is configured to acquire the target visual content associated with the question-and-answer task data; The visual processing module is configured to use the visual perception unit of the visual question answering model to call visual tools to locate the target entity in the target visual content, and to crop the image region corresponding to the target entity based on the location result to obtain visual context information. The search module is configured to perform network retrieval based on the visual context information through the network search unit of the visual question answering model to obtain retrieval evidence information. The generation module is configured to generate question-and-answer results based on the visual context information, the retrieval evidence information, and the question-and-answer task data.

17. A visual question-answering model training device, characterized in that, include: The second acquisition module is configured to acquire sample question-and-answer task data, sample visual content associated with the sample question-and-answer task data, and acquire question-and-answer decision trajectory, wherein the question-and-answer decision trajectory includes an action sequence of first using visual tools to perform entity localization and visual search to obtain visual context, and then using network tools to perform text retrieval to obtain retrieval evidence and generate an answer. The first training module is configured to supervise the training of the initial model using the sample question-answering task data, the sample visual content, and the question-answering decision trajectory to obtain an intermediate model. The intermediate model includes a visual perception unit and a network exploration unit, and the intermediate model is used to call the visual perception unit and the network exploration unit according to the action sequence. The second training module is configured to use the sample question-answering task data and the sample visual content to perform reinforcement learning training on the intermediate model, so as to obtain a trained visual question-answering model.

18. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the question-answering task processing method according to any one of claims 1-14 or the visual question-answering model training method according to claim 15.

19. A computer-readable storage medium, characterized in that, It stores a computer program / instruction that, when executed by a processor, implements the steps of the question-answering task processing method of any one of claims 1-14 or the visual question-answering model training method of claim 15.

20. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the steps of the question-answering task processing method of any one of claims 1-14 or the visual question-answering model training method of claim 15.

Images