Question and answer method and device, equipment and medium

Abstract

The invention relates to the field of question answering, in particular to a question answering method and device, equipment and a medium, and the method comprises the steps: obtaining multi-modal question information; disassembling the multi-mode question information into reference content and task keywords; according to the method, a summary model, a selection model and an extraction model are utilized to obtain summary content, selection content and extraction content according to reference content and the relation with task keywords, different models process information from different angles, useful information in the reference content is comprehensively mined, information compression is achieved, and meanwhile high-quality information is reserved; and performing structured rewriting on the summarized content, the selected content and the extracted content by using the structured rewriting model, and reasoning the rewritten content by using the target model to obtain a reply result. Through the synergistic effect of multiple models, effective information is compressed from multi-modal information, multi-modal context is compressed while core semantics are reserved, and finally obtained reply results are more accurate and comprehensive.

Description

Technical Field

[0001] This application relates to the field of question answering, and in particular to a question answering method, apparatus, device, and medium. Background Technology

[0002] In multimodal question-answering systems, models need to simultaneously understand information from multiple sources, such as video, images, and text, and generate accurate answers based on user questions. However, raw audio and video data is usually lengthy and information-dense, and directly inputting it into a large model will lead to two major problems: excessively long context, resulting in high computational and inference costs; and dilution of key information, leading to decreased relevance of the answer.

[0003] However, conventional text compression or summarization models are mostly based on single-modal (plain text) implementation. When processing multimodal data such as audio and video, there are the following pain points: different types of questions require different information depths; when users input a large amount of content, key information is easily diluted in a large amount of irrelevant information, resulting in a decrease in the relevance of the answer.

[0004] Therefore, how to compress multimodal context while preserving core semantics has become a key aspect of improving the performance of audio and video intelligent question answering systems. Summary of the Invention

[0005] The purpose of this application is to provide a question-answering method, apparatus, device, and medium that can compress multimodal context while preserving core semantics.

[0006] In a first aspect, a question-answering method is provided, comprising: acquiring multimodal question information, wherein the multimodal question information represents question information in text form determined based on multimodal information; separating the multimodal question information to obtain reference content and task keywords; obtaining summary content using a summary model based on the reference content; obtaining selected content using a selection model based on the reference content and the task keywords; obtaining extracted content using an extraction model based on the reference content and the task keywords; performing structured rewriting using a structured rewriting model based on the summary content, the selected content, and the extracted content to obtain rewritten content; and performing reasoning using a target model based on the rewritten content to obtain a response result.

[0007] In a preferred embodiment, this application can be further configured to: obtain multimodal question information, including: obtaining multimodal information and task data input by the user, wherein the multimodal information includes at least one of the following: text data and visual data; inputting the visual data into a multimodal parsing module to obtain visual text; and obtaining multimodal question information based on at least one of the text data and the visual text, and the task data.

[0008] In a preferred embodiment, this application can be further configured to: input the visual data into a multimodal analysis module to obtain visual text, including: if the visual data is video data, extracting keyframes according to the multimodal analysis module; identifying the first image information and the first text information of the audio subtitles of the keyframes; generating visual text based on the first image information and the first text information; if the visual data is image data, identifying the second image information and the second text information of the image data according to the multimodal analysis module; generating visual text based on the second image information and the second text information.

[0009] In a preferred embodiment, this application can be further configured as follows: if the multimodal information includes text data and visual data; accordingly, based on at least one of the text data and the visual text, and task data, multimodal question information is obtained, including: determining the task target type according to the task data; determining text weight and visual weight according to the task target type; performing a weighted calculation on the text data and the visual text according to the text weight and the visual weight, and combining it with the task data to obtain multimodal question information.

[0010] In a preferred embodiment, this application can be further configured as follows: Based on the reference content, a summary model is used to obtain summary content, including: using the summary model to extract key information from the reference content and summarizing the key information to obtain summary content; based on the reference content and the task keywords, a selection model is used to obtain selected content, including: using the selection model to filter out fragment content associated with the task keywords from the reference content; performing semantic reconstruction based on the fragment content to obtain selected content; based on the reference content and the task keywords, an extraction model is used to obtain extracted content, including: using the extraction model to match detailed content from the reference content as extracted content.

[0011] In a preferred embodiment, this application can be further configured to: filter out fragment content associated with the task keyword from the reference content, including: filtering out a first sub-fragment content associated with the task keyword from the reference content; filtering out a second sub-fragment content associated with the first sub-fragment content from the reference content; and obtaining fragment content based on the first sub-fragment content and the second sub-fragment content.

[0012] In a preferred embodiment, this application may be further configured as follows: after obtaining a response result by reasoning using the target model based on the rewritten content, it further includes: periodically determining the differences between the response result and the expected result; and optimizing the structured rewriting model and the target model based on the differences.

[0013] Secondly, a question-answering device is provided, comprising: an acquisition module for acquiring multimodal question information, wherein the multimodal question information represents question information in text form determined based on multimodal information; a task content separation module for separating the multimodal question information to obtain reference content and task keywords; a context compression module for obtaining summary content using a summary model based on the reference content; obtaining selected content using a selection model based on the reference content and the task keywords; obtaining extracted content using an extraction model based on the reference content and the task keywords; a structured reconstruction module for performing structured rewriting using a structured rewriting model based on the summary content, the selected content, and the extracted content to obtain rewritten content; and an inference module for performing inference using a target model based on the rewritten content to obtain a response result.

[0014] Thirdly, an electronic device is provided, the electronic device including a memory and a processor, the memory storing a computer program, the processor executing the method of any one of the first aspects when running the computer program.

[0015] Fourthly, a computer-readable storage medium is provided, wherein at least one piece of program code is stored therein, the program code being loaded and executed by a processor to implement the method as described in any of the first aspects.

[0016] Fifthly, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the method as described in any of the first aspects.

[0017] In summary, the question-and-answer method provided in this application has the following beneficial technical effects:

[0018] This process involves acquiring multimodal question information, where the multimodal question information represents textual question information determined based on multimodal information. The multimodal question information is decomposed into reference content and task keywords. A summarization model, a selection model, and an extraction model are used to obtain summarized content, selected content, and extracted content based on the reference content and its relationship with the task keywords, respectively. Different models process the information from different perspectives, comprehensively mining useful information from the reference content, achieving information compression while retaining high-quality information. A structured rewriting model is used to rewrite the summarized content, selected content, and extracted content, and then a target model is used to reason about the rewritten content to obtain the response result. Through the synergistic effect of multiple models, effective information is compressed from multimodal information, preserving core semantics while compressing the multimodal context, resulting in a more accurate and comprehensive response.

[0019] In addition, this application also provides a question-and-answer device, equipment, and medium, all of which have the aforementioned beneficial technical effects. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a question-and-answer method provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a scenario provided in an embodiment of this application;

[0023] Figure 3 This is a flowchart illustrating another question-and-answer method provided in an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of a question-and-answer device provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0027] It should be noted that, in the optional embodiments of this application, the data related to object information, when applied to specific products or technologies, requires the permission or consent of the object. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, it must be obtained with the permission and consent of the object, the permission and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the permission and consent of the object.

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0030] This application proposes a task-driven context compression method that maximizes the retention of key content related to the problem within a limited context window through task identification and semantic filtering mechanisms.

[0031] This technology is primarily applied to the following types of intelligent audio and video systems: 1. Video-based question-answering systems: These systems allow users to ask questions about video clips, requiring the system to quickly locate keyframes. 2. Knowledge extraction systems: These systems extract accurate and effective knowledge from massive amounts of text and multimodal content references to feed into downstream large-scale models for answering questions. 3. Interactive systems with extensive memory capabilities: For intelligent interactive systems with long-term contextual memory, the core issue limiting memory effectiveness is how to more effectively extract contextual information to complete question-and-answer sessions.

[0032] This application provides a question-and-answer method, such as... Figure 1 As shown, the method provided in this application embodiment can be executed by an electronic device, which is a server. This server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The terminal device and electronic device can be directly or indirectly connected via wired or wireless communication. This application embodiment does not impose any limitations on this connection. The question-and-answer method includes:

[0033] S101. Obtain multimodal question information, wherein the multimodal question information represents question information in text form determined based on multimodal information;

[0034] In this embodiment of the application, multimodal data includes at least two different types of data, such as text, images, voice, and video; the question information refers to the information raised by the user, which may include reference information and task information.

[0035] The core objective of a typical question-and-answer system is to convey the user's intent through diverse input methods (such as speaking a sentence, typing a text, uploading an image, recording a voice message, or a combination of these methods).

[0036] In this application, the user input and the task are used for recognition. For images, voice and video, the data are extracted and generated into text format to obtain the question information.

[0037] It is noteworthy that in the automated context optimization algorithm provided in this application, the type of reference information directly affects the subsequent compression and rewriting effects. For multi-source heterogeneous input data, this application further introduces a reference information classification mechanism in the extraction stage, such as... Figure 2 As shown, the reference content is categorized into three typical scenarios based on modal type: pure text scenarios; visual scenarios primarily consisting of video or images; and mixed text and image scenarios. This design allows the system to invoke different processing paths based on content type during information extraction to obtain the query information, thereby improving the accuracy and robustness of information compression while maintaining semantic consistency.

[0038] Specifically, the system begins with the user's multimodal input, which can be text, image, or video descriptions. Image or video input undergoes unified formatting processing by a multimodal parsing module. This module, based on a multimodal fusion model, performs semantic transcription of speech, scene annotation on images, and extracts keyframes and semantic segments from videos, uniformly converting different modalities into textual expressions to obtain the query information. This process ensures the structural consistency of the information input, enabling downstream models to operate within a unified semantic space.

[0039] S102. Separate the multimodal question information to obtain reference content and task keywords;

[0040] The query text, after multimodal parsing, enters the task content separation model. This model is the core entry point of the entire system, employing an explicit task identification and content differentiation mechanism. It is a fine-tuned Large Language Model (LLM) that categorizes user input into two types: task requirement description and reference content. The task requirement description typically includes user intent, task objectives, constraints, and other information, leading to task keywords. The reference content contains knowledge fragments, contextual corpora, or document summaries required for processing or answering. Through explicit decoupling of task and content, differentiated information compression strategies can be adopted for different types of tasks, rather than simply relying on the concatenation of the original long context.

[0041] In one feasible approach, a large dataset containing multimodal information and explicitly labeled with reference content and task keywords is acquired; the dataset is then input into a large language model for training, enabling the large language model to learn how to accurately separate reference content and task keywords from multimodal information.

[0042] S103. Based on the reference content, use the summary model to obtain the summary content; based on the reference content and task keywords, use the selection model to obtain the selection content; based on the reference content and task keywords, use the extraction model to obtain the extraction content.

[0043] After task separation, three context optimization models can be applied sequentially to the reference content: a summary model, a selection model, and an extraction model. These three constitute a hierarchical information compression system that is both independent and collaborative. One possible implementation of this application involves using the summary model to extract key information from the reference content and summarizing this key information to obtain the summary content.

[0044] The summary model is trained using a large dataset of texts containing detailed descriptions and corresponding summaries, allowing it to learn how to extract key points from detailed text. Then, the input reference content is preprocessed, including noise removal, word segmentation, and part-of-speech tagging, before being fed into the summary model. The model generates summaries based on the learned patterns and rules, outputting the summary content. The summary model can perform global semantic compression at a macro level, extracting key information such as core viewpoints, timelines, and event sequences from the reference content, condensing long texts into short summaries.

[0045] One possible implementation of this application involves using a selection model to filter out fragment content associated with task keywords from reference content; and then performing semantic reconstruction based on the fragment content to obtain the selected content.

[0046] The selection model uses a task relevance assessment mechanism to filter out the segments most directly related to the task and performs semantic reconstruction to ensure logical consistency between the selected content and the task objective. The selection model evaluates the semantics expressed by task keywords in the reference content, essentially representing a biased summary by the larger model based on these task keywords.

[0047] One possible implementation of this application embodiment is to use an extraction model to match detailed content from reference content as extracted content.

[0048] In one feasible approach, the extraction model employs a language model-based syntactic parsing and semantic matching mechanism to automatically identify entities, events, and topics in the original text, extracting sentence segments highly relevant to the task. For textual content, the key lies in preserving semantic logic and maintaining contextual coherence. Therefore, the system uses a Transformer-based semantic segmentation algorithm to break down long texts into several semantic units. The importance of each unit is calculated using a task relevance scoring function; high-scoring portions are prioritized and low-scoring portions are either included in the summary or discard channel, thus achieving lightweight information filtering. After obtaining the semantic units, semantic reconstruction can be performed to obtain the selected content.

[0049] In another possible implementation, the fragment content associated with the task keywords is filtered from the reference content, including: filtering out a first sub-fragment content associated with the task keywords from the reference content; filtering out a second sub-fragment content associated with the first sub-fragment content from the reference content; and obtaining the fragment content based on the first and second sub-fragment content.

[0050] The extraction model operates at a finer granular level, extracting speech from the reference content based on task keywords. Understandably, it utilizes entity recognition, task keyword matching, and dependency semantic structure analysis to extract the most representative sentences, numerical values, or factual information from the reference content, ensuring that important details are not lost during compression.

[0051] In this embodiment of the application, in order to enable the three models to work together, the present invention designs an automated scheduling module for dynamically selecting model combinations based on task characteristics.

[0052] To address different task requirements, this application provides three context optimization strategies: summarization, selection, and extraction. An automated context optimization algorithm is also constructed. By explicitly separating the reference information of audio and video content from the task, and through information compression using three models, information efficiency is significantly improved. This not only reduces the context length but also increases the relevance of the reference content. Further structured rewriting makes the module input structure of each reference information module clearer. Through multi-dimensional and effective context engineering, the response quality of the subsequent target large model is significantly improved.

[0053] S104. Based on the summarized content, selected content, and extracted content, use the structured rewriting model to perform structured rewriting to obtain the rewritten content;

[0054] In this embodiment, the text, after three layers of optimization, is fed into the structured rewriting module. The main function of the structured rewriting module is to reorganize the compressed content into structured corpus conforming to the input format of the target large model. Through template-based reconstruction and hierarchical semantic alignment mechanisms, the structured rewriting module integrates reference information from different sources (such as document summaries, question-answering context, and task conditions) into a unified input structure. For example, for multi-document question-answering tasks, the system automatically generates an input framework containing sub-modules such as background, contextual information, timeline, and data references, enabling the target model to reason and generate more efficiently within ordered information. The structured rewriting process not only improves the readability and logical coherence of the input but also provides a more stable contextual feature space for subsequent models.

[0055] Furthermore, during the structured rewriting process, we can also consider summarizing, selecting, and extracting the importance of each content, so that high-importance content can be prioritized when performing structured integration.

[0056] The relative importance of summarizing, selecting, and extracting content can be determined based on the task type in the question information. For example, if the task requires a global understanding of an open-ended problem, the summarizing model is the most important; if the task is an instruction-based or retrieval-based task, the selecting model is the most important; and if the task involves generating precise structured results (such as reports, tables, or summary cards), the extracting model is the most important. Content corresponding to models with higher importance should be listed first.

[0057] Furthermore, the three models are used in cascades to gradually compress and refine the content. This dynamic routing mechanism gives the system good adaptability and scalability, and can automatically allocate the optimal processing path based on task complexity and context length.

[0058] S105. Based on the rewritten content, use the target model to reason and obtain the response result.

[0059] After structured rewriting, the optimized context is input into the target large model for final inference and generation. Because the previous information compression significantly reduced the input length, the target large model can receive more task-relevant content within a limited context window, thus reducing redundant computation while ensuring information integrity. Experiments show that, at the same model size, this method can reduce the input length by an average of 40%–60%, while improving the relevance score and factual consistency index of the generated results by more than 30%. Especially in audio and video scenarios, due to the multimodal, long time-series, and high information redundancy characteristics of the input content, the context optimization strategy of this invention can effectively filter irrelevant segments, retaining only the parts closely related to the task's semantics, significantly improving question-answering accuracy and response speed.

[0060] As can be seen, in this embodiment, multimodal question information is obtained, wherein the multimodal question information represents textual question information determined based on multimodal information; the multimodal question information is decomposed into reference content and task keywords; a summary model, a selection model, and an extraction model are used to obtain summary content, selection content, and extracted content respectively based on the reference content and its relationship with the task keywords. Different models process the information from different perspectives, comprehensively mining useful information in the reference content, achieving information compression, and retaining high-quality information; a structured rewriting model is used to rewrite the summary content, selection content, and extracted content in a structured manner, and then a target model is used to reason about the rewritten content to obtain the answer result. Through the synergistic effect of multiple models, effective information is compressed from multimodal information, retaining the core semantics while compressing the multimodal context, and the final answer result is more accurate and comprehensive.

[0061] One possible implementation of this application embodiment is to obtain multimodal question information, including: obtaining multimodal information and task data input by the user, wherein the multimodal information includes at least one of the following: text data and visual data; inputting the visual data into a multimodal parsing module to obtain visual text; and obtaining multimodal question information based on at least one of the text data, visual text, and task data.

[0062] In the embodiments of this application, multimodal information includes text data; visual data; and a combination of text data and visual data.

[0063] For visual data, it can be a multimodal scene dominated by video or images (e.g., Figure 2 In scenarios 2 and 3), electronic devices can enable the multimodal analysis module for more complex extraction and translation.

[0064] The visual data is input into the multimodal parsing module to obtain visual text, including:

[0065] If the visual data is video data, then according to the multimodal analysis module, key frames are extracted; and the first screen information of the key frames and the first text information of the audio subtitles are identified; and visual text is generated based on the first screen information and the first text information.

[0066] If the visual data is image data, then according to the multimodal analysis module, the second image information and the second text information of the image data are identified; and visual text is generated based on the second image information and the second text information.

[0067] Specifically, in video processing scenarios (i.e., visual data is video data), the electronic device first selects keyframes and segments shots. Then, based on a temporal attention mechanism, it identifies the first frame information in the image. This first frame information includes, but is not limited to, salient objects, actions, and event nodes. Subsequently, it combines this information with audio subtitles or OCR text to form a multimodal semantic fragment, i.e., the first text information. In image scenarios (i.e., visual data is image data), a visual recognition and text generation model is used to generate content descriptions, i.e., the second frame information, and the second text information, explicitly converting the visual information into a textual expression. After this process, all multimodal content is ultimately mapped to a unified textual semantic space, providing standardized input for subsequent summarization, selection, and structured rewriting modules.

[0068] One possible implementation of this application embodiment is as follows: if the multimodal information includes text data and visual data; accordingly, based on at least one of the text data and visual text, and task data, multimodal question information is obtained, including: determining the task target type according to the task data; determining text weight and visual weight according to the task target type; performing weighted calculation based on the text weight, visual weight, text data, and visual text, and combining it with the task data to obtain multimodal question information.

[0069] Specifically, based on the task data, the task objective type is analyzed. The task data includes information such as task description, application scenario, and user needs. Natural language processing is then performed on the task data to extract key entities, verbs, and contextual information. Based on predefined task types (question-answering, summary generation, image description), a classification model is used to determine the task objective type. For example, if the task description is "generate questions based on images and titles," then the objective type is "question-answering generation."

[0070] Furthermore, after determining the task target type, the text weight and visual weight corresponding to the task target type can be determined based on the task target type and pre-defined corresponding information. This correspondence is between the task target type and its weight. For example, in question-answering generation: text weight = 0.7, visual weight = 0.3; in image description: text weight = 0.3, visual weight = 0.7.

[0071] Furthermore, the question information is obtained by weighting the text vector corresponding to the text data and the visual vector corresponding to the visual data.

[0072] As can be seen, in the embodiments of this application, in composite scenarios involving mixed text and images, the electronic device can simultaneously consider the weight allocation of both modalities. To this end, this invention designs a task-driven fusion strategy: when the task objective leans more towards semantic understanding (such as question answering, summary generation), text information has a higher weight; when the task emphasizes visual perception (such as video content analysis, scene recognition), the weight of image or video information is correspondingly increased. The fused result not only retains the key semantic elements at the visual level but also maintains the logical consistency of the language expression, making the generated result both visible and clear.

[0073] One possible implementation of this application embodiment, after obtaining the response result by reasoning using the target model based on the rewritten content, further includes: periodically determining the differences between the response result and the expected result; and optimizing the structured rewriting model and the target model based on the differences.

[0074] The implementation of this invention also includes a task-driven context feedback mechanism. After completing a task, the output of the target model is aligned with the task expectation, and the differences are fed back to the optimization module. Through this closed-loop optimization, the system can continuously learn task features during multiple task executions, adjust the weights or thresholds of the summarized, selected, and extracted models, and gradually bring the algorithm to a steady state.

[0075] This mechanism performs particularly well in long-term dependent tasks such as multi-turn dialogues, document question answering, and report generation. It can adaptively learn which types of information are most easily ignored or redundant, and prioritize retaining key parts in subsequent processing. In addition, to support interpretability and traceability, the system generates intermediate result logs for the output of each stage, which can be used for subsequent analysis and optimization.

[0076] It is understood that the language model involved in the embodiments of this application can be a trained model with corresponding functions. The specific training process and model structure are not limited in the embodiments of this application, and users can set them according to the actual situation.

[0077] In one possible scenario, the summary model, selection model, extraction model, structured rewriting model, target model, and multimodal parsing module are all large language models. Their basic structures can be the same or different, and they can be commercial models or self-trained models. For example, the large language model comprising the summary model, selection model, and extraction model could be a dense large language model, qwen3-32B, based on a Transformer Decoder-only architecture, containing 32.5B parameters. Its core components integrate GQA, SwiGLU, RoPE, RMSnorm, and QKV biases, with the maximum context parameter supporting ultra-long text processing. It could be a model with corresponding functionality obtained by fine-tuning LoRa using Llama Factory and training accordingly. The structured rewriting model could be qwen3-8B; the target model could be qwen3-max; and the multimodal parsing module could be Qwen3-VL-32B.

[0078] Based on any of the above embodiments, the automated context optimization algorithm of the present invention achieves end-to-end information flow from raw input—semantic decoupling—information compression—structured reconstruction—target reasoning in its architecture. After explicitly separating the task content, it introduces it into a three-layer compression model. Through multi-dimensional filtering, the reference information becomes more focused and efficient. Then, through structured rewriting to unify the input format, it significantly improves the understanding depth and generation accuracy of the target large model. This method not only optimizes the efficiency of context processing but also provides a general context engineering paradigm for multimodal intelligent agents, enabling the implementation of interpretable, scalable, and iterative intelligent question answering and content generation systems in real-world business scenarios.

[0079] For details, see Figure 3 The system employs a context optimization mechanism that separates tasks from content. By structurally decoupling the "task objective" and "reference information" in user input, the system can dynamically match appropriate compression paths for different task types, achieving full-process control from semantic recognition to information extraction, and improving task matching accuracy and generation precision. It adopts a multimodal collaborative information compression and reconstruction architecture, integrating three optimization models: summarization, selection, and extraction. Differentiated processing paths are designed for multi-source data such as text, video, and images, achieving cross-modal semantic alignment and unified representation, significantly reducing context length, and increasing the density and relevance of key information. Using a structured rewriting and adaptive feedback mechanism, after compression, the system reconstructs information into modular corpus templates and continuously optimizes the weights of the three models through task-driven feedback, achieving a dynamic balance between information compression, generation quality, and interpretability. This provides a highly efficient and reusable context engineering solution for multimodal intelligent agents.

[0080] The following describes a question-and-answer device provided by an embodiment of this application. The question-and-answer device described below can be referred to in correspondence with the question-and-answer method described above. The device in this embodiment is installed in an electronic device. Figure 4 , Figure 4 This is a structural block diagram of a question-answering device according to one embodiment of this application, including: an acquisition module 210, used to acquire multimodal question information, wherein the multimodal question information represents question information in text form determined based on multimodal information; a task content separation module 220, used to separate the multimodal question information to obtain reference content and task keywords; a context compression module 230, used to obtain summary content using a summary model based on the reference content; obtain selected content using a selection model based on the reference content and task keywords; obtain extracted content using an extraction model based on the reference content and task keywords; a structured reconstruction module 240, used to perform structured rewriting using a structured rewriting model based on the summary content, selected content, and extracted content to obtain rewritten content; and a reasoning module 250, used to perform reasoning using a target model based on the rewritten content to obtain a response result.

[0081] In one possible implementation, the acquisition module 210 is configured to: acquire multimodal information and task data input by the user, wherein the multimodal information includes at least one of the following: text data and visual data; input the visual data into the multimodal parsing module to obtain visual text; and obtain multimodal question information based on at least one of the text data, visual text, and task data.

[0082] In one possible implementation, the acquisition module 210 is configured to: if the visual data is video data, extract keyframes according to the multimodal analysis module; and identify the first screen information and the first text information of the audio subtitles of the keyframes; generate visual text based on the first screen information and the first text information; if the visual data is image data, identify the second screen information and the second text information of the image data according to the multimodal analysis module; and generate visual text based on the second screen information and the second text information.

[0083] In one feasible approach, if the multimodal information includes text data and visual data, then the acquisition module 210 is configured to: determine the task target type based on the task data; determine the text weight and visual weight based on the task target type; perform weighted calculations based on the text weight, visual weight text data, and visual text, and combine them with the task data to obtain multimodal question information.

[0084] In one possible implementation, the context compression module 230 is used to extract key information from the reference content using a summary model, and summarize the key information to obtain summary content; to select fragment content related to the task keywords from the reference content using a selection model; to perform semantic reconstruction based on the fragment content to obtain selected content; and to match detailed content from the reference content using an extraction model as extracted content.

[0085] In one possible implementation, the context compression module 230 is used to: filter out a first sub-fragment content associated with the task keyword from the reference content; filter out a second sub-fragment content associated with the first sub-fragment content from the reference content; and obtain fragment content based on the first and second sub-fragment content.

[0086] In one feasible approach, it also includes: an optimization module for periodically identifying the differences between the response results and the expected results; and optimizing the structured rewrite model and the target model based on the differences.

[0087] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the electronic device includes: a memory 60 for storing a computer program; and a processor 61 for executing the computer program to implement the steps of the method as described in the above embodiments.

[0088] The electronic devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0089] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0090] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 60 is used to store at least the following computer program 601, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps of the method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. The operating system 602 may include Windows, Unix, Linux, etc.

[0091] In some embodiments, the electronic device may further include a display screen 62, an input / output interface 63, a communication interface 64, a power supply 65, and a communication bus 66.

[0092] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0093] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, magnetic disks, or optical disks, and other media capable of storing program code.

[0094] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described above.

[0095] Based on this, embodiments of the present invention also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described method. It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0096] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A question-and-answer method, characterized in that, include: Obtain multimodal question information, wherein the multimodal question information represents question information in text form determined based on multimodal information; Separate the multimodal question information to obtain reference content and task keywords; Based on the reference content, a summary model is used to obtain the summary content; based on the reference content and the task keywords, a selection model is used to obtain the selection content; based on the reference content and the task keywords, an extraction model is used to obtain the extraction content. Based on the summarized content, the selected content, and the extracted content, a structured rewriting model is used to perform structured rewriting to obtain the rewritten content; Based on the rewritten content, the target model is used to perform reasoning to obtain the response result.

2. The question-and-answer method according to claim 1, characterized in that, Obtain multimodal question information, including: Acquire multimodal information and task data input by the user, wherein the multimodal information includes at least one of the following: text data and visual data; The visual data is input into the multimodal parsing module to obtain visual text; Based on at least one of the text data, the visual text, and the task data, multimodal question information is obtained.

3. The question-and-answer method according to claim 2, characterized in that, The visual data is input into the multimodal parsing module to obtain visual text, including: If the visual data is video data, then according to the multimodal analysis module, key frames are extracted; and the first image information of the key frames and the first text information of the audio subtitles are identified; and visual text is generated based on the first image information and the first text information. If the visual data is image data, then according to the multimodal analysis module, the second image information and the second text information of the image data are identified; and visual text is generated based on the second image information and the second text information.

4. The question-and-answer method according to claim 2, characterized in that, If the multimodal information includes text data and visual data; Accordingly, based on at least one of the text data and the visual text, and the task data, multimodal question information is obtained, including: Based on the task data, determine the task objective type; Determine text weights and visual weights based on the task objective type; The text data and visual text are weighted and calculated based on the text weight, the visual weight, and the visual text, and combined with the task data to obtain multimodal question information.

5. The question-and-answer method according to claim 1, characterized in that, Based on the reference content, the summary content is obtained using a summary model, including: using the summary model to extract key information from the reference content, and summarizing the key information to obtain the summary content; Based on the reference content and the task keywords, a selection model is used to obtain selected content, including: using the selection model to filter out fragment content associated with the task keywords from the reference content; and performing semantic reconstruction based on the fragment content to obtain the selected content. Based on the reference content and the task keywords, the extraction model is used to obtain the extracted content, including: using the extraction model to match detailed content from the reference content as the extracted content.

6. The question-and-answer method according to claim 5, characterized in that, Filtering out content segments associated with the task keywords from the reference content includes: Filter out the first sub-fragment content associated with the task keyword from the reference content; Filter out the second sub-fragment content that is associated with the first sub-fragment content from the reference content; The fragment content is obtained based on the content of the first sub-fragment and the content of the second sub-fragment.

7. The question-and-answer method according to any one of claims 1 to 6, characterized in that, After obtaining the response result by reasoning using the target model based on the rewritten content, the process also includes: Regularly identify the differences between the responses and the expected results; The structured rewriting model and the target model are optimized based on the differences in content.

8. A question-and-answer device, characterized in that, include: The acquisition module is used to acquire multimodal question information, wherein the multimodal question information represents question information in text form determined based on multimodal information; The task content separation module is used to separate the multimodal question information to obtain reference content and task keywords; The context compression module is used to obtain summarized content using a summary model based on the reference content; to obtain selected content using a selection model based on the reference content and the task keywords; and to obtain extracted content using an extraction model based on the reference content and the task keywords. The structured reconstruction module is used to perform structured rewriting based on the summarized content, the selected content, and the extracted content using a structured rewriting model to obtain the rewritten content. The reasoning module is used to perform reasoning based on the rewritten content using the target model to obtain the response result.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the question-and-answer method according to any one of claims 1 to 7 when running the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the question-and-answer method as described in any one of claims 1 to 7.

Images