Information processing method and device in intelligent question-answering system, equipment and medium

Abstract

The invention discloses an information processing method and device in an intelligent question-answering system, equipment and a medium, and relates to the technical field of artificial intelligence and multi-modal information processing. The method comprises the following steps: receiving user data from multiple modes, and respectively determining significance scores of the user data of each mode and user query; processing the significance score through a nonlinear fusion function so as to generate an adaptive weight corresponding to each mode; and performing weighted fusion on the user data of each modal according to the self-adaptive weight to generate a unified fusion semantic representation, and inputting the fusion semantic representation into a large language model to perform inference and decision of downstream tasks so as to output a task inference result for user query. Through the technical scheme provided by the invention, the key mode can be dynamically highlighted, irrelevant information can be suppressed, the robustness and accuracy of the intelligent agent in a complex multi-mode scene can be remarkably improved, and the universality and expansibility are relatively high.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal information processing technology, and in particular to an information processing method, apparatus, device and medium in an intelligent question-and-answer system. Background Technology

[0002] In recent years, intelligent agents have become an important form of large-scale model applications, widely used in fields such as dialogue and question answering, knowledge retrieval, human-computer interaction, and automated decision-making. Existing intelligent agents typically need to process data input from multiple modalities, such as user natural language commands (text modality), user-uploaded images or video frames (visual modality), knowledge base or internet search results (structured / unstructured text modality), or data returned by external tools (numerical values, API call results, etc.). However, current mainstream methods generally employ linear weighting or simple concatenation for multimodal information fusion. Linear weighting assigns fixed weights to different modalities and sums them proportionally. This approach lacks adaptive modeling of task relevance and input variability, easily leading to low-relevance modalities "dragging down" the overall decision. Simple concatenation directly concatenates features from different modalities before inputting them into the downstream model. This method does not consider the importance of each modality in different tasks, resulting in high sensitivity of the model to irrelevant information and insufficient inference accuracy.

[0003] While existing research has introduced attention mechanisms, most are limited to single modalities (such as image saliency detection and text attention distribution), and a cross-modal, interpretable saliency-driven fusion framework has not yet been formed. In particular, in agent scenarios, how to dynamically amplify modalities that are strongly relevant to the task and suppress weakly relevant or noisy modalities is a pressing problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an information processing method, apparatus, device, and medium in an intelligent question-answering system, which can dynamically highlight key modalities, suppress irrelevant information, significantly improve the robustness and accuracy of the intelligent agent in complex multimodal scenarios, and solve the problems of insufficient linear weighted fusion, imbalanced weight distribution between modalities, and interference from low-related information in existing intelligent agents under multi-source input scenarios. The specific solution is as follows:

[0005] In a first aspect, this application discloses an information processing method for an intelligent question-answering system, including:

[0006] Receive user data from multiple modalities and determine the significance score of the user data and the user query for each modality; wherein the significance score is used to measure the relevance of the user data and the user query;

[0007] The saliency scores are processed by a preset nonlinear fusion function to generate adaptive weights corresponding to each mode; wherein, the preset nonlinear fusion function is used to amplify or suppress the weights of the modes corresponding to the saliency scores based on the magnitude relationship between the saliency scores.

[0008] The user data of each modality is weighted and fused according to the adaptive weights to generate a unified fused semantic representation. The fused semantic representation is then input into a large language model for inference and decision-making in downstream tasks, so as to output the task inference result for the user query.

[0009] Optionally, the multimodal user data includes at least two of the following: text modal data, image modal data, search result data, and tool return result data.

[0010] The determination of the salience scores of the user data and user queries for each modality includes:

[0011] When the user data is the text modal data, the first similarity between the semantic embedding vector of the text modal data and the semantic embedding vector of the user query is determined by the semantic matching model to obtain the text saliency score;

[0012] When the user data is the image modality data, a saliency map of the image modality data is generated through a saliency detection model, and salient region features of the saliency map are extracted; a second similarity is determined between the embedding vector of the salient region features and the semantic embedding vector of the user query, and a weighted average of the second similarity is performed based on the saliency weight assigned to each salient region feature to obtain a visual saliency score;

[0013] When the user data is the search result data, determine the BM25 score of each searched document in the search result data, and the third similarity between the embedding vector of the search result data and the semantic embedding vector of the user query; weight the third similarity and sum it with the BM25 score, and take the maximum value as the document saliency score;

[0014] When the user data is the result data returned by the tool, the tool significance score is determined by weighted summation based on the tool's prediction success rate and the fourth similarity between the tool output and the user query.

[0015] Optionally, the step of processing the saliency score using a preset nonlinear fusion function to generate adaptive weights corresponding to each modality includes:

[0016] The significance scores of each modality are input into an exponential weighting function to generate adaptive weights corresponding to each modality;

[0017] The formula for calculating the exponential weighting function is as follows: ;in, The amplification factor is i, and j represents different modes, respectively. , The significance score of the modality.

[0018] Optionally, the step of processing the saliency score using a preset nonlinear fusion function to generate adaptive weights corresponding to each modality includes:

[0019] The significance score of each modality is compared with a first preset threshold.

[0020] By using a gating function, the weights corresponding to modes with saliency scores below the first preset threshold are suppressed, while the weights corresponding to modes with saliency scores above the first preset threshold are retained or amplified, in order to generate adaptive weights corresponding to each mode.

[0021] The formula for calculating the gate function is as follows: ;in, It is the Sigmoid function. The significance score of the modality. The first preset threshold is used.

[0022] Optionally, the step of processing the saliency score using a preset nonlinear fusion function to generate adaptive weights corresponding to each modality includes:

[0023] The saliency scores of all modalities are concatenated into an input vector, and the input vector is fed into a multi-layer nonlinear neural network for nonlinear transformation to generate adaptive weights corresponding to each modality.

[0024] Optionally, the information processing method in the intelligent question-answering system further includes:

[0025] If the target saliency score is greater than the second preset threshold, and the target saliency score is greater than the saliency score of any other modality by a preset proportion, then the modality-first inference path is triggered.

[0026] Based on the modality-first reasoning path, the user data of the modality corresponding to the target saliency score is directly input into the large language model for downstream task reasoning and decision-making, so as to output the task reasoning result for the user query.

[0027] Optionally, the step of inputting the fused semantic representation into a large language model for inference and decision-making in downstream tasks, to output task inference results for the user query, includes:

[0028] If the downstream task is a question-answering generation task, the fused semantic representation is input as a conditional vector into the decoding process of the large language model to generate a natural language answer for the user query.

[0029] If the downstream task is a tool invocation task, the fused semantic representation is input into the policy network to select the target tool to be invoked for the user query and determine the corresponding invocation parameters;

[0030] If the downstream task is a retrieval enhancement task, the fused semantic representation is used as a query extension vector, and the knowledge base is re-retrieved or the retrieved results are reordered based on the query extension vector to filter out retrieval documents that are more relevant to the user's query.

[0031] Secondly, this application discloses an information processing device for an intelligent question-answering system, comprising:

[0032] A saliency score calculation module is used to receive user data from multiple modalities and determine the saliency score of the user data and the user query for each modality; wherein the saliency score is used to measure the relevance between the user data and the user query;

[0033] The weight allocation module is used to process the saliency scores through a preset nonlinear fusion function to generate adaptive weights corresponding to each mode; wherein, the preset nonlinear fusion function is used to amplify or suppress the weights of the modes corresponding to the saliency scores based on the magnitude relationship between the saliency scores.

[0034] The inference module is used to perform weighted fusion of the user data of each modality according to the adaptive weights to generate a unified fusion semantic representation, and input the fusion semantic representation into the large language model for inference and decision-making of downstream tasks to output the task inference result for the user query.

[0035] Thirdly, this application discloses an electronic device, which includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the information processing method in the intelligent question-answering system as described above.

[0036] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the information processing method in the intelligent question-answering system described above.

[0037] The beneficial effects of this application are as follows: By linking saliency scores with a nonlinear fusion function, dynamic weight allocation is achieved, overcoming the limitations of traditional linear weighted fusion which relies on fixed weight allocation. This allows for automatic adjustment of modality weights in different task scenarios, effectively avoiding bias caused by fixed weights. Thus, when a modality has a significant advantage in a specific task, the system can automatically amplify its contribution, highlighting its dominant role and improving the credibility of the results, while weakly correlated modalities are suppressed, avoiding the dilution of important information through "averaging." It also possesses strong versatility and can be widely applied to various agent tasks such as text-image question answering, cross-modal retrieval, and tool scheduling. Furthermore, the fused semantic representation supports flexible decoupling from downstream tasks, exhibits good scalability, and can be embedded as an independent fusion layer into agents driven by any large model, demonstrating strong compatibility. Therefore, through the organic combination of saliency calculation, nonlinear weight allocation, and large-model inference, the system can generate a unified representation that better matches user intent and demonstrates superior performance in inference and decision-making. This method can dynamically highlight key modalities and suppress irrelevant information, significantly improving the robustness and accuracy of the agent in complex multimodal scenarios, and has strong universality and scalability.

[0038] Furthermore, the information processing device, equipment, and storage medium provided in this application for an intelligent question-and-answer system correspond to the information processing method in the aforementioned intelligent question-and-answer system and have the same effect. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1 This application discloses a flowchart of an information processing method in an intelligent question-answering system.

[0041] Figure 2 This is a schematic diagram of the information processing device structure in an intelligent question-answering system disclosed in this application;

[0042] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Currently, intelligent agents in multi-source input scenarios suffer from problems such as insufficient linear weighted fusion, imbalanced weight allocation between modalities, and interference from low-related information. To address these issues, this application provides an information processing scheme for an intelligent question-answering system that enables dynamic and accurate modal weight allocation when processing multi-modal inputs, thereby improving reasoning ability and robustness in complex scenarios.

[0045] Example 1 discloses an information processing method in an intelligent question-answering system, using this typical application scenario to illustrate the implementation of the invention. Those skilled in the art should understand that this is merely a specific application example of the invention and not a limitation thereof. See also Figure 1 As shown, the method includes:

[0046] Step S11: Receive user data from multimodal sources and determine the significance score of the user data and user query for each modality; wherein the significance score is used to measure the relevance of the user data and the user query.

[0047] In this embodiment, user data from multiple modalities is first received during the input phase. This data includes not only natural language and images, but may also contain document information returned by the retrieval system and execution results provided by external tools. Specifically, the multimodal user data includes at least two of the following: text modal data, image modal data, retrieval result data, and tool-returned result data.

[0048] Furthermore, to address the issue of ineffective weight allocation due to a lack of deep understanding of task context and user intent, a method is used to accurately assess modal relevance by calculating saliency scores on multimodal user data. Each modality of user data corresponds to a saliency score, representing its "usefulness" in completing the user query task. This reduces interference from invalid information, ensuring that subsequent inference processes align with the user's actual needs and enhancing the reliability and accuracy of the inference results.

[0049] It is understood that although the core technical framework of this invention is described using an intelligent question-answering system as a preferred embodiment, it essentially provides a general multimodal information processing paradigm. Therefore, it can be widely applied to various intelligent agent systems that require the fusion of multimodal information. As a technical entity carrying artificial intelligence capabilities, an intelligent agent's core characteristic lies in its ability to perceive the environment, perform reasoning, and execute decisions. Therefore, this invention can provide a unified multimodal information processing foundation for various intelligent agent systems, directly supporting the competitiveness enhancement of product lines such as intelligent question-answering platforms, knowledge retrieval systems, and enterprise-level multimodal assistants. It can achieve cross-domain implementation in multiple industry scenarios such as finance, healthcare, education, and security, creating new market space and service value.

[0050] Step S12: The saliency scores are processed by a preset nonlinear fusion function to generate adaptive weights corresponding to each mode; the preset nonlinear fusion function is used to amplify or suppress the weights of the modes corresponding to the saliency scores based on the magnitude relationship between the saliency scores.

[0051] When users input multimodal user data, each modality corresponds to a saliency score. In this embodiment, a saliency-driven nonlinear amplification mechanism is used to achieve a "winner-takes-all" effect when the modality saliency scores differ significantly, fundamentally avoiding information loss caused by averaging. Specifically, a nonlinear fusion function is used to amplify or suppress the saliency scores. The weights of modalities with high importance are automatically amplified, while the weights of modalities with low importance are suppressed.

[0052] It is understandable that in a dynamic environment, the modal content and quality of each user query and input may differ. For example, an image might be crucial this time, but text might be more critical next time. Therefore, this embodiment does not rely on preset fixed weights, but instead determines the final adaptive weights in real time based on the salience scores of each modality in the current input (calculated in real time). In this way, the system can adaptively adjust modal weights in a dynamic environment, effectively avoiding the problems of single modality dominance or modal conflict, and ensuring stability and accuracy under complex tasks and uncertain conditions.

[0053] Step S13: The user data of each modality is weighted and fused according to the adaptive weights to generate a unified fused semantic representation, and the fused semantic representation is input into the large language model for inference and decision-making of downstream tasks to output the task inference result for the user query.

[0054] In this embodiment, based on the generated modality weight vector, weighted fusion is performed on the user data of each modality to obtain a unified fused semantic representation. The fusion method can be weighted summation, weighted concatenation, or dynamic aggregation based on an attention mechanism.

[0055] Furthermore, the unified representation after nonlinear fusion is fed into the large model inference layer. At this stage, it supports inference and decision-making for various downstream tasks, such as cross-modal retrieval, speech-assisted image recognition, and video semantic understanding.

[0056] In the first specific implementation, in a natural language question-answering generation scenario, if the downstream task is question-answering generation, the fused semantic representation is used as a conditional vector input to the decoding process of the large language model to generate a natural language answer for the user query. An attention mechanism dynamically guides the model to focus more on information corresponding to salient modalities when generating answers, thereby improving the relevance and accuracy of the responses.

[0057] In the second specific implementation, in a tool invocation decision scenario, if the downstream task is a tool invocation task, the fused semantic representation is input into the policy network to select the target tool to be invoked based on the user query and determine the corresponding invocation parameters. It can be seen that the policy network is used to analyze the user query requirements based on the fused semantic representation, select the most suitable external tool from a pre-defined external tool library, and determine the invocation parameters. The policy network can be an end-to-end trained neural network, and its output can be a tool identifier and its corresponding parameter set.

[0058] In the third specific implementation, if the downstream task is a retrieval enhancement task, the fused semantic representation is used as the query expansion vector, and the knowledge base is re-retrieved or the retrieved results are reordered based on the query expansion vector to filter out retrieved documents that are more relevant to the user's query and obtain more relevant knowledge support. In the image and text retrieval task, the final joint representation can be input into the contrastive learning loss function for optimization; in the classification task, it can be input into the classifier for prediction.

[0059] Finally, the system outputs the inference results to the user, which can be in the form of natural language answers, specific actions, or multimodal results displayed in a visual format. The entire process of this invention is highly flexible, applicable not only to typical modalities such as text-image-document-tool, but also extending to more complex multimodal inputs such as voice and sensor data. Notably, a feedback mechanism can be introduced during the inference phase to further suppress low-quality modalities and optimize system robustness.

[0060] The beneficial effects of this application are as follows: By linking saliency scores with a nonlinear fusion function, dynamic weight allocation is achieved, overcoming the limitations of traditional linear weighted fusion which relies on fixed weight allocation. This allows for automatic adjustment of modality weights in different task scenarios, effectively avoiding bias caused by fixed weights. Thus, when a modality has a significant advantage in a specific task, the system can automatically amplify its contribution, highlighting its dominant role and improving the credibility of the results, while weakly correlated modalities are suppressed, avoiding the dilution of important information through "averaging." It also possesses strong versatility and can be widely applied to various agent tasks such as text-image question answering, cross-modal retrieval, and tool scheduling. Furthermore, the fused semantic representation supports flexible decoupling from downstream tasks, exhibits good scalability, and can be embedded as an independent fusion layer into agents driven by any large model, demonstrating strong compatibility. Therefore, through the organic combination of saliency calculation, nonlinear weight allocation, and large-model inference, the system can generate a unified representation that better matches user intent and demonstrates superior performance in inference and decision-making. This method can dynamically highlight key modalities and suppress irrelevant information, significantly improving the robustness and accuracy of the agent in complex multimodal scenarios, and has strong universality and scalability.

[0061] Example 2: Compared with traditional linear weighting or simple concatenation methods, the core innovation of this application's embodiment lies in introducing the concept of "saliency-driven," which not only relies on the feature strength of the modality itself, but also uses the degree of matching between the modality and task requirements and contextual intent as the core indicator, and then adjusts the weights through a nonlinear mechanism. Therefore, this embodiment specifically describes the process of calculating the saliency of multimodal user data.

[0062] First, raw input data from different modalities, such as image data, text descriptions, or speech signals, is received. Preprocessing operations such as normalization, noise reduction, and feature standardization are performed for each modality. Second, the task relevance of features for each modality is calculated to obtain a saliency score. The saliency measure considers not only a single similarity index but also incorporates multi-dimensional factors: semantic relevance to the target task label or query, the discriminative power of features in the candidate space, and intra-modal confidence or quality assessment metrics. For example, in cross-modal retrieval tasks, the contribution of text and image modalities to the query can be calculated by combining cosine similarity with a cross-attention mechanism.

[0063] (1) When the user data is the text modal data, the first similarity between the semantic embedding vector of the text modal data and the semantic embedding vector of the user query is determined by the semantic matching model to obtain the text saliency score.

[0064] In this embodiment, for text modal data, a pre-trained language model is used to extract context embedding vectors. A semantic matching model is introduced to perform deep interaction modeling between the input text and the user query, thereby obtaining a relevance score. Its calculation can be formalized as .

[0065] Here, Q (Query) represents the user query, such as the question or command text entered by the user; T (Text) represents the text modal data to be evaluated (such as the documents returned by the retrieval, the content of historical conversations, the text results returned by the tool, etc.). , Let T and Q represent the semantic embedding vectors of the query and the text, respectively. As can be seen, this formula measures the semantic relevance between the text input T and the user query Q, using the semantic embedding vectors... and This is achieved through similarity.

[0066] Furthermore, this invention introduces contextual intent matching degree into the saliency calculation of text modality. By analyzing the user's historical dialogue, scene context and task goal, it determines whether the text truly matches the user's intent, thereby making the saliency score more reflective of task relevance.

[0067] (2) When the user data is the image modality data, a saliency map of the image modality data is generated by the saliency detection model, and the saliency region features of the saliency map are extracted; the second similarity between the embedding vector of the saliency region features and the semantic embedding vector of the user query is determined, and the second similarity is weighted and averaged based on the saliency weight assigned to each saliency region feature to obtain a visual saliency score.

[0068] In this embodiment, a saliency map is generated for image modality data using a saliency detection model. Features of salient regions are extracted using a convolutional neural network or a visual Transformer, and then mapped to low-dimensional embedding vectors to obtain visual saliency scores. Its calculation method can be expressed as follows:

[0069] Where r (Region) refers to a salient region (e.g., an object or region in the image) identified by the salientity detection model, and R represents the set of salient regions. For the significance weight of the region, It is the feature embedding vector obtained after feature extraction of region r. This represents the semantic embedding vector of the user query. This is the similarity function to the user query. As can be seen, this formula is a weighted sum of all salient regions r.

[0070] Furthermore, for speech modalities, Mel-frequency transform can be performed first, followed by extraction of temporal features using an acoustic encoder. The calculation method is similar to that for image modalities, and will not be elaborated upon here.

[0071] (3) When the user data is the search result data, determine the BM25 score of each searched document in the search result data, and the third similarity between the embedding vector of the search result data and the semantic embedding vector of the user query; weight the third similarity and sum it with the BM25 score, and take the maximum value as the document saliency score.

[0072] In this embodiment of the application, for the retrieval result data, a document saliency score is obtained through a document-query relevance scoring mechanism. Formalization .

[0073] Where D represents the set of documents returned by the retrieval, and d is a single document in set D; This is a balancing factor used to adjust the weight ratio between the BM25 score and the semantic similarity score; This represents the semantic embedding vector of the user query. Let d represent the semantic embedding vector of document d; BM25 is an information retrieval scoring function used to calculate the statistical relevance of word frequencies between a query and a document. As can be seen, this formula combines the traditional retrieval metric BM25 with semantic similarity, enhancing its ability to distinguish long documents.

[0074] (4) When the user data is the result data returned by the tool, the tool significance score is determined by weighted summation based on the tool's prediction success rate and the fourth similarity between the tool output and the user query.

[0075] In this embodiment, confidence and execution reliability indicators are defined for the tool's returned results data according to task requirements, and a tool saliency score is generated by combining logical rules and a learning model. It can be written as:

[0076] ;in, It is a balancing hyperparameter used to adjust the prediction success rate of the tool. The semantic matching degree between the tool output and the user query The weights between them. It is evident that this formula considers both the reliability of the tool's execution and the relevance of the tool's returned result O to the user's query Q. It is used to control which side is more dominant.

[0077] Example 3: After obtaining the above-mentioned multimodal significance score set... Subsequently, this invention proposes several nonlinear fusion methods for dynamic weighting as optional implementation paths. Modal weights are generated using a nonlinear mapping function. This function amplifies the influence of highly significant modes and suppresses the interference of low-significance modes. Commonly used functions include: the sigmoid function, the softmax function with temperature adjustment, and the exponential function; when the significance of a certain mode is much higher than that of other modes, the function nonlinearly increases the weight of that mode, rather than a simple linear proportion.

[0078] In the first specific implementation, an exponential weighting mechanism based on nonlinear fusion function amplification is used: specifically, the significance scores of each modality are input into the exponential weighting function to generate adaptive weights corresponding to each modality.

[0079] The formula for calculating the exponential weighting function is as follows: ;in, The amplification factor can be adjusted to change the degree of emphasis on highly significant modes; i and j represent different modes, respectively. The significance score can represent the current modality. is the significance score for all other modalities (j≠i).

[0080] This mechanism is based on amplification factor. The exponential function is used to nonlinearly transform the saliency scores of each modality, and then the weight of each modality is calculated through normalization. The core of this mechanism is that small differences in modality saliency scores are amplified through exponential operations, significantly increasing the weight of modalities more relevant to the task. If the saliency scores of different modalities are similar, the weights are smoothly distributed, avoiding extreme skewness or averaging. When the saliency score of one modality is significantly higher than that of other modalities, the exponential weighting mechanism allows its weight to grow exponentially, thus dominating the unified representation; while when the saliency scores are similar, the mechanism provides a smooth weighting effect, achieving dynamic fusion of multimodal inputs. This approach not only balances flexibility and robustness but also provides core support for inference in complex scenarios.

[0081] In the second specific implementation, a threshold selection mechanism based on a gating function is used: specifically, the significance score of each modality is compared with a first preset threshold; the weights corresponding to modalities with significance scores lower than the first preset threshold are suppressed by the gating function, while the weights corresponding to modalities with significance scores higher than the first preset threshold are retained or amplified, so as to generate adaptive weights corresponding to each modality.

[0082] The formula for calculating the gate function is as follows: ;in, It is the Sigmoid function, which maps the difference between the significance score and the threshold τ to a weight value between 0 and 1; The significance score of the modality. The first preset threshold is used.

[0083] As can be seen, based on the Sigmoid gating function, a preset significance threshold is introduced. The saliency scores of each modality are compared with a threshold and converted into weights. Modal weights below the threshold are significantly suppressed, while those above the threshold are retained or amplified. This gating mechanism is particularly suitable for scenarios with weak modal interference, effectively eliminating low-relevance or even noisy modes, thus ensuring the purity and high relevance of the final fused representation. This implementation demonstrates significant advantages in adversarial environments or under noisy input conditions.

[0084] For example, in a "noisy cross-modal retrieval task", the user query is "find range data for new energy vehicles", and the multimodal input includes "new energy vehicle documents (retrieval result modality)". =0.8), "Blurred landscape image (noise mode, =0.2). Set the threshold. =0.5, calculate the weight according to the formula: Modal weight of retrieval results Noise mode weights That is, the noise mode weight is only 0.425, which is much lower than the 0.57 of the search result mode, thus achieving noise suppression.

[0085] In a third specific implementation, a deep nonlinear structure is used for combination. Specifically, the saliency scores of all modes are concatenated into an input vector, and this input vector is fed into a multi-layer nonlinear neural network for nonlinear transformation to generate adaptive weights corresponding to each mode.

[0086] This method concatenates all modality saliency scores into an input vector, which is then fed into a multi-layer nonlinear network structure. After nonlinear transformation, the final modality weight vector is output. Unlike the previous two predefined functions, this method automatically learns the optimal fusion strategy through end-to-end training, without relying on manually defined function forms, thus exhibiting stronger expressive power and generalization ability. This implementation is particularly suitable for scenarios with diverse tasks and complex input distributions, and can adaptively capture nonlinear dependencies between modalities.

[0087] In the fourth specific implementation, a dynamic routing mechanism is used. Specifically, if the target salience score is greater than a second preset threshold, and the target salience score is greater than the salience score of any other modality by a preset proportion, a modality-first inference path is triggered. Based on the modality-first inference path, the user data of the modality corresponding to the target salience score is directly input into the large language model for inference and decision-making of downstream tasks, so as to output the task inference result for the user query.

[0088] In this approach, when the saliency score of a certain modality is significantly higher than that of other modalities and exceeds a preset threshold, the system can directly trigger a "modality-priority inference path," bypassing the weighted fusion process and directly sending the modality's features into the inference and decision-making module. This mechanism can significantly improve inference efficiency while ensuring the accuracy of the results, reducing unnecessary computation time, and preventing the information of salient modalities from being diluted by averaging. It has unique value for scenarios with high real-time requirements or tasks that heavily rely on a single modality.

[0089] It should be noted that the nonlinear fusion strategy uses an exponential weighting mechanism as its core implementation method, while also providing a gating function threshold mechanism, a deep nonlinear structure, and a dynamic routing mechanism as alternative implementation methods. The unified representation after nonlinear fusion is fed into the large model inference layer, and finally, the system outputs the inference results to the user. This method has broad application prospects in multiple fields such as intelligent question answering, cross-modal retrieval, tool invocation, and multimodal interaction, demonstrating significant originality and practical value.

[0090] Accordingly, embodiments of this application also disclose an information processing device for an intelligent question-and-answer system, see [link to relevant documentation]. Figure 2 As shown, the device includes:

[0091] The saliency score calculation module 11 is used to receive user data from multiple modalities and determine the saliency score of the user data and the user query for each modality; wherein the saliency score is used to measure the relevance between the user data and the user query;

[0092] The weight allocation module 12 is used to process the saliency scores through a preset nonlinear fusion function to generate adaptive weights corresponding to each mode; wherein, the preset nonlinear fusion function is used to amplify or suppress the weights of the modes corresponding to the saliency scores based on the magnitude relationship between the saliency scores.

[0093] The reasoning module 13 is used to perform weighted fusion of the user data of each modality according to the adaptive weights to generate a unified fusion semantic representation, and input the fusion semantic representation into the large language model for reasoning and decision-making of downstream tasks to output the task reasoning result for the user query.

[0094] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0095] Therefore, the above-described scheme in this embodiment achieves dynamic weight allocation through the linkage mechanism of saliency scores and nonlinear fusion functions. This overcomes the limitations of traditional linear weighted fusion, which relies on fixed weight allocation, and can automatically adjust modal weights in different task scenarios, effectively avoiding the bias caused by fixed weights. In this way, when a modality has a significant advantage in a specific task, the system can automatically amplify its contribution, highlighting its dominant role and improving the credibility of the results, while weakly correlated modalities are suppressed, avoiding the dilution of important information through "averaging." It also has strong versatility and can be widely applied to various intelligent agent tasks such as text-image question answering, cross-modal retrieval, and tool scheduling. Furthermore, the fused semantic representation supports flexible decoupling of downstream tasks, has good scalability, and can be embedded as an independent fusion layer into intelligent agents driven by any large model, exhibiting strong compatibility. Thus, through the organic combination of saliency calculation, nonlinear weight allocation, and large model inference, the system can generate a unified representation that better matches user intent and demonstrates superior performance in inference and decision-making. This method can dynamically highlight key modalities and suppress irrelevant information, significantly improving the robustness and accuracy of the agent in complex multimodal scenarios, and has strong universality and scalability.

[0096] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0097] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the information processing method of the intelligent question-answering system disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0099] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it can include an operating system 221, computer programs 222, and data 223, etc. The data 223 can include various types of data. The storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the information processing methods of the intelligent question-answering system executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0101] Furthermore, this application also discloses a computer-readable storage medium, which includes random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, magnetic disks, optical disks, or any other form of storage medium known in the art. When the computer program is executed by a processor, it implements the information processing method in the aforementioned intelligent question-answering system. The specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0102] Furthermore, embodiments of this application also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements any of the information processing methods in the aforementioned intelligent question-and-answer system.

[0103] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0104] The steps of the information processing methods or algorithms in the intelligent question-answering system described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The above provides a detailed description of the information processing method, apparatus, device, and medium in an intelligent question-and-answer system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An information processing method in an intelligent question-answering system, characterized by, The method comprises the steps of: receiving user data from multiple modalities, and determining the saliency scores of the user data of each modality and a user query respectively; wherein the saliency score is used to measure the relevance of the user data and the user query; processing the saliency scores through a preset nonlinear fusion function to generate adaptive weights corresponding to each modality; wherein the preset nonlinear fusion function is used to amplify or suppress the weight of the modality corresponding to the saliency score based on the size relationship between the saliency scores; weighting and fusing the user data of each modality according to the adaptive weights to generate a unified fused semantic representation, and inputting the fused semantic representation into a large language model for inference and decision of a downstream task to output a task inference result for the user query. 2.The information processing method in the intelligent question-answering system according to claim 1, characterized in that, The user data of the multiple modalities includes at least two of text modality data, image modality data, search result data, and tool return result data; The step of determining the saliency scores of the user data of each modality and the user query respectively comprises the steps of: when the user data is the text modality data, determining a first similarity between the semantic embedding vector of the text modality data and the semantic embedding vector of the user query through a semantic matching model to obtain a text saliency score; when the user data is the image modality data, generating a saliency map of the image modality data through a saliency detection model, and extracting salient region features of the saliency map; determining a second similarity between the embedding vector of the salient region features and the semantic embedding vector of the user query, and weighting and averaging the second similarity based on the saliency weight assigned to each salient region feature to obtain a visual saliency score; when the user data is the search result data, determining the BM25 score of each search document in the search result data, and a third similarity between the embedding vector of the search result data and the semantic embedding vector of the user query; weighting the third similarity and summing it with the BM25 score to take the maximum value as a document saliency score; when the user data is the tool return result data, determining a tool saliency score by weighting and summing based on the prediction success rate of the tool and the fourth similarity between the tool output and the user query. 3.The information processing method in the intelligent question-answering system according to claim 1, wherein, The step of processing the saliency scores through a preset nonlinear fusion function to generate adaptive weights corresponding to each modality comprises the steps of: inputting the saliency scores of each modality into an exponential weighting function to generate adaptive weights corresponding to each modality; The calculation formula of the exponential weighting function is: ; wherein, is an amplification coefficient, i and j respectively represent different modalities, , is the significance score of the modality. 4.The information processing method in the intelligent question-answering system according to claim 1, wherein, The step of processing the saliency scores through a preset nonlinear fusion function to generate adaptive weights corresponding to each modality comprises the steps of: comparing the saliency scores of each modality with a first preset threshold; suppressing the weight corresponding to the modality whose saliency score is lower than the first preset threshold through a gating function, and retaining or amplifying the weight corresponding to the modality whose saliency score is higher than the first preset threshold to generate adaptive weights corresponding to each modality; The calculation formula of the gating function is: ; wherein, is a Sigmoid function, is a saliency score of a modality, is the first preset threshold. 5.The information processing method in the intelligent question-answering system according to claim 1, wherein, The significant score is processed by a preset nonlinear fusion function to generate adaptive weights corresponding to each modality, including: The significant scores of all modalities are spliced into an input vector, and the input vector is input into a multi-layer nonlinear neural network for nonlinear transformation to generate adaptive weights corresponding to each modality. 6.The information processing method in the intelligent question-answering system according to claim 1, characterized in that, Also includes: If the target significant score is greater than a second preset threshold, and the target significant score is greater than the significant score of any other modality by a preset proportion, trigger the modality priority reasoning path; Based on the modality priority reasoning path, the user data of the modality corresponding to the target significant score is directly input into the large language model for reasoning and decision-making of the downstream task to output the task reasoning result for the user query.

7. The information processing method in the intelligent question-answering system according to any one of claims 1 to 6, characterized in that, The fusion semantic representation is input into a large language model for reasoning and decision-making of a downstream task to output a task reasoning result for the user query, including: If the downstream task is a question and answer generation task, the fusion semantic representation is input into the decoding process of the large language model as a condition vector to generate a natural language answer for the user query; If the downstream task is a tool calling task, the fusion semantic representation is input into a policy network to select a target tool to be called and determine the corresponding calling parameters for the user query; If the downstream task is a retrieval enhancement task, the fusion semantic representation is used as a query expansion vector, and the knowledge base is re-retrieved or the retrieved results are re-ranked based on the query expansion vector to filter retrieval documents with higher relevance to the user query. 8.An information processing device in an intelligent question-answering system, comprising: Including: A significant score calculation module for receiving user data from multiple modalities and determining the significant score of the user data and user query for each modality; wherein the significant score is used to measure the relevance of the user data and the user query; A weight distribution module for processing the significant score by a preset nonlinear fusion function to generate adaptive weights corresponding to each modality; wherein the preset nonlinear fusion function is used to amplify or suppress the weight of the modality corresponding to the significant score based on the size relationship between the significant scores. An inference module for weighting and fusing the user data of each modality according to the adaptive weights to generate a unified fusion semantic representation, and inputting the fusion semantic representation into a large language model for reasoning and decision-making of a downstream task to output a task reasoning result for the user query.

9. An electronic device, comprising: The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, the computer program is loaded and executed by the processor to realize the information processing method in the intelligent question and answer system as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing a computer program; wherein the computer program is executed by the processor to realize the information processing method in the intelligent question and answer system as claimed in any one of claims 1 to 7.

Images