Homogeneous heterogeneous and attribute signal decoupling representation learning scientific question and answer method and system

Abstract

The application discloses a homogeneous heterogeneous and attribute signal decoupling representation learning scientific question and answer method and system, and belongs to the field of visual question and answer, and comprises the following steps: obtaining initial embedding of text and visual modalities through an encoder; utilizing a first decoupling module to decouple the embedding of each modality into homogeneous embedding and heterogeneous embedding, and performing fusion to obtain preliminary representation of each modality; utilizing a second decoupling module to further decouple the fused multi-modal embedding according to the attribute signal of the question, and enhancing the distinguishability of the embedding under different attribute signals; fusing the decoupled representation with original information, and predicting an answer through a decoder. The application solves the problems of high entanglement of multi-modal information homogeneity and heterogeneity and high entanglement between various attribute signals in the existing scientific question and answer model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual question answering, and particularly relates to a homogeneous and heterogeneous attribute signal decoupling representation learning scientific question answering method and system. BACKGROUND

[0002] Existing science QA technology mainly relies on the fusion and reasoning of multi-modal information, predicting answers by combining text and visual information. Current mainstream methods usually use pre-trained Transformer models (such as BERT, FLAN-T5) to process text input, and combine visual encoders (such as DETR, ResNet) to extract image features, then realize multi-modal fusion through attention mechanism or simple splicing. In addition, some researches try to introduce the Chain-of-Thought (CoT) mechanism or external knowledge base to enhance the reasoning ability of the model. Although these methods improve the performance of scientific question answering to some extent, they often model multi-modal information as a whole, without explicitly distinguishing shared semantics (homogeneous information) and modality-specific features (heterogeneous information) between modalities, leading to the model being easily disturbed by redundant or noisy information when dealing with complex problems.

[0003] The main problem faced by existing science QA models is the high entanglement of homogeneous and heterogeneous multi-modal information, as well as the confusion of various attribute signals (such as subject categories). Specifically, existing methods fail to effectively decouple shared semantics and unique features in text and visual modalities, making it difficult for the model to focus on core information and affecting the accuracy of reasoning. In addition, problems in different disciplines (such as natural sciences and social sciences) often have unique background knowledge and logical structures, but existing technologies lack explicit decoupling mechanisms for attribute signals, making it difficult for the model to distinguish feature representations in different disciplines. This information entanglement not only limits the discriminative ability of the model, but also reduces the robustness and interpretability of scientific question answering. SUMMARY

[0004] To solve the above technical problems, the present application proposes a homogeneous and heterogeneous attribute signal decoupling representation learning scientific question answering method and system to solve the problems existing in the prior art.

[0005] To achieve the above purpose, in a first aspect, the present application provides a homogeneous and heterogeneous attribute signal decoupling representation learning scientific question answering method, comprising:

[0006] obtaining text input and visual input of a scientific question;

[0007] text encoding the text input to obtain an initial text embedding; visual encoding the visual input to obtain an initial visual embedding;

[0008] decoupling the initial text embedding into a homogeneous text embedding and a heterogeneous text embedding, and decoupling the initial visual embedding into a homogeneous visual embedding and a heterogeneous visual embedding by a first decoupling module;

[0009] fusing the homogeneous text embedding and the heterogeneous text embedding to obtain a text modal representation, and fusing the homogeneous visual embedding and the heterogeneous visual embedding to obtain a visual modal representation;

[0010] fusing the text modal representation and the visual modal representation to obtain a preliminary multi-modal embedding;

[0011] performing attribute signal decoupling representation learning on the preliminary multi-modal embedding based on a predefined attribute signal by a second decoupling module;

[0012] fusing the embedding after attribute signal decoupling learning and the embedding of the original information to obtain a final comprehensive representation;

[0013] predicting an answer to the scientific question based on the final comprehensive representation by a decoder.

[0014] Preferably, the text input includes question text, context text, multiple-choice option text, and thought chain text.

[0015] Preferably, the text encoding adopts a Transformer model, and the visual encoding adopts one of a DETR model, a ResNet model, a CLIP model, or a ViT model or a combination thereof.

[0016] Preferably, the first decoupling module includes a shared homogeneous encoder and independent heterogeneous encoders for text and vision respectively, and utilizes an autoencoder for reconstruction learning.

[0017] Preferably, in the first decoupling module, a cosine similarity-based loss function is used to distinguish homogeneous embeddings and heterogeneous embeddings within each modality, and a reconstruction loss function is introduced.

[0018] Preferably, the attribute signal includes natural science, social science, and linguistic science.

[0019] Preferably, the second decoupling module adopts a contrast learning mechanism, considers sample pairs belonging to the same attribute signal as positive sample pairs and considers sample pairs belonging to different attribute signals as negative sample pairs, and narrows the distance between positive sample pairs and widens the distance between negative sample pairs based on the decoupling loss of the attribute signal.

[0020] Preferably, the fusion of the homogeneous text embedding and the heterogeneous text embedding, the fusion of the homogeneous visual embedding and the heterogeneous visual embedding, the fusion of the text modal representation and the visual modal representation, and the fusion of the embedding learned by attribute signal decoupling and the original information embedding all adopt a fusion strategy based on a multi-head attention mechanism.

[0021] In a second aspect, the present application further provides a homogeneous-heterogeneous and attribute signal decoupling representation learning scientific question answering system, comprising:

[0022] an input interface configured to obtain text input and visual input of a scientific question;

[0023] an encoding module configured to perform text encoding on the text input to obtain an initial text embedding, and perform visual encoding on the visual input to obtain an initial visual embedding;

[0024] a first decoupling module configured to decouple the initial text embedding into a homogeneous text embedding and a heterogeneous text embedding, and decouple the initial visual embedding into a homogeneous visual embedding and a heterogeneous visual embedding;

[0025] a first fusion module configured to fuse the homogeneous text embedding and the heterogeneous text embedding to obtain a text modal representation, and fuse the homogeneous visual embedding and the heterogeneous visual embedding to obtain a visual modal representation;

[0026] a second fusion module configured to fuse the text modal representation and the visual modal representation to obtain a preliminary multi-modal embedding;

[0027] a second decoupling module configured to perform attribute signal decoupling representation learning on the preliminary multi-modal embedding based on a pre-defined attribute signal;

[0028] a final representation fusion module configured to fuse the embedding learned by attribute signal decoupling and the embedding of the original information to obtain a final comprehensive representation;

[0029] an answer prediction module configured to predict an answer to the scientific question by a decoder based on the final comprehensive representation.

[0030] In a third aspect, the present application further discloses a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method of the first aspect.

[0031] Compared with the prior art, the present application has the following advantages and technical effects:

[0032] The application provides a scientific question and answer method for homogenous and heterogeneous attribute signal decoupling representation learning, comprising the following steps: first, obtaining text input and visual input of a scientific question; performing text encoding on the text input to obtain an initial text embedding; performing visual encoding on the visual input to obtain an initial visual embedding; second, decoupling the initial text embedding into a homogenous text embedding and a heterogeneous text embedding by a first decoupling module, and decoupling the initial visual embedding into a homogenous visual embedding and a heterogeneous visual embedding; fusing the homogenous text embedding and the heterogeneous text embedding to obtain a text modal representation; fusing the homogenous visual embedding and the heterogeneous visual embedding to obtain a visual modal representation; fusing the text modal representation and the visual modal representation to obtain a preliminary multi-modal embedding; third, performing attribute signal decoupling representation learning on the preliminary multi-modal embedding based on a predefined attribute signal by a second decoupling module; finally, fusing the embedding obtained by attribute signal decoupling learning and the embedding of the original information to obtain a final comprehensive representation; and predicting an answer to the scientific question by a decoder based on the final comprehensive representation.

[0033] The application explicitly decouples the text / visual embedding into a homogenous embedding (modal shared semantics) and a heterogeneous embedding (modal unique information) by the first decoupling module, and fuses them step by step, solves the problem of entanglement of homogeneity and heterogeneity of multi-modal information, can force the model to separate the common features and modal unique features across modalities, avoid the interference of heterogeneous noise on core semantic understanding, and thus improve the discriminability and robustness of multi-modal representation.

[0034] The application performs contrastive learning on the multi-modal embedding based on the predefined attribute signal (such as discipline category) by the second decoupling module, solves the problem of confusion of multi-attribute signal features, can enhance the distinguishability of the embedding under different attribute signals (such as natural science vs. social science), and makes the model be able to learn discipline-related features, and improve the field adaptability.

[0035] The application fuses the decoupled embedding and the original information embedding again, and then decodes and predicts the answer, supplements the original detail information on the basis of decoupling, avoids the information loss in the decoupling process, ensures that the final representation has both discriminability and integrity, and improves the answer prediction accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the application illustrated in the drawings, and their description, are used to explain the application and are not intended to limit the application. In the drawings:

[0037] Figure 1 The method flowchart of the embodiment of the application;

[0038] Figure 2 The system structure schematic diagram of the embodiment of the application. Detailed Implementation

[0039] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0041] Example 1

[0042] like Figure 1 As shown, this embodiment provides a scientific question-answering method based on the decoupling representation of homogeneous and heterogeneous elements and attribute signals, including:

[0043] S1. Obtain text and visual input for scientific questions;

[0044] Specifically, the input for the science question-answering task includes textual input (such as questions, context, options, and optional thought chain text) and visual input (such as images).

[0045] S2. Perform text encoding on the text input to obtain an initial text embedding; perform visual encoding on the visual input to obtain an initial visual embedding;

[0046] Furthermore, the text input is encoded using a pre-trained text encoder (such as a Transformer) to obtain the initial text embedding. .

[0047] The visual input is encoded using a pre-trained visual encoder (such as DETR) to obtain the initial visual embedding. .

[0048] Specifically, text input QCM text consisting of questions, context, and options With MindChain CoT Text To splice, that is .

[0049] Using a pre-trained text encoder (For example, Transformer models, such as FLAN-T5) will Encoded as initial text embedding . ,in It is the text length. It is the embedded dimension.

[0050] visual input : using a pre-trained visual extractor (e.g. a DETR model) to map the visual input into initial visual embeddings . The initial visual embeddings are unified into by a linear transformation operation where is the number of image patches, is the embedding dimension, For samples in the dataset without visual input, a zero vector is used as their visual embedding.

[0051] S3, decoupling the initial text embeddings into homogeneous text embeddings and heterogeneous text embeddings by a first decoupling module, and decoupling the initial visual embeddings into homogeneous visual embeddings and heterogeneous visual embeddings by a second decoupling module;

[0052] Specifically, the embeddings of each modality are adaptively decoupled into homogeneous and heterogeneous parts;

[0053] Homogeneous encoding: using a parameter-shared homogeneous encoder to process the initial text embeddings and the initial visual embeddings respectively, to obtain homogeneous text embeddings and homogeneous visual embeddings . These embeddings represent modality-agnostic semantic information. The homogeneous embeddings include homogeneous visual embeddings and homogeneous text embeddings , both of which are semantic information that is not interfered by modalities.

[0054] ;

[0055] .

[0056] Heterogeneous encoding: using two parameter-independent heterogeneous encoders, i.e. a text heterogeneous encoder and a visual heterogeneous encoder to process the initial text embeddings and the initial visual embeddings respectively, to obtain heterogeneous text embeddings and heterogeneous visual embeddings These embeddings represent the information specific to each modality. For example, in a science question about “plant cell structure” with a microscope image, the core concept of “nucleus” can be encoded as homogeneous information because it exists in both the textual description and the visual image; the specific phrasing of the question or the logical relationship between the options are heterogeneous information specific to the textual modality; and the staining of the nucleus in the image, the magnification of the microscope, and other visual details are heterogeneous information specific to the visual modality.

[0057] ;

[0058] .

[0059] S4, fuse the homogeneous text embedding and the heterogeneous text embedding to obtain a text modality representation; fuse the homogeneous visual embedding and the heterogeneous visual embedding to obtain a visual modality representation;

[0060] Specifically, the homogeneous embedding and the heterogeneous embedding of each modality are fused to obtain a comprehensive representation of the modality. The fusion strategy A specific structure is adopted, which combines multi-head attention (MHA) and a gating mechanism. Specifically, for two inputs and , first apply MHA to to obtain an intermediate output Output1. At the same time, and (or only ) are passed through a linear layer and an activation function (such as Sigmoid) to obtain a gating scalar (or vector) . The final fusion output can be in the form of , or other forms that use to weight and combine and the original after MHA processing, thereby dynamically adjusting the contributions of the two information sources.

[0061] ;

[0062] ;

[0063] wherein and are the two inputs of the fusion operation.

[0064] The fused modality representations and are respectively sent to the corresponding decoders and for reconstruction.

[0065] Cosine similarity loss To ensure the effective separation of homogeneous and heterogeneous embeddings, even if they are semantically different.

[0066] ;

[0067] This loss encourages homogeneous and heterogeneous embeddings to point in different directions in the vector space (ideally orthogonal).

[0068] Reconstruction loss and Used to ensure that the decoupled embedding can recover the original information and heterogeneous information, and to reduce the ambiguity of the embedding.

[0069] ;

[0070] ;

[0071] in It is the decoding (reconstruction) of the comprehensive representation of text. It is the decoding of visual comprehensive representation. Ensure that the heterogeneous encoder can extract heterogeneous information again from the decoded synthetic representation.

[0072] Total loss of homogeneous-heterogeneous decoupling :

[0073] .

[0074] This embodiment separates modality-shared information (homogeneous) and modality-specific information (heterogeneous) within each modality (text and vision).

[0075] S5. The text modal representation and the visual modal representation are fused to obtain a preliminary multimodal embedding;

[0076] The learned decoupled visual representation and text representation Through fusion strategy Fusion into homogeneous and heterogeneous decoupled embedding .

[0077] S6. Through the second decoupling module, based on the predefined attribute signals, the initial multimodal embedding is subjected to attribute signal decoupling representation learning;

[0078] Comparative learning:

[0079] Assuming that scientific question-and-answer pairs can be based on their subject-specific signals (such as natural sciences) Social Sciences Language Science ) are classified.

[0080] The embedding set of all samples is . According to the attribute signal, can be divided into (natural science), (social science), (linguistic science).

[0081] Decoupling loss based on attribute signal The core idea is: for a certain sample, other samples with the same attribute signal are positive samples, and samples with different attribute signals are negative samples. The goal is to pull the embeddings of the same samples closer and push the embeddings of different samples further apart.

[0082] ;

[0083] Decoupling loss of natural science attribute signal The mathematical expression is:

[0084] ;

[0085] where, is the embedding of the th natural science sample, is the other natural science sample (positive sample), and are the social science and linguistic science samples (negative samples), respectively, is the temperature hyperparameter.

[0086] Decoupling loss of social science attribute signal The mathematical expression is:

[0087] ;

[0088] where, is the embedding of the th social science sample, is the other social science sample (positive sample), and are the natural science and linguistic science samples (negative samples), respectively, is the temperature hyperparameter.

[0089] Decoupling loss of linguistic science attribute signal The mathematical expression is:

[0090] ;

[0091] where, is the embedding of the th linguistic science sample, are other language science class samples (positive samples), and are natural science and social science class samples (negative samples), respectively, is a temperature hyperparameter.

[0092] This embodiment enhances the distinguishability of multi-modal embeddings under different attributes according to the attribute of the visual-text pair (attribute signal). The “predefined attribute signal” is usually determined according to the meta-information provided by the dataset itself, for example, each question of the ScienceQA dataset is labeled with its subject, topic and specific skill. These labeled information can be directly used as pre-defined attribute signal. In the training process, these attribute signals are used by the model as the label or metadata of the sample, and play a key role especially in constructing positive and negative sample pairs in contrast learning: sample pairs with the same attribute signal (for example, belonging to the category of “natural science- physics-mechanics”) are regarded as positive samples, while sample pairs with different attribute signals are regarded as negative samples.

[0093] S7, fuse the embedding learned by attribute signal decoupling learning with the embedding of the original information to obtain a final comprehensive representation;

[0094] In order to supplement the original, non-complex decoupled modal information at the final prediction, first, the initial visual embedding and the initial text embedding are fused to obtain .

[0095] ;

[0096] Then, the embedding learned by attribute signal decoupling and this original supplementary information embedding are finally fused to obtain .

[0097] .

[0098] S8, based on the final comprehensive representation, predict the answer to the scientific question through a decoder.

[0099] Specifically, the final comprehensive representation is input into a Transformer decoder to generate a predicted answer .

[0100] ;

[0101] Calculate the predicted answer cross-entropy loss between the true answer label and the model's prediction .

[0102] ;

[0103] The total loss function of the model is a weighted sum of the individual losses .

[0104] ;

[0105] The total loss is minimized by backpropagation and gradient descent algorithms (e.g., AdamW), thus optimizing all learnable parameters of the model. Hyperparameters and are used to balance the contributions of different loss terms, for example, they can be set to , , .

[0106] Embodiment Two

[0107] As shown in Figure 2 , a heterogeneous and attribute signal decoupled representation learning scientific question answering system includes:

[0108] An input interface for obtaining text input and visual input of a scientific question;

[0109] Specifically, the starting point of the system is the input interface, which is responsible for receiving scientific questions provided by users or external data sources. These questions usually contain text parts (such as question descriptions, related background knowledge, and possible options) and visual parts (such as charts related to the question, experimental device diagrams, natural phenomenon pictures, etc.). For example, an actual scientific question may be: "[Picture: circuit diagram] In the circuit shown in the picture, when switch S is closed, how do the brightness of bulbs L1 and L2 change? A. L1 becomes brighter, L2 becomes dimmer B. L1 becomes dimmer, L2 becomes brighter C. L1 and L2 both become brighter D. L1 and L2 both become dimmer". The input interface passes the text "circuit diagram" and options, and visual information "circuit diagram" to the subsequent modules, respectively.

[0110] After receiving the original input, the data preprocessing module standardizes it so that the model can effectively utilize it. For text input, possible operations include: text cleaning (removing irrelevant characters), word segmentation (dividing sentences into words or sub-word units), building a vocabulary, and converting text into a unified numerical sequence. For visual input, operations may include: image size normalization, cropping, data augmentation (such as rotation, flipping, to increase training data diversity), and pixel value normalization, etc. This module ensures the consistency and quality of the input data format.

[0111] a coding module, configured to perform text encoding on the text input to obtain an initial text embedding, and perform visual encoding on the visual input to obtain an initial visual embedding;

[0112] Specifically, this module is the core of information extraction, and specifically includes:

[0113] The text encoder (e.g., a FLAN-T5 model based on Transformer) is responsible for processing the preprocessed text sequence. It captures the complex semantic relationships and contextual dependencies within the text through self-attention mechanisms, maps each word or subword to a high-dimensional vector space, and finally outputs the initial text embedding of the entire text input . This embedding captures the semantic information of the question, the meaning of the options, and the contextual knowledge.

[0114] The visual encoder (e.g., a DETR model) is responsible for processing the preprocessed image. It usually first divides the image into multiple image patches, then extracts the features of each image patch through convolutional layers or Transformer structures, and further aggregates them into an initial visual embedding representing the entire image or key regions in the image . This embedding captures visual information such as objects, scenes, and spatial relationships between them in the image.

[0115] a first decoupling module, configured to decouple the initial text embedding into a homogeneous text embedding and a heterogeneous text embedding, and decouple the initial visual embedding into a homogeneous visual embedding and a heterogeneous visual embedding;

[0116] Specifically, the first decoupling module is a homogeneous and heterogeneous decoupling module, which is the first innovation core of the present application, aiming to solve the problem of entanglement between homogeneity and heterogeneity of multi-modal information.

[0117] This module receives the initial text embedding and the initial visual embedding output by the coding module.

[0118] Through a shared homogeneous encoder , the core semantic information shared by the modal and independent of the specific modal is extracted from and , respectively, to obtain a homogeneous text embedding and a homogeneous visual embedding . For example, in the above circuit problem, concepts such as "bulb", "switch", and "closed" are shared by text and vision.

[0119] Through independent text heterogeneous encoders and visual heterogeneous encoders , respectively, and extract information specific to each modality. For example, the question logic of a question in a text, the subtle differences between options are text heterogeneous information; the specific connection method of an element in a circuit diagram, the standard drawing of a symbol is visual heterogeneous information.

[0120] Through the optimization of the carefully designed loss function (including cosine similarity loss and reconstruction loss , , the homogeneous information and the heterogeneous information are effectively separated, and the decoupled information can still well represent the original information.

[0121] The first fusion module is configured to fuse the homogeneous text embedding and the heterogeneous text embedding to obtain a text modality representation, and fuse the homogeneous visual embedding and the heterogeneous visual embedding to obtain a visual modality representation.

[0122] Specifically, within each modality, the homogeneous embedding and the heterogeneous embedding are fused (by a fusion strategy ) to obtain an enhanced text modality representation and a visual modality representation . This fusion makes the representation of each modality contain both core shared semantics and its uniqueness.

[0123] The second fusion module is configured to fuse the text modality representation and the visual modality representation to obtain a preliminary multi-modal embedding.

[0124] Specifically, the enhanced text modality representation and the visual modality representation are fused across modalities (also by a fusion strategy ) to obtain a preliminary multi-modal embedding . This embedding is the basis for subsequent attribute signal decoupling.

[0125] In this embodiment, the first fusion module and the second fusion module are responsible for effectively integrating the decoupled different representations.

[0126] The second decoupling module is configured to perform attribute signal decoupling representation learning on the preliminary multi-modal embedding based on a predefined attribute signal.

[0127] Specifically, the second decoupling module is an attribute signal decoupling module, which is the second core innovation of the present application, and aims to solve the problem of knowledge entanglement in different disciplines (attribute signals).

[0128] The module receives the preliminary multi-modal embedding .

[0129] based on a predefined attribute signal (e.g. natural science, social science, linguistics, or more specific physics, chemistry, biology, etc.), using a contrastive learning mechanism . The goal is to make embeddings of questions from the same subject area closer in the feature space, while embeddings of questions from different subject areas are further apart. This is achieved by minimizing a decoupling loss function based on the attribute signal. For example, the embedding of a physics circuit problem would be more similar to other physics problems, but different from a history problem.

[0130] a final representation fusion module that fuses the embeddings learned by the attribute signal decoupling with the embeddings of the original information to obtain a final comprehensive representation;

[0131] Specifically, to utilize both the deeply decoupled features and preserve some original, untransformed information for final prediction, this module performs a last fusion.

[0132] First, the initial text embeddings and visual embeddings are directly fused to obtain a supplementary embedding .

[0133] Then, the embeddings learned by the attribute signal decoupling are fused with this supplementary embedding (by a fusion strategy ), to obtain a final comprehensive representation . This representation is considered to contain the most informative and effective features for answer prediction.

[0134] an answer prediction module that predicts the answer of the scientific question based on the final comprehensive representation, by a decoder.

[0135] Specifically, this module receives the final comprehensive representation and utilizes a Transformer decoder to generate or select a final answer . For multiple-choice questions, the decoder can output scores for each option, and select the option with the highest score as the answer; for open-ended questions, the decoder generates a piece of text as the answer.

[0136] This embodiment also includes a model training and optimization module:

[0137] In the training phase, the entire system is optimized end-to-end by a unified total loss function . This loss function combines the accuracy of answer prediction (by a cross-entropy loss function measures), the effect of homophily-heterophily decoupling (through the homophily-heterophily decoupling total loss function measures), and the effect of attribute signal decoupling (through the attribute signal-based decoupling loss function measures). Hyperparameters and (experimentally set to 0.1) are used to balance the contribution weights of different loss terms. All learnable parameters in the model are constantly adjusted by the backpropagation algorithm and gradient descent optimizer (e.g., AdamW) to minimize the total loss.

[0138] The system can be deployed on a server and provide scientific Q&A services for various educational applications (such as online learning platforms, intelligent tutoring systems) through API interfaces, helping students understand scientific knowledge and answer questions. It can also be integrated into research assistance tools to help researchers quickly obtain information from literature and experimental data.

[0139] The embodiment solves the problem of entanglement of multi-modal information and attribute signals, thereby improving the accuracy of scientific Q&A.

[0140] Embodiment Three

[0141] The embodiment also discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method in the embodiment one are implemented.

[0142] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application can be easily thought of by those skilled in the art, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for scientific question answering with heterogeneous attribute signal decoupled representation learning, characterized in that, The method comprises the following steps: obtaining text input and visual input of a scientific question; text encoding is performed on the text input to obtain an initial text embedding, and visual encoding is performed on the visual input to obtain an initial visual embedding; the initial text embedding is decoupled into homogeneous text embedding and heterogeneous text embedding by a first decoupling module, and the initial visual embedding is decoupled into homogeneous visual embedding and heterogeneous visual embedding; the homogeneous text embedding and the heterogeneous text embedding are fused to obtain a text modal representation; the homogeneous visual embedding and the heterogeneous visual embedding are fused to obtain a visual modal representation; the text modal representation and the visual modal representation are fused to obtain a preliminary multi-modal embedding; the preliminary multi-modal embedding is subjected to attribute signal decoupling representation learning based on a predefined attribute signal by a second decoupling module; the embedding subjected to attribute signal decoupling learning and the embedding of the original information are fused to obtain a final comprehensive representation; an answer to the scientific question is predicted by a decoder based on the final comprehensive representation.

2. The method of claim 1, wherein, The text input comprises question text, context text, multiple-choice option text and thought chain text.

3. The method of claim 1, wherein, The text encoding adopts a Transformer model, and the visual encoding adopts one of a DETR model, a ResNet model, a CLIP model or a ViT model or a combination thereof.

4. The method of claim 1, wherein, The first decoupling module comprises a shared homogeneous encoder and independent heterogeneous encoders for text and vision respectively, and utilizes a self-encoder for reconstruction learning.

5. The method of claim 1, wherein, In the first decoupling module, a cosine similarity-based loss function is used to distinguish homogeneous embedding and heterogeneous embedding within each modality, and a reconstruction loss function is introduced.

6. The method of claim 1, wherein, The attribute signal comprises natural science, social science and language science.

7. The method of claim 1, wherein, The second decoupling module adopts a contrast learning mechanism, regards sample pairs belonging to the same attribute signal as positive sample pairs, regards sample pairs belonging to different attribute signals as negative sample pairs, and narrows the distance of the positive sample pairs and widens the distance of the negative sample pairs based on the decoupling loss of the attribute signal.

8. The method of claim 1, wherein, The fusion of the homogeneous text embedding and the heterogeneous text embedding, the fusion of the homogeneous visual embedding and the heterogeneous visual embedding, the fusion of the text modal representation and the visual modal representation, and the fusion of the embedding subjected to attribute signal decoupling learning and the embedding of the original information all adopt a fusion strategy based on a multi-head attention mechanism.

9. A heterogeneous and attribute signal decoupled representation learning scientific question answering system, characterized in that, The method comprises the following steps: an input interface is configured to obtain text input and visual input of a scientific question; an encoding module is configured to perform text encoding on the text input to obtain an initial text embedding, and perform visual encoding on the visual input to obtain an initial visual embedding; a first decoupling module is configured to decouple the initial text embedding into homogeneous text embedding and heterogeneous text embedding, and decouple the initial visual embedding into homogeneous visual embedding and heterogeneous visual embedding; a first fusion module is configured to fuse the homogeneous text embedding and the heterogeneous text embedding to obtain a text modal representation; and fuse the homogeneous visual embedding and the heterogeneous visual embedding to obtain a visual modal representation; a second fusion module is configured to fuse the text modal representation and the visual modal representation to obtain a preliminary multi-modal embedding; a second decoupling module configured to perform attribute signal decoupled representation learning on the preliminary multi-modal embedding based on a predefined attribute signal; a final representation fusion module configured to fuse the attribute signal decoupled embedding with an embedding of the original information to obtain a final comprehensive representation; an answer prediction module configured to predict an answer to the scientific question based on the final comprehensive representation via a decoder.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method of any one of claims 1-8.

Images