Multi-modal alignment evidence selection method based on heterogeneous evidence graph

Abstract

The invention discloses a multi-modal alignment evidence selection method based on a heterogeneous evidence graph, which comprises the following steps of: constructing a multi-modal alignment evidence selector model based on the heterogeneous evidence graph, firstly utilizing an alignment-based modal integration evidence selector, combining a cross-modal retrieval and alignment strategy, and jointly modeling table lines, table headers, text paragraphs and sentences to obtain a multi-modal alignment evidence selector model; the method comprises the following steps of: constructing a heterogeneous evidence graph by using a personnel domain knowledge graph and a coarse and fine granularity relationship of the evidence in a re-ordering module based on a heterogeneous graph attention network, and ordering the evidence based on the heterogeneous graph attention network to obtain a final screening evidence; the final screening evidence is obtained by performing joint training with a question-header alignment task and a question-paragraph abstract task; and generating an optimized reordering evidence. According to the method, the problems of asymmetry of modality and granularity, inconsistency of semantics, insufficient evidence relationship modeling, data missing and the like of heterogeneous personnel data in question and answer retrieval are solved, and an efficient and accurate evidence selection method is provided for a personnel intelligent question and answer system.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a multimodal alignment evidence selection method based on heterogeneous evidence graphs. Background Art

[0002] Enterprises often need to extract key information from heterogeneous data sources to support decision-making. These problems involve the combined reasoning of tables and text, known as hybrid table-text question answering. During the retrieval process, the answer to a question may come from cross-modal evidence sources, including the header information for a row or column in a personnel-related table, or the paragraph text associated with a table cell.

[0003] By studying relevant methods, we found that there are still some problems in the existing evidence retrieval methods for heterogeneous personnel data. (1) Asymmetry of heterogeneous personnel data in modality and granularity. Heterogeneous personnel data consists of tables and texts, and data of different modalities have significant differences in semantic complexity. If the text processing method is directly applied to table data, the inherent structural information of the table will be lost. In addition, data of different modalities also have differences in granularity. (2) Modeling the semantic consistency between personnel-related evidence and questions. In the personnel multi-hop question-answering task, the question often does not directly contain the table header name of the relevant table or the title of the text, making it difficult for the model to directly match key evidence. In addition, personnel text paragraphs usually lack titles and have long content. If similarity matching is performed directly, the performance is poor. (3) Relationship modeling of heterogeneous personnel evidence. When modeling the relationship between table rows, table headers and text paragraphs and sentences, the current retrieval model usually regards all evidence as isomorphic text modality nodes and only establishes evidence associations through simple entity links. This method makes it difficult to capture the structural relationship and semantic association between tables and texts, resulting in insufficient heterogeneous information fusion capabilities and affecting the reasoning quality. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a multimodal alignment evidence selection method based on heterogeneous evidence graphs, providing an efficient and accurate evidence selection method for the human resources intelligent question answering system.

[0005] The technical solution adopted by the present invention is: a multimodal alignment evidence selection method based on heterogeneous evidence graph, the specific steps are as follows:

[0006] S1. Construct a multimodal alignment evidence selector model based on heterogeneous evidence graph;

[0007] The selector model includes: a modality integration retrieval framework based on joint representation learning, a heterogeneous graph construction module based on personnel knowledge graph, and a reordering module based on heterogeneous graph attention network.

[0008] The modality integration retrieval framework includes an alignment-based modality integration evidence selector and a question-evidence alignment subtask module. The evidence includes headers and paragraph summaries, which are generated from the original paragraphs using the pre-trained language model T5.

[0009] The selector model adopts multi-task joint learning to uniformly train the modality integration evidence selector, question-evidence alignment subtask and re-ranking module based on heterogeneous graph attention network.

[0010] S2. Based on the selector model constructed in step S1, the table and text data in the retrieval library, as well as the question, are input into the modality integration retrieval framework. The alignment-based modality integration evidence selector is used in combination with cross-modal retrieval and alignment strategies to jointly model table rows, table headers, text paragraphs, and sentences to obtain preliminary screening evidence. The model is then jointly trained with the question-header alignment task and the question-paragraph summarization task to optimize semantic consistency.

[0011] S21. Segment the table and text data in the search database to obtain table row evidence, table header evidence, paragraph evidence, and sentence evidence, jointly model them, and perform preliminary screening and evidence selection based on the unified representation of evidence and the input question;

[0012] S22. Jointly semantically align questions with table headers and paragraph summaries to improve the semantic understanding ability of the modality integration evidence selector.

[0013] S3. Based on step S2, after obtaining the preliminary screening evidence, input it into the heterogeneous graph construction module based on the personnel knowledge graph, use the personnel domain knowledge graph and the coarse-grained and fine-grained relationship of the evidence to construct a heterogeneous evidence graph, and then use the reordering module based on the heterogeneous graph attention network to sort the evidence, and finally generate the optimized reordered evidence as the final evidence output.

[0014] Furthermore, the step S21 is specifically as follows:

[0015] S211, independently represent heterogeneous personnel data;

[0016] Define an evidence set C, including four types of heterogeneous evidence, namely, table row evidence C r 、Header Evidence C h 、Paragraph Evidence C p and sentence evidence C s .

[0017] S212, Encoding cross-modal evidence and questions;

[0018] Based on the evidence definition in step S211, the table and text data in the search database are segmented, and a joint probability modeling method is used to represent the relationship between question Q and table T, paragraph P, and sentence S. The joint probability of modality integration evidence is expressed as follows:

[0019] P e (C|Q,T,P,S)=P e (C r |Q,T)·P e (C h |Q,T)·P e (C p ∣∣Q,P)·P e (C s ∣Q,S)

[0020] Among them, P e represents the probability of evidence selection, Q represents a question, T represents a table, P represents a paragraph, and S represents a sentence.

[0021] For each question Q, modal type modal, specific type type and candidate evidence C, combine them into the following format:

[0022] A=[CLS]modal[SEP]type[SEP]Q

[0023] Among them, A represents the input form of the question; [CLS] represents the first word vector symbol in BERT; [SEP] represents the separator symbol in BERT; the modal type parameter modal includes table t1 and text t2; the specific type type of table t1 includes table row r and table header h; the specific type type of text t2 includes paragraph p and sentence s.

[0024] For each row r in the table i , and all cells c ij Paragraphs that may be relevant ij Concatenate representations and construct an embedding representation C of row evidence hr , integrating table structure information and related text content, the expression is as follows:

[0025] C hr =[Concat(c i1 ,c i2 ,…,c i|h| );Concat(p i1 ,p i2 ,…,p i|h| )]

[0026] Among them, |h| represents the number of columns in the table; cell c ij and paragraph p ijThe embedding of is obtained through a separate encoder.

[0027] Header evidence directly uses the header h j The embedding representation C hh , the expression is as follows:

[0028] C hh =BERT(h j )

[0029] For each paragraph p k , represent the sentences in the paragraph as s kj Perform average pooling to obtain the global embedding representation C of the paragraph hp , the expression is as follows:

[0030]

[0031] Among them, M k Indicates paragraph p k The total number of sentences in .

[0032] For each sentence evidence s f , directly use the original embedding representation C of each sentence hs , the expression is as follows:

[0033] C hs =BERT(s f )

[0034] Then the sequence A is input into the BERT encoder of the pre-trained language model to obtain the embedding representation H A , which is embedded with the evidence to represent C hn Connect and compress it using attention pooling, denoted as h A , the process expression is as follows:

[0035] H A =BERT(A)

[0036] H AC =Concat(H A ,C hn )

[0037] h A =Attention-Pooling(H)

[0038] Concat represents the connection function, n represents the type of evidence, which are r, h, p, s, H AC A connected representation of the question and the evidence.

[0039] The weight of the attention pool is calculated by the relevance score of each token, and the expression is as follows:

[0040]

[0041] Among them, w i represents the attention weight, u i represents the i-th output vector of BERT, represents transpose; v Q represents the global vector representation of question Q; L represents the sequence length.

[0042] S213, modeling evidence selection task;

[0043] Based on step S212, the table row evidence C r 、Header Evidence C h 、Paragraph Evidence C p and sentence evidence C s Each piece is processed independently and its vectorized representation is concatenated. Then, combined with the modal information, evidence type, and question vector, it is input into the pre-trained language model BERT encoder for binary classification prediction, screening out the most relevant evidence, i.e., the primary screening evidence.

[0044] The coded representation h obtained in step S212 A , use binary classification to predict whether a given candidate evidence C can support the answer generation of question Q, and the classification probability P of evidence selection s The expression is as follows:

[0045] P s (C|Q)=σ(W·h A +b)

[0046] Among them, W and b represent learnable parameters, and σ represents the Sigmoid function.

[0047] Then the matching score P of the paragraph is s (C|Q) Set threshold S p When the score of the paragraph where the sentence is located is less than the threshold, P s The value of (C|Q) is set to 0. That is, for the i-th paragraph When s j ∈P i And P s (C|Q) p When P s Set to 0.

[0048] Among them, SL represents the number of sentences in the paragraph, s i Represents each sentence in a paragraph.

[0049] Finally, the probability of evidence classification P s Sort from high to low and select the top E with the highest score initial ​A set of evidence is used as preliminary screening evidence.

[0050] Among them, E initial The value is set according to the actual situation.

[0051] Focal Loss is used as the loss function during training. The loss function expression for evidence selection is as follows:

[0052]

[0053] Where N is the total number of samples; p i =P(C i |Q) represents the i-th sample C i The classification probability of y i represents the binary classification label; α and γ represent the adjustment parameters.

[0054] Furthermore, the step S22 is specifically as follows:

[0055] S221, independent coding of questions and evidence;

[0056] In the heterogeneous personnel data question answering task, given a length of Q L Natural language problems First, use the existing BM25 search method to preliminarily filter the headers and paragraphs to obtain the H L A collection of headers and P L A collection of paragraphs The generative summary method based on the T5 model is used to extract the main semantic information of paragraph p, and the length is limited to p according to the actual situation. count Get paragraph summary collection within words

[0057] The question, table header, and paragraph summary are input into the BERT encoder of the pre-trained language model respectively to obtain their hidden representations, which are expressed as follows:

[0058] h q =BERT(Q)

[0059]

[0060] Among them, h q Represents the global vector representation of the problem Q; h H and h DP Represent the embedding representations of the table header and paragraph summary respectively.

[0061] S222. Calculate the similarity score between the question and the evidence;

[0062] For the question, header and paragraph summary codes h obtained in step S221 q 、h H and h DP , calculate the similarity score S respectively h (h j ,Q) and S dp (dp k ,Q).

[0063] For each header h i , calculate its semantic similarity score with question Q, the expression is as follows:

[0064]

[0065] Among them, W h Represents a learnable weight matrix; ReLU(·) represents a nonlinear activation function; ⊕ represents a vector concatenation operation, Indicates the header h i Embedded.

[0066] For each paragraph summary dp k , calculate its similarity score with question Q, the expression is as follows:

[0067]

[0068] Among them, W dp A learnable weight matrix representing the paragraph summary alignment; Indicates paragraph summary dp k Embedding.

[0069] S223, training for the task of aligning question headers and question paragraph summaries;

[0070] For the similarity results obtained in step S222, a multi-task learning approach is used to model the alignment task into two subtasks, corresponding to header alignment and paragraph summary alignment. The final alignment loss function is defined as follows:

[0071]

[0072] Among them, the Focal loss function is used to deal with the problem of unbalanced sample distribution; and Respectively represent the model prediction of the table header and paragraph summary selection probability; y h and y dp denote the true labels of the header and paragraph summary respectively; h and λ dp represents the weight hyperparameter.

[0073] Furthermore, the step S3 is specifically as follows:

[0074] S31. Coarse-grained mapping based on knowledge graph in personnel field;

[0075] The output of step S2 is E initial Screening evidence set D initial As the input of heterogeneous evidence graph construction, D initial All candidate evidences in construct heterogeneous evidence graph G evid =(V,E).

[0076] Where V represents the node set and E represents the edge set. The node set V is divided into four categories according to the modality and granularity, namely: table row node set V r , cell node set V c , paragraph node set V p and sentence node set V s The edge set E includes two main types of relations: (1) relations E guided by the knowledge graph in the personnel field k , (2) Internal relations E of evidence granularity refinement m .

[0077] First, the table row node V r and paragraph node V p Mapping to the entity space ε in the knowledge graph KG , through the mapping function f:V→ε KG , mapping the nodes in the graph to entities in the knowledge graph. i ,v j ∈V is mapped to the entity entity i ,entity j ∈ε KG Heterogeneous evidence graph of coarse-grained mapping in the presence of connections re Establish corresponding relationship edges e between nodes ij ∈E k , the expression is as follows:

[0078]

[0079] S32, fine-grained expansion based on evidence granularity;

[0080] Based on the coarse-grained heterogeneous evidence graph obtained in step S31 For each connected table row node GR in the graph i ∈V r and its corresponding cell node c j ∈V c , the structure of the graph is refined through the natural mapping relationship between table row nodes and cell nodes. The mapping relationship is achieved through the edge E m To represent, each edge eij ∈E m Indicates the containment relationship between table rows and cells. Similarly, for each connected paragraph node GP in the graph i ∈V p Each sentence node s in k ∈V s , the relationship between sentence and paragraph nodes is also through E m Edge to connect.

[0081] S33, secondary mapping based on knowledge graph in personnel field;

[0082] After completing the coarse-grained and fine-grained relationship mapping in steps S31 and S32, all nodes in the heterogeneous evidence graph are traversed and the knowledge graph is used again. Calculate whether there is a valid mapping relationship between them.

[0083] Among them, when two nodes v i and v j The number of relationships between corresponding entities |E ij |>S e , no further mapping and connection is performed on these two nodes.

[0084] S34, Evidence ranking based on heterogeneous graph attention network;

[0085] Construct the evidence graph G through steps S31, S32 and S33 evid For heterogeneous graphs, a heterogeneous graph attention network is used to process multiple node types and edge types. For each node type and edge type, the heterogeneous graph attention network HGAN learns independent linear transformations and attention mechanisms respectively, and defines an independent linear transformation matrix W for the node type t∈{r,c,p,s} t , for edge type e∈{E k ,E m}Define different attention mechanism parameters.

[0086] Build L g The HGAN of the layer is embedded by an independent BERT encoder to obtain the representation of the 0th layer, and each of the remaining layers updates the node representation through linear transformation, attention mechanism and weighted aggregation.

[0087] First, perform a linear transformation, for each node v i , according to its type t, apply the corresponding linear transformation to obtain the node representation of the lth layer The expression is as follows:

[0088]

[0089] Among them, each node v iThe type is t∈{r,c,p,s}, then

[0090] Then use the attention mechanism to calculate the node v for each edge type e i and its neighbors v j The attention weight between The expression is as follows:

[0091]

[0092] Among them, a e represents the attention parameter associated with edge type e, represents transpose, ‖ represents vector concatenation, and d represents the dimension of node representation.

[0093] Finally, according to the attention weight, the information of the neighbor nodes is weighted and aggregated to update The expression is as follows:

[0094]

[0095] in, Indicates that it is connected to node v through edge type e i The set of neighbor nodes.

[0096] Passing L g After the heterogeneous graph attention network layer, each node v i The final representation embedding will be obtained

[0097] After completing node representation learning, the final evidence retrieved is at the table row level and paragraph level. The reordering scores of table rows and paragraphs are calculated respectively by using the representation of the evidence itself and the representation of the finer-grained cells and sentences associated with it through the fully connected layer.

[0098] For each table row node v r ∈V r , its reordering score s r It is determined by its own representation and the representation of the cell node associated with it, and the expression is as follows:

[0099]

[0100] Among them, h q Representation embedding for representation problems; and Represent the final representation of the table row node and its adjacent cell nodes respectively; By edge type E m Connect to node v r The set of neighbor nodes.

[0101] For each paragraph node v p ∈V p , its reordering score s p It is also determined by its own representation and the representation of the sentence node associated with it. The expression is as follows:

[0102]

[0103] in, and Represent the final representation of the paragraph node and its adjacent sentence nodes respectively; By edge type E m Connect to node v p The set of neighbor nodes of .

[0104] Sort the scores of table rows and paragraphs uniformly, starting from E initial Select the top E with the highest score among the initial screening evidences enhance The evidence is used as the final screening result of the retrieval stage.

[0105] Among them, E enhanced The value is set according to the actual situation.

[0106] S35, retrieval phase training;

[0107] Define the loss function Optimize the reordering module based on the heterogeneous graph attention network. The expression is as follows:

[0108]

[0109] Among them, y v Represents the binary classification label of the evidence node v. If node v is the gold standard evidence containing the answer, y v =1, otherwise y v =0; represents all candidate evidence nodes in the primary set; s v represents the prediction score of the candidate evidence node v generated by the heterogeneous graph attention network.

[0110] Combined with step S2, the final retrieval stage includes the joint training of three subtasks, namely the evidence selection loss, alignment loss, and re-ranking loss of the modality-integrated evidence selector, which are expressed as follows:

[0111]

[0112] Among them, λ e ,λ a and λ rThey represent the weight coefficients of the evidence selection loss term, alignment loss term, and re-ranking loss term, respectively.

[0113] Beneficial effects of the present invention: The method of the present invention constructs a multimodal alignment evidence selector model based on a heterogeneous evidence graph, and efficiently retrieves relevant evidence from a candidate evidence library through a two-stage screening mechanism. In the initial stage, an alignment-based modality integration evidence selector is used, combined with cross-modal retrieval and alignment strategies, to jointly model table rows, headers, text paragraphs and sentences, and jointly train with question-header alignment tasks and question-paragraph summary tasks to optimize semantic consistency and obtain preliminary screening evidence. Then, in a re-ranking module based on a heterogeneous graph attention network, a knowledge graph in the personnel field and the coarse- and fine-grained relationships of evidence are used to construct a heterogeneous evidence graph, and evidence is ranked based on the heterogeneous graph attention network, ultimately generating optimized re-ranked evidence. The method of the present invention effectively solves the problems of modality and granularity asymmetry, semantic inconsistency, insufficient evidence relationship modeling, and data missing in heterogeneous personnel data in question-answering retrieval, and provides an efficient and accurate evidence selection method for personnel intelligent question-answering systems, which has broad application value in the fields of intelligent personnel management, automated knowledge question-answering, and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] Figure 1 This is a flowchart of a multimodal alignment evidence selection method based on heterogeneous evidence graphs of the present invention.

[0115] Figure 2 This is a diagram of the architecture of a multimodal alignment evidence selector model based on heterogeneous evidence graphs in an embodiment of the present invention.

[0116] Figure 3 Schematic diagram of the modality integration retrieval framework structure and processing flow in an embodiment of the present invention.

[0117] Figure 4 This is a schematic diagram of the process of aligning questions with table headers and paragraph summaries in an embodiment of the present invention.

[0118] Figure 5 A schematic diagram of constructing a heterogeneous evidence graph in an embodiment of the present invention.

[0119] Figure 6 Schematic diagram of the heterogeneous graph attention network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0120] The method of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0121] like Figure 1 As shown in FIG, a flowchart of a multimodal alignment evidence selection method based on a heterogeneous evidence graph of the present invention is shown, and the specific steps are as follows:

[0122] S1. Construct a multimodal alignment evidence selector model based on heterogeneous evidence graph;

[0123] like Figure 2 As shown, the selector model includes: a modality integration retrieval framework based on joint representation learning, a heterogeneous graph construction module based on personnel knowledge graph, and a reordering module based on heterogeneous graph attention network. Figure 2 Here, E1 represents the evidence with the highest score after sorting, E2 represents the second highest, and so on.

[0124] The modality integration retrieval framework includes an alignment-based modality integration evidence selector and a question-evidence alignment subtask module. The evidence includes headers and paragraph summaries, which are generated from the original paragraphs using the pre-trained language model T5.

[0125] In order to improve the performance of the model, multi-task joint learning is adopted to uniformly train the modality integration evidence selector, question-evidence alignment subtask and the reranking module based on heterogeneous graph attention network.

[0126] S2. Based on the selector model constructed in step S1, the table and text data in the retrieval library, as well as the question, are input into the modality integration retrieval framework. The alignment-based modality integration evidence selector is used in combination with cross-modal retrieval and alignment strategies to jointly model table rows, table headers, text paragraphs, and sentences to obtain preliminary screening evidence. The model is then jointly trained with the question-header alignment task and the question-paragraph summarization task to optimize semantic consistency.

[0127] S21. Segment the table and text data in the search database to obtain table row evidence, table header evidence, paragraph evidence, and sentence evidence, jointly model them, and perform preliminary screening and evidence selection based on the unified representation of evidence and the input question;

[0128] S22. Jointly semantically align questions with table headers and paragraph summaries to improve the semantic understanding ability of the modality integration evidence selector.

[0129] S3. Based on step S2, after obtaining the preliminary screening evidence, input it into the heterogeneous graph construction module based on the personnel knowledge graph, use the personnel domain knowledge graph and the coarse-grained and fine-grained relationship of the evidence to construct a heterogeneous evidence graph, and then use the reordering module based on the heterogeneous graph attention network to sort the evidence, and finally generate the optimized reordered evidence as the final evidence output.

[0130] In this embodiment, step S21 is specifically as follows:

[0131] S211, independently represent heterogeneous personnel data;

[0132] In order to comprehensively and accurately represent the cross-modal evidence in heterogeneous personnel data, an evidence set C is defined, which includes four types of heterogeneous evidence, namely, table row evidence C r 、Header Evidence C h 、Paragraph Evidence C p and sentence evidence C s , providing support from different modalities and granularities to enhance the model's understanding and reasoning ability of question Q.

[0133] Table row evidence refers to a complete row in a personnel table, typically containing multiple data points relevant to question Q. Header evidence corresponds to the column headings or labels of the personnel table, identifying the attributes or field information of the table column. Paragraph evidence is a coarse-grained form of textual information that may be directly linked to a table cell via a hyperlink or may simply provide semantic contextual support. Sentence evidence is a fine-grained form of textual information that is more semantically precise and localized.

[0134] For a heterogeneous personnel problem involving reasoning across tables and paragraphs, the table row evidence C r Provides the basis for structured data, while paragraph evidence C p By providing context, the coherence of the reasoning is further enhanced. s The fine-grained representation of can significantly improve the accuracy of answer generation.

[0135] S212, Encoding cross-modal evidence and questions;

[0136] Based on the evidence definition in step S211, in order to fully integrate evidence of different modalities and granularities, the table and text data in the retrieval database are segmented, and a joint probability modeling method is used to represent the relationship between question Q and table T, paragraph P, and sentence S. The joint probability of modality-integrated evidence is expressed as follows:

[0137] P e (C|Q,T,P,S)=P e (C r |Q,T)·P e (C h |Q,T)·P e (C p ∣∣Q,P)·P e (C s ∣Q,S)

[0138] Among them, P e represents the probability of evidence selection, Q represents a question, T represents a table, P represents a paragraph, and S represents a sentence.

[0139] The core task of the modal integration retrieval framework is to efficiently select evidence in cross-modal data. Therefore, the modal integration retrieval framework described in this embodiment is based on joint representation learning, which can simultaneously process tabular evidence (including table rows and headers) and text evidence (including paragraphs and fine-grained sentences), and realize consistent modeling of multimodal data through shared representation.

[0140] For each question Q, modal type modal, specific type type and candidate evidence C, combine them into the following format:

[0141] A=[CLS]modal[SEP]type[SEP]Q

[0142] Among them, A represents the input form of the question; [CLS] represents the first word vector symbol in BERT; [SEP] represents the separator symbol in BERT; the modal type parameter modal includes table t1 and text t2; the specific type type of table t1 includes table row r and table header h; the specific type type of text t2 includes paragraph p and sentence s.

[0143] Since the modalities and granularities of candidate evidence C are different, different types of evidence require different processing methods. Therefore, question Q and candidate evidence C are encoded separately and spliced after encoding.

[0144] In order to cover the different granularity of information in tables and texts, the following definitions and representation structures are given for the four modal evidences:

[0145] For each row r in the table i , and all cells c ij Paragraphs that may be relevant ij Concatenate representations and construct an embedding representation C of row evidence hr , integrating table structure information and related text content, the expression is as follows:

[0146] C hr =[Concat(c i1 ,c i2 ,…,c i|h| );Concat(p i1 ,p i2 ,…,p i|h| )]

[0147] Among them, |h| represents the number of columns in the table; cell c ij and paragraph p ij The embedding of is obtained through a separate encoder.

[0148] Header evidence directly uses the header h j The embedding representation C hh, the expression is as follows:

[0149] C hh =BERT(h j )

[0150] For each paragraph p k , represent the sentences in the paragraph as s kj Perform average pooling to obtain the global embedding representation C of the paragraph hp , the expression is as follows:

[0151]

[0152] Among them, M k Indicates paragraph p k The total number of sentences in .

[0153] For each sentence evidence s f , directly use the original embedding representation C of each sentence hs , the expression is as follows:

[0154] C hs =BERT(s f )

[0155] Then the sequence A is input into the BERT encoder of the pre-trained language model to obtain the embedding representation H A , which is embedded with the evidence to represent C hn Connect and compress it using attention pooling, denoted as h A , the process expression is as follows:

[0156] H A =BERT(A)

[0157] H AC =Concat(H A ,C hn )

[0158] h A =Attention-Pooling(H)

[0159] Concat represents the connection function, n represents the type of evidence, which are r, h, p, s, H AC A connected representation of the question and the evidence.

[0160] The weight of the attention pool is calculated by the relevance score of each token, and the expression is as follows:

[0161]

[0162] Among them, w i represents the attention weight, ui represents the i-th output vector of BERT, represents transpose; v Q represents the global vector representation of question Q; L represents the sequence length.

[0163] S213, modeling evidence selection task;

[0164] The modality integration retrieval framework structure and processing flow are as follows: Figure 3 As shown, in heterogeneous personnel data, evidence of different modalities has different expression forms and granularity in the question-answering task. Based on step S212, in this embodiment, for the table modality, the table header (such as "department") is used as the table header evidence, and the row where He Lin is located is used as the table row evidence. For the text modality, the entire text is regarded as paragraph evidence, and each sentence in it is regarded as sentence evidence, that is, for the table row evidence C r 、Header Evidence C h 、Paragraph Evidence C p and sentence evidence C s Each piece is processed independently and its vectorized representation is concatenated. Then, combined with the modal information, evidence type, and question vector, it is input into the pre-trained language model BERT encoder for binary classification prediction, screening out the most relevant evidence, i.e., the primary screening evidence.

[0165] The coded representation h obtained in step S212 A , use binary classification to predict whether a given candidate evidence C can support the answer generation of question Q, and the classification probability P of evidence selection s The expression is as follows:

[0166] P s (C|Q)=σ(W·h A +b)

[0167] Where W and b represent learnable parameters, and σ represents the Sigmoid function, which is used to normalize the score.

[0168] Because there is a possibility that the sentence is a pseudo-label, such as some sentences include the key words of the answer, but are not in the correct reasoning chain, the matching score P of the paragraph is s (C|Q) Set threshold S p When the score of the paragraph where the sentence is located is less than the threshold, P s The value of (C|Q) is set to 0. That is, for the i-th paragraph When s j ∈P i And P s (C|Q) p When P s Set it to 0 to avoid the impact of incorrect positioning on the subsequent reasoning process. ​

[0169] Among them, SL represents the number of sentences in the paragraph, s i Represents each sentence in a paragraph.

[0170] Finally, the probability of evidence classification P s Sort from high to low and select the top E with the highest score initial A set of evidence is used as preliminary screening evidence.

[0171] Among them, E initial The value is set according to the actual situation. After experiments, in this embodiment, E initial The optimal value is 100.

[0172] Due to the uneven distribution of positive and negative samples in heterogeneous personnel data retrieval tasks, the model can easily identify answers that appear only once as golden evidence, but has difficulty accurately locating answers that appear multiple times. Therefore, Focal Loss is used as the loss function during training. The loss function for evidence selection is expressed as follows:

[0173]

[0174] Where N is the total number of samples; p i =P(C i |Q) represents the i-th sample C i The classification probability of y i represents the binary classification label; α and γ represent adjustment parameters used to reduce the focus on easy-to-classify samples.

[0175] In this embodiment, step S22 is specifically as follows:

[0176] In question-answering tasks involving heterogeneous personnel data, in addition to modal consistency, semantic consistency is also a key factor for efficient evidence selection. To ensure that the model accurately captures the structural relationship between table headers and questions, as well as the semantic relationship between paragraph text and questions, this implementation proposes a framework based on question-evidence alignment and multi-task learning. By filtering and selecting table headers and paragraph summaries, the model's understanding of table structure and text semantics is enhanced, thereby improving retrieval performance in question-answering tasks involving heterogeneous personnel data.

[0177] S221, independent coding of questions and evidence;

[0178] In the heterogeneous personnel data question answering task, given a length of Q L Natural language problems First, use the existing BM25 search method to preliminarily filter the headers and paragraphs to obtain the H L A collection of headers and P LA collection of paragraphs Since the length of the personnel paragraph text may exceed the model input limit, in order to effectively compress the information while retaining the key information, a generative summary method based on the T5 model is used to summarize the paragraph p k Extract the main semantic information. In this embodiment, the length is limited to p count =Within 50 words, get a collection of paragraph summaries To avoid long input sequences.

[0179] like Figure 4 As shown in the figure, in order to achieve aligned modeling of the header and paragraph summary with the question, the question, header, and paragraph summary are respectively input into the BERT encoder of the pre-trained language model to obtain their hidden representations. Inspired by human cognitive behavior, three independent BERT encoders are used to encode the question, header, and paragraph summary respectively. The expression is as follows:

[0180] h q =BERT(Q)

[0181]

[0182] Among them, h q Represents the global vector representation of the problem Q; h H and h DP Represent the embedding representations of the table header and paragraph summary respectively.

[0183] S222. Calculate the similarity score between the question and the evidence;

[0184] For the question, header and paragraph summary codes h obtained in step S221 q 、h H and h DP , calculate the similarity score S respectively h (h j ,Q) and S dp (dp k ,Q).

[0185] For each header h i , calculate its semantic similarity score with question Q, the expression is as follows:

[0186]

[0187] Among them, W h Represents a learnable weight matrix; ReLU(·) represents a nonlinear activation function; ⊕ represents a vector concatenation operation, Indicates the header h i Embedding.

[0188] For each paragraph summary dp k, calculate its similarity score with question Q, the expression is as follows:

[0189]

[0190] Among them, W dp A learnable weight matrix representing the paragraph summary alignment; Indicates paragraph summary dp k Embedding.

[0191] S223, training for the task of aligning question headers and question paragraph summaries;

[0192] For the similarity results obtained in step S222, in order to simultaneously optimize the alignment of the header and question, and the paragraph summary and question, a multi-task learning approach is adopted, modeling the alignment task as two subtasks, corresponding to the header alignment and paragraph summary alignment respectively. The final alignment loss function is defined as follows:

[0193]

[0194] Among them, the Focal loss function is used to deal with the problem of unbalanced sample distribution; and Respectively represent the model prediction of the table header and paragraph summary selection probability; y h and y dp denote the true labels of the header and paragraph summary respectively; h and λ dp represents the weight hyperparameter, which is used to adjust the relative importance of the two subtasks.

[0195] In this embodiment, step S3 is specifically as follows:

[0196] S31. Coarse-grained mapping based on knowledge graph in personnel field;

[0197] The output of step S2 is E intial Screening evidence set D initial As the input of heterogeneous evidence graph construction, D initial All candidate evidences in construct heterogeneous evidence graph G evid =(V,E).

[0198] Where V represents the node set and E represents the edge set. The node set V is divided into four categories according to the modality and granularity, namely: table row node set V r , cell node set V c , paragraph node set V p and sentence node set V s The edge set E includes two main types of relations: (1) relations E guided by the knowledge graph in the personnel field k, represents the cross-node relationship derived from the domain knowledge graph, (2) the internal relationship E with refined evidence granularity m , describes the process of further refinement and refinement from coarse-grained evidence to fine-grained evidence.

[0199] First, the table row node V r and paragraph node V p Mapping to the entity space ε in the knowledge graph KG , through the mapping function f:V→ε KG , mapping the nodes in the graph to entities in the knowledge graph. i ,v j ∈V is mapped to the entity entity i ,entity j ∈ε KG Heterogeneous evidence graph of coarse-grained mapping in the presence of connections re Establish corresponding relationship edges e between nodes ij ∈E k , the expression is as follows:

[0200] .

[0201] However, a single table row or paragraph may correspond to multiple entities. Therefore, coarse-grained mapping based on knowledge graphs is only a preliminary stage and still requires further expansion and graph structure optimization at a finer granularity.

[0202] S32, fine-grained expansion based on evidence granularity;

[0203] Based on the coarse-grained heterogeneous evidence graph obtained in step S31 In the fine-grained expansion phase, the relationship between nodes is further refined, that is, for each connected table row node GR in the graph i ∈V r and its corresponding cell node c j ∈V c , the structure of the graph is refined through the natural mapping relationship between table row nodes and cell nodes. The mapping relationship is achieved through the edge E m To represent, each edge e ij ∈E m Indicates the containment relationship between table rows and cells. Similarly, for each connected paragraph node GP in the graph i ∈V p Each sentence node s in k ∈V s , the relationship between sentence and paragraph nodes is also through E m The edges are connected to further enhance the refinement of the graph structure. Figure 5 shown.

[0204] S33, secondary mapping based on knowledge graph in personnel field;

[0205] After completing the coarse-grained and fine-grained relationship mapping in steps S31 and S32, all nodes in the heterogeneous evidence graph are traversed and the knowledge graph is used again. Calculate whether there is a valid mapping relationship between them.

[0206] In order to avoid the decrease in computational efficiency due to the excessive number of edges in the graph structure, when two nodes v i and v j The number of relationships between corresponding entities |E ij |>S e (That is, when the number of relationships between nodes exceeds the preset threshold S e ), no further mapping and connection is performed on these two nodes.

[0207] Because this filtering process occurs during the global connection phase, connections between table row nodes and paragraphs, which are more weighted for inference, are retained, while connections between more fine-grained cell nodes and sentence nodes may be filtered. This approach reduces redundant information, avoids wasted computational resources due to overly complex graph structures, and ensures efficient representation of heterogeneous evidence graphs.

[0208] S34, Evidence ranking based on heterogeneous graph attention network;

[0209] Construct the evidence graph G through steps S31, S32 and S33 evid For heterogeneous graphs, in order to effectively capture the complex information of different types of nodes and their relationships, a heterogeneous graph attention network is used to process multiple node types and edge types. For each node type and edge type, the heterogeneous graph attention network HGAN learns independent linear transformations and attention mechanisms respectively, and defines an independent linear transformation matrix W for each node type t∈{r,c,p,s} t , for edge type e∈{E k ,E m} Define different attention mechanism parameters to capture the information interaction under different relationship types of heterogeneous personnel data. The heterogeneous graph attention network structure of this embodiment is as follows Figure 6 shown.

[0210] Build L g The HGAN of the layer is embedded by an independent BERT encoder to obtain the representation of the 0th layer, and each of the remaining layers updates the node representation through linear transformation, attention mechanism and weighted aggregation.

[0211] First, perform a linear transformation, for each node v i, according to its type t, apply the corresponding linear transformation to obtain the node representation of the lth layer The expression is as follows:

[0212]

[0213] Among them, each node v i The type is t∈{r,c,p,s}, then

[0214] Then use the attention mechanism to calculate the node v for each edge type e i and its neighbors v j The attention weight between The expression is as follows:

[0215]

[0216] Among them, a e represents the attention parameter associated with edge type e, represents transpose, ‖ represents vector concatenation, and d represents the dimension of node representation.

[0217] Finally, according to the attention weight, the information of the neighbor nodes is weighted and aggregated to update The expression is as follows:

[0218]

[0219] in, Indicates that it is connected to node v through edge type e i The set of neighbor nodes.

[0220] Passing L g After the heterogeneous graph attention network layer, each node v i The final representation embedding will be obtained

[0221] After learning node representations, the final evidence retrieved is at the table row and paragraph level. Using the evidence's own representation, along with its associated finer-grained cell and sentence representations, a fully connected layer is used to calculate reranking scores for the table row and paragraph, respectively. This process allows for a more precise measurement of the relevance of evidence to the query, improving the quality of retrieval results and ensuring the selection of highly relevant evidence.

[0222] For each table row node v r ∈V r , its reordering score s r It is determined by its own representation and the representation of the cell node associated with it, and the expression is as follows:

[0223]

[0224] Among them, h q Representation embedding for representation problems; and Represent the final representation of the table row node and its adjacent cell nodes respectively; By edge type E m Connect to node v r The set of neighbor nodes.

[0225] For each paragraph node v p ∈V p , its reordering score s p It is also determined by its own representation and the representation of the sentence node associated with it. The expression is as follows:

[0226]

[0227] in, and Represent the final representation of the paragraph node and its adjacent sentence nodes respectively; By edge type E m Connect to node v p The set of neighbor nodes of .

[0228] Sort the scores of table rows and paragraphs uniformly, starting from E initial Select the top E with the highest score among the initial screening evidences enhanced The evidence (including table rows and paragraphs) is used as the final screening result of the retrieval stage.

[0229] Among them, E enhanced The value is set according to the actual situation. In this embodiment, when E initial =100, E enhanced A value of 30 works best.

[0230] S35, retrieval phase training;

[0231] In order to optimize the re-ranking module based on the heterogeneous graph attention network, the loss function is defined as follows:

[0232]

[0233] Among them, y v Represents the binary classification label of the evidence node v. If node v is the gold standard evidence containing the answer, y v =1, otherwise y v =0; represents all candidate evidence nodes in the primary set; s v represents the prediction score of the candidate evidence node v generated by the heterogeneous graph attention network.

[0234] Combined with step S2, the final retrieval stage includes the joint training of three subtasks, namely the evidence selection loss, alignment loss, and re-ranking loss of the modality-integrated evidence selector, which are expressed as follows:

[0235]

[0236] Among them, λ e ,λ a and λ r They represent the weight coefficients of the evidence selection loss term, alignment loss term, and re-ranking loss term, respectively.

[0237] In summary, the method of the present invention effectively solves the problems of modality and granularity asymmetry, semantic inconsistency, insufficient evidence relationship modeling, and data missing in heterogeneous personnel data in question-answering retrieval, and provides an efficient and accurate evidence selection method for personnel intelligent question-answering systems. It has broad application value in the fields of intelligent personnel management, automated knowledge question-answering, etc.

[0238] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A multimodal alignment evidence selection method based on heterogeneous evidence graph, the specific steps are as follows: S1. Construct a multimodal alignment evidence selector model based on heterogeneous evidence graph; The selector model includes: A modality integration retrieval framework based on joint representation learning, a heterogeneous graph construction module based on personnel knowledge graph, and a reranking module based on heterogeneous graph attention network; The modality integration retrieval framework includes: an alignment-based modality integration evidence selector and a question-evidence alignment subtask module; the evidence includes a header and a paragraph summary, and a pre-trained language model T5 is used to generate a paragraph summary from the original paragraph; The selector model adopts multi-task joint learning to uniformly train the modality integration evidence selector, question-evidence alignment subtask, and the re-ranking module based on heterogeneous graph attention network; S2. Based on the selector model constructed in step S1, the table and text data in the retrieval library, as well as the question, are input into the modality integration retrieval framework. The alignment-based modality integration evidence selector is used in combination with cross-modal retrieval and alignment strategies to jointly model table rows, table headers, text paragraphs, and sentences to obtain preliminary screening evidence. The model is then jointly trained with the question-header alignment task and the question-paragraph summarization task to optimize semantic consistency. S21. Segment the table and text data in the search database to obtain table row evidence, table header evidence, paragraph evidence, and sentence evidence, jointly model them, and perform preliminary screening and evidence selection based on the unified representation of evidence and the input question; S22. Joint semantic alignment of questions, table headers, and paragraph summaries to improve the semantic understanding capability of the modality integration evidence selector. S3. Based on step S2, after obtaining the preliminary screening evidence, input it into the heterogeneous graph construction module based on the personnel knowledge graph, use the personnel domain knowledge graph and the coarse-grained and fine-grained relationship of the evidence to construct a heterogeneous evidence graph, and then use the reordering module based on the heterogeneous graph attention network to sort the evidence, and finally generate the optimized reordered evidence as the final evidence output.

2. A method for selecting evidence based on a multimodal alignment of heterogeneous evidence graphs according to claim 1, characterized in that: The step S21 is specifically as follows: S211, independently represent heterogeneous personnel data; Define an evidence set C, including four types of heterogeneous evidence, namely, table row evidence C r 、Header Evidence C h 、Paragraph Evidence C p and sentence evidence C s ; S212, Encoding cross-modal evidence and questions; Based on the evidence definition in step S211, the table and text data in the search database are segmented, and a joint probability modeling method is used to represent the relationship between question Q and table T, paragraph P, and sentence S. The joint probability of modal integration evidence is expressed as follows: P e (C∣Q,T,P,S)=P e (C r ∣Q,T)·P e (C h ∣Q,T)·P e (C p ∣∣Q,P)·P e (C s ∣Q,S) Among them, P e represents the probability of evidence selection, Q represents a question, T represents a table, P represents a paragraph, and S represents a sentence. For each question Q, modal type modal, specific type type and candidate evidence C, combine them into the following format: A=[CLS]modal[SEP]type[SEP]Q Where A represents the input form of the question; [CLS] represents the first word vector symbol in BERT; [SEP] represents the separator symbol in BERT; the modal type parameter modal includes table t1 and text t2; the specific type of table t1 includes table row r and table header h; the specific type of text t2 includes paragraph p and sentence s; For each row r in the table i , and all cells c ij Paragraphs that may be relevant ij Concatenate representations and construct an embedding representation C of row evidence hr , integrating table structure information and related text content, the expression is as follows: C hr =[Concat(c i1 ,c i2 ,…,c i|h| );Concat(p i1 ,p i2 ,…,p i|h| )] Among them, |h| represents the number of columns in the table; cell c ij and paragraph p ij The embedding of is obtained through a separate encoder; Header evidence directly uses the header h j The embedding representation C hh , the expression is as follows: C hh =BERT(h j ) For each paragraph p k , represent the sentences in the paragraph as s kj Perform average pooling to obtain the global embedding representation C of the paragraph hp , the expression is as follows: Among them, M k Indicates paragraph p k The total number of sentences in For each sentence evidence s f , directly use the original embedding representation C of each sentence hs , the expression is as follows: C hs =BERT(s f ) Then the sequence A is input into the BERT encoder of the pre-trained language model to obtain the embedding representation H A , which is embedded with the evidence to represent C hn Connect and compress it using attention pooling, denoted as h A , the process expression is as follows: H A =BERT(A) H AC =Concat(H A ,C hn ) h A =Attention-Pooling(H) Concat represents the connection function, n represents the type of evidence, which are r, h, p, s, H AC A representation of the connection between the question and the evidence; The weight of the attention pool is calculated by the relevance score of each token, and the expression is as follows: Among them, w i represents the attention weight, u i represents the i-th output vector of BERT, represents transpose; v Q Represents the global vector representation of question Q; L represents the sequence length; S213, modeling evidence selection task; Based on step S212, the table row evidence C r 、Header Evidence C h 、Paragraph Evidence C p and sentence evidence C s The data are processed independently and their vectorized representations are concatenated. The data are then combined with the modal information, evidence type, and question vector and fed into the pre-trained language model BERT encoder for binary classification prediction to select the most relevant evidence, i.e., the primary screening evidence. The coded representation h obtained in step S212 A , use binary classification to predict whether a given candidate evidence C can support the answer generation of question Q, and the classification probability P of evidence selection s The expression is as follows: P s (C∣Q)=σ(W·h A +b) Where W and b represent learnable parameters, and σ represents the Sigmoid function; Then the matching score P of the paragraph is s (C|Q) Set threshold S p ; When the score of the paragraph where the sentence is located is less than the threshold, P s The value of (C|Q) is set to 0; that is, for the i-th paragraph When s j ∈P i And P s (C|Q) p When P s Set to 0;​ Among them, SL represents the number of sentences in the paragraph, s i Represents each sentence in the paragraph; Finally, the probability of evidence classification P s Sort from high to low and select the top E with the highest score initial A set of evidence as preliminary screening evidence; Among them, E initial The value is set according to the actual situation; Focal Loss is used as the loss function during training. The loss function expression for evidence selection is as follows: Where N is the total number of samples; p i =P(C i |Q) represents the i-th sample C i The classification probability of y i represents the binary classification label; α and γ represent the adjustment parameters.

3. The method for selecting multimodal aligned evidence based on heterogeneous evidence graph according to claim 1, characterized in that: The step S22 is specifically as follows: S221, independent coding of questions and evidence; In the heterogeneous personnel data question answering task, given a length of Q L Natural language problems First, use the existing BM25 search method to preliminarily filter the headers and paragraphs to obtain the H L A collection of headers and P L A collection of paragraphs The generative summarization method based on T5 model is used to summarize the paragraph p k Extract the main semantic information and limit the length to p according to the actual situation. count Get paragraph summary collection within words The question, table header, and paragraph summary are input into the BERT encoder of the pre-trained language model respectively to obtain their hidden representations, which are expressed as follows: h q =BERT(Q) Among them, h q Represents the global vector representation of the problem Q; h H and h DP Represent the embedding representations of the table header and paragraph summary respectively; S222. Calculate the similarity score between the question and the evidence; For the question, header and paragraph summary codes h obtained in step S221 q 、h H and h DP , calculate the similarity score S respectively h (h j ,Q) and S dp (dp k ,Q); For each header h i , calculate its semantic similarity score with question Q, the expression is as follows: Among them, W h Represents a learnable weight matrix; ReLU(·) represents a nonlinear activation function; Represents vector concatenation operation, Indicates the header h i Embedding For each paragraph summary dp k , calculate its similarity score with question Q, the expression is as follows: Among them, W dp A learnable weight matrix representing the paragraph summary alignment; Indicates paragraph summary dp k Embedding S223, training for the task of aligning question headers and question paragraph summaries; For the similarity result obtained in step S222, a multi-task learning approach is used to model the alignment task into two subtasks, corresponding to header alignment and paragraph summary alignment respectively; the final alignment loss function is defined as follows: Among them, the Focal loss function is used to deal with the problem of unbalanced sample distribution; and Respectively represent the model prediction of the table header and paragraph summary selection probability; y h and y dp denote the true labels of the header and paragraph summary respectively; h and λ dp represents the weight hyperparameter.

4. The method for selecting multimodal aligned evidence based on heterogeneous evidence graph according to claim 1, characterized in that: The step S3 is specifically as follows: S31. Coarse-grained mapping based on knowledge graph in personnel field; The output of step S2 is E intial Screening evidence set D intial As the input of heterogeneous evidence graph construction, D intial All candidate evidences in construct heterogeneous evidence graph G evid =(V,E); Among them, V represents the node set, E represents the edge set; the node set V is divided into four categories according to the mode and granularity, namely: table row node set V r , cell node set V c , paragraph node set V p and sentence node set V s ; The edge set E includes two main types of relations: (1) Relationship E guided by the knowledge graph in the personnel field k , (2) Internal relations E of evidence granularity refinement m ; First, the table row node V r and paragraph node V p Mapping to the entity space ε in the knowledge graph KG , through the mapping function f:V→ε KG , mapping the nodes in the graph to entities in the knowledge graph; two nodes v i ,v j ∈V is mapped to the entity entity i ,entity j ∈ε KG Heterogeneous evidence graph of coarse-grained mapping in the presence of connections re Establish corresponding relationship edges e between nodes ij ∈E k , the expression is as follows: S32, fine-grained expansion based on evidence granularity; Based on the coarse-grained heterogeneous evidence graph obtained in step S31 For each connected table row node GR in the graph i ∈V r and its corresponding cell node c j ∈V c , the structure of the graph is refined through the natural mapping relationship between table row nodes and cell nodes; the mapping relationship is achieved through the edge E m To represent, each edge e ij ∈E m Indicates the inclusion relationship between table rows and cells; similarly, for each connected paragraph node GP in the graph i ∈V p Each sentence node s in k ∈V s , the relationship between sentence and paragraph nodes is also through E m Edge to connect; S33, secondary mapping based on knowledge graph in personnel field; After completing the coarse-grained and fine-grained relationship mapping in steps S31 and S32, all nodes in the heterogeneous evidence graph are traversed and the knowledge graph is used again. Calculate whether there is a valid mapping relationship between them; Among them, when two nodes v i and v j The number of relationships between corresponding entities |E ij |>S e , no further mapping and connection will be performed on these two nodes; S34, Evidence ranking based on heterogeneous graph attention network; Construct the evidence graph G through steps S31, S32 and S33 evid For heterogeneous graphs, a heterogeneous graph attention network is used to process multiple node types and edge types. For each node type and edge type, the heterogeneous graph attention network HGAN learns independent linear transformations and attention mechanisms respectively, and defines an independent linear transformation matrix W for the node type t∈{r,c,p,s} t , for edge type e∈{E k ,E m }Define different attention mechanism parameters; Build L g The HGAN of the layer is embedded by an independent BERT encoder to obtain the representation of the layer 0, and each of the remaining layers updates the node representation through linear transformation, attention mechanism and weighted aggregation; First, perform a linear transformation, for each node v i , according to its type t, apply the corresponding linear transformation to obtain the node representation of the lth layer The expression is as follows: Among them, each node v j The type is t∈{r,c,p,s}, then Then use the attention mechanism to calculate the node v for each edge type e i and its neighbors v j The attention weight between The expression is as follows: Among them, a e represents the attention parameter associated with edge type e, represents transpose, ‖ represents vector concatenation, and d represents the dimension of node representation; Finally, according to the attention weight, the information of the neighbor nodes is weighted and aggregated to update The expression is as follows: in, Indicates that it is connected to node v through edge type e i The set of neighbor nodes of Passing L g After the heterogeneous graph attention network layer, each node v i The final representation embedding will be obtained After learning the node representations, the final evidence retrieved is at the table row and paragraph level. Using the evidence representation itself and its associated finer-grained cell and sentence representations, we calculate the reranking scores for the table row and paragraph, respectively, through a fully connected layer. For each table row node v r ∈V r , its reordering score s r It is determined by its own representation and the representation of the cell node associated with it, and the expression is as follows: Among them, h q Representation embedding for representation problems; and Represent the final representation of the table row node and its adjacent cell nodes respectively; By edge type E m Connect to node v r The set of neighbor nodes of For each paragraph node v p ∈V p , its reordering score s p It is also determined by its own representation and the representation of the sentence node associated with it. The expression is as follows: in, and Represent the final representation of the paragraph node and its adjacent sentence nodes respectively; By edge type E m Connect to node v p The set of neighbor nodes; Sort the scores of table rows and paragraphs uniformly, starting from E initial Select the top E with the highest score among the initial screening evidences enhance The evidence is used as the final screening result of the retrieval stage; Among them, E enhanced The value is set according to the actual situation; S35, retrieval phase training; Define the loss function Optimize the reordering module based on the heterogeneous graph attention network. The expression is as follows: Among them, y v Represents the binary classification label of the evidence node v. If node v is the gold standard evidence containing the answer, y v =1, otherwise y v =0; represents all candidate evidence nodes in the primary set; s v represents the prediction score of the candidate evidence node v generated by the heterogeneous graph attention network; Combined with step S2, the final retrieval stage includes the joint training of three subtasks, namely the evidence selection loss, alignment loss, and re-ranking loss of the modality-integrated evidence selector, which are expressed as follows: Among them, λ e ,λ a and λ r They represent the weight coefficients of the evidence selection loss term, alignment loss term, and re-ranking loss term, respectively.

Images