Automatic question answering methods and devices, storage media, electronic devices
By using reverse feature aggregation and encoding, the prediction accuracy of the automatic question answering system is improved by leveraging BERT and self-attention mechanisms, solving the problem of semantic feature loss in existing systems and achieving more accurate information extraction and prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG MENTAL HEALTH CENT
- Filing Date
- 2023-06-15
- Publication Date
- 2026-05-26
AI Technical Summary
Existing automated question-answering systems struggle to effectively integrate preceding information during multi-round conversations, leading to the loss of semantic features and impacting prediction accuracy.
The pre-trained language model BERT is used to encode the question-and-answer text sequence and extract semantic features in reverse order. The features are aggregated by combining self-attention and cross-attention mechanisms, and the question-and-answer semantic representation is generated by summation operation. The model is trained using cross-entropy loss and AdamW optimization function.
It improves the prediction accuracy of automatic question answering models, reduces the loss of semantic features during information extraction, and enhances the richness and accuracy of encoded features.
Smart Images
Figure CN117093678B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of natural language processing and artificial intelligence, specifically to an automatic question-answering method and apparatus, storage medium, and electronic device. Background Technology
[0002] In today's society, an increasing number of people suffer from mental illness, some even to the point of despair and suicidal thoughts. Many of them are unwilling to acknowledge their condition as a mental illness, let alone seek treatment at a mental health center. They fear being ridiculed if others know they have a mental illness, thus suppressing their feelings and missing the optimal window for treatment. This is a deeply ingrained prejudice against psychiatry, difficult to change in a short time. However, with the rapid development of artificial intelligence, automated question-and-answer systems are increasingly entering various industries. If patients can understand that they can determine their condition through conversations with robots, it can effectively reduce their denial of seeking treatment for mental illness and avoid the stigma associated with discussing mental health issues in front of real people.
[0003] Automatic question answering (AQI) technology is generally used to extract and fuse semantic features from the conversation between the system and the patient. Based on these features, the system assesses the patient's condition and predicts the most likely topics and content for the next conversation, thereby assisting doctors in diagnosis and avoiding patient stigma. In AQI tasks, because the conversation between the system and the patient is multi-turn, the system may not be able to effectively integrate and utilize the information mentioned earlier in the later stages of the conversation, leading to incorrect predictions. In a normal question-and-answer conversation, the sequential appearance of questions and answers follows logical consistency and semantic coherence. However, the sentence most relevant to the potential conversation content is the last sentence in the question-and-answer history. Similarly, the sentence most relevant to the penultimate sentence in the AQI text is the second-to-last sentence in the question-and-answer history, and so on. Therefore, to extract effective information related to the potential conversation content, a reasonable approach would be to filter and aggregate information from the AQI text in reverse order of the question-and-answer statements. However, existing methods do not address this direction, resulting in unsatisfactory performance of current models.
[0004] Therefore, how to reasonably extract effective semantic information from automatic question-answering text sequences and how to reduce the loss of semantic features during the information extraction process, thereby improving the prediction accuracy of automatic question answering, are technical problems that urgently need to be solved. Summary of the Invention
[0005] The technical objective of this invention is to provide an automatic question-answering method, apparatus, storage medium, and electronic device to address the problems of how to reasonably extract effective semantic information from automatic question-answering text sequences and how to reduce the loss of semantic features during the information extraction process, thereby improving the prediction accuracy of automatic question answering.
[0006] The technical objective of this invention is achieved in the following manner: an automatic question-answering method, the specific method of which is as follows:
[0007] Obtain the automatic question-answering dataset: Use web crawling technology to crawl question-answering data from online public medical question-answering platforms. For each automatic question-answering data, construct positive example data and negative example data, and divide the training set and test set according to a set ratio;
[0008] Constructing an automatic question-answering model: Each sentence in the automatic question-answering data is used to construct an automatic question-answering text sequence according to the question-answering order. Each statement in the automatic question-answering text sequence and the candidate conversation content are encoded separately to obtain the encoded representation of each statement in the automatic question-answering text and the encoded representation of the candidate conversation content. Semantic features are extracted and aggregated from the automatic question-answering text sequence in reverse order of the question-answering statements. A summation operation is then used to obtain the question-answering semantic representation, which is mapped to a floating-point value within a specified range. This floating-point value is used as the matching degree between the candidate conversation content and the automatic question-answering text. The matching degrees of different candidate conversation contents are then compared, and the candidate conversation content with the highest matching degree is pushed to the user as the correct conversation content.
[0009] Training the automatic question answering model: Train the automatic question answering model using the automatic question answering dataset, and construct the loss function and optimization function to predict the content of the candidate conversation.
[0010] As a preferred option, the automatic question-answering model is constructed as follows:
[0011] Constructing the input data: For each data point in the dataset, construct an automatic question-and-answer text sequence for each sentence in the automatic question-and-answer text, denoted as h1, h2, ..., h3. n Select one candidate conversation item from multiple candidate conversation items as the current candidate conversation item, formalized as r; determine the label of this data item based on whether it is the correct candidate conversation item, that is, if the candidate conversation item is the correct candidate conversation item, it is recorded as 1; otherwise, it is recorded as 0; the automatic question and answer text sequence, the current candidate conversation item, and the label together constitute a single input data item, in the form of (h1, h2, ..., h...). n (r, tag);
[0012] Encoding Processing: The pre-trained language model BERT is used to encode the input data, resulting in the encoded representation of each sentence in the automatic question-answering text and the encoded representation of the candidate dialogue content, denoted as... and The formula is as follows:
[0013]
[0014]
[0015] Where h1, h2, ..., h n-1 ,h n The text represents statement 1, statement 2, ..., statement n-1, statement n in the automatic question and answer text; r represents the dialogue options.
[0016] Reverse word order feature aggregation: Semantic features of the automatic question-and-answer text sequence are extracted and aggregated by reversing the order in which the question and answer statements appear, and then the question-and-answer semantic representation is obtained through summation.
[0017] Tag prediction: The question-answering semantic representation is processed by a fully connected network with a dimension of 1 and an activation function of Sigmoid, thereby obtaining the probability that the current candidate conversation content is the correct candidate conversation content;
[0018] Train the automatic question answering model before it has been trained to optimize its parameters;
[0019] Once the automatic question-answering model has been trained, it predicts which of the candidate conversation topics is the correct one.
[0020] More specifically, the inverse word order feature aggregation is as follows:
[0021] A self-attention mechanism is used to filter features from the encoded representations of the candidate conversation content, thereby obtaining key representations of the candidate conversation content, denoted as . The formula is as follows:
[0022]
[0023] in, Encode the content to be selected for conversation;
[0024] The key representation of the candidate conversation content is used to filter the encoded representation n, resulting in key representation 1, denoted as , and the formula is as follows:
[0025]
[0026] in, Key indications of the topics to be discussed; The encoding represents n;
[0027] Using key representation 1 to filter information from encoded representation n-1, we obtain key representation 2, denoted as... Rejoin key representation 1 and key representation 2 to obtain aggregate representation 1, denoted as The formula is as follows:
[0028]
[0029]
[0030] in, The key is represented by 1; The encoding represents n-1;
[0031] Using key representation 2 to filter information from encoded representation n-2, we obtain key representation 3, denoted as... Rejoining aggregate representation 1 and key representation 3 yields aggregate representation 2, denoted as The formula is as follows:
[0032]
[0033]
[0034] in, The key is represented as 2; The encoding represents n-2; For aggregation, 1 is represented.
[0035] This process continues until key representation n-1 is used to filter information from encoded representation 1, thus obtaining key representation n, denoted as... Next, we connect the aggregate representation n-2 and the key representation n to obtain the aggregate representation n-1, denoted as . The formula is as follows:
[0036]
[0037]
[0038] in, The key is represented as n-1; The encoding represents 1; The aggregation is represented as n-2;
[0039] A fully connected network layer called Dense with ReLU activation function is used to map the aggregated representation, thus obtaining the semantic representation of the automatic question-answering text, denoted as . Finally, the semantic representation of the automatic question-answering text will be... Adding the key representations of the candidate conversation content to obtain the question-and-answer semantic representation, denoted as The formula is as follows:
[0040]
[0041] in, The aggregation is represented as n-1; This indicates the key points of the conversation to be discussed.
[0042] As a preferred option, the automatic question answering model is trained as follows:
[0043] Constructing the loss function: Cross-entropy is used as the loss function, as shown in the following formula:
[0044]
[0045] Among them, y true For real labels; y pred This represents the correct probability output by the model.
[0046] Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate, which was set to 2e-5, all other hyperparameters of AdamW were set to the default values in PyTorch.
[0047] An automatic question-and-answer device, the device comprising:
[0048] The automatic question-answering dataset acquisition unit is used to crawl question-answering data from online public medical question-answering platforms using web crawling technology. For each automatic question-answering data, positive example data and negative example data are constructed, and the training set and test set are divided according to a set ratio.
[0049] The automatic question-answering model construction unit is used to construct an automatic question-answering text sequence from each sentence in the automatic question-answering data according to the question-answering order. Each statement in the automatic question-answering text sequence and the candidate conversation content are encoded separately to obtain the encoded representation of each statement in the automatic question-answering text and the encoded representation of the candidate conversation content. Semantic features of the automatic question-answering text sequence are extracted and aggregated in reverse order of the question-answering statements. Then, a summation operation is used to obtain the question-answering semantic representation, which is mapped to a floating-point value in a specified range. This floating-point value is used as the matching degree between the candidate conversation content and the automatic question-answering text. The matching degrees of different candidate conversation contents are then compared, and the candidate conversation content with the highest matching degree is pushed to the user as the correct conversation content.
[0050] The automatic question answering model training unit is used to train the automatic question answering model using an automatic question answering dataset, and to construct the loss function and optimization function to predict the content of the candidate conversation.
[0051] Preferably, the automatic question-answering model building unit includes:
[0052] The input building block is used to preprocess the original dataset and build the input data;
[0053] The encoding module is used to encode the input data using the pre-trained language model BERT, thereby obtaining the encoded representation of each sentence in the automatic question-answering text and the encoded representation of the candidate conversation content;
[0054] The reverse word order feature aggregation module is used to extract and aggregate semantic features of the automatic question and answer text sequence by using the encoded representation of each sentence in the automatic question and answer text and the encoded representation of the candidate conversation content in reverse order of the order in which the question and answer sentences appear. Then, the semantic representation of the question and answer is obtained by summing.
[0055] The tag prediction unit is used to determine whether the current candidate conversation content is the correct candidate conversation content based on the question-and-answer semantic representation.
[0056] More preferably, the implementation process of the reverse word order feature aggregation module is as follows:
[0057] (1) A self-attention mechanism is used to filter features from the encoded representation of the candidate conversation content, thereby obtaining the key representation of the candidate conversation content, denoted as . The formula is as follows:
[0058]
[0059] in, Encode the content to be selected for conversation;
[0060] (2) Use the key representation of the candidate conversation content to filter the information of the encoded representation n, so as to obtain the key representation 1, denoted as, and the formula is as follows:
[0061]
[0062] in, Key indications of the topics to be discussed; The encoding represents n;
[0063] (3) Use key representation 1 to filter information from encoded representation n-1 to obtain key representation 2, denoted as... Rejoin key representation 1 and key representation 2 to obtain aggregate representation 1, denoted as The formula is as follows:
[0064]
[0065]
[0066] in, The key is represented by 1; The encoding represents n-1;
[0067] (4) Use key representation 2 to filter information from encoded representation n-2 to obtain key representation 3, denoted as... Rejoining aggregate representation 1 and key representation 3 yields aggregate representation 2, denoted as The formula is as follows:
[0068]
[0069]
[0070] in, The key is represented as 2; The encoding represents n-2; For aggregation, 1 is represented.
[0071] (5) This process continues until key representation n-1 is used to filter information from encoded representation 1, thus obtaining key representation n, denoted as... Next, we connect the aggregate representation n-2 and the key representation n to obtain the aggregate representation n-1, denoted as . The formula is as follows:
[0072]
[0073]
[0074] in, The key is represented as n-1; The encoding represents 1; The aggregation is represented as n-2;
[0075] (6) A fully connected network layer called Dense with ReLU activation function is used to map the aggregated representation to obtain the semantic representation of the automatic question-answering text, denoted as . Finally, the semantic representation of the automatic question-answering text will be... Adding the key representations of the candidate conversation content to obtain the question-and-answer semantic representation, denoted as The formula is as follows:
[0076]
[0077] in, The aggregation is represented as n-1; This indicates the key points of the conversation to be discussed.
[0078] Preferably, the automatic question answering model training unit includes:
[0079] The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function.
[0080] The optimization function building module is used to train and adjust the parameters in the training of the automatic question answering model to reduce prediction error.
[0081] An electronic device, characterized in that it comprises: a memory and at least one processor;
[0082] The memory contains computer programs;
[0083] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the automatic question-and-answer method as described above.
[0084] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the automatic question-answering method as described above.
[0085] The automatic question-answering method, apparatus, storage medium, and electronic device of the present invention have the following advantages:
[0086] (i) This invention can reasonably extract effective semantic information from automatic question-answering text sequences, reduce the loss of semantic features in the information extraction process, thereby obtaining accurate semantic features and improving the prediction accuracy of automatic question-answering models.
[0087] (ii) This invention can capture the encoded features in the automatic question-and-answer text and the content of the candidate conversation through encoding processing, thereby obtaining a richer and more accurate encoded representation;
[0088] (III) This invention can reasonably extract effective semantic information from automatic question-and-answer text sequences through reverse word order feature aggregation, thereby obtaining richer encoded feature representations;
[0089] (iv) This invention can effectively reduce the problem of semantic feature loss in the information extraction process by using reverse word order feature aggregation, thereby obtaining accurate encoded feature representation;
[0090] (v) The present invention combines reverse word order feature aggregation to effectively improve the prediction accuracy of automatic question answering models. Attached Figure Description
[0091] The invention will be further described below with reference to the accompanying drawings.
[0092] Appendix Figure 1 A flowchart of an automatic question-answering method;
[0093] Appendix Figure 2 A flowchart for building an automatic question-answering model;
[0094] Appendix Figure 3 Flowchart for training an automatic question-answering model;
[0095] Appendix Figure 4 This is a block diagram of an automatic question-and-answer device.
[0096] Appendix Figure 5 A flowchart illustrating the implementation process of the reverse word order feature aggregation module;
[0097] Appendix Figure 6 This is a schematic diagram of the framework for the reverse word order feature aggregation module. Detailed Implementation
[0098] The automatic question-answering method and apparatus, storage medium, and electronic device of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0099] Example 1:
[0100] As attached Figure 1 As shown, this embodiment provides an automatic question-answering method, characterized in that the method is as follows:
[0101] S1. Obtain the automatic question answering dataset: Details are as follows:
[0102] S101. Use web crawlers to obtain data: Write a web crawler program to crawl question and answer data from online public medical question and answer platforms to obtain an automatic question and answer knowledge base.
[0103] For example, online public medical Q&A sharing platforms contain a large amount of Q&A data, all of which is open to the public. Therefore, we can design corresponding web crawlers based on the characteristics of these platforms to obtain automated Q&A data and thus create an automated Q&A knowledge base.
[0104] S102. Construct an automatic question-answering dataset: For each question-answering data, construct positive and negative example data, and divide the training and test sets according to a fixed ratio.
[0105] For example, taking the question and answer example from a medical question and answer platform in step S101 as an example, S1 to S4 are taken as the automatic question and answer text sequence, S5 is taken as the correct candidate conversation content and marked as 1; and 3 question and answer statements are randomly selected from other data and taken as the incorrect candidate conversation content of that data and marked as 0.
[0106] The dataset was divided into training and test sets in a 4:1 ratio.
[0107] S2. Constructing an Automatic Question Answering Model: For each sentence in the automatic question answering data, construct an automatic question answering text sequence according to the question-and-answer order. Encode each statement in the automatic question answering text sequence and the candidate conversation content separately to obtain the encoded representation of each statement in the automatic question answering text and the encoded representation of the candidate conversation content. Extract and aggregate semantic features of the automatic question answering text sequence in reverse order of the question and answer statements. Then, obtain the question-and-answer semantic representation through a summation operation. Map the question-and-answer semantic representation to a floating-point value in a specified range. Use this floating-point value as the matching degree between the candidate conversation content and the automatic question answering text. Compare the matching degrees of different candidate conversation contents and push the candidate conversation content with the highest matching degree to the user as the correct conversation content.
[0108] S3. Training the automatic question answering model: Train the automatic question answering model using the automatic question answering dataset, and construct the loss function and optimization function to predict the content of the candidate conversation.
[0109] As attached Figure 2 As shown, the construction of the automatic question-answering model in step S2 of this embodiment is as follows:
[0110] S201. Constructing Input Data: For each data point in the dataset, construct an automatic question-and-answer text sequence for each sentence in the automatic question-and-answer text according to the question-and-answer order, denoted as h1, h2, ..., h... n Select one candidate conversation item from multiple candidate conversation items as the current candidate conversation item, formalized as r; determine the label of this data item based on whether it is the correct candidate conversation item, that is, if the candidate conversation item is the correct candidate conversation item, it is recorded as 1; otherwise, it is recorded as 0; the automatic question and answer text sequence, the current candidate conversation item, and the label together constitute a single input data item, in the form of (h1, h2, ..., h...). n (r, tag);
[0111] S202. Encoding Processing: The pre-trained language model BERT is used to encode the input data, resulting in the encoded representation of each sentence in the automatic question-and-answer text and the encoded representation of the candidate dialogue content, denoted as... and The formula is as follows:
[0112]
[0113]
[0114] Where h1, h2, ..., h n-1 ,h n The text represents statement 1, statement 2, ..., statement n-1, statement n in the automatic question and answer text; r represents the dialogue options.
[0115] For example, when implementing this invention on a dataset, the pre-trained language model BERT is called to complete the operation of this module, and all settings are based on the default settings of BERT in PyTorch. In PyTorch, the code implementation described above is as follows:
[0116] # Encode the input data using BERT's encoding layer
[0117] h_encoder_list = []
[0118] for iin h_embed_list:
[0119] h_encoder_list.append(BERT(i)[1])
[0120] r_embed = BERT(r)[1]
[0121] Here, h_embed_list represents each statement in the automatic question-and-answer text, r is the candidate dialogue content, h_encoder_list represents the encoded representation of each statement in the automatic question-and-answer text, response represents the sequence of options, and response_embed is the encoded representation of the candidate dialogue content.
[0122] S203, Reverse word order feature aggregation: Semantic features of the automatic question and answer text sequence are extracted and aggregated by reversing the order in which the question and answer statements appear, and then the question and answer semantic representation is obtained through summation operation;
[0123] S204, Tag Prediction: The question-answering semantic representation is processed by a fully connected network with a dimension of 1 and an activation function of Sigmoid, thereby obtaining the probability that the current candidate conversation content is the correct candidate conversation content;
[0124] Train the automatic question answering model before it has been trained to optimize its parameters;
[0125] Once the automatic question-answering model has been trained, it predicts which of the candidate conversation topics is the correct one.
[0126] As attached Figure 5 and 6 As shown, the inverse word order feature aggregation in step S203 of this embodiment is as follows:
[0127] S20301. Use a self-attention mechanism to perform feature filtering on the encoded representation of the candidate conversation content, thereby obtaining the key representation of the candidate conversation content, denoted as... The formula is as follows:
[0128]
[0129] in, Encode the content to be selected for conversation;
[0130] S20302. Use the key representation of the candidate conversation content to filter the information of the encoded representation n, thereby obtaining the key representation 1, denoted as , with the following formula:
[0131]
[0132] in, Key indications of the topics to be discussed; The encoding represents n;
[0133] S20303. Use key representation 1 to filter information from encoded representation n-1 to obtain key representation 2, denoted as... Rejoin key representation 1 and key representation 2 to obtain aggregate representation 1, denoted as The formula is as follows:
[0134]
[0135]
[0136] in, The key is represented by 1; The encoding represents n-1;
[0137] S20304. Use key representation 2 to filter information from encoded representation n-2 to obtain key representation 3, denoted as... Rejoining aggregate representation 1 and key representation 3 yields aggregate representation 2, denoted as The formula is as follows:
[0138]
[0139]
[0140] in, The key is represented as 2; The encoding represents n-2; For aggregation, 1 is represented.
[0141] S20305, and so on, until the key representation n-1 is used to filter the information of the encoded representation 1, thus obtaining the key representation n, denoted as Next, we connect the aggregate representation n-2 and the key representation n to obtain the aggregate representation n-1, denoted as . The formula is as follows:
[0142]
[0143]
[0144] in, The key is represented as n-1; The encoding represents 1; The aggregation is represented as n-2;
[0145] S20306. A fully connected network layer called Dense with ReLU activation function is used to map the aggregated representation to obtain the semantic representation of the automatic question-answering text, denoted as . Finally, the semantic representation of the automatic question-answering text will be... Adding the key representations of the candidate conversation content to obtain the question-and-answer semantic representation, denoted as The formula is as follows:
[0146]
[0147] in, The aggregation is represented as n-1; This indicates the key points of the conversation to be discussed.
[0148] For example, the self-attention mechanism Self-Attention() uses the Dot-Product Self Attention calculation method, with the following formula:
[0149]
[0150] in, This indicates that the interaction calculation of the candidate conversation content is implemented by using dot product multiplication operations to encode the interaction calculation of itself. This indicates the encoding of the candidate conversation content; This represents the dot product multiplication operation;
[0151]
[0152] in, This indicates that the attention weight α is obtained through a normalization operation. i , i and i' represent the indexes of the elements in the corresponding input tensor; l represents the input tensor The number of elements in;
[0153]
[0154] in, This indicates the use of the already obtained attention weight α. i The feature filtering of the encoded representation of the candidate conversation content is completed, thus obtaining the candidate conversation content filtering representation; l represents and α iThe number of elements in;
[0155] When filtering information at each layer, a cross-attention mechanism is used. The Cross-Attention mechanism uses the Dot-Product Attention calculation method. Taking the calculation of the key representation 1 as an example, the calculation process is as follows:
[0156]
[0157] This indicates that the interaction between the key representation of the candidate conversation content and the encoded representation n is achieved through dot product multiplication. This indicates the key points of the conversation to be discussed. The encoding represents n, This represents the dot product multiplication operation;
[0158]
[0159] This indicates that the attention weight α is obtained through a normalization operation. i , i and i' represent the indexes of the elements in the corresponding input tensor; l represents the input tensor The number of elements in;
[0160]
[0161] This indicates the use of the already obtained attention weight α. i By completing the feature selection of the encoded representation n, the candidate key representations 1 and l are obtained. and the number of elements in α.
[0162] In PyTorch, the code described above is implemented as follows:
[0163] # Define the calculation process for cross attention
[0164] def cross_attention(s1,s2):
[0165] s1_s2_dot=tf.expand_dims(s1,axis=1)*tf.expand_dims(s2,axis=2)sd1=tf.multiply(tf.tanh(K.dot(s1_s2_dot,self.Wd)),self.vd)sd2=tf.squeeze(sd1,axis=-1)
[0166] ad = tf.nn.softmax(sd2)
[0167] qdq6K.batch_dot(ad,s2)
[0168] return qdq
[0169] #Your self attention quote
[0170] def self_attention(s3):
[0171] s4 / s3
[0172] s3_s4_dot6tf.expand_dims(s3,axis1)*tf.expand_dims(s4,axis12)sd1Tf.multi ply ( tf . tanh ( K . dot ( s3_s4_dot , self . Wd ) ) , self . vd ) sd2 ( tf . squeeze ( sd1 , axis 1 - 1 ) ;
[0173] adTf.nn.softmax(sd2)
[0174] qdq6K.batch_dot(ad,s4)
[0175] return qdq
[0176] #MyFatherFace
[0177] dense_reluxnn.Linear(768,768)
[0178] e_r_response_embed
[0179] e_h_list_history_embed_list
[0180] f_r_self_attention(e_r)
[0181] lin_f_r
[0182] key_infor”
[0183] for e_h in e_h_list:
[0184] line_cross_attention ( line , e_h ) ;
[0185] if key_infor”:
[0186] key_infor(link).
[0187] else:
[0188] key_infor=torch.cat((key_infor,lin),dim=1)
[0189] S=nn.ReLU()(dense_relu(key_infor))
[0190] Q = S + f_r
[0191] Here, history_embed_list represents a list of all coded representations of statements in the automatic question-and-answer text; response_embed represents the coded representation of the candidate conversation content; and Q represents the question-and-answer semantic representation.
[0192] As attached Figure 3 As shown, the training of the automatic question-answering model in step S3 of this embodiment is as follows:
[0193] S301. Constructing the loss function: Cross-entropy is used as the loss function, as shown in the following formula:
[0194]
[0195] Among them, y true For real labels; y pred This represents the correct probability output by the model.
[0196] For example, in PyTorch, the code implementation described above would be as follows:
[0197] #Calculate the error between the predicted value and the label using the cross-entropy loss function.
[0198] loss_fct = CrossEntropyLoss()
[0199] loss=loss_fct(logits.view(-1,self.num_labels),labels.view(-1))
[0200] Here, labels are the true labels, and logits are the correct probabilities of the model output.
[0201] S302. Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate being set to 2e-5, all other hyperparameters of AdamW were set to the default values in PyTorch.
[0202] For example, in PyTorch, the code implementation described above would be as follows:
[0203] #Optimize model parameters using the AdamW optimizer
[0204] optimizer=AdamW(optimizer_grouped_parameters,lr=2e-5)
[0205] Here, optimizer_grouped_parameters are the parameters to be optimized, which by default are all the parameters in the automatic question answering model.
[0206] If the model has not yet been trained, step S3 needs to be executed to train it and optimize the model parameters; once the model has been trained, step S204 can be executed to predict which of the candidate conversation topics is the correct candidate conversation topic.
[0207] Example 2:
[0208] As attached Figure 4 As shown, this embodiment provides an automatic question-and-answer device, characterized in that the device includes:
[0209] The automatic question-answering dataset acquisition unit is used to crawl question-answering data from online public medical question-answering platforms using web crawling technology. For each automatic question-answering data, positive example data and negative example data are constructed, and the training set and test set are divided according to a set ratio.
[0210] The automatic question-answering model construction unit is used to construct an automatic question-answering text sequence from each sentence in the automatic question-answering data according to the question-answering order. Each statement in the automatic question-answering text sequence and the candidate conversation content are encoded separately to obtain the encoded representation of each statement in the automatic question-answering text and the encoded representation of the candidate conversation content. Semantic features of the automatic question-answering text sequence are extracted and aggregated in reverse order of the question-answering statements. Then, a summation operation is used to obtain the question-answering semantic representation, which is mapped to a floating-point value in a specified range. This floating-point value is used as the matching degree between the candidate conversation content and the automatic question-answering text. The matching degrees of different candidate conversation contents are then compared, and the candidate conversation content with the highest matching degree is pushed to the user as the correct conversation content.
[0211] The automatic question answering model training unit is used to train the automatic question answering model using an automatic question answering dataset, and to construct the loss function and optimization function to predict the content of the candidate conversation.
[0212] The automatic question-answering model construction unit in this embodiment includes:
[0213] The input building block is used to preprocess the original dataset and build the input data;
[0214] The encoding module is used to encode the input data using the pre-trained language model BERT, thereby obtaining the encoded representation of each sentence in the automatic question-answering text and the encoded representation of the candidate conversation content;
[0215] The reverse word order feature aggregation module is used to extract and aggregate semantic features of the automatic question and answer text sequence by using the encoded representation of each sentence in the automatic question and answer text and the encoded representation of the candidate conversation content in reverse order of the order in which the question and answer sentences appear. Then, the semantic representation of the question and answer is obtained by summing.
[0216] The tag prediction unit is used to determine whether the current candidate conversation content is the correct candidate conversation content based on the question-and-answer semantic representation.
[0217] As attached Figure 5 and 6 As shown, the implementation process of the reverse word order feature aggregation module in this embodiment is as follows:
[0218] (1) A self-attention mechanism is used to filter features from the encoded representation of the candidate conversation content, thereby obtaining the key representation of the candidate conversation content, denoted as . The formula is as follows:
[0219]
[0220] in, Encode the content to be selected for conversation;
[0221] (2) Use the key representation of the candidate conversation content to filter the information of the encoded representation n, so as to obtain the key representation 1, denoted as, and the formula is as follows:
[0222]
[0223] in, Key indications of the topics to be discussed; The encoding represents n;
[0224] (3) Use key representation 1 to filter information from encoded representation n-1 to obtain key representation 2, denoted as... Rejoin key representation 1 and key representation 2 to obtain aggregate representation 1, denoted as The formula is as follows:
[0225]
[0226]
[0227] in, The key is represented by 1; The encoding represents n-1;
[0228] (4) Use key representation 2 to filter information from encoded representation n-2 to obtain key representation 3, denoted as... Rejoining aggregate representation 1 and key representation 3 yields aggregate representation 2, denoted as The formula is as follows:
[0229]
[0230]
[0231] in, The key is represented as 2; The encoding represents n-2; For aggregation, 1 is represented.
[0232] (5) This process continues until key representation n-1 is used to filter information from encoded representation 1, thus obtaining key representation n, denoted as... Next, we connect the aggregate representation n-2 and the key representation n to obtain the aggregate representation n-1, denoted as . The formula is as follows:
[0233]
[0234]
[0235] in, The key is represented as n-1; The encoding represents 1; The aggregation is represented as n-2;
[0236] (6) A fully connected network layer called Dense with ReLU activation function is used to map the aggregated representation to obtain the semantic representation of the automatic question-answering text, denoted as . Finally, the semantic representation of the automatic question-answering text will be... Adding the key representations of the candidate conversation content to obtain the question-and-answer semantic representation, denoted as The formula is as follows:
[0237]
[0238] in, The aggregation is represented as n-1; This indicates the key points of the conversation to be discussed.
[0239] The automatic question-answering model training unit in this embodiment includes:
[0240] The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function.
[0241] The optimization function building module is used to train and adjust the parameters in the training of the automatic question answering model to reduce prediction error.
[0242] Example 3:
[0243] This embodiment also provides an electronic device, including: a memory and a processor;
[0244] The memory stores the instructions executed by the computer.
[0245] The processor executes computer execution instructions stored in the memory, causing the processor to perform the automatic question-answering method in any embodiment of the present invention.
[0246] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.
[0247] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.
[0248] Example 4:
[0249] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the automatic question-answering method in any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.
[0250] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0251] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0252] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0253] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0254] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not render the corresponding technology ineffective.
Claims
1. An automatic question-answering method, characterized in that, The method is as follows: Obtain the automatic question-answering dataset: Use web crawling technology to crawl question-answering data from online public medical question-answering platforms. For each automatic question-answering data, construct positive example data and negative example data, and divide the training set and test set according to a set ratio; Constructing an automatic question-answering model: Each sentence in the automatic question-answering data is used to construct an automatic question-answering text sequence according to the question-answering order. Each statement in the automatic question-answering text sequence and the candidate conversation content are encoded separately to obtain the encoded representation of each statement in the automatic question-answering text and the encoded representation of the candidate conversation content. Semantic features are extracted and aggregated from the automatic question-answering text sequence in reverse order of the question-answering statements. A summation operation is then used to obtain the question-answering semantic representation, which is mapped to a floating-point value within a specified range. This floating-point value is used as the matching degree between the candidate conversation content and the automatic question-answering text. The matching degrees of different candidate conversation contents are then compared, and the candidate conversation content with the highest matching degree is pushed to the user as the correct conversation content. Training the automatic question answering model: Train the automatic question answering model using the automatic question answering dataset, and construct the loss function and optimization function to predict the content of the candidate conversation; The inverse word order feature aggregation is as follows: A self-attention mechanism is used to filter features from the encoded representations of the candidate conversation content, thereby obtaining key representations of the candidate conversation content, denoted as . The formula is as follows: ； in, Encode the content to be selected for conversation; The key representation of the candidate conversation content is used to filter the encoded representation n, resulting in key representation 1, denoted as , and the formula is as follows: ； in, Key indications of the topics to be discussed; The encoding represents n; Using key representation 1 to filter information from encoded representation n-1, we obtain key representation 2, denoted as... Rejoin key representation 1 and key representation 2 to obtain aggregate representation 1, denoted as The formula is as follows: ； ； in, The key is represented by 1; The encoding represents n-1; Using key representation 2 to filter information from encoded representation n-2, we obtain key representation 3, denoted as... Rejoin aggregate representation 1 and key representation 3 to obtain aggregate representation 2, denoted as... The formula is as follows: ； ； in, The key is represented as 2; The encoding represents n-2; The aggregation is represented by 1; This process continues until key representation n-1 is used to filter information from encoded representation 1, thus obtaining key representation n, denoted as... Next, the aggregate representation n-2 and the key representation n are joined to obtain the aggregate representation n-1, denoted as . The formula is as follows: ； ； in, The key is represented as n-1; The encoding represents 1; The aggregation is represented as n-2; A fully connected network layer called Dense with ReLU activation function is used to map the aggregated representation, thus obtaining the semantic representation of the automatic question-answering text, denoted as . Finally, the semantic representation of the automatic question-and-answer text will be... Adding the key representations of the candidate conversation content to obtain the question-and-answer semantic representation, denoted as The formula is as follows: ； in, The aggregation is represented as n-1; This indicates the key points of the conversation to be discussed.
2. The automatic question-answering method according to claim 1, characterized in that, The automatic question answering model is constructed as follows: Constructing Input Data: For each data point in the dataset, construct an automatic question-and-answer text sequence for each sentence in the automatic question-and-answer text, denoted as h1, h2, ..., h3. n Select one candidate conversation item from multiple candidate conversation items as the current candidate conversation item, formalized as r; determine the label of this data item based on whether it is the correct candidate conversation item, that is, if the candidate conversation item is the correct candidate conversation item, it is recorded as 1; otherwise, it is recorded as 0; the automatic question and answer text sequence, the current candidate conversation item, and the label together constitute a single input data item, in the form of (h1, h2, ..., h...). n (r, tag); Encoding Processing: The pre-trained language model BERT is used to encode the input data, resulting in the encoded representation of each sentence in the automatic question-answering text and the encoded representation of the candidate dialogue content, denoted as... , , ..., , and The formula is as follows: ； ； Where h1, h2, …, h n-1 , h n The text represents statement 1, statement 2, ..., statement n-1, statement n in the automatic question and answer text; r represents the dialogue options. Reverse word order feature aggregation: Semantic features of the automatic question-and-answer text sequence are extracted and aggregated by reversing the order in which the question and answer statements appear, and then the question-and-answer semantic representation is obtained through summation. Tag prediction: The question-answering semantic representation is processed by a fully connected network with a dimension of 1 and an activation function of Sigmoid, thereby obtaining the probability that the current candidate conversation content is the correct candidate conversation content; Train the automatic question answering model before it has been trained to optimize its parameters; Once the automatic question-answering model has been trained, it predicts which of the candidate conversation topics is the correct one.
3. The automatic question-answering method according to claim 1, characterized in that, The training of the automatic question answering model is as follows: Constructing the loss function: Cross-entropy is used as the loss function, as shown in the following formula: ； Among them, y true For real labels; y pred This represents the correct probability output by the model. Constructing the optimization function: After testing various optimization functions, AdamW was finally selected as the optimization function. Except for its learning rate, which was set to 2e-5, all other hyperparameters of AdamW were set to the default values in PyTorch.
4. An automatic question-and-answer device, characterized in that, The device includes: The automatic question-answering dataset acquisition unit is used to crawl question-answering data from online public medical question-answering platforms using web crawling technology. For each automatic question-answering data, positive example data and negative example data are constructed, and the training set and test set are divided according to a set ratio. The automatic question-answering model construction unit is used to construct an automatic question-answering text sequence from each sentence in the automatic question-answering data according to the question-answering order. Each statement in the automatic question-answering text sequence and the candidate conversation content are encoded separately to obtain the encoded representation of each statement in the automatic question-answering text and the encoded representation of the candidate conversation content. Semantic features of the automatic question-answering text sequence are extracted and aggregated in reverse order of the question-answering statements. Then, a summation operation is used to obtain the question-answering semantic representation, which is mapped to a floating-point value in a specified range. This floating-point value is used as the matching degree between the candidate conversation content and the automatic question-answering text. The matching degrees of different candidate conversation contents are then compared, and the candidate conversation content with the highest matching degree is pushed to the user as the correct conversation content. The automatic question answering model training unit is used to train the automatic question answering model using an automatic question answering dataset, and to construct the loss function and optimization function to predict the content of the candidate conversation. The implementation process of the reverse word order feature aggregation module is as follows: (1) Use a self-attention mechanism to perform feature filtering on the encoding representation of the candidate conversation content, thereby obtaining the key representation of the candidate conversation content, denoted as The formula is as follows: ； in, Encode the content to be selected for conversation; (2) Use the key representation of the candidate conversation content to filter the information of the encoded representation n, so as to obtain the key representation 1, denoted as, and the formula is as follows: ； in, Key indications of the topics to be discussed; The encoding represents n; (3) Use key representation 1 to filter information from encoded representation n-1 to obtain key representation 2, denoted as Rejoin key representation 1 and key representation 2 to obtain aggregate representation 1, denoted as The formula is as follows: ； ； in, The key is represented by 1; The encoding represents n-1; (4) Use key representation 2 to filter information from encoded representation n-2 to obtain key representation 3, denoted as Rejoin aggregate representation 1 and key representation 3 to obtain aggregate representation 2, denoted as... The formula is as follows: ； ； in, The key is represented as 2; The encoding represents n-2; The aggregation is represented by 1; (5) This process continues until key representation n-1 is used to filter information from encoded representation 1, thus obtaining key representation n, denoted as n-1. Next, the aggregate representation n-2 and the key representation n are joined to obtain the aggregate representation n-1, denoted as . The formula is as follows: ； ； in, The key is represented as n-1; The encoding represents 1; The aggregation is represented as n-2; (6) A fully connected network layer called Dense with ReLU activation function is used to map the aggregated representation to obtain the semantic representation of the automatic question answering text, denoted as . Finally, the semantic representation of the automatic question-and-answer text will be... Adding the key representations of the candidate conversation content to obtain the question-and-answer semantic representation, denoted as The formula is as follows: ； in, The aggregation is represented as n-1; This indicates the key points of the conversation to be discussed.
5. The automatic question-and-answer device according to claim 4, characterized in that, The automatic question answering model construction unit includes: The input building block is used to preprocess the original dataset and build the input data; The encoding module is used to encode the input data using the pre-trained language model BERT, thereby obtaining the encoded representation of each sentence in the automatic question-answering text and the encoded representation of the candidate conversation content; The reverse word order feature aggregation module is used to extract and aggregate semantic features of the automatic question and answer text sequence by using the encoded representation of each sentence in the automatic question and answer text and the encoded representation of the candidate conversation content in reverse order of the order in which the question and answer sentences appear. Then, the semantic representation of the question and answer is obtained by summing. The tag prediction unit is used to determine whether the current candidate conversation content is the correct candidate conversation content based on the question-and-answer semantic representation.
6. The automatic question-and-answer device according to claim 4, characterized in that, The automatic question answering model training unit includes: The loss function building module is used to calculate the error between the prediction results and the actual data using the cross-entropy loss function. The optimization function building module is used to train and adjust the parameters in the training of the automatic question answering model to reduce prediction error.
7. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the automatic question-answering method as described in any one of claims 1 to 3.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the automatic question-answering method as described in any one of claims 1 to 3.