Model training and question answering method and device for retrieval enhancement, equipment and medium

Abstract

The invention discloses a model training and question answering method and device for retrieval enhancement, equipment and a medium, a Double-Dueling DQN model is trained by adopting reinforcement learning to serve as a retriever, the retrieval effect is more accurate and reliable by continuously exploring and testing in different states, namely, questions, selected documents and actions of selecting new documents, and the retrieval efficiency is improved. And finally, the document most suitable for the answer is selected, the situation that a traditional method depends on the similarity between calculation questions and document vectors is avoided, the retrieval effect is improved, meanwhile, the training retriever can reduce the calculation cost brought by training a large generator, and the retrieval efficiency is improved. The technical problems that in a current retrieval enhancement model, retrieval tasks of a retriever in a specific field are poor in performance, the capability of fine adjustment of the specific field is lacked, and meanwhile deep association of the document and the problem in semantics is ignored in the document selection process, so that the selected document and the problem are not completely related are solved.

Description

Technical Field

[0001] The present application relates to the field of search and question answering technology, and in particular to a model training, question answering method, device, equipment and medium for search enhancement. Background Art

[0002] Retrieval-Augmented Question Answering (RAQA) is an advanced question answering system that combines traditional information retrieval technology with natural language processing technology. The core idea of ​​this technology is to retrieve a large amount of text data to find the most relevant information fragments to the user's question, and then use this information to generate accurate and natural answers.

[0003] The working principle of retrieval-enhanced question-answering technology is as follows:

[0004] Retrieval stage: The system first uses a retriever to retrieve the documents or information fragments most relevant to the user's question from a large-scale text database. These retrievers are usually based on pre-trained language models, which can understand the semantic content of the question and retrieve documents based on semantic similarity.

[0005] Generation phase: The retrieved relevant documents or information fragments are then fed into the generator, which is responsible for integrating this information and generating a fluent and accurate natural language answer.

[0006] The current mainstream retrieval enhancement models include:

[0007] Retrieval-Augmented Generation (RAG): The RAG model obtains relevant information through the retrieval stage and combines this information to generate answers. It usually uses a pre-trained model such as BERT as a retriever and uses a sequence-to-sequence model in the generation stage.

[0008] Fusion in Decoder (FiD): The FiD model fuses the retrieved information in the decoder so that the generator can directly utilize the retrieved document content when generating the answer.

[0009] Atlas: Atlas is a large-scale retrieval-augmented model that handles complex question-answering tasks through sophisticated retrieval and generation mechanisms.

[0010] The current retrieval enhancement model has the following technical problems:

[0011] Limitations of Retrievers: Current retrievers often rely on pre-trained language models, which may perform poorly on domain-specific retrieval tasks because they are not fine-tuned for the specific domain.

[0012] Accuracy of document selection: During the document selection process, some search engines may only focus on word meaning and frequency, while ignoring the deep semantic connection between the document and the question, which may result in the selected documents not being completely relevant.

[0013] Computational resource requirements: Generators based on large pre-trained models require a lot of computing resources and time during fine-tuning, which poses a challenge for application scenarios with limited resources or that require fast response.

[0014] Therefore, it is particularly important to find a method that can maintain high retrieval effect while reducing computing requirements. Summary of the invention

[0015] The present application provides a model training, question-answering method, apparatus, device and medium for retrieval enhancement, which solves the technical problems existing in the current retrieval enhancement model, that is, the retriever performs poorly in retrieval tasks in specific fields, lacks the ability to fine-tune specific fields, and ignores the deep semantic connection between documents and questions in the document selection process, resulting in the selected documents being not completely relevant to the questions.

[0016] In view of this, the first aspect of the present application provides a model training method for retrieval enhancement, the method comprising:

[0017] Constructing a Double-Dueling DQN model to be trained, wherein the Double-Dueling DQN model includes an online network module and a target network module;

[0018] Acquire training data, the training data including training questions, a number of corresponding training documents, and standard answers corresponding to the training questions;

[0019] The Double-Dueling DQN model is trained using the training data, wherein:

[0020] The Double-Dueling DQN model performs a selection action in the current state through the online network module, and calculates the first action value of the current selection action, wherein the selection action is to select a new training document and add it to the current string consisting of the training question in the current state and the selected training document, so as to obtain a new state of the online network module;

[0021] The Double-Dueling DQN model performs an estimation action through the target network module, wherein the estimation action is to calculate a second action value of the new state based on the instantaneous reward submodule and the penalty item submodule according to the new state;

[0022] According to the preset loss function, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, thereby obtaining the trained Double-Dueling DQN model.

[0023] Optionally, the online network module includes a state value function and an advantage function;

[0024] The state value function is used to estimate the overall value under the current state;

[0025] The advantage function is used to estimate the relative value of the current selected action relative to all other selected actions;

[0026] The first action value is the sum of the first value of the state value function and the second value of the advantage function calculated according to the current selected action.

[0027] Optionally, the target network module includes an instantaneous reward submodule, a penalty term submodule and an estimation submodule;

[0028] The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a gap value between the generated answer to the training question generated in the new state and the standard answer;

[0029] The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty item submodule calculated according to the new state, and the estimated value of the best action in the new state by the estimation submodule.

[0030] Optionally, the penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein:

[0031] The first penalty formula is to calculate the first penalty value based on the absolute value of the change in the model parameter before and after the generator training and based on the first preset coefficient;

[0032] The second penalty formula is to calculate the second penalty value based on the square of the number of the selected training documents and based on a second preset coefficient.

[0033] Optionally, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated according to a preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, and the trained Double-Dueling DQN model is obtained as follows:

[0034] Calculate the second action value of the new state by using the target network model;

[0035] Taking the new state as the current state, and calculating the first action value of the current state through the online network module;

[0036] Calculate a loss value according to a mean square error between the first action value and the second action value;

[0037] According to the loss value, the network parameters of the online network module and the target network module are updated based on the gradient descent method until the loss value converges or the number of bands reaches a preset threshold, thereby obtaining the trained Double-Dueling DQN model.

[0038] The second aspect of the present application provides a question-answering method based on retrieval enhancement, using a Double-Dueling DQN model trained by the model training method for retrieval enhancement described in any one of the first aspects of the present application, and the question-answering method based on retrieval enhancement includes:

[0039] Get questions to be answered;

[0040] Taking the question to be answered and a document database containing a plurality of question-answer documents as inputs of the Double-Dueling DQN model, and obtaining an optimal question-answer document output by the Double-Dueling DQN model;

[0041] Based on the question to be answered and the optimal question-answer document, an answer is generated through a preset generator output.

[0042] Optionally, after generating the answer by outputting the preset generator, the method further includes:

[0043] Obtaining a corrected answer from the user after optimizing the generated answer;

[0044] The corrected answer is used as the standard answer to the question to be answered, combined with the optimal question and answer document, and added to the database used to train the Double-Dueling DQN model.

[0045] The third aspect of the present application provides a model training device for retrieval enhancement, the device comprising:

[0046] A model building unit, used to build a Double-Dueling DQN model to be trained, wherein the Double-Dueling DQN model includes an online network module and a target network module;

[0047] A training data acquisition unit, used to acquire training data, wherein the training data includes training questions, a number of corresponding training documents, and standard answers corresponding to the training questions;

[0048] A model training unit, configured to train the Double-Dueling DQN model using the training data, wherein:

[0049] The Double-Dueling DQN model performs a selection action in the current state through the online network module, and calculates the first action value of the current selection action, wherein the selection action is to select a new training document and add it to the current string consisting of the training question in the current state and the selected training document, so as to obtain a new state of the online network module;

[0050] The Double-Dueling DQN model performs an estimation action through the target network module, wherein the estimation action is to calculate a second action value of the new state based on the instantaneous reward submodule and the penalty item submodule according to the new state;

[0051] The loss optimization unit is used to iteratively update the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to a preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, thereby obtaining the trained Double-Dueling DQN model.

[0052] Optionally, the online network module includes a state value function and an advantage function;

[0053] The state value function is used to estimate the overall value under the current state;

[0054] The advantage function is used to estimate the relative value of the current selected action relative to all other selected actions;

[0055] The first action value is the sum of the first value of the state value function and the second value of the advantage function calculated according to the current selected action.

[0056] Optionally, the target network module includes an instantaneous reward submodule, a penalty term submodule and an estimation submodule;

[0057] The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a gap value between the generated answer to the training question generated in the new state and the standard answer;

[0058] The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty item submodule calculated according to the new state, and the estimated value of the best action in the new state by the estimation submodule.

[0059] Optionally, the penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein:

[0060] The first penalty formula is to calculate the first penalty value based on the absolute value of the change in the model parameter before and after the generator training and based on the first preset coefficient;

[0061] The second penalty formula is to calculate the second penalty value based on the square of the number of the selected training documents and based on a second preset coefficient.

[0062] Optionally, the loss optimization unit is specifically used to:

[0063] Calculate the second action value of the new state by using the target network model;

[0064] Taking the new state as the current state, and calculating the first action value of the current state through the online network module;

[0065] Calculate a loss value according to a mean square error between the first action value and the second action value;

[0066] According to the loss value, the network parameters of the online network module and the target network module are updated based on the gradient descent method until the loss value converges or the number of bands reaches a preset threshold, thereby obtaining the trained Double-Dueling DQN model.

[0067] A fourth aspect of the present application provides a model training device for retrieval enhancement, the device comprising a processor and a memory:

[0068] The memory is used to store program code and transmit the program code to the processor;

[0069] The processor is used to execute the steps of the method for retrieving enhanced model training as described in the first aspect above according to the instructions in the program code.

[0070] A fifth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0071] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0072] In the present application, a model training, question-answering method, apparatus, device and medium for retrieval enhancement are provided. A Double-Dueling DQN model is trained as a retriever using reinforcement learning. By continuously exploring and experimenting between different states, namely, question + selected document, and the action of selecting a new document, the retrieval effect is made more accurate and reliable, and the document that best suits the answer is finally selected, rather than relying on calculating the similarity between the question and document vectors as in traditional methods, thereby improving the retrieval effect. At the same time, training the retriever can reduce the computational cost of training a large generator, and solves the problems existing in the current retrieval enhancement model, such as poor performance of the retriever in retrieval tasks in specific fields, lack of ability to fine-tune specific fields, and ignoring the deep semantic connection between the document and the question in the document selection process, resulting in the selected document being not completely relevant to the question. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A method flow chart of a model training method for retrieval enhancement in an embodiment of the present application;

[0074] Figure 2 A method flow chart of a question-answering method based on retrieval enhancement in an embodiment of the present application;

[0075] Figure 3 It is a structural schematic diagram of a model training device for retrieval enhancement in an embodiment of the present application;

[0076] Figure 4 This is a schematic diagram of the structure of a model training device for retrieval enhancement in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] The present application designs a model training, question-answering method, apparatus, device and medium for retrieval enhancement, which solves the technical problems that the retriever of the current retrieval enhancement model performs poorly in retrieval tasks in specific fields, lacks the ability to fine-tune specific fields, and ignores the deep semantic connection between documents and questions in the document selection process, resulting in the selected documents being not completely relevant to the questions.

[0079] For easier understanding, see Figure 1 , Figure 1 This is a method flow chart of a model training method for retrieval enhancement in an embodiment of the present application, such as Figure 1 As shown, specifically:

[0080] S101, constructing a Double-Dueling DQN model to be trained, where the Double-Dueling DQN model includes an online network module and a target network module;

[0081] It should be noted that the Double-Dueling DQN model is designed and initialized, including the online network and the target network.

[0082] Embodiment: Both the online network and the target network adopt deep neural networks and have similar structures but different parameters, wherein: the online network is used for decision-making, and the target network is used for stabilizing the training process.

[0083] S102, obtaining training data, where the training data includes training questions, a number of corresponding training documents, and standard answers corresponding to the training questions;

[0084] It should be noted that the questions, documents and standard answers required for training should be prepared.

[0085] Example: 1000 questions and their related documents and standard answers are randomly selected from the question-answering database as a training set.

[0086] S103, training the Double-Dueling DQN model using training data, wherein:

[0087] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string consisting of the training question and the selected training document in the current state to obtain the new state of the online network module.

[0088] The Double-Dueling DQN model performs an estimated action through the target network module. The estimated action is to calculate the second action value of the new state based on the new state, based on the instantaneous reward submodule and the penalty submodule;

[0089] It should be noted that the model is trained using training data so that it can select the most relevant documents.

[0090] Example:

[0091] For each question, the model selects the most relevant documents from the document corpus.

[0092] Calculate the immediate reward for selecting a document, including the similarity between the question and the document, the similarity between the document and the standard answer, etc.

[0093] Further, the online network module includes a state value function and an advantage function;

[0094] The state value function is used to estimate the overall value in the current state;

[0095] The advantage function is used to estimate the relative value of the current selected action relative to all other selected actions;

[0096] The first action value is the sum of the first value of the state value function and the second value of the advantage function calculated according to the currently selected action.

[0097] It should be noted that the online network module included in the embodiment of the present application is specifically:

[0098] Q(s,a;θ)=(A(s,a;θ a )-mean(A(s,a';θ a )))+V(s;θ v )

[0099] Among them, s represents the current state, that is, a combination of question + selected document. At the beginning, s0 = question, that is, in the initial state, there is only the string corresponding to the question; a represents the current action, that is, selecting a new document to add to s; a' represents all possible actions; s' represents the new state, that is, question + selected documents, and the selected documents include the new document; θ, θ a and θ v They represent the network parameters of the online network module, state value function and advantage function respectively.

[0100] The above formula shows that a dueling Q-Network Q(s,a;θ) is decomposed into a state value function V(s;θ v ) and advantage function A(s,a;θ a ). The mean of the advantage function is subtracted to increase stability.

[0101] The state value function estimates the overall value of being in state s, and the advantage function estimates the relative value of taking action a compared to other actions.

[0102] The Double-DuelingDQN model not only utilizes Double Q-Learning to improve learning stability by reducing estimation errors, but also improves the model's ability to understand the environment by accurately separating and processing state values ​​and advantage functions based on the Dueling network architecture. This combination can achieve more efficient and accurate decision-making in complex reinforcement learning tasks.

[0103] Furthermore, the target network module includes an instantaneous reward submodule, a penalty term submodule, and an estimation submodule;

[0104] The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a gap value between the generated answer to the training question generated in the new state and the standard answer;

[0105] The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty item submodule calculated according to the new state, and the estimated value of the best action in the new state by the estimation submodule.

[0106] It should be noted that the target network module Q target (s, a; θ') includes an instantaneous reward submodule, a penalty term submodule, and an estimation submodule, which can be specifically expressed as:

[0107]

[0108] r=α1Sim(new_doc,question)+α2Sim(new_doc,std_answer)+α3R(gen_answer,std_answer)-P

[0109] Where r represents the instantaneous reward, which includes the instantaneous reward submodule and the penalty submodule P; the instantaneous reward submodule is composed of the similarity between the selected document and the question α1Sim(new_doc,question), the similarity between the selected document and the standard answer α2Sim(new_doc,std_answer), and the difference value between the generated answer and the standard answer α3R(gen_answer,std_answer); δ∈[0,1] is the discount factor, which is used to adjust the importance of future rewards; It is the estimation submodule, which is the Q value estimation of the target network for the best action in the next state s'; is the best action chosen by the online network based on the current state s'.

[0110] Further, the penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein:

[0111] The first penalty formula is to calculate the first penalty value based on the absolute value of the model parameter change before and after the generator training and based on the first preset coefficient;

[0112] The second penalty formula is to calculate the second penalty value based on the square of the number of selected training documents and based on a second preset coefficient.

[0113] It should be noted that the setting of the penalty submodule is to prevent the algorithm from selecting too many documents, thereby affecting the learning efficiency.

[0114] For the two situations where generator training is required or not, two penalty value calculation methods are provided for the penalty submodule:

[0115] When no training is performed, the penalty term is proportional to the square of the number of selected documents, that is, P(train=False)=c(num selected_doc ) 2 , num selected_doc Indicates the number of selected documents;

[0116] When training, the penalty term is proportional to the size of the change in the parameters of the large language model (LLM), i.e. represents the parameters of the Large Language Model (LLM) after training, Represents the pre-trained parameters of the Large Language Model (LLM).

[0117] S104, iteratively updating the network parameters of the online network module and the target network module in the Double-DuelingDQN model according to a preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, thereby obtaining a trained Double-DuelingDQN model;

[0118] Furthermore, according to the preset loss function, the network parameters of the online network module and the target network module in the Double-DuelingDQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, and the trained Double-DuelingDQN model is obtained as follows:

[0119] Calculate the second action value of the new state through the target network model;

[0120] Taking the new state as the current state, the first action value of the current state is calculated through the online network module;

[0121] Calculate the loss value according to the mean square error between the first action value and the second action value;

[0122] According to the loss value, the network parameters of the online network module and the target network module are updated based on the gradient descent method until the loss value converges or the number of gains reaches the preset threshold, and the trained Double-DuelingDQN model is obtained.

[0123] It should be noted that the model performance can be improved by iteratively optimizing the model parameters.

[0124] Example: Using mean square error as the error between the online network Q(s,a;θ) and the target network Q target The loss function between (s,a;θ') is L(θ) = E[(Q(s,a;θ)-Q target (s,a;θ')) 2 ] and update the network parameters by gradient descent.

[0125] In addition, during the interaction process, the experience tuple (state s, action a, reward r, new state s', whether it ends) can also be stored in the experience replay buffer.

[0126] Periodically, a batch of experience tuples are randomly drawn from the buffer for training.

[0127] In the prior art, there is a reinforcement learning algorithm Double Q-Learning for reducing estimation errors and a Dueling network architecture for deep reinforcement learning. Specifically:

[0128] Double Q-Learning is a reinforcement learning technique used to reduce estimation errors in the Q-Learning algorithm. In the standard Q-Learning algorithm, a Q network is used to estimate the value of each action (i.e., Q value). However, this single estimation may lead to overestimation or underestimation of the value of certain actions, thus affecting the learning effect.

[0129] How it works: Double Q-Learning solves this problem by using two separate Q networks. These two networks are called the online network and the target network. The online network is used to select actions, while the target network is used to estimate the Q value of the next state.

[0130] Action selection: The online network selects the action with the highest Q value based on the current state.

[0131] Q-value estimation: The target network is used to estimate the Q-value of the selected action.

[0132] The key to this approach is that even if the online network is biased in selecting actions, the independent estimation of the target network can reduce the impact of this bias, thereby improving the stability and accuracy of the learning process.

[0133] Application scenarios: Double Q-Learning is widely used in reinforcement learning problems that need to deal with high estimation errors, especially in continuous action space or high-dimensional state space.

[0134] DuelingNetworkArchitectures is a deep neural network structure designed specifically to handle the estimation of value function and advantage function in reinforcement learning. This structure is particularly suitable for problems where there is a complex relationship between state value and action advantage.

[0135] How it works: The Dueling network architecture treats the state-value function and the advantage function separately and then combines them to compute the final Q-value. This design is based on the observation that in many environments, some states may be universally more valuable than other states, and some actions may be more advantageous in specific states relative to other actions.

[0136] State value function: estimates the expected reward on average for a given state, regardless of the action taken.

[0137] Advantage function: estimates the extra reward of taking an action relative to the average action.

[0138] The outputs of these two functions are combined through a combination layer to produce the Q-value for each action.

[0139] Application scenarios: Dueling network architecture performs well in a variety of reinforcement learning tasks, especially in those tasks with complex relationships between state values ​​and action advantages. This structure helps the model better understand and learn the dynamic characteristics of the environment.

[0140] In the embodiments of the present application, Double Q-Learning is creatively combined with Dueling Network Architectures to construct a Double-Dueling DQN model, so that the Double-Dueling DQN model can further optimize the performance of the reinforcement learning model.

[0141] Based on the first embodiment of this application, please refer to Figure 2 , Figure 2 is a method flow chart of the question-answering method based on retrieval enhancement in the embodiment of the present application, such as Figure 2As shown, the Double-Dueling DQN model trained by any model training method for retrieval enhancement in the first embodiment of the present application, the question-answering method based on retrieval enhancement includes:

[0142] S201, obtaining questions to be answered;

[0143] It should be noted that the questions input by the user are received and prepared for retrieval and answer generation.

[0144] Example: A user enters a question into the bank’s online customer service system: “Which fintech companies’ stocks have performed best recently?”

[0145] S202, taking the question to be answered and the document database containing several question-answer documents as inputs of the Double-DuelingDQN model, and obtaining the optimal question-answer document output by the Double-Dueling DQN model;

[0146] It should be noted that the trained Double-Dueling DQN model is used to retrieve the documents most relevant to the question from the document database.

[0147] Example:

[0148] The model is fed with user questions and a document database containing financial news, stock market reports, and company analysis.

[0149] The model selects the most relevant documents from the online web, such as a recent analyst report on the stock performance of a fintech company.

[0150] S203. Based on the question to be answered and the optimal question-answer document, an answer is generated through a preset generator output.

[0151] It should be noted that the answer is generated by a preset generator using the selected optimal document.

[0152] Example:

[0153] The answers are generated by Seq2Seq model using selected FinTech company stock analysis reports.

[0154] The generated answer might be: "Recently, the stocks of financial technology companies such as Square, PayPal, and Robinhood have performed well, rising more than the industry average."

[0155] Furthermore, after generating the answer through the preset generator output, it also includes:

[0156] Obtain the corrected answer after the user optimizes the generated answer;

[0157] The corrected answer is used as the standard answer to the question to be answered, combined with the optimal question and answer document, and added to the database used to train the Double-Dueling DQN model.

[0158] It should be noted that in order to improve the quality of answers, optimization is performed through user feedback, and the training of the Double-Dueling DQN model is strengthened to obtain user corrections to the generated answers.

[0159] Users can modify the generated answers to more accurately reflect their needs or provide additional information.

[0160] Example:

[0161] The user reviews the generated answer and suggests a revision: “Please provide a chart of the stock price trends of these companies over the past year.”

[0162] The user's corrected answer is added to the training database along with the relevant documents for use in future question answering.

[0163] Example:

[0164] Based on the user's revised request, the bank's analysts provide the corresponding stock price chart and add it to the training database along with the user's original question and selected documents.

[0165] In the financial field, training a Double-Dueling DQN model for retrieval-enhanced question answering can be applied to automated financial consulting services to help customers obtain detailed information about financial products such as stocks, funds, bonds, foreign exchange, etc. The following is a specific application example:

[0166] Application scenario: Automated financial consulting services

[0167] Objective: To develop an intelligent question-and-answer system that can answer customers' questions about various financial products and market dynamics, and provide timely and accurate financial information.

[0168] Step 1. Data collection: Financial product information: Collect data on various financial products, such as stock prices, fund net value, bond interest rates, etc.; Market news: Collect data from financial news websites and market analysis reports; Regulations and policies: Collect relevant information on financial regulations and policy changes.

[0169] Step 2: Data preprocessing: Clean the collected text data and remove irrelevant information; extract keywords, entities (such as company names, product names), and topics from the text; and annotate the data to train the model, for example, annotate the association between certain news reports and specific financial products.

[0170] Step 3: Model training:

[0171] Building the Double-Dueling DQN architecture:

[0172] Online Network: Used to select financial documents and news most relevant to the client's question.

[0173] Target network: used to stabilize the training process and reduce estimation errors.

[0174] Design a reward function: The reward function might be based on how relevant the documents selected by the model are to the customer’s question.

[0175] Training process:

[0176] Using financial product information, market news, and regulatory policies as input, the model is trained to learn how to retrieve the most relevant information based on customer questions.

[0177] The model learns which types of information are most relevant to the customer's question by constantly trying different combinations of information.

[0178] Step 4: Model evaluation and optimization:

[0179] Offline testing: Test the retrieval accuracy of the model on historical data.

[0180] Online simulation: Test the actual application effect of the model in a simulated consulting environment.

[0181] Parameter tuning: Adjust model parameters based on test results to optimize model performance.

[0182] Step 5: Deployment and monitoring:

[0183] System deployment: Deploy the trained model to the financial consulting platform.

[0184] Performance monitoring: Continuously monitor the performance of the model in actual applications and collect user feedback.

[0185] Model iteration: Regularly update models and data based on market changes and user feedback.

[0186] Step 6: System initialization: The financial consulting platform deploys the question-answering system and loads the trained Double-DuelingDQN model and the document database containing financial data.

[0187] Step 7: The customer asks questions about specific financial products through the platform, such as "Which technology stocks have performed best recently?"

[0188] Step 8. Document Retrieval: The Double-Dueling DQN model receives the question and the document database and performs a retrieval process to select the most relevant financial documents and news.

[0189] Step 9, answer generation: Using the retrieved information, generate answers through the generator, such as providing recent performance analysis of technology stocks.

[0190] Step 10: Client Feedback: The client reviews the answers and provides feedback, such as requesting more analysis on a specific company.

[0191] Step 11: Data update: Add customer feedback and new financial data to the training database.

[0192] Step 12: Model fine-tuning: In the next model training cycle, use the updated data to fine-tune the Double-Dueling DQN model.

[0193] Step 13: Continuous optimization: The system continuously monitors market changes and model performance, and performs optimization and updates regularly.

[0194] Through this embodiment, automated financial consulting services can provide customers with fast and accurate financial information, improving customer satisfaction and consulting efficiency.

[0195] See also Figure 3 , Figure 3 Schematic diagram of the structure of the model training device for retrieval enhancement in the embodiment of the present application, such as Figure 3 As shown, specifically:

[0196] A model building unit 301 is used to build a Double-Dueling DQN model to be trained, where the Double-Dueling DQN model includes an online network module and a target network module;

[0197] A training data acquisition unit 302 is used to acquire training data, where the training data includes training questions, a number of corresponding training documents, and standard answers corresponding to the training questions;

[0198] The model training unit 303 is used to train the Double-Dueling DQN model using training data, wherein:

[0199] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string consisting of the training question and the selected training document in the current state to obtain the new state of the online network module.

[0200] The Double-Dueling DQN model performs an estimated action through the target network module. The estimated action is to calculate the second action value of the new state based on the new state, based on the instantaneous reward submodule and the penalty submodule;

[0201] The loss optimization unit 304 is used to iteratively update the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to a preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, thereby obtaining a trained Double-Dueling DQN model.

[0202] Another embodiment of the present invention provides a model training device for retrieval enhancement, such as Figure 4 As shown, the device 10 includes:

[0203] One or more processors 110 and memory 120, Figure 3 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0204] The processor 110 is used to complete various control logics of the device 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0205] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for constructing a multilingual phoneme representation model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for constructing a multilingual phoneme representation model in the above method embodiment.

[0206] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0207] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0208] Build a Double-Dueling DQN model to be trained. The Double-Dueling DQN model includes an online network module and a target network module.

[0209] Obtain training data, which includes training questions, several corresponding training documents, and standard answers corresponding to the training questions;

[0210] The Double-Dueling DQN model is trained using training data, where:

[0211] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string consisting of the training question and the selected training document in the current state to obtain the new state of the online network module.

[0212] The Double-Dueling DQN model performs an estimated action through the target network module. The estimated action is to calculate the second action value of the new state based on the new state, based on the instantaneous reward submodule and the penalty submodule;

[0213] According to the preset loss function, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, thereby obtaining a trained Double-Dueling DQN model.

[0214] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, any one of the implementation methods of a model training method for retrieval enhancement described in the above embodiments is implemented.

[0215] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0216] In the embodiments of the present application, a model training, question-answering method, apparatus, device and medium for retrieval enhancement are provided, and a Double-Dueling DQN model is trained as a retriever using reinforcement learning. By continuously exploring and experimenting between different states, i.e., question + selected document, and the action of selecting a new document, the retrieval effect is made more accurate and reliable, and the document that best suits the answer is ultimately selected, rather than relying on calculating the similarity between the question and document vectors as in traditional methods, thereby improving the retrieval effect. At the same time, training the retriever can reduce the computational cost of training a large generator, and solves the problems existing in the current retrieval enhancement model, such as poor performance of the retriever in retrieval tasks in specific fields, lack of ability to fine-tune specific fields, and the technical problem that the deep semantic association between the document and the question is ignored in the document selection process, resulting in the selected document being not completely relevant to the question.

[0217] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0218] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0219] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0220] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0221] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0222] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0223] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0224] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0225] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

Claims

1. A model training method for retrieval enhancement, It is characterized in that include: Constructing a Double-Dueling DQN model to be trained, wherein the Double-Dueling DQN model includes an online network module and a target network module; Acquire training data, the training data including training questions, a number of corresponding training documents, and standard answers corresponding to the training questions; The Double-Dueling DQN model is trained using the training data, wherein: The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates a first action value of the selection action, wherein the selection action is to select a new training document and add it to the current string consisting of the training question in the current state and the selected training document to obtain a new state of the online network module; The Double-Dueling DQN model performs an estimation action through the target network module, wherein the estimation action is to calculate a second action value of the new state based on the instantaneous reward submodule and the penalty item submodule according to the new state; According to the preset loss function, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, thereby obtaining the trained Double-Dueling DQN model.

2. The model training method for retrieval enhancement according to claim 1, It is characterized in that The online network module includes a state value function and an advantage function; The state value function is used to estimate the overall value under the current state; The advantage function is used to estimate the relative value of the current selected action relative to all other selected actions; The first action value is the sum of the first value of the state value function and the second value of the advantage function calculated according to the selected action.

3. The model training method for retrieval enhancement according to claim 2, It is characterized in that The target network module includes an instantaneous reward submodule, a penalty term submodule and an estimation submodule; The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a gap value between the generated answer to the training question generated in the new state and the standard answer; The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty item submodule calculated according to the new state, and the estimated value of the best action under the new state by the estimation submodule.

4. The model training method for retrieval enhancement according to claim 3, It is characterized in that The penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein: The first penalty formula is to calculate the first penalty value based on the absolute value of the change in the model parameter before and after the generator training and based on the first preset coefficient; The second penalty formula is to calculate the second penalty value based on the square of the number of the selected training documents and based on a second preset coefficient.

5. The model training method for retrieval enhancement according to claim 4, It is characterized in that The method of iteratively updating the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to the preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, and obtaining the trained Double-Dueling DQN model is specifically as follows: Calculate the second action value of the new state by using the target network model; Taking the new state as the current state, and calculating the first action value of the current state through the online network module; Calculate a loss value according to a mean square error between the first action value and the second action value; According to the loss value, the network parameters of the online network module and the target network module are updated based on the gradient descent method until the loss value converges or the number of bands reaches a preset threshold, thereby obtaining the trained Double-Dueling DQN model.

6. A question answering method based on retrieval enhancement, It is characterized in that A Double-Dueling DQN model trained by the model training method for retrieval enhancement according to any one of claims 1 to 5, wherein the question-answering method based on retrieval enhancement comprises: Get questions to be answered; Taking the question to be answered and a document database containing a plurality of question-answer documents as inputs of the Double-DuelingDQN model, and obtaining an optimal question-answer document output by the Double-Dueling DQN model; Based on the question to be answered and the optimal question-answer document, an answer is generated through a preset generator output.

7. The question answering method based on retrieval enhancement according to claim 6, It is characterized in that After generating the answer by outputting the preset generator, the following steps are further included: Obtaining a corrected answer from the user after optimizing the generated answer; The corrected answer is used as the standard answer to the question to be answered, combined with the optimal question and answer document, and added to the database used to train the Double-Dueling DQN model.

8. A model training device for retrieval enhancement, It is characterized in that include: A model building unit, used to build a Double-Dueling DQN model to be trained, wherein the Double-Dueling DQN model includes an online network module and a target network module; A training data acquisition unit, used to acquire training data, wherein the training data includes training questions, a number of corresponding training documents, and standard answers corresponding to the training questions; A model training unit, configured to train the Double-Dueling DQN model using the training data, wherein: The Double-Dueling DQN model performs a selection action in the current state through the online network module, and calculates the first action value of the current selection action, wherein the selection action is to select a new training document and add it to the current string consisting of the training question in the current state and the selected training document, so as to obtain a new state of the online network module; The Double-Dueling DQN model performs an estimation action through the target network module, wherein the estimation action is to calculate a second action value of the new state based on the instantaneous reward submodule and the penalty item submodule according to the new state; The loss optimization unit is used to iteratively update the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to a preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, thereby obtaining the trained Double-Dueling DQN model.

9. A model training device for retrieval enhancement, It is characterized in that The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the model training method for retrieval enhancement described in any one of claims 1-5 according to the instructions in the program code.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the model training method for retrieval enhancement as described in any one of claims 1-5.

Images