Systems, methods, and media for retrieving enhanced model training, question answering methods, apparatuses, devices, and media

Abstract

The application discloses a kind of for retrieving enhanced model training, question and answer method, device, equipment and medium, using reinforcement learning has trained a Double-Dueling DQN model as retriever, by constantly exploring and testing between different states, i.e. question+selected document, and the action of selecting new document, make retrieval effect more accurate and reliable, finally select the document most suitable for answer, not like traditional method, rely on the similarity between the calculation question and document vector, improve retrieval effect, at the same time, training retriever can reduce the calculation cost brought by training large generator, solve the technical problems that the retriever of current retrieval enhancement model performs poorly in specific domain retrieval task, lack of fine-tuning ability for specific domain, while the deep semantic association between document and question is ignored in the document selection process, resulting in the selected document and question are not completely relevant.

Description

Technical Field

[0001] This application relates to the field of retrieval and question answering technology, and in particular to a model training, question answering method, apparatus, device, and medium for retrieval enhancement. Background Technology

[0002] Retrieval-Augmented Question Answering (RAQA) is an advanced question-answering system that combines traditional information retrieval techniques with natural language processing techniques. The core idea of ​​this technology is to retrieve the most relevant information fragments to the user's question from a large amount of text data, and then use this information to generate accurate and natural answers.

[0003] The working principle of search-enhanced question answering technology is as follows:

[0004] Retrieval Phase: The system first uses retrieval tools to retrieve documents or information fragments most relevant to the user's question from a large-scale text database. These retrieval tools are typically based on pre-trained language models, capable of understanding the semantic content of the question and retrieving documents based on semantic similarity.

[0005] Generation phase: The retrieved relevant documents or information fragments are then fed into the generator, which is responsible for integrating this information and generating a fluent and accurate natural language answer.

[0006] Currently, the mainstream retrieval enhancement models include:

[0007] Retrieval-Augmented Generation (RAG): RAG models acquire relevant information during the retrieval phase and combine this information to generate answers. It typically uses pre-trained models such as BERT as the retrieval engine and a sequence-to-sequence model during the generation phase.

[0008] Fusion in Decoder (FiD): The FiD model integrates retrieved information into the decoder, allowing the generator to directly utilize the retrieved document content when generating answers.

[0009] Atlas: Atlas is a large-scale retrieval augmentation model that handles complex question-answering tasks through sophisticated retrieval and generation mechanisms.

[0010] Current retrieval enhancement models have the following technical problems:

[0011] Limitations of retrieval machines: Current retrieval machines typically rely on pre-trained language models, which may perform poorly in domain-specific retrieval tasks because they are not fine-tuned for that domain.

[0012] Accuracy of document selection: In the process of document selection, some search engines may only focus on word meaning and frequency, while ignoring the deep semantic relationship between the document and the question, which may result in the selected documents not being completely relevant.

[0013] Computational resource requirements: Generators based on large pre-trained models require a lot of computational resources and time during fine-tuning, which poses a challenge for application scenarios with limited resources or those requiring rapid response.

[0014] Therefore, it is particularly important to find a method that can maintain high retrieval efficiency while reducing computational requirements. Summary of the Invention

[0015] This application provides a model training, question answering method, apparatus, device, and medium for retrieval enhancement, which solves the technical problems of current retrieval enhancement models, such as poor performance of the retrieval machine in specific domain retrieval tasks, lack of ability to fine-tune for specific domains, and neglect of the deep semantic relationship between documents and questions during document selection, resulting in the selected documents not being completely relevant to the questions.

[0016] In view of this, the first aspect of this application provides a model training method for retrieval enhancement, the method comprising:

[0017] Construct a Double-Dueling DQN model to be trained, the Double-Dueling DQN model including an online network module and a target network module;

[0018] Acquire training data, which includes training questions, several corresponding training documents, and standard answers to the training questions;

[0019] The Double-Dueling DQN model is trained using the training data, wherein:

[0020] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thereby obtaining the new state of the online network module.

[0021] The Double-Dueling DQN model performs an estimation action through the target network module. The estimation action is to calculate the second action value of the new state based on the new state, the instantaneous reward submodule, and the penalty term submodule.

[0022] According to the preset loss function, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, thus obtaining the trained Double-Dueling DQN model.

[0023] Optionally, the online network module includes a state value function and a dominance function;

[0024] The state value function is used to estimate the overall value in the current state;

[0025] The advantage function is used to estimate the relative value of the current choice action relative to all other choices action;

[0026] The first action value is the sum of the first value of the state value function calculated based on the current selected action and the second value of the advantage function.

[0027] Optionally, the target network module includes an instantaneous reward submodule, a penalty term submodule, and an estimation submodule;

[0028] The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a difference value between the generated answer of the training question generated in the new state and the standard answer;

[0029] The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty item submodule calculated based on the new state, and the estimate value of the optimal action in the new state by the estimation submodule.

[0030] Optionally, the penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein:

[0031] The first penalty formula is to calculate a first penalty value based on the absolute value of the change in model parameters before and after generator training, using a first preset coefficient.

[0032] The second penalty formula is to calculate a second penalty value based on the square of the number of selected training documents and a second preset coefficient.

[0033] Optionally, the step of iteratively updating the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to a preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, to obtain the trained Double-Dueling DQN model, specifically involves:

[0034] The second action value of the new state is calculated using the target network model;

[0035] The new state is taken as the current state, and the first action value of the current state is calculated through the online network module;

[0036] The loss value is calculated based on the mean square error between the first action value and the second action value;

[0037] Based on the loss value, the network parameters of the online network module and the target network module are updated using gradient descent until the loss value converges or the number of iterations reaches a preset threshold, thus obtaining the trained Double-Dueling DQN model.

[0038] A second aspect of this application provides a retrieval-enhanced question answering method, using a Double-Dueling DQN model trained with the retrieval-enhanced model training method described in any one of the first aspects of this application. The retrieval-enhanced question answering method includes:

[0039] Get the questions to be answered;

[0040] The question to be answered and a document database containing several question-and-answer documents are used as input to the Double-Dueling DQN model to obtain the optimal question-and-answer document output by the Double-Dueling DQN model.

[0041] Based on the question to be answered and the optimal question-and-answer document, the answer is generated by a preset generator.

[0042] Optionally, after generating the answer through the preset generator, the process further includes:

[0043] Obtain the user's corrected answer after optimizing the generated answer;

[0044] The corrected answer is used as the standard answer to the question to be answered, and combined with the optimal question-and-answer document, it is added to the database used to train the Double-Dueling DQN model.

[0045] A third aspect of this application provides a model training apparatus for retrieving enhancements, the apparatus comprising:

[0046] The model building unit is used to build a Double-Dueling DQN model to be trained, wherein the Double-Dueling DQN model includes an online network module and a target network module;

[0047] The training data acquisition unit is used to acquire training data, which includes training questions, several corresponding training documents, and standard answers to the training questions.

[0048] The model training unit is used to train the Double-Dueling DQN model using the training data, wherein:

[0049] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thereby obtaining the new state of the online network module.

[0050] The Double-Dueling DQN model performs an estimation action through the target network module. The estimation action is to calculate the second action value of the new state based on the new state, the instantaneous reward submodule, and the penalty term submodule.

[0051] The loss optimization unit is used to iteratively update the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to a preset loss function, until the preset loss function converges or the number of iterations reaches a preset threshold, so as to obtain the trained Double-Dueling DQN model.

[0052] Optionally, the online network module includes a state value function and a dominance function;

[0053] The state value function is used to estimate the overall value in the current state;

[0054] The advantage function is used to estimate the relative value of the current choice action relative to all other choices action;

[0055] The first action value is the sum of the first value of the state value function calculated based on the current selected action and the second value of the advantage function.

[0056] Optionally, the target network module includes an instantaneous reward submodule, a penalty term submodule, and an estimation submodule;

[0057] The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a difference value between the generated answer of the training question generated in the new state and the standard answer;

[0058] The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty item submodule calculated based on the new state, and the estimate value of the optimal action in the new state by the estimation submodule.

[0059] Optionally, the penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein:

[0060] The first penalty formula is to calculate a first penalty value based on the absolute value of the change in model parameters before and after generator training, using a first preset coefficient.

[0061] The second penalty formula is to calculate a second penalty value based on the square of the number of selected training documents and a second preset coefficient.

[0062] Optionally, the loss optimization unit is specifically used for:

[0063] The second action value of the new state is calculated using the target network model;

[0064] The new state is taken as the current state, and the first action value of the current state is calculated through the online network module;

[0065] The loss value is calculated based on the mean square error between the first action value and the second action value;

[0066] Based on the loss value, the network parameters of the online network module and the target network module are updated using gradient descent until the loss value converges or the number of iterations reaches a preset threshold, thus obtaining the trained Double-Dueling DQN model.

[0067] A fourth aspect of this application provides a model training apparatus for retrieving enhancements, the apparatus comprising a processor and a memory:

[0068] The memory is used to store program code and transmit the program code to the processor;

[0069] The processor is configured to execute, according to instructions in the program code, the steps of the method for retrieving enhanced model training as described in the first aspect above.

[0070] A fifth aspect of this application provides a computer-readable storage medium for storing program code for performing the method described in the first aspect above.

[0071] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0072] This application provides a model training, question answering method, apparatus, device, and medium for retrieval enhancement. It employs reinforcement learning to train a Double-Dueling DQN model as the retrieval engine. By continuously exploring and experimenting with different states—question + selected document and the action of selecting a new document—the retrieval results are made more accurate and reliable. Ultimately, it selects the document most suitable for the answer, rather than relying on calculating the similarity between the question and document vectors as in traditional methods, thus improving retrieval performance. Simultaneously, training the retrieval engine reduces the computational cost of training large generators. This addresses the technical problems of current retrieval enhancement models, such as poor performance of the retrieval engine in specific domain retrieval tasks, lack of ability to fine-tune for specific domains, and neglect of the deep semantic relationship between documents and questions during document selection, leading to selected documents that are not entirely relevant to the question. Attached Figure Description

[0073] Figure 1 This is a flowchart of a method for retrieving enhanced model training methods in an embodiment of this application;

[0074] Figure 2 This is a flowchart of the retrieval-enhanced question-answering method in the embodiments of this application;

[0075] Figure 3 This is a schematic diagram of the structure of the model training device used for retrieval enhancement in an embodiment of this application;

[0076] Figure 4 This is a schematic diagram of the structure of the model training device used for retrieval enhancement in an embodiment of this application. Detailed Implementation

[0077] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0078] This application designs a model training, question answering method, device, equipment, and medium for retrieval enhancement, which solves the technical problems of current retrieval enhancement models, such as poor performance of the retrieval machine in specific domain retrieval tasks, lack of ability to fine-tune for specific domains, and neglect of the deep semantic relationship between documents and questions during document selection, resulting in the selected documents not being completely relevant to the questions.

[0079] For easier understanding, please refer to Figure 1 , Figure 1 This is a flowchart of a method for retrieving and enhancing model training methods in an embodiment of this application, such as... Figure 1 As shown, specifically:

[0080] S101. Construct the Double-Dueling DQN model to be trained. The Double-Dueling DQN model includes an online network module and a target network module.

[0081] It should be noted that the Double-Dueling DQN model is designed and initialized, including the online network and the target network.

[0082] Example: Both the online network and the target network use deep neural networks and have similar structures, but different parameters. The online network is used for decision-making, and the target network is used to stabilize the training process.

[0083] S102. Obtain training data, which includes training questions, several corresponding training documents, and standard answers to the training questions.

[0084] It should be noted that the training requires questions, documents, and standard answers.

[0085] Example: Randomly select 1000 questions, their related documents, and standard answers from the question-and-answer database as the training set.

[0086] S103. Train the Double-Dueling DQN model using training data, where:

[0087] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thus obtaining the new state of the online network module.

[0088] The Double-Dueling DQN model performs an estimated action through the target network module. The estimated action is calculated based on the new state, using the instantaneous reward submodule and the penalty term submodule to obtain the second action value of the new state.

[0089] It should be noted that the training data is used to train the model so that it can select the most relevant documents.

[0090] Example:

[0091] For each problem, the model selects the most relevant document from the document library.

[0092] The system calculates immediate rewards for selecting documents, including the similarity between the question and the document, and the similarity between the document and the standard answer.

[0093] Furthermore, the online network module includes a state value function and a dominance function;

[0094] The state-value function is used to estimate the total value in the current state;

[0095] The advantage function is used to estimate the relative value of the current action choice relative to all other actions choice.

[0096] The first action value is the sum of the first value of the state value function calculated based on the currently selected action and the second value of the advantage function.

[0097] It should be noted that the online network module included in the embodiments of this application specifically includes:

[0098] Q(s,a;θ)=(A(s,a;θ a )-mean(A(s,a';θ a )))+V(s;θ v )

[0099] Where s represents the current state, which is a combination of the question and the selected document. Initially, s0 = question, meaning the initial state only contains the string corresponding to the question; a represents the current action, which is to select a new document and add it to s; a' represents all possible actions; s' represents the new state, which is the question plus the selected document, including the new document; θ, θ' a and θ v These represent the network parameters for the online network module, the state value function, and the dominance function, respectively.

[0100] The above formula indicates that a dueling Q-Network Q(s, a; θ) is decomposed into a state-value function V(s; θ). v ) and the advantage function A(s,a;θ) a In this process, the mean of the dominance function is subtracted to increase stability.

[0101] The state value function estimates the overall value in state s, while the dominance function estimates the relative value of taking action a relative to other actions.

[0102] The Double-Dueling DQN model leverages Double Q-Learning to improve learning stability by reducing estimation errors, while also utilizing the Dueling network architecture to enhance the model's understanding of the environment by accurately separating and processing state values ​​and advantage functions. This combination enables more efficient and accurate decision-making in complex reinforcement learning tasks.

[0103] Furthermore, the target network module includes an instantaneous reward submodule, a penalty term submodule, and an estimation submodule;

[0104] The instantaneous reward submodule includes: the first similarity between the training question and the new training document, the second similarity between the new training document and the standard answer, and the difference between the generated answer of the training question generated in the new state and the standard answer;

[0105] The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty item submodule calculated based on the new state, and the estimate value of the optimal action in the new state by the estimation submodule.

[0106] It should be noted that the target network module Q target (s,a;θ') includes an instantaneous reward submodule, a penalty term submodule, and an estimation submodule, specifically represented as follows:

[0107]

[0108] r=α1Sim(new_doc,question)+α2Sim(new_doc,std_answer)+α3R(gen_answer,std_answer)-P

[0109] Where r represents the instantaneous reward, and r includes an instantaneous reward submodule and a penalty submodule P; the instantaneous reward submodule consists of the similarity between the selected document and the question α1Sim(new_doc,question), the similarity between the selected document and the standard answer α2Sim(new_doc,std_answer), and the difference between the generated answer and the standard answer α3R(gen_answer,std_answer); δ∈[0,1] is a discount factor used to adjust the importance of future rewards; The estimation submodule is the Q-value estimate of the target network for the best action in the next state s'; It is the best action chosen by the online network based on the current state s'.

[0110] Furthermore, the penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein:

[0111] The first penalty formula is to calculate the first penalty value based on the absolute value of the change in model parameters before and after generator training, using a first preset coefficient.

[0112] The second penalty formula is to calculate the second penalty value based on the square of the number of selected training documents and a second preset coefficient.

[0113] It should be noted that the penalty item submodule is set to prevent the algorithm from selecting too many documents, thereby affecting learning efficiency.

[0114] For the two scenarios of whether generator training is required or not, two methods for calculating the penalty value are provided in the penalty term submodule:

[0115] When no training is performed, the penalty term is proportional to the square of the number of selected documents, i.e., P(train=False)=c(num) selected_doc ) 2 ,num selected_doc Indicates the number of documents selected;

[0116] During training, the penalty term is proportional to the magnitude of the change in the parameters of the Large Language Model (LLM), i.e. This represents the parameters after training a large language model (LLM). This represents the parameters before training a large language model (LLM).

[0117] S104. According to the preset loss function, iteratively update the network parameters of the online network module and the target network module in the Double-DuelingDQN model until the preset loss function converges or the number of iterations reaches the preset threshold, and obtain the trained Double-DuelingDQN model.

[0118] Furthermore, based on the preset loss function, the network parameters of the online network module and the target network module in the Double-DuelingDQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, resulting in the trained Double-DuelingDQN model as follows:

[0119] Calculate the second action value of the new state using the target network model;

[0120] The new state is used as the current state, and the first action value of the current state is calculated through the online network module.

[0121] The loss value is calculated based on the mean square error between the first action value and the second action value.

[0122] Based on the loss value, the network parameters of the online network module and the target network module are updated using the gradient descent method until the loss value converges or the number of iterations reaches a preset threshold, thus obtaining the trained Double-DuelingDQN model.

[0123] It should be noted that model performance is improved by iteratively optimizing model parameters.

[0124] Example: Using mean squared error as the online network Q(s,a;θ) and the target network Q target The loss function between (s,a;θ') is L(θ) = E[(Q(s,a;θ) - Q]. target (s,a;θ')) 2 The network parameters are updated using gradient descent.

[0125] In addition, during the interaction, the experience tuple (state s, action a, reward r, new state s', whether it has ended) can be stored in the experience replay buffer.

[0126] A batch of experience tuples is randomly selected from the buffer periodically for training.

[0127] In the prior art, there exists a reinforcement learning algorithm called Double Q-Learning to reduce estimation errors and a Dueling network architecture for deep reinforcement learning, specifically:

[0128] Double Q-Learning is a reinforcement learning technique used to reduce estimation errors in Q-Learning algorithms. In standard Q-Learning algorithms, a single Q-network is used to estimate the value of each action (i.e., the Q-value). However, this single estimation may lead to overestimating or underestimating the value of some actions, thus affecting the learning outcome.

[0129] How it works: Double Q-Learning solves this problem by using two independent Q-networks. These two networks are called the online network and the target network, respectively. The online network is used to select the action, while the target network is used to estimate the Q-value of the next state.

[0130] Action selection: The online network selects the action with the highest Q value based on the current state.

[0131] Q-value estimation: The target network is used to estimate the Q-value of the selected action.

[0132] The key to this approach is that even if the online network is biased in selecting actions, the independent estimation of the target network can reduce the impact of this bias, thereby improving the stability and accuracy of the learning process.

[0133] Application scenarios: Double Q-Learning is widely used in reinforcement learning problems that require handling high estimation errors, especially in problems with continuous action spaces or high-dimensional state spaces.

[0134] Dueling Network Architectures are deep neural network architectures specifically designed to handle the estimation of value functions and advantage functions in reinforcement learning. This architecture is particularly suitable for problems where there are complex relationships between state values ​​and action advantages.

[0135] How it works: The Dueling network architecture processes the state-value function and the advantage function separately, and then combines them to calculate the final Q-value. This design is based on the observation that in many environments, certain states may be generally more valuable than others, and certain actions may be more advantageous in a particular state compared to others.

[0136] State-value function: estimates the expected return under average conditions, regardless of the action taken, in a given state.

[0137] Advantage function: estimates the additional reward of taking a certain action relative to the average action.

[0138] The outputs of these two functions are combined through a combination layer to produce the Q-value for each action.

[0139] Application scenarios: The Dueling network architecture performs exceptionally well across a variety of reinforcement learning tasks, particularly those with complex relationships between state values ​​and action advantages. This structure helps the model better understand and learn the dynamic characteristics of the environment.

[0140] In this embodiment, Double Q-Learning is creatively combined with Dueling Network Architectures to construct a Double-Dueling DQN model, which allows the Double-Dueling DQN model to further optimize the performance of reinforcement learning models.

[0141] Based on the first embodiment of this application, please refer to further details. Figure 2 , Figure 2 This is a flowchart of the retrieval-enhanced question-answering method in the embodiments of this application, such as... Figure 2As shown, the Double-Dueling DQN model trained using any of the retrieval enhancement model training methods of the first embodiment of this application includes the following retrieval enhancement-based question answering methods:

[0142] S201. Obtain the questions to be answered;

[0143] It should be noted that the system receives user input questions and prepares them for retrieval and answer generation.

[0144] Example: A user enters the question "Which fintech companies have had the best stock performance recently?" into a bank's online customer service system.

[0145] S202. Take the questions to be answered and the document database containing several question-and-answer documents as input to the Double-Dueling DQN model to obtain the optimal question-and-answer document output by the Double-Dueling DQN model;

[0146] It should be noted that a pre-trained Double-Dueling DQN model is used to retrieve the most relevant documents from the document database.

[0147] Example:

[0148] Input users' questions and a database of documents containing financial news, stock market reports, and company analyses into the model.

[0149] The model selects the most relevant documents from online networks, such as recent analysis reports on the stock performance of fintech companies.

[0150] S203. Based on the question to be answered and the optimal question-and-answer document, generate the answer through a preset generator.

[0151] It should be noted that the answer is generated using the selected optimal document and a preset generator.

[0152] Example:

[0153] Using selected fintech company stock analysis reports, generate answers using a Seq2Seq model.

[0154] The generated answer might be: "Recently, stocks of fintech companies such as Square, PayPal, and Robinhood have outperformed the industry average."

[0155] Furthermore, after generating the answer using the preset generator, it also includes:

[0156] Obtain the user's corrected answer after optimizing the generated answer;

[0157] The corrected answers are used as the standard answers to the questions to be answered, and combined with the best question-and-answer document, they are added to the database used to train the Double-Dueling DQN model.

[0158] It should be noted that, in order to improve the quality of answers, optimization is carried out based on user feedback, and the training of the Double-Dueling DQN model is strengthened to obtain user corrections on the generated answers.

[0159] Users can modify the generated answers to more accurately reflect their needs or provide additional information.

[0160] Example:

[0161] Users reviewed the generated answers and suggested revisions: "Please provide stock price charts for these companies over the past year."

[0162] The user's corrected answers and related documents are added to the training database for use in future question-and-answer sessions.

[0163] Example:

[0164] Based on the user's correction request, the bank's analysts provide corresponding stock price charts and add them, along with the user's original question and selected documents, to the training database.

[0165] In the financial sector, training a Double-Dueling DQN model for enhanced question answering retrieval can be applied to automate financial advisory services, helping clients obtain detailed information about financial products such as stocks, funds, bonds, and foreign exchange. Here is a specific application example:

[0166] Application scenario: Automated financial advisory services

[0167] Objective: To develop an intelligent question-and-answer system capable of answering customers' questions about various financial products and market dynamics, providing timely and accurate financial information.

[0168] Step 1, Data Collection: Financial Product Information: Collect data on various financial products, such as stock prices, fund net asset values, bond interest rates, etc.; Market News: Collect data from financial news websites and market analysis reports; Laws and Policies: Collect information on relevant financial laws and policy changes.

[0169] Step 2, Data Preprocessing: Clean the collected text data to remove irrelevant information; extract keywords, entities (such as company names and product names) and topics from the text; label the data to train the model, for example, label the association between certain news reports and specific financial products.

[0170] Step 3, Model Training:

[0171] Building a Double-Dueling DQN architecture:

[0172] Online network: Used to select financial documents and news most relevant to customer issues.

[0173] Target network: Used to stabilize the training process and reduce estimation errors.

[0174] Design a reward function: The reward function may be based on the relevance of the document selected by the model to the customer's problem.

[0175] Training process:

[0176] Using financial product information, market news, and regulations as input, the model is trained to learn how to retrieve the most relevant information based on customer questions.

[0177] The model learns which types of information are most relevant to customer problems by constantly trying different combinations of information.

[0178] Step 4: Model Evaluation and Optimization

[0179] Offline testing: Test the model's retrieval accuracy on historical data.

[0180] Online simulation: Test the model's practical application effectiveness in a simulated consulting environment.

[0181] Parameter tuning: Adjust model parameters based on test results to optimize model performance.

[0182] Step 5, Deployment and Monitoring:

[0183] System Deployment: Deploy the trained model to the financial advisory platform.

[0184] Performance monitoring: Continuously monitor the model's performance in real-world applications and collect user feedback.

[0185] Model iteration: Update the model and data regularly based on market changes and user feedback.

[0186] Step 6: System Initialization: The financial consultation platform deploys the question-and-answer system, loads the trained Double-Dueling DQN model and the document database containing financial data.

[0187] Step 7: Customers inquire about specific financial products through the platform, such as "Which tech stocks have performed best recently?"

[0188] Step 8, Document Retrieval: The Double-Dueling DQN model receives a database of questions and documents, performs a retrieval process, and selects the most relevant financial documents and news.

[0189] Step 9, Answer Generation: Using the retrieved information, generate an answer through a generator, such as providing an analysis of the recent performance of technology stocks.

[0190] Step 10, Customer Feedback: The customer reviews the answer and provides feedback, such as requesting more analysis about a specific company.

[0191] Step 11, Data Update: Add customer feedback and new financial data to the training database.

[0192] Step 12, Model Fine-tuning: In the next model training cycle, fine-tune the Double-Dueling DQN model using the updated data.

[0193] Step 13, Continuous Optimization: The system continuously monitors market changes and model performance, and performs optimization and updates regularly.

[0194] Through this example, automated financial advisory services can provide customers with fast and accurate financial information, improving customer satisfaction and advisory efficiency.

[0195] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of the model training device used for retrieval enhancement in an embodiment of this application, as shown below. Figure 3 As shown, specifically:

[0196] Model building unit 301 is used to build the Double-Dueling DQN model to be trained. The Double-Dueling DQN model includes an online network module and a target network module.

[0197] The training data acquisition unit 302 is used to acquire training data, which includes training questions, several corresponding training documents, and standard answers to the training questions.

[0198] Model training unit 303 is used to train the Double-Dueling DQN model using training data, wherein:

[0199] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thus obtaining the new state of the online network module.

[0200] The Double-Dueling DQN model performs an estimated action through the target network module. The estimated action is calculated based on the new state, using the instantaneous reward submodule and the penalty term submodule to obtain the second action value of the new state.

[0201] The loss optimization unit 304 is used to iteratively update the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to the preset loss function until the preset loss function converges or the number of iterations reaches the preset threshold, so as to obtain the trained Double-Dueling DQN model.

[0202] Another embodiment of the present invention provides a model training device for retrieval enhancement, such as... Figure 4 As shown, device 10 includes:

[0203] One or more processors 110 and memory 120, Figure 3 The following description uses a processor 110 as an example. The processor 110 and the memory 120 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0204] Processor 110 is used to perform various control logics of device 10, and can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 110 can also be any conventional processor, microprocessor, or state machine. Processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0205] The memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the method for constructing the multilingual phoneme representation model in this embodiment of the invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions, and units stored in the memory 120, thereby implementing the method for constructing the multilingual phoneme representation model in the above-described method embodiment.

[0206] The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10. Furthermore, the memory 120 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include memory remotely located relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0207] One or more units are stored in memory 120, and when executed by one or more processors 110, perform the following steps:

[0208] Construct a Double-Dueling DQN model to be trained. The Double-Dueling DQN model includes an online network module and a target network module.

[0209] Acquire training data, which includes training questions, several corresponding training documents, and standard answers to the training questions;

[0210] The Double-Dueling DQN model is trained using training data, where:

[0211] The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thus obtaining the new state of the online network module.

[0212] The Double-Dueling DQN model performs an estimated action through the target network module. The estimated action is calculated based on the new state, using the instantaneous reward submodule and the penalty term submodule to obtain the second action value of the new state.

[0213] Based on the preset loss function, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated until the preset loss function converges or the number of iterations reaches the preset threshold, thus obtaining the trained Double-Dueling DQN model.

[0214] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by one or more processors, they implement any of the above-described methods for retrieval enhancement model training.

[0215] As examples, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The memory components or memories disclosed in the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0216] This application provides a model training, question answering method, apparatus, device, and medium for retrieval enhancement. A Double-Dueling DQN model is trained using reinforcement learning as the retrieval engine. By continuously exploring and experimenting between different states—question + selected document and the action of selecting a new document—the retrieval effect becomes more accurate and reliable. Ultimately, it selects the document most suitable for the answer, rather than relying on calculating the similarity between the question and document vectors as in traditional methods, thus improving retrieval performance. Simultaneously, training the retrieval engine reduces the computational cost of training large generators. This addresses the technical problems of current retrieval enhancement models, such as poor performance of the retrieval engine in specific domain retrieval tasks, lack of ability to fine-tune for specific domains, and neglect of the deep semantic relationship between documents and questions during document selection, resulting in selected documents that are not entirely relevant to the question.

[0217] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0218] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0219] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0220] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0221] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0222] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0223] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0224] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0225] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A model training method for retrieval enhancement, characterized in that, include: Construct a Double-Dueling DQN model to be trained, the Double-Dueling DQN model including an online network module and a target network module; Acquire training data, which includes training questions, several corresponding training documents, and standard answers to the training questions; The Double-Dueling DQN model is trained using the training data, wherein: The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thereby obtaining the new state of the online network module. The Double-Dueling DQN model performs an estimation action through the target network module. The estimation action is to calculate a second action value for the new state based on the instantaneous reward submodule and the penalty submodule. The target network module includes an instantaneous reward submodule, a penalty submodule, and an estimation submodule. The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a difference between the generated answer of the training question generated in the new state and the standard answer. The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty submodule calculated based on the new state, and the estimated value of the best action in the new state by the estimation submodule. According to the preset loss function, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, thus obtaining the trained Double-Dueling DQN model.

2. The model training method for retrieval enhancement according to claim 1, characterized in that, The online network module includes a state value function and a dominance function; The state value function is used to estimate the overall value in the current state; The advantage function is used to estimate the relative value of the current choice action relative to all other choices action; The first action value is the sum of the first value of the state value function calculated based on the selected action and the second value of the advantage function.

3. The model training method for retrieval enhancement according to claim 1, characterized in that, The penalty term submodule includes a first penalty formula for when the generator performs training and a second penalty formula for when the generator does not perform training, wherein: The first penalty formula is to calculate a first penalty value based on the absolute value of the change in model parameters before and after generator training, using a first preset coefficient. The second penalty formula is to calculate a second penalty value based on the square of the number of selected training documents and a second preset coefficient.

4. The model training method for retrieval enhancement according to claim 1, characterized in that, The step of iteratively updating the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to a preset loss function until the preset loss function converges or the number of iterations reaches a preset threshold, to obtain the trained Double-Dueling DQN model, is as follows: The second action value of the new state is calculated using the target network model; The new state is taken as the current state, and the first action value of the current state is calculated through the online network module; The loss value is calculated based on the mean square error between the first action value and the second action value; Based on the loss value, the network parameters of the online network module and the target network module are updated using gradient descent until the loss value converges or the number of iterations reaches a preset threshold, thus obtaining the trained Double-Dueling DQN model.

5. A retrieval-enhanced question-answering method, characterized in that, The Double-Dueling DQN model trained using the retrieval enhancement model training method according to any one of claims 1 to 4, wherein the retrieval enhancement-based question answering method comprises: Get the questions to be answered; The question to be answered and a document database containing several question-and-answer documents are used as input to the Double-Dueling DQN model to obtain the optimal question-and-answer document output by the Double-Dueling DQN model. Based on the question to be answered and the optimal question-and-answer document, the answer is generated using a preset generator. The training of the Double-Dueling DQN model includes: The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thereby obtaining the new state of the online network module. The Double-Dueling DQN model performs an estimation action through the target network module. The estimation action is to calculate a second action value for the new state based on the instantaneous reward submodule and the penalty submodule. The target network module includes an instantaneous reward submodule, a penalty submodule, and an estimation submodule. The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a difference between the generated answer of the training question generated in the new state and the standard answer. The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty submodule calculated based on the new state, and the estimated value of the best action in the new state by the estimation submodule. According to the preset loss function, the network parameters of the online network module and the target network module in the Double-Dueling DQN model are iteratively updated until the preset loss function converges or the number of iterations reaches a preset threshold, thus obtaining the trained Double-Dueling DQN model.

6. The question-answering method based on retrieval enhancement according to claim 5, characterized in that, After generating the answer through the preset generator, the process also includes: Obtain the user's corrected answer after optimizing the generated answer; The corrected answer is used as the standard answer to the question to be answered, and combined with the optimal question-and-answer document, it is added to the database used to train the Double-Dueling DQN model.

7. A model training apparatus for retrieval enhancement, characterized in that, include: The model building unit is used to build a Double-Dueling DQN model to be trained, wherein the Double-Dueling DQN model includes an online network module and a target network module; The training data acquisition unit is used to acquire training data, which includes training questions, several corresponding training documents, and standard answers to the training questions. The model training unit is used to train the Double-Dueling DQN model using the training data, wherein: The Double-Dueling DQN model performs a selection action in the current state through the online network module and calculates the first action value of the current selection action. The selection action is to select a new training document and add it to the current string composed of the training question and the selected training document in the current state, thereby obtaining the new state of the online network module. The Double-Dueling DQN model performs an estimation action through the target network module. The estimation action is to calculate a second action value for the new state based on the instantaneous reward submodule and the penalty submodule. The target network module includes an instantaneous reward submodule, a penalty submodule, and an estimation submodule. The instantaneous reward submodule includes: a first similarity between the training question and the new training document, a second similarity between the new training document and the standard answer, and a difference between the generated answer of the training question generated in the new state and the standard answer. The second action value is the sum of the difference between the reward value of the instantaneous reward submodule and the penalty value of the penalty submodule calculated based on the new state, and the estimated value of the best action in the new state by the estimation submodule. The loss optimization unit is used to iteratively update the network parameters of the online network module and the target network module in the Double-Dueling DQN model according to a preset loss function, until the preset loss function converges or the number of iterations reaches a preset threshold, so as to obtain the trained Double-Dueling DQN model.

8. A model training device for retrieval enhancement, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the model training method for retrieving enhancements as described in any one of claims 1-4 according to instructions in the program code.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the model training method for retrieving enhancements as described in any one of claims 1-4.

Images