Power business scene question and answer method and system based on reinforcement learning

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly discloses a power business scene question and answer method and system based on reinforcement learning, and the method comprises the steps: training each pre-configured reward function through obtaining a sorted and labeled power question and answer training data set; the method comprises the following steps: selecting reward functions with better performance for weighted combination according to the loss value condition of each reward function, constructing a multi-dimensional dynamic reward model, then simulating a human sorting thought by utilizing the multi-dimensional dynamic reward model, and optimizing a large model for electric power customer service questions and answers in combination with a corresponding enhancement strategy, so as to improve the power customer service questions and answers. The large model is driven to realize accurate parameter adjustment in a power business question-answering scene, and the power customer service question-answering performance is improved. According to the method, the reward models of the corresponding question and answer quality evaluation dimensions are dynamically combined, and a dynamic weight mechanism is introduced, so that a large-model power business question and answer result can better meet user requirements, and the question and answer accuracy and the user interaction experience are improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a question-answering method and system for power business scenarios based on reinforcement learning. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, customer service question-and-answer services in the power industry are gradually shifting from manual operations to large-scale model-based services. However, large-scale models require iterative optimization during their implementation to better adapt to specific business scenarios and output Q&A results that better meet user needs. Optimizing large-scale customer service Q&A can be accomplished using a reward model. This is a mechanism that quantitatively assesses and scores the quality of large-scale model outputs, aiming to guide the large-scale model in optimizing its behavior. By incorporating human feedback into the reinforcement learning process, it helps the large-scale model learn how to make optimal decisions. Traditional reward models rely on a single metric (such as manual labeling matching) for large-scale model strategy optimization, resulting in poor balance. The accuracy of optimized Q&A and the user interaction experience also need to be improved. Summary of the Invention

[0003] The purpose of the present invention is to provide a power business scenario question-answering method and system based on reinforcement learning to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: First, we provide a reinforcement learning-based question-answering method for power business scenarios, including: Obtaining a set of electric power question and answer training data groups, wherein the set of electric power question and answer training data groups includes a plurality of electric power question and answer training data groups, each of which includes a corresponding electric power business scenario question sample and a plurality of answer samples associated with the electric power business scenario question sample; Retrieve several pre-configured reward functions from the reward function library, and train each reward function using each power question-answering training data set to obtain the corresponding trained reward function, and determine the target loss value corresponding to each trained reward function; Select several reward functions from all the post-training reward functions as target reward functions according to the target loss value of each post-training reward function; Perform a weighted combination of each objective reward function based on its target loss value to obtain a multi-dimensional dynamic reward model; When the pre-trained large model answers questions in power business scenarios, a multi-dimensional dynamic reward model is used to evaluate the rewards of the large model's output answer instances and obtain the corresponding reward evaluation results. Perform reinforcement learning on the large model based on the reward evaluation results to obtain an optimized large model; The optimized large model is used to output question and answer examples of power business scenario problems.

[0005] In one possible design, each answer sample associated with the corresponding power business scenario question sample in each power question and answer training data group has been sorted and labeled.

[0006] In one possible design, the reward functions are trained using the power question-answering training data sets to obtain corresponding trained reward functions, including: Based on the order of the answer samples associated with the power business scenario question samples in the power question answering training data set, a triple data set (x, y q ,y h ), where x represents the problem sample of the power business scenario, y q Represents the top answer samples in the power question answering training data set, y h Represents the answer samples ranked last in the power question answering training data set; By contrastive learning and training the corresponding reward function, it generates a corresponding scalar reward value for each answer sample associated with the power business scenario question sample in the power question answering training data set, thereby determining y q The corresponding scalar reward value R(x, y q ) and y h The corresponding scalar reward value R(x, y h ); Determine the loss function L corresponding to the reward function during the training process of the power question answering training data set RM = -logσ[R(x,y q )-R(x,y h )], where σ represents the Sigmoid function. The training of the reward function for the power question-answering training data set ends until the loss function meets the set convergence condition, and the corresponding convergence loss value is obtained. After completing the training of the reward function for each power question-answering training data set, the corresponding trained reward function is obtained.

[0007] In one possible design, determining the target loss value corresponding to each post-training reward function includes: The convergence loss values ​​obtained by training the reward function for each power question-answering training data set are averaged to obtain the target loss value of the corresponding reward function after training.

[0008] In one possible design, the target loss value of each post-training reward function is used to select several reward functions from all post-training reward functions as target reward functions, including: Arrange the post-training reward functions in ascending order of target loss value, and select the top post-training reward functions as the target reward functions.

[0009] In one possible design, the target loss values ​​of the target reward functions are weightedly combined to obtain a multi-dimensional dynamic reward model, including: Arrange each objective reward function in ascending order of target loss value, and assign a weight coefficient according to the order of the corresponding objective reward function; Based on the weight coefficients of each objective reward function, each objective reward function is weightedly combined to obtain a multidimensional dynamic reward model.

[0010] In one possible design, the reward evaluation result is the instance reward value R θ (x', y'), where x' represents the question instance of the power business scenario and y' represents the answer instance.

[0011] In one possible design, the large model is subjected to reinforcement learning based on the reward evaluation results to obtain an optimized large model, including: The proximal strategy optimization algorithm is used to optimize the model parameters of the large model to obtain the optimized large model. The advantage function of the proximal strategy optimization algorithm is A(x', y') = R θ (x', y')-V ϕ (x') where R θ (x', y') is the instance reward value, V ϕ (x') is the set value function.

[0012] Secondly, a question-answering system for power business scenarios based on reinforcement learning is provided, including a data acquisition unit, a function training unit, a target selection unit, a model building unit, a reward evaluation unit, a model optimization unit, and a question-answering application unit, wherein: A data acquisition unit is used to acquire a set of electric power question and answer training data groups, wherein the set of electric power question and answer training data groups includes a plurality of electric power question and answer training data groups, each of which includes a corresponding electric power business scenario question sample and a plurality of answer samples associated with the electric power business scenario question sample; A function training unit is used to call a number of pre-configured reward functions from the reward function library, and train each reward function using each power question-answering training data set to obtain the corresponding trained reward function, and determine the target loss value corresponding to each trained reward function; A target selection unit, configured to select a number of target reward functions from all the trained reward functions according to the target loss value of each trained reward function; A model building unit, configured to perform a weighted combination of each target reward function based on its target loss value to obtain a multi-dimensional dynamic reward model; A reward evaluation unit is used to evaluate the rewards of the pre-trained large model's output answer instances for power business scenario questions using a multi-dimensional dynamic reward model to obtain corresponding reward evaluation results; The model optimization unit is used to perform reinforcement learning on the large model based on the reward evaluation results to obtain the optimized large model; The question-and-answer application unit is used to output question and answer examples of power business scenarios using the optimized large model.

[0013] Thirdly, we provide a reinforcement learning-based question-answering system for power business scenarios, including: a memory for storing instructions; A processor is used to read the instructions stored in the memory and execute any one of the methods described in the first aspect according to the instructions.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform any one of the methods described in the first aspect. Furthermore, a computer program product is provided that, when executed on a computer, performs any one of the methods described in the first aspect.

[0015] Beneficial effects: The present invention trains each pre-configured reward function by obtaining a set of sorted and labeled electric power question-and-answer training data groups, selects reward functions with better performance according to the loss value of each reward function, and weightedly combines them to construct a multi-dimensional dynamic reward model. The multi-dimensional dynamic reward model is then used to simulate human sorting ideas, and combined with corresponding reinforcement strategies to optimize the large model used for electric power customer service question-and-answer, driving the large model to achieve precise parameter adjustment in the electric power business question-and-answer scenario, thereby improving its electric power customer service question-and-answer performance. The present invention dynamically combines the reward models of the corresponding question-and-answer quality evaluation dimensions and introduces a dynamic weight mechanism to achieve the optimal comprehensive evaluation of the large model question-and-answer, so as to balance different optimization objectives, so that the large model electric power business question-and-answer results are more in line with user needs, and improve its question-and-answer accuracy and user interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 Schematic diagram of the steps of the method in Example 1 of the present invention; Figure 2 Schematic diagram of the system structure in Example 2 of the present invention; Figure 3 This is a schematic diagram of the system structure in Example 3 of the present invention. DETAILED DESCRIPTION

[0018] It should be noted that the description of these embodiments is intended to help understand the present invention, but does not constitute a limitation of the present invention. The specific structural and functional details disclosed herein are merely intended to describe exemplary embodiments of the present invention. However, the present invention may be embodied in a variety of alternative forms, and should not be construed as being limited to the embodiments set forth herein.

[0019] It should be understood that, unless otherwise expressly specified or limited, the corresponding terms should be understood in a broad sense. For example, "connection" can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to an electrical connection, a direct connection, an indirect connection through an intermediate medium, or internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the embodiments based on specific circumstances.

[0020] In the following description, certain details are provided to facilitate a thorough understanding of the example embodiments. However, one skilled in the art will appreciate that the example embodiments may be practiced without these specific details. For example, devices may be shown in block diagrams to avoid obscuring the examples with unnecessary detail. In other embodiments, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0021] Example 1: This embodiment provides a power business scenario question-answering method based on reinforcement learning, which can be applied to corresponding power business question-answering servers, such as Figure 1 As shown, the method includes the following steps: S1. Obtain a set of electric power question and answer training data groups, where the set of electric power question and answer training data groups includes several electric power question and answer training data groups, each of which includes a corresponding electric power business scenario question sample and several answer samples associated with the electric power business scenario question sample.

[0022] During specific implementation, the server first obtains a set of power question-and-answer training data sets, which includes several power question-and-answer training data sets. Each power question-and-answer training data set contains corresponding power business scenario question samples and several answer samples associated with the power business scenario question samples. Each answer sample associated with the corresponding power business scenario question sample in each power question-and-answer training data set has been sorted and labeled, and the labeled order represents the user's preference for each answer sample. Through such preference labeling, the reward function can be subsequently inferred from user preferences, improving the interpretability of the reward model. The reward model can then be used to optimize the large model so that when it performs power scenario business question-and-answer, it outputs question-and-answer results that are more in line with user preferences.

[0023] S2. Retrieve several pre-configured reward functions from the reward function library, and use each power question-answering training data set to train each reward function to obtain the corresponding trained reward function, and determine the target loss value corresponding to each trained reward function.

[0024] During specific implementation, the server calls the reward function library and retrieves several pre-configured reward functions from the reward function library. Several reward functions can be pre-selected and configured in the reward function library. Each reward function focuses on different reward dimensions, such as semantic accuracy dimension, emotional adaptability dimension, information diversity dimension, business goal orientation dimension and interaction efficiency dimension, etc. Users can configure them according to actual needs. After retrieving each reward function, the server can use each power question and answer training data group to train each reward function to obtain the corresponding trained reward function. The process includes: randomly constructing a triple data set (x, y based on the sorting of each answer sample associated with the power business scenario question sample in the power question and answer training data group) q ,y h ), where x represents the problem sample of the power business scenario, y q Represents the top answer samples in the power question answering training data set, y h Characterize the answer samples ranked later in the power question-answering training data set; optimize the corresponding reward function through comparative learning training, so that it generates a corresponding scalar reward value for each answer sample associated with the power business scenario question sample in the power question-answering training data set, thereby determining y q The corresponding scalar reward value R(x, y q ) and y h The corresponding scalar reward value R(x, y h ); Determine the loss function L corresponding to the reward function in the training process of the power question answering training data set RM = -logσ[R(x,y q )-R(x,y h)], where σ represents the Sigmoid function, until the loss function meets the set convergence conditions. If the standard deviation of the loss function values ​​for multiple consecutive times is within the set range, the training of the reward function for the power question and answer training data group is terminated, and the loss function value at the final convergence is obtained as the corresponding convergence loss value; after completing the training of the reward function for each power question and answer training data group, the corresponding post-training reward function is obtained.

[0025] S3. Select several target reward functions from all the trained reward functions according to the target loss value of each trained reward function.

[0026] In specific implementation, the convergence loss values ​​obtained by training the reward function of each power question-and-answer training data group can be averaged to obtain the target loss value of the corresponding trained reward function. Then, the trained reward functions are arranged in ascending order according to the target loss value, and several trained reward functions with the highest ranking are selected as the target reward function.

[0027] S4. Perform a weighted combination of the target reward functions based on the target loss values ​​of the target reward functions to obtain a multidimensional dynamic reward model.

[0028] In specific implementation, each target reward function can be arranged in ascending order of target loss value, and a weight coefficient can be assigned according to the ranking of the corresponding target reward function; then, each target reward function is weightedly combined based on the weight coefficient of each target reward function to obtain a multi-dimensional dynamic reward model.

[0029] S5. When the pre-trained large model answers questions in power business scenarios, a multi-dimensional dynamic reward model is used to evaluate the rewards of the large model’s output answer instances to obtain corresponding reward evaluation results.

[0030] In the specific implementation, when using the pre-trained large model to answer questions in the power customer service question-answering scenario, the multi-dimensional dynamic reward model can be used to evaluate the output answer instance of the large model. The reward evaluation result is the instance reward value R θ (x', y'), where x' represents the question instance of the power business scenario and y' represents the answer instance.

[0031] S6. Perform reinforcement learning on the large model based on the reward evaluation results to obtain an optimized large model.

[0032] In specific implementation, the proximal policy optimization algorithm (PPO algorithm) can be used to iteratively optimize the model parameters of the large model to obtain the optimized large model. The advantage function of the proximal policy optimization algorithm is A(x', y') = R θ (x', y')-V ϕ (x') where Rθ (x', y') is the instance reward value, V ϕ (x') is the set value function.

[0033] S7. Use the optimized large model to output question and answer examples for power business scenario problems.

[0034] In specific implementation, after the iterative optimization is completed, the server finally uses the optimized large model to output questions and answers for power business scenario problem instances to better meet the user's question and answer needs.

[0035] This method dynamically combines reward models corresponding to question-answering quality assessment dimensions and introduces a dynamic weighting mechanism to achieve optimal comprehensive evaluation of large-scale question-answering. This balances different optimization objectives, making the large-scale power business question-answering results more in line with user needs and improving its question-answering accuracy and user interaction experience.

[0036] Example 2: This embodiment provides a power business scenario question-answering system based on reinforcement learning, such as Figure 2 As shown, it includes a data acquisition unit, a function training unit, a target selection unit, a model building unit, a reward evaluation unit, a model optimization unit and a question-and-answer application unit, wherein: A data acquisition unit is used to acquire a set of electric power question and answer training data groups, wherein the set of electric power question and answer training data groups includes a plurality of electric power question and answer training data groups, each of which includes a corresponding electric power business scenario question sample and a plurality of answer samples associated with the electric power business scenario question sample; A function training unit is used to call a number of pre-configured reward functions from the reward function library, and train each reward function using each power question-answering training data set to obtain the corresponding trained reward function, and determine the target loss value corresponding to each trained reward function; A target selection unit, configured to select a number of target reward functions from all the trained reward functions according to the target loss value of each trained reward function; A model building unit, configured to perform a weighted combination of each target reward function based on its target loss value to obtain a multi-dimensional dynamic reward model; A reward evaluation unit is used to evaluate the rewards of the pre-trained large model's output answer instances for power business scenario questions using a multi-dimensional dynamic reward model to obtain corresponding reward evaluation results; The model optimization unit is used to perform reinforcement learning on the large model based on the reward evaluation results to obtain the optimized large model; The question-and-answer application unit is used to output question and answer examples of power business scenarios using the optimized large model.

[0037] Example 3: This embodiment provides a power business scenario question-answering system based on reinforcement learning, such as Figure 3 As shown, at the hardware level, it includes: Data interface, used to establish data connection between the processor and the external data terminal; a memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the power business scenario question and answer method based on reinforcement learning in Example 1 according to the instructions.

[0038] Optionally, the system further includes an internal bus, through which the processor, memory, and data interface can be interconnected. The internal bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0039] The memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out memory (FIFO) and / or first-in-last-out memory (FILO), etc. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0040] Example 4: This embodiment provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the computer is caused to execute the reinforcement learning-based question-answering method for power business scenarios in Example 1. The computer-readable storage medium refers to a data storage medium, which may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0041] This embodiment further provides a computer program product, which, when executed on a computer, executes the power business scenario question-answering method based on reinforcement learning in Example 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0042] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A question-answering method for power business scenarios based on reinforcement learning, characterized by: include: Obtaining a set of electric power question and answer training data groups, wherein the set of electric power question and answer training data groups includes a plurality of electric power question and answer training data groups, each of which includes a corresponding electric power business scenario question sample and a plurality of answer samples associated with the electric power business scenario question sample; Retrieve several pre-configured reward functions from the reward function library, and train each reward function using each power question-answering training data set to obtain the corresponding trained reward function, and determine the target loss value corresponding to each trained reward function; Select several reward functions from all the post-training reward functions as target reward functions according to the target loss value of each post-training reward function; Perform a weighted combination of each objective reward function based on its target loss value to obtain a multi-dimensional dynamic reward model; When the pre-trained large model answers questions in power business scenarios, a multi-dimensional dynamic reward model is used to evaluate the rewards of the large model's output answer instances and obtain the corresponding reward evaluation results. Perform reinforcement learning on the large model based on the reward evaluation results to obtain an optimized large model; The optimized large model is used to output question and answer examples of power business scenario problems.

2. The power business scenario question-answering method based on reinforcement learning according to claim 1 is characterized in that: Each answer sample associated with the corresponding power business scenario question sample in each power question and answer training data group has been sorted and labeled.

3. The power business scenario question-answering method based on reinforcement learning according to claim 2 is characterized in that: The method of training each reward function using each power question-answering training data set to obtain a corresponding trained reward function includes: Based on the order of the answer samples associated with the power business scenario question samples in the power question answering training data set, a triple data set (x, y q ,y h ), where x represents the problem sample of the power business scenario, y q Represents the top answer samples in the power question answering training data set, y h Represents the answer samples ranked last in the power question answering training data set; By contrastive learning and training the corresponding reward function, it generates a corresponding scalar reward value for each answer sample associated with the power business scenario question sample in the power question answering training data set, thereby determining y q The corresponding scalar reward value R(x, y q ) and y h The corresponding scalar reward value R(x, y h ); Determine the loss function L corresponding to the reward function during the training process of the power question answering training data set RM = -logσ[R(x,y q )-R(x,y h )], where σ represents the Sigmoid function. The training of the reward function for the power question-answering training data set ends until the loss function meets the set convergence condition, and the corresponding convergence loss value is obtained. After completing the training of the reward function for each power question-answering training data set, the corresponding trained reward function is obtained.

4. The power business scenario question-answering method based on reinforcement learning according to claim 3 is characterized in that: Determining the target loss value corresponding to each post-training reward function includes: The convergence loss values ​​obtained by training the reward function for each power question-answering training data set are averaged to obtain the target loss value of the corresponding reward function after training.

5. The power business scenario question-answering method based on reinforcement learning according to claim 1 is characterized in that: The method selects several target reward functions from all the trained reward functions according to the target loss value of each trained reward function, including: Arrange the post-training reward functions in ascending order of target loss value, and select the top post-training reward functions as the target reward functions.

6. The power business scenario question-answering method based on reinforcement learning according to claim 5 is characterized in that: The target loss value of each target reward function is used to perform a weighted combination of each target reward function to obtain a multi-dimensional dynamic reward model, including: Arrange each objective reward function in ascending order of target loss value, and assign a weight coefficient according to the order of the corresponding objective reward function; Based on the weight coefficients of each objective reward function, each objective reward function is weightedly combined to obtain a multidimensional dynamic reward model.

7. The power business scenario question-answering method based on reinforcement learning according to claim 1 is characterized in that: The reward evaluation result is the instance reward value R θ (x', y'), where x' represents the question instance of the power business scenario and y' represents the answer instance.

8. The power business scenario question-answering method based on reinforcement learning according to claim 7 is characterized in that: The reinforcement learning of the large model based on the reward evaluation results is performed to obtain an optimized large model, including: The proximal strategy optimization algorithm is used to optimize the model parameters of the large model to obtain the optimized large model. The advantage function of the proximal strategy optimization algorithm is A(x', y') = R θ (x', y')-V ϕ (x') where R θ (x', y') is the instance reward value, V ϕ (x') is the set value function.

9. The power business scenario question-answering system based on reinforcement learning is characterized by: It includes data acquisition unit, function training unit, target selection unit, model building unit, reward evaluation unit, model optimization unit and question-answering application unit, among which: A data acquisition unit is used to acquire a set of electric power question and answer training data groups, wherein the set of electric power question and answer training data groups includes a plurality of electric power question and answer training data groups, each of which includes a corresponding electric power business scenario question sample and a plurality of answer samples associated with the electric power business scenario question sample; A function training unit is used to call a number of pre-configured reward functions from the reward function library, and train each reward function using each power question-answering training data set to obtain the corresponding trained reward function, and determine the target loss value corresponding to each trained reward function; A target selection unit, configured to select a number of target reward functions from all the trained reward functions according to the target loss value of each trained reward function; A model building unit, configured to perform a weighted combination of each target reward function based on its target loss value to obtain a multi-dimensional dynamic reward model; A reward evaluation unit is used to evaluate the rewards of the pre-trained large model's output answer instances for power business scenario questions using a multi-dimensional dynamic reward model to obtain corresponding reward evaluation results; The model optimization unit is used to perform reinforcement learning on the large model based on the reward evaluation results to obtain the optimized large model; The question-and-answer application unit is used to output question and answer examples of power business scenarios using the optimized large model.

10. The power business scenario question-answering system based on reinforcement learning is characterized by: include: a memory for storing instructions; A processor is used to read the instructions stored in the memory and execute the power business scenario question and answer method based on reinforcement learning as described in any one of claims 1-8 according to the instructions.

Images