Text processing method and device, computer device and storage medium

Abstract

This application relates to a text processing method, apparatus, computer device, and storage medium. The method includes: acquiring text to be parsed; converting the text to be parsed into a text string; the text string including prompt information and answer information from the text to be parsed; filling the prompt information and answer information from the text string into a prompt template according to a pre-determined prompt template; inputting the filled prompt template into a pre-trained reinforcement learning agent; outputting the answer information from the prompt template; and converting the answer information into a prediction result of the reinforcement learning agent. This method can process the text to be parsed into a standardized format using prompt templates, ensuring that the reinforcement learning agent can handle new scenarios with unlabeled or poorly labeled text, allowing the reinforcement learning agent to adapt to new text processing scenarios and guaranteeing the recognition accuracy of new texts to be parsed.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a text processing method, apparatus, computer device, and storage medium. Background Technology

[0002] In handling non-performing assets, each transaction involves dozens to hundreds of documents, including PDFs, Word documents, JPGs, screenshots, photocopies, and handwritten documents (such as credit reports, due diligence reports, write-off reports, and litigation documents). These documents cover a wide range of areas and are diverse in type, spanning the front, middle, and back offices of the business. The review scope extends beyond write-off conditions to include the completeness of the asset preservation and disposal process, the compliance of the branch's review and application procedures, the standardization of application documents and certificates, and the robustness of the branch's basic asset preservation management. The sheer volume of documents and the immense workload are significant.

[0003] Currently, the conventional method is to use pre-trained language models to extract features from multiple documents. To ensure that the recognition accuracy of the pre-trained language model meets business requirements (92%+), the conventional pre-trained language model training process is repeated for each type of file, namely the annotation and parameter tuning process of the pre-trained language model. Manual annotation is time-consuming and labor-intensive, and it is difficult to cover all existing file types. Furthermore, in some business scenarios, file types that the pre-trained language model has not processed may be encountered, which cannot guarantee the recognition accuracy of newly added files to be parsed. Summary of the Invention

[0004] Therefore, it is necessary to provide a text processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can adapt to file feature extraction scenarios with a small amount of labeled data or no labeled data, and improve the recognition accuracy of the file to be parsed, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a text processing method, the method comprising:

[0006] Get the text to be parsed; the text to be parsed can be either unannotated or annotated.

[0007] Convert the text to be parsed into a text string; the text string includes the hints and answers in the file to be parsed;

[0008] According to the pre-determined prompt template, fill in the prompt information and answer information from the text string into the prompt template respectively;

[0009] The filled-in prompt template is input into the pre-trained reinforcement learning agent, the answer information in the prompt template is output, and the answer information is converted into the prediction result of the reinforcement learning agent.

[0010] In one embodiment, the method further includes:

[0011] If the prompt template is a continuous prompt template, and the prediction result of the reinforcement learning agent does not match the answer information in the prompt template, then freeze the network parameters of the reinforcement learning agent and update the weight matrix parameters of the prompt template until the prediction result of the reinforcement learning agent matches the answer information in the prompt template.

[0012] In one embodiment, the method further includes:

[0013] Obtain multiple samples; each sample includes text data and labels that annotate the answer information in the text data.

[0014] Multiple samples are input into a reinforcement learning agent, which outputs prediction results. The reward information of the prediction results output by the reinforcement learning agent is determined based on the difference between the prediction results and the labels of the samples.

[0015] Based on the reward information and the preset loss function, update the network parameters of the reinforcement learning agent to complete one training cycle;

[0016] The training process is iterated multiple times. When the loss function meets the target value, the training stops, and a well-trained reinforcement learning agent is obtained.

[0017] In one embodiment, multiple samples are input into a reinforcement learning agent, and a prediction result is output. Based on the difference between the prediction result and the label of the sample, reward information for the prediction result output by the reinforcement learning agent is determined, including:

[0018] Multiple samples are input into the reinforcement learning agent, and the reward error value between the first reward value of the reinforcement learning agent in response to the prediction result of each sample in the current environmental state and the second reward value of the reinforcement learning agent in response to the prediction result of each sample in the next environmental state is determined.

[0019] The reward design function is determined based on the first potential function of the reinforcement learning agent in the current environment and the second potential function in the next environment; the first potential function is used to evaluate the quality of the current environment; the second potential function is used to evaluate the quality of the next environment.

[0020] Based on the reward error value and the reward design function, the reward information of the prediction result output by the reinforcement learning agent is determined.

[0021] In one embodiment, determining the reward error value between a first reward value for the reinforcement learning agent's output prediction results for each sample in the current environmental state and a second reward value for the reinforcement learning agent's output prediction results for each sample in the next environmental state includes:

[0022] Multiple samples are input into the reinforcement learning agent. The reinforcement learning agent responds to the current environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored and the score value is used as the first reward value for the reinforcement learning agent to output the prediction result for each sample in the current environmental state.

[0023] The reinforcement learning agent responds to the next environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the second reward value for the reinforcement learning agent to output the prediction result for each sample in the next environmental state.

[0024] The reward error value between the first reward value and the second reward value is determined based on the difference between the first reward value and the second reward value.

[0025] In one embodiment, a reward design function is determined based on the first potential function of the reinforcement learning agent in the current environment and the second potential function in the next environment, including:

[0026] For the transfer learning process of a reinforcement learning agent in the current environment, the first potential function of the reinforcement learning agent in the current environment is determined based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0027] For the transfer learning process of a reinforcement learning agent in the next environment, the second potential function of the reinforcement learning agent in the next environment is determined based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0028] The reward design function is determined based on the second potential function and the first potential function.

[0029] Secondly, this application also provides a text processing apparatus. The apparatus includes:

[0030] The acquisition module is used to acquire the text to be parsed; the text to be parsed can be either unannotated or annotated text.

[0031] The conversion module is used to convert the text to be parsed into a text string; the text string includes the hints and answer information in the file to be parsed.

[0032] The fill module is used to fill the prompt information and answer information in the text string into the prompt template according to the pre-defined prompt template;

[0033] The prediction module is used to input the filled prompt template into the pre-trained reinforcement learning agent, output the answer information in the prompt template, and convert the answer information into the prediction result of the reinforcement learning agent.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0035] Get the text to be parsed; the text to be parsed can be either unannotated or annotated.

[0036] Convert the text to be parsed into a text string; the text string includes the hints and answers in the file to be parsed;

[0037] According to the pre-determined prompt template, fill in the prompt information and answer information from the text string into the prompt template respectively;

[0038] The filled-in prompt template is input into the pre-trained reinforcement learning agent, the answer information in the prompt template is output, and the answer information is converted into the prediction result of the reinforcement learning agent.

[0039] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0040] Get the text to be parsed; the text to be parsed can be either unannotated or annotated.

[0041] Convert the text to be parsed into a text string; the text string includes the hints and answers in the file to be parsed;

[0042] According to the pre-determined prompt template, fill in the prompt information and answer information from the text string into the prompt template respectively;

[0043] The filled-in prompt template is input into the pre-trained reinforcement learning agent, the answer information in the prompt template is output, and the answer information is converted into the prediction result of the reinforcement learning agent.

[0044] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0045] Get the text to be parsed; the text to be parsed can be either unannotated or annotated.

[0046] Convert the text to be parsed into a text string; the text string includes the hints and answers in the file to be parsed;

[0047] According to the pre-determined prompt template, fill in the prompt information and answer information from the text string into the prompt template respectively;

[0048] The filled-in prompt template is input into the pre-trained reinforcement learning agent, the answer information in the prompt template is output, and the answer information is converted into the prediction result of the reinforcement learning agent.

[0049] The aforementioned text processing method, apparatus, computer equipment, and storage medium convert the text to be parsed into a text string, which includes prompt information and answer information from the file to be parsed. The prompt information and answer information from the text string are then filled into a prompt template. The filled prompt template is input into a pre-trained reinforcement learning agent, and the answer information from the prompt template is output. The answer information is then converted into the prediction result of the reinforcement learning agent. By using prompt templates to process the text to be parsed into a standardized format, it is possible to ensure that the reinforcement learning agent can handle new scenarios with unlabeled or poorly labeled text. Even if a file type is encountered that the reinforcement learning agent has never processed before, the weight matrix parameters of the prompt template can be fine-tuned to allow the reinforcement learning agent to adapt to new text processing scenarios, ensuring the recognition accuracy of new files to be parsed. Attached Figure Description

[0050] Figure 1 This is a diagram illustrating the application environment of a text processing method in one embodiment.

[0051] Figure 2 This is a flowchart illustrating a text processing method in one embodiment;

[0052] Figure 3 This is a schematic diagram of the parsing process for an approval opinion parsing scenario in one embodiment;

[0053] Figure 4 This is a flowchart illustrating an example of training a reinforcement learning agent using labeled litigation documents in another embodiment.

[0054] Figure 5 This is a flowchart illustrating the process of a reinforcement learning agent predicting unlabeled text in one embodiment.

[0055] Figure 6 This is a schematic diagram of the training process of a reinforcement learning agent in one embodiment;

[0056] Figure 7 This is a flowchart illustrating the process of obtaining reward information from the prediction results output by a reinforcement learning agent in one embodiment.

[0057] Figure 8 This is a structural block diagram of a text processing device in one embodiment;

[0058] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] The text processing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 acquires the text to be parsed; the text to be parsed can be unlabeled or labeled; the text to be parsed is converted into a text string; the text string includes prompt information and answer information from the text to be parsed; according to a pre-determined prompt template, the prompt information and answer information from the text string are filled into the prompt template respectively; the filled prompt template is input into a pre-trained reinforcement learning agent, the answer information in the prompt template is output, and the answer information is converted into the prediction result of the reinforcement learning agent. Terminal 102 communicates with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Portable wearable devices can be head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0061] In one embodiment, such as Figure 2 As shown, a text processing method is provided, which can be applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps:

[0062] Step 202: Obtain the text to be parsed; the text to be parsed can be either unannotated or annotated text.

[0063] The text to be parsed includes both unannotated and annotated text within the business scenario. For example, in a bank special asset write-off scenario, an example of annotated text would be: A certain resource originally consisted of 35 residential units and 10 commercial units under Company A's name. After being handled by the borrower and mortgagor themselves and through judicial auction, the remaining 6 commercial units have failed to sell at auction and have not yet been disposed of. The annotations are: Resource Requester: Company A; Resource Item: 35 residential units and 10 commercial units; Resource Status: Auction.

[0064] Step 204: Convert the text to be parsed into a text string; the text string includes the hints and answers from the file to be parsed.

[0065] Converting the text to be parsed into a text string serves two purposes: firstly, it facilitates the terminal's understanding of the text; secondly, it ensures that the reinforcement learning agent reads the dataset in a consistent manner. The text string includes the text content, labeled data, and the current text string's position within the parsed text.

[0066] Taking the text to be parsed as "Borrower Wang Er borrowed 40 yuan from the target unit using collateral" as an example, the corresponding text string is as follows:

[0067] Borrow B-TYPE

[0068] I-TYPE

[0069] I-TYPE

[0070] Wang B-NAME

[0071] II-NAME

[0072] In O

[0073] B-MECHANISM

[0074] I-MECHANISM

[0075] Single I-MECHANISM

[0076] I-MECHANISM

[0077] Based on O

[0078] Borrow O

[0079] O

[0080] Bet O

[0081] Item O

[0082] Borrow O

[0083] Model B-AMOUNT

[0084] 4I-AMOUNT

[0085] Pick I-AMOUNT

[0086] Yuan I-AMOUNT

[0087] I-AMOUNT

[0088] O

[0089] In each line, the first element is the document content, the second is its tag, separated by spaces, and sentences are separated by blank lines.

[0090] Hint information and answer information are crucial in hint learning, a method in natural language processing that reconstructs the task to better suit a pre-trained model, thereby improving the overall performance of the model. The general process of hint learning is as follows: The input to the task is transformed into hints using a hint function, and then filled with alternative answers; the pre-trained model scores the filled hints, and the highest-scoring hint is used as the answer; finally, the answer is converted into the task output.

[0091] Therefore, the prompt message is information in a standardized format corresponding to the processing requirements of the text to be parsed, with reserved space for the reinforcement learning agent to fill in predicted information. For example, when the text to be parsed is "Borrower Wang Er borrowed 40 yuan from the target unit with collateral," the prompt template can be designed as follows: (Borrower) With (collateral) exist (Target unit) loan (Action volume) in," (Borrower) "", (collateral) "", (Target unit) "and" (Action volume) "This is the answer information."

[0092] Answer information is used to extract predictive information from answers with a standardized format for the reinforcement learning agent. Answer information is typically associated with prompt information. For example, when the prompt information is "[The text to be parsed] is __borrowed __ by __", the role of answer information is to extract the information from the answer to fill in the blanks in "(borrower)", ... (collateral) "", (Target List) Bit) "and" (Action volume) Information at the location.

[0093] Optionally, the terminal converts the text to be parsed into a text string according to a predetermined format, and annotates the prompt information and answer information in the text string.

[0094] Step 206: Fill the prompt information and answer information from the text string into the prompt template according to the pre-determined prompt template.

[0095] The prompt template converts a text string into a standardized format adapted to the task requirements of the text to be parsed. This standardized format includes reserved spaces for pre-trained models to fill in predicted information. For example, if the text to be parsed is "Borrower Wang Er borrowed forty yuan from the target unit using collateral," the prompt template could be designed as follows: (Borrower) With (collateral) exist (Target unit) loan (Action volume) in," (Borrower) "", (collateral) "", (Target unit) "and" (Action volume) This is a reserved space for the reinforcement learning agent to fill in the predicted information. When any text to be parsed is transformed through this prompt template, the resulting prompt text can be: "[Text to be parsed] is __borrowed __ by __". Therefore, regardless of how the input text to be parsed changes or how its format is designed, the transformed prompt information will have the same standardized format.

[0096] The prompt template frame settings in this embodiment are as follows:

[0097]

[0098] Optionally, the terminal fills the prompt information and answer information from the text string into the prompt template according to the predetermined prompt template.

[0099] Step 208: Input the filled prompt template into the pre-trained reinforcement learning agent, output the answer information in the prompt template, and convert the answer information into the prediction result of the reinforcement learning agent.

[0100] In this context, a reinforcement learning agent is a system that exists within and is part of an environment. It can sense the environment at any time and execute corresponding actions, while gradually establishing its own activity rules to cope with possible future environmental changes. This embodiment employs a dual Actor-Critic network architecture, and the initial reinforcement learning agent has pre-set network parameters.

[0101] Deep Reinforcement Learning (DRL) is a novel algorithm that combines deep learning and reinforcement learning to achieve direct mapping from perception to action. It takes perceptual information (such as vision) as input and then outputs actions directly through a deep neural network, without any hard-coding process in between. LRL combines the advantages of deep neural networks and reinforcement learning, effectively solving perceptual decision-making problems for intelligent agents in high-dimensional and complex situations. It is a cutting-edge research direction in the field of general artificial intelligence and has broad application prospects.

[0102] The key to deep reinforcement learning is to train a deep policy network by having an agent continuously interact with the environment to obtain training samples. In this deep policy network, the agent receives data representing the current state of the environment and executes actions from a continuous action space in response to the received environment state data, attempting to perform the corresponding task in the environment.

[0103] Compared to supervised learning, reinforcement learning's main characteristics are: (a) reinforcement learning optimizes the entire sequential decision-making process with the overall reward as the objective; and (b) reinforcement learning optimizes the policy through interaction with the environment without the need for labeled samples. The key idea here is that evolutionary algorithms can be used to address the core challenges of deep reinforcement learning without being inferior to deep reinforcement learning methods that utilize gradients to improve sample efficiency. Evolutionary reinforcement learning inherits the fitness approach of evolutionary algorithms, handling temporary credit assignment by consolidating the return values ​​throughout the training episode. The selection operator in evolutionary reinforcement learning operates based on this fitness, applying selection pressure to regions of the policy space, resulting in higher rewards across the training episodes. Furthermore, evolutionary reinforcement learning inherits the population-based approach of evolutionary algorithms, leading to redundancy, which stabilizes convergence properties and makes the learning process more robust. Second, by using prompting learning to process the input text to be parsed according to a specific template, the task is restructured into a form that can make better use of the pre-trained language model, thereby solving various NLP tasks (i.e., automatic summarization, machine translation, named body recognition (NER), relation extraction, sentiment analysis, speech recognition, topic segmentation, etc.).

[0104] In the scenario of special bank asset write-offs, the text to be parsed includes due diligence reports, credit reports, and litigation documents such as complaints, judgments, enforcement applications, enforcement rulings, and preservation rulings. To ensure that the model's recognition accuracy meets business requirements (92%+), the conventional pre-trained language model training process is repeated for each type of document, namely, the annotation and parameter tuning process of the pre-trained language model. Manual annotation is time-consuming and labor-intensive, and it is difficult to cover all existing document types. Furthermore, in some business scenarios, document types that the pre-trained language model has not processed may be encountered, which cannot guarantee the recognition accuracy of newly added documents to be parsed. Therefore, to solve the above problems, this embodiment introduces prompt learning. The prompt template is used to process the text to be parsed into a standardized format, which can ensure that the reinforcement learning agent can perform new scenarios with zero or few annotations for the text to be parsed. Even if a document type that the reinforcement learning agent has never processed before appears, the weight matrix parameters of the prompt template can be fine-tuned to allow the reinforcement learning agent to adapt to the new text processing scenario and ensure the recognition accuracy of newly added documents to be parsed.

[0105] Optionally, the terminal uses the filled-in prompt template as input to a pre-trained reinforcement learning agent. The reinforcement learning agent scores the prompts in the filled-in prompt template, and the one with the highest score is used as the answer information. The answer information is then converted into the prediction result of the reinforcement learning agent.

[0106] In the above text processing method, the text to be parsed is converted into a text string, which includes prompt information and answer information from the file to be parsed. The prompt information and answer information from the text string are then filled into a prompt template. The filled prompt template is input into a pre-trained reinforcement learning agent, and the answer information from the prompt template is output. The answer information is then converted into the prediction result of the reinforcement learning agent. By using prompt templates to process the text to be parsed into a standardized format, it is possible to ensure that the reinforcement learning agent can handle new scenarios with zero or few annotations for the text to be parsed. Even if a file type is encountered that the reinforcement learning agent has never processed before, the weight matrix parameters of the prompt template can be fine-tuned to allow the reinforcement learning agent to adapt to the new text processing scenario and ensure the recognition accuracy of new texts to be parsed.

[0107] In one embodiment, the prompt template can be either manually designed or automatically learned. Manually designed templates are generally based on human natural language knowledge, resulting in semantically fluent and efficient templates, but require extensive experimentation, experience, and linguistic expertise. Automatically learned templates are further divided into discrete and continuous prompt templates. Discrete prompt templates are automatically generated from words in natural language, and their search space is discrete. Continuous prompt templates are not limited to human-understandable natural language; they can use any language that machines or computers can understand. Therefore, the parameters of continuous templates are not directly taken from the language model but have their own independent weight matrix parameters, which can be adjusted using training data.

[0108] When a new recognition scenario for a file to be parsed is introduced, the traditional approach is to retrain the reinforcement learning agent based on this new scenario, updating its network parameters. Since the network parameters of a reinforcement learning agent have tens of millions of dimensions, updating them to adapt to the new scenario increases training costs and reduces recognition efficiency. Therefore, adjusting the weight matrix parameters of the prompt template requires significantly fewer dimensions than adjusting the reinforcement learning agent's parameters. In this embodiment, when a new recognition scenario for a file to be parsed is introduced, only a small amount of annotation is needed on the file, the network parameters of the reinforcement learning agent are frozen, and the weight matrix parameters of the prompt template are fine-tuned simultaneously. This allows the trained knowledge to be transferred to the new scenario. Specifically, if the prompt template is discrete and the reinforcement learning agent's prediction does not match the answer information in the prompt template, the network parameters of the reinforcement learning agent are frozen, and the discrete prompt template is replaced until the reinforcement learning agent's prediction matches the answer information in the prompt template. If the prompt template is a continuous prompt template, and the prediction result of the reinforcement learning agent does not match the answer information in the prompt template, then freeze the network parameters of the reinforcement learning agent and update the weight matrix parameters of the prompt template until the prediction result of the reinforcement learning agent matches the answer information in the prompt template.

[0109] For example, such as Figure 3 As shown, in actual production, if a new document parsing scenario needs to be added, such as approval opinions, the document to be parsed is: a resource originally consisted of 35 residential units and 10 commercial units under Company A's name. After being disposed of by the borrower and mortgagor themselves and through judicial auction, the remaining 6 commercial units have failed to sell at auction and have not yet been disposed of. The annotations are: Resource Requester: Company A; Resource Item: 35 residential units and 10 commercial units; Resource Status: Auction. In this case, only a small amount of annotation is needed. The neural network parameters of the agent are frozen, and the weight matrix parameters of the prompt template are fine-tuned simultaneously. This allows the knowledge trained in the reinforcement learning agent to be transferred to the approval opinion scenario. Although the trained reinforcement learning agent does not possess the specific business knowledge of approval opinions, it already contains traditional Chinese knowledge, such as part-of-speech analysis and text reasoning, as well as some business knowledge. Only the business knowledge of approval opinions needs to be supplemented.

[0110] In some embodiments, the reinforcement learning agent is trained on labeled documents, thus the agent possesses background knowledge corresponding to the labeled documents. For example, such as... Figure 4As shown, a reinforcement learning agent is trained using labeled litigation documents (complaints, judgments, enforcement applications, enforcement rulings, preservation rulings, property preservation guarantees, tax bureau coordination letters, mediation agreements, powers of attorney, notices of case acceptance, enforcement case filing acceptance notices, and collection letters). The labeled litigation documents could be: a resource originally consisted of 35 residential properties and 10 commercial properties under Company A's name. After being disposed of by the borrower and mortgagor themselves and through judicial auction, the remaining 6 commercial properties failed to sell at auction and have not yet been disposed of. The labeling is as follows: Resource Requester: Company A; Resource Items: 35 residential properties and 10 commercial properties; Resource Status: Auction. At this point, the reinforcement learning agent possesses external background knowledge of litigation law and meets the needs of the business department, recognizing asset details and individual asset information within the litigation documents.

[0111] In some embodiments, if a new file parsing scenario is added, such as a judgment of dishonesty in litigation, even though this type of file is not present in the training dataset of the reinforcement learning agent, the agent has already been trained on litigation files and possesses litigation background knowledge, so it can be directly transferred and used. For example, as... Figure 5 As shown, the input is an unlabeled document: "Company A's 2932 square meters of land located in location A, valued at 90 million yuan, is mortgaged for 50 million yuan, and the original mortgage guarantee is released." Using a discrete prompt template, the reinforcement learning agent predicts the answer to the discrete prompt template based on the input question "What are the individual resources described in the input text?" and converts the answer into the prediction result.

[0112] In this embodiment, the scenario of generalizing the reinforcement learning agent through prompt templates is used to train the reinforcement learning agent using a large amount of training text and resources. After a small amount of model modification, the inherent potential of the reinforcement learning agent (a large number of parameters of the pre-trained language model) is stimulated by prompt information, and the knowledge learned by the reinforcement learning agent in the large-scale pre-training stage is mined.

[0113] In one embodiment, the reinforcement learning agent is trained using a hybrid algorithm combining evolutionary reinforcement learning, evolutionary strategy, and prompt-based learning. Specifically, it combines the evolutionary algorithm's generation-based approach to generate (text prediction) experience to train the reinforcement learning agent, and periodically transfers the learned behavioral strategies (strategy experience for predicting target words) from the reinforcement learning agent into the evolutionary algorithm's population, thereby injecting gradient information into the evolutionary algorithm. Simultaneously, the evolutionary strategy is used for black-box optimization of the large-scale parameters of high-dimensional text problems, avoiding the common problem of high-dimensional vector and high sample complexity in evolutionary algorithms. For example, Figure 6 As shown, the training process of a reinforcement learning agent specifically includes the following steps:

[0114] Step 602: Obtain multiple samples; each sample includes text data and labels that annotate the answer information in the text data.

[0115] This embodiment combines evolutionary algorithms and deep reinforcement learning, leveraging the strengths of both. Evolutionary algorithms and strategies are used to breed initial samples in the evolutionary population, allowing these bred samples to explore and learn in the interactive environment. This increases the frequency of interaction between the reinforcement learning agent and the environment, enabling it to learn more predictive actions. Furthermore, training the reinforcement learning agent with predicted actions learned from interactions between multiple target samples in the evolutionary population and the environment improves the learning efficiency and effectiveness of deep reinforcement learning, thereby better controlling the deep reinforcement learning agent to complete continuous control tasks.

[0116] Optionally, the terminal performs breeding processing on samples in the evolutionary population through an evolutionary strategy to obtain breeding samples; and performs breeding processing on the breeding samples through an evolutionary algorithm to obtain the final samples.

[0117] Step 604: Input multiple samples into the reinforcement learning agent, output prediction results, and determine the reward information of the prediction results output by the reinforcement learning agent based on the difference between the prediction results and the labels of the samples.

[0118] Here, reward information refers to the reward value of the prediction action for the reinforcement learning agent's output prediction result. The network parameters of the reinforcement learning agent can be updated based on the reward information. In this embodiment, the reward information can be the cumulative reward value of the prediction action for the reinforcement learning agent's output prediction result.

[0119] Optionally, the terminal inputs multiple samples into the reinforcement learning agent, the reinforcement learning agent outputs the prediction result corresponding to each sample, scores the prediction result output by the reinforcement learning agent each time, uses the score value as the reward value, and uses the cumulative reward value corresponding to each sample as the reward information of the reinforcement learning agent.

[0120] Step 606: Update the network parameters of the reinforcement learning agent based on the reward information and the preset loss function to complete one training cycle.

[0121] In deep reinforcement learning, network parameters are updated using a gradient strategy.

[0122] Optionally, the terminal updates the network parameters of the reinforcement learning agent based on the error between the reward information and the preset reward information, as well as the preset loss function, to complete one training cycle.

[0123] Step 608: Iterate the training process multiple times. When the loss function meets the target value, stop training and obtain the trained reinforcement learning agent.

[0124] Optionally, if the loss function does not meet the target value, multiple samples are obtained from the sample set again, and steps 602-606 above are executed repeatedly until the loss function meets the target value. Then, the update of the network parameters of the reinforcement learning agent is terminated, and the trained reinforcement learning agent is obtained.

[0125] In this embodiment, the reinforcement learning agent is trained by learning the predicted actions through the interaction between multiple target samples in the evolutionary population and the environment, which can improve the learning efficiency and effectiveness of deep reinforcement learning.

[0126] In one embodiment, such as Figure 7 As shown, multiple samples are input into a reinforcement learning agent, which outputs prediction results. Based on the difference between the prediction results and the sample labels, the reward information of the prediction results output by the reinforcement learning agent is determined, including:

[0127] Step 702: Input multiple samples into the reinforcement learning agent and determine the reward error value between the first reward value of the reinforcement learning agent in response to the prediction result of each sample in the current environmental state and the second reward value of the reinforcement learning agent in response to the prediction result of each sample in the next environmental state.

[0128] In this context, the current environment state and the next environment state of the reinforcement learning agent are simulated environments, and the reinforcement learning agent is implemented as one or more computer programs that interact with these simulated environments. As an example, the simulated environment could be a question-answering system, and the reinforcement learning agent could be a simulated user responding to input questions. Since the reinforcement learning agent outputs actions in a continuous action space, the current environment state and the next environment state are consecutive environment states.

[0129] In some embodiments, step 702 specifically includes the following steps:

[0130] Step 1: Input multiple samples into the reinforcement learning agent. The reinforcement learning agent responds to the current environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the first reward value for the reinforcement learning agent to output the prediction result for each sample in the current environmental state.

[0131] Step 2: The reinforcement learning agent responds to the next environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the second reward value for the reinforcement learning agent to output the prediction result for each sample in the next environmental state.

[0132] Step 3: Determine the reward error value between the first reward value and the second reward value based on the difference between the first reward value and the second reward value.

[0133] The reward error value can be determined by the following formula:

[0134]

[0135] Among them, y i This represents the reward error value between the first reward value and the second reward value; r i Indicates the first reward value; s i +1 represents the next environment state; γ represents the hyperparameter; the reinforcement learning agent in this embodiment adopts a dual Actor-Critic network architecture, Q′ represents the reward function of the target Critic neural network, and θ Q′ π′ represents the target network parameters of the Critic neural network, θ′ represents the target Actor neural network, and θ represents the target Actor neural network. π′ This represents the target network parameters of the target Actor neural network.

[0136] 704. Based on the first potential function of the reinforcement learning agent in the current environment and the second potential function in the next environment, determine the reward design function; the first potential function is used to evaluate the quality of the current environment; the second potential function is used to evaluate the quality of the next environment.

[0137] As business scenarios become increasingly complex and accuracy requirements rise, each time a new problem or scenario is encountered, the processes of data cleaning, augmentation, model training, and deployment must be repeatedly executed to ensure that the pre-trained language model possesses the background knowledge of the current scenario. This process is time-consuming and labor-intensive. To address these issues, a transfer learning method is added to the existing deep reinforcement learning algorithm to avoid the high-cost data annotation work. This involves adding a reward design to the reinforcement learning agent. When the reinforcement learning agent is applied to a new scenario, its original reward function is modified, and a new reward function replaces the target domain.

[0138] In some embodiments, step 704 specifically includes the following steps:

[0139] Step 1: During the transfer learning process of the reinforcement learning agent in the current environment, determine the first potential function of the reinforcement learning agent in the current environment based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0140] Transfer learning, in this context, refers to the learning of previously learned knowledge and its transfer to another unknown domain, i.e., from a source domain to a target domain. The source domain refers to the existing knowledge of the reinforcement learning agent before transfer learning; the target domain refers to the new knowledge the agent needs to learn after transfer learning. The reward function is used to evaluate the reward value for the reinforcement learning agent's prediction action of outputting a prediction result. This embodiment uses... This represents the reward function corresponding to the source domain during the transfer learning process of a reinforcement learning agent in the current environment; using This represents the reward function corresponding to the target domain during the transfer learning process of a reinforcement learning agent in the current environment; using Let represent the first potential function under the current environment. Then, the mathematical expression of the first potential function is:

[0141]

[0142] Step 2: During the transfer learning process of the reinforcement learning agent in the next environment, determine the second potential function of the reinforcement learning agent in the next environment based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0143] In this embodiment, the following is used: Let represent the second potential function under the next environment. The mathematical expression of the second potential function is similar to that of the first potential function, and will not be repeated here.

[0144] Step 3: Determine the reward design function based on the second potential function and the first potential function.

[0145] In this embodiment, F is used. i (s,a,s′,a′) represents the reward design function. The mathematical expression of the reward design function is as follows:

[0146]

[0147] Where γ represents the hyperparameter.

[0148] Step 706: Determine the reward information of the prediction result output by the reinforcement learning agent based on the reward error value and the reward design function.

[0149] Here, the sum of the reward error value and the reward design function represents the reward information of the prediction result output by the reinforcement learning agent. Let R′ represent the reward information, then the mathematical expression of the reward information R′ is:

[0150] R′=y i +F i (s,a,s′,a′)

[0151] Optionally, the sum of the reward error value and the reward design function can be used as the reward information for the prediction result output by the reinforcement learning agent.

[0152] In this embodiment, a transfer learning method is added to the original deep reinforcement learning algorithm. Based on the transfer learning method, the reward design of the reinforcement learning agent is added. When the reinforcement learning agent is applied to a new scenario, the original reward function of the reinforcement learning agent is modified and a new reward function is used to replace the target domain, thus avoiding the problem that the reinforcement learning agent needs to be repeatedly trained to adapt to the new scenario.

[0153] In one embodiment, a detailed text processing method is provided, specifically including the following steps:

[0154] Step 1: Obtain multiple samples; each sample includes text data and labels that annotate the answer information in the text data.

[0155] Step 2: Input multiple samples into the reinforcement learning agent. The reinforcement learning agent responds to the current environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the first reward value for the reinforcement learning agent to output the prediction result for each sample in the current environmental state.

[0156] Step 3: The reinforcement learning agent responds to the next environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the second reward value for the reinforcement learning agent to output the prediction result for each sample in the next environmental state.

[0157] Step 4: Determine the reward error value between the first reward value and the second reward value based on the difference between the first reward value and the second reward value.

[0158] Step 5: During the transfer learning process of the reinforcement learning agent in the current environment, determine the first potential function of the reinforcement learning agent in the current environment based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0159] Step 6: During the transfer learning process of the reinforcement learning agent in the next environment, determine the second potential function of the reinforcement learning agent in the next environment based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0160] Step 7: Determine the reward design function based on the second potential function and the first potential function.

[0161] Step 8: Determine the reward information for the prediction results output by the reinforcement learning agent based on the reward error value and the reward design function.

[0162] Step 9: Update the network parameters of the reinforcement learning agent based on the reward information and the preset loss function to complete one training cycle.

[0163] Step 10: Iterate through the training process multiple times. When the loss function meets the target value, stop training and obtain the trained reinforcement learning agent.

[0164] Step 11, obtain the text to be parsed; the text to be parsed is either unannotated or annotated text.

[0165] Step 12: Convert the text to be parsed into a text string; the text string includes the hints and answers from the file to be parsed.

[0166] Step 13: Fill the prompt information and answer information from the text string into the prompt template according to the pre-determined prompt template.

[0167] Step 14: Input the filled prompt template into the pre-trained reinforcement learning agent, output the answer information in the prompt template, and convert the answer information into the prediction result of the reinforcement learning agent.

[0168] Step 15: If the prompt template is a continuous prompt template and the prediction result of the reinforcement learning agent does not match the answer information in the prompt template, then update the weight matrix parameters of the prompt template until the prediction result of the reinforcement learning agent matches the answer information in the prompt template.

[0169] In this embodiment, a hybrid algorithm combining evolutionary reinforcement learning and evolutionary strategies with cue learning and transfer learning is used to train the reinforcement learning agent. This allows for minimal modification of the reinforcement learning agent, leveraging cue information to mine the knowledge learned during large-scale pre-training. Transfer learning is then used to transfer the source domain knowledge of the reinforcement learning agent to the target domain. This avoids the need to repeatedly perform data cleaning, augmentation, and reinforcement learning agent training and deployment processes when introducing new file parsing scenarios to ensure the agent possesses background knowledge for the current scenario, thus reducing the modification and training costs of the reinforcement learning agent. Furthermore, this application defines a new cue function through cue learning, enabling the reinforcement learning agent to perform zero-label or minimally labeled data learning, adapting to new scenarios with zero or minimally labeled data.

[0170] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0171] Based on the same inventive concept, this application also provides a text processing apparatus for implementing the text processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more text processing apparatus embodiments provided below can be found in the limitations of the text processing method described above, and will not be repeated here.

[0172] In one embodiment, such as Figure 8 As shown, a text processing apparatus is provided, including: an acquisition module 100, a conversion module 200, a filling module 300, and a prediction module 400, wherein:

[0173] The acquisition module 100 is used to acquire the text to be parsed; the text to be parsed is either unannotated text or annotated text.

[0174] The conversion module 200 is used to convert the text to be parsed into a text string; the text string includes the prompt information and answer information in the file to be parsed.

[0175] The fill module 300 is used to fill the prompt information and answer information in the text string into the prompt template according to the pre-determined prompt template.

[0176] The prediction module 400 is used to input the filled prompt template into the pre-trained reinforcement learning agent, output the answer information in the prompt template, and convert the answer information into the prediction result of the reinforcement learning agent.

[0177] In one embodiment, the prediction module 400 further includes:

[0178] If the prompt template is a continuous prompt template, and the prediction result of the reinforcement learning agent does not match the answer information in the prompt template, then freeze the network parameters of the reinforcement learning agent and update the weight matrix parameters of the prompt template until the prediction result of the reinforcement learning agent matches the answer information in the prompt template.

[0179] In one embodiment, the text processing device further includes a training module 500, which is used to: acquire multiple samples; each sample includes text data and labels that annotate the answer information in the text data;

[0180] Multiple samples are input into a reinforcement learning agent, which outputs prediction results. The reward information of the prediction results output by the reinforcement learning agent is determined based on the difference between the prediction results and the labels of the samples.

[0181] Based on the reward information and the preset loss function, update the network parameters of the reinforcement learning agent to complete one training cycle;

[0182] The training process is iterated multiple times. When the loss function meets the target value, the training stops, and a well-trained reinforcement learning agent is obtained.

[0183] In one embodiment, the training module 500 is further configured to: input multiple samples into the reinforcement learning agent, and determine a reward error value between a first reward value of the reinforcement learning agent in response to each sample in the current environmental state and a second reward value of the reinforcement learning agent in response to each sample in the next environmental state.

[0184] The reward design function is determined based on the first potential function of the reinforcement learning agent in the current environment and the second potential function in the next environment; the first potential function is used to evaluate the quality of the current environment; the second potential function is used to evaluate the quality of the next environment.

[0185] Based on the reward error value and the reward design function, the reward information of the prediction result output by the reinforcement learning agent is determined.

[0186] In one embodiment, the training module 500 is further configured to: input multiple samples into a reinforcement learning agent, the reinforcement learning agent responding to the current environmental state of each sample, and outputting prediction results for each sample based on the network parameters initialized by the reinforcement learning agent, scoring the prediction results for each sample, and using the score value as the first reward value for the reinforcement learning agent to output prediction results for each sample in the current environmental state.

[0187] The reinforcement learning agent responds to the next environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the second reward value for the reinforcement learning agent to output the prediction result for each sample in the next environmental state.

[0188] The reward error value between the first reward value and the second reward value is determined based on the difference between the first reward value and the second reward value.

[0189] In one embodiment, the training module 500 is further configured to: determine the first potential function of the reinforcement learning agent in the current environment based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively, during the transfer learning process of the reinforcement learning agent in the current environment;

[0190] For the transfer learning process of a reinforcement learning agent in the next environment, the second potential function of the reinforcement learning agent in the next environment is determined based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0191] The reward design function is determined based on the second potential function and the first potential function.

[0192] Each module in the aforementioned text processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can invoke and execute the operations corresponding to each module.

[0193] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a text processing method. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0194] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0195] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0196] Get the text to be parsed; the text to be parsed can be either unannotated or annotated.

[0197] Convert the text to be parsed into a text string; the text string includes the hints and answers in the file to be parsed;

[0198] According to the pre-determined prompt template, fill in the prompt information and answer information from the text string into the prompt template respectively;

[0199] The filled-in prompt template is input into the pre-trained reinforcement learning agent, the answer information in the prompt template is output, and the answer information is converted into the prediction result of the reinforcement learning agent.

[0200] In one embodiment, when the processor executes the computer program, it further implements the following steps: if the prompt template is a continuous prompt template and the prediction result of the reinforcement learning agent does not match the answer information in the prompt template, then freeze the network parameters of the reinforcement learning agent and update the weight matrix parameters of the prompt template until the prediction result of the reinforcement learning agent matches the answer information in the prompt template.

[0201] In one embodiment, the processor, when executing the computer program, further performs the following steps: acquiring multiple samples; each sample includes text data and labels that annotate answer information in the text data;

[0202] Multiple samples are input into a reinforcement learning agent, which outputs prediction results. The reward information of the prediction results output by the reinforcement learning agent is determined based on the difference between the prediction results and the labels of the samples.

[0203] Based on the reward information and the preset loss function, update the network parameters of the reinforcement learning agent to complete one training cycle;

[0204] The training process is iterated multiple times. When the loss function meets the target value, the training stops, and a well-trained reinforcement learning agent is obtained.

[0205] In one embodiment, when the processor executes the computer program, it further performs the following steps: inputting multiple samples into the reinforcement learning agent, and determining a reward error value between a first reward value of the reinforcement learning agent in response to each sample in the current environmental state and a second reward value of the reinforcement learning agent in response to each sample in the next environmental state.

[0206] The reward design function is determined based on the first potential function of the reinforcement learning agent in the current environment and the second potential function in the next environment; the first potential function is used to evaluate the quality of the current environment; the second potential function is used to evaluate the quality of the next environment.

[0207] Based on the reward error value and the reward design function, the reward information of the prediction result output by the reinforcement learning agent is determined.

[0208] In one embodiment, when the processor executes the computer program, it further implements the following steps: inputting multiple samples into a reinforcement learning agent, the reinforcement learning agent responding to the current environmental state of each sample, and outputting a prediction result for each sample based on the network parameters initialized by the reinforcement learning agent, scoring the prediction result for each sample, and using the score value as the first reward value for the reinforcement learning agent to output the prediction result for each sample in the current environmental state.

[0209] The reinforcement learning agent responds to the next environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the second reward value for the reinforcement learning agent to output the prediction result for each sample in the next environmental state.

[0210] The reward error value between the first reward value and the second reward value is determined based on the difference between the first reward value and the second reward value.

[0211] In one embodiment, when the processor executes the computer program, it further performs the following steps: for the transfer learning process of the reinforcement learning agent in the current environment, based on the reward functions of the reinforcement learning agent in the source domain and the target domain respectively, determine the first potential function of the reinforcement learning agent in the current environment;

[0212] For the transfer learning process of a reinforcement learning agent in the next environment, the second potential function of the reinforcement learning agent in the next environment is determined based on the reward functions of the reinforcement learning agent in the source domain and the target domain, respectively.

[0213] The reward design function is determined based on the second potential function and the first potential function.

[0214] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0215] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0217] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0218] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0219] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A text processing method, characterized in that, The method includes: Obtain the text to be parsed; the text to be parsed can be either unannotated or annotated text. The text to be parsed is converted into a text string; the text string includes the hint information and answer information in the text to be parsed. According to the predetermined prompt template, fill the prompt information and answer information in the text string into the prompt template respectively; The filled-in prompt template is input into a pre-trained reinforcement learning agent, and the answer information in the prompt template is output. The answer information is converted into the prediction result of the reinforcement learning agent. If the prompt template is a continuous prompt template and the prediction result of the reinforcement learning agent does not match the answer information in the prompt template, the network parameters of the reinforcement learning agent are frozen, and the weight matrix parameters of the prompt template are updated until the prediction result of the reinforcement learning agent matches the answer information in the prompt template. The training process of the reinforcement learning agent includes: Multiple samples are acquired; each sample includes text data and labels that annotate the answer information in the text data; the multiple samples are input into the reinforcement learning agent, and the prediction result is output. Based on the difference between the prediction result and the label of the sample, the reward information of the prediction result output by the reinforcement learning agent is determined; the network parameters of the reinforcement learning agent are updated according to the reward information and the preset loss function to complete one training cycle; the training process is iterated multiple times, and when the loss function meets the target value, the training is stopped to obtain the trained reinforcement learning agent; The process of inputting multiple samples into a reinforcement learning agent, outputting prediction results, and determining reward information for the prediction results output by the reinforcement learning agent based on the difference between the prediction results and the labels of the samples includes: Multiple samples are input into the reinforcement learning agent. A reward error value is determined between a first reward value (for each sample in the current environment) and a second reward value (for each sample in the next environment). A reward design function is determined based on the first potential function in the current environment and the second potential function in the next environment. The first potential function is used to evaluate the quality of the current environment, and the second potential function is used to evaluate the quality of the next environment. The reward information for the prediction results output by the reinforcement learning agent is determined based on the reward error value and the reward design function.

2. The method according to claim 1, characterized in that, The step of inputting multiple samples into a reinforcement learning agent and determining the reward error value between a first reward value (the reward value of the reinforcement learning agent outputting a prediction result for each sample in the current environmental state) and a second reward value (the reward value of the reinforcement learning agent outputting a prediction result for each sample in the next environmental state) includes: Multiple samples are input into a reinforcement learning agent. The reinforcement learning agent responds to the current environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the first reward value for the reinforcement learning agent to output the prediction result for each sample in the current environmental state. The reinforcement learning agent responds to the next environmental state of each sample and outputs the prediction result for each sample based on the network parameters initialized by the reinforcement learning agent. The prediction result for each sample is scored, and the score value is used as the second reward value for the reinforcement learning agent to output the prediction result for each sample in the next environmental state. The reward error value between the first reward value and the second reward value is determined based on the difference between the first reward value and the second reward value.

3. The method according to claim 1, characterized in that, The step of determining the reward design function based on the first potential function of the reinforcement learning agent in the current environment and the second potential function in the next environment includes: For the transfer learning process of the reinforcement learning agent in the current environment, the first potential function of the reinforcement learning agent in the current environment is determined according to the reward function of the reinforcement learning agent in the source domain and the target domain respectively. For the transfer learning process of the reinforcement learning agent in the next environment, the second potential function of the reinforcement learning agent in the next environment is determined according to the reward function of the reinforcement learning agent in the source domain and the target domain respectively. The reward design function is determined based on the second potential function and the first potential function.

4. A text processing device, characterized in that, The device includes: The acquisition module is used to acquire the text to be parsed; the text to be parsed can be unlabeled text or labeled text. A conversion module is used to convert the text to be parsed into a text string; the text string includes the prompt information and answer information in the text to be parsed. The fill module is used to fill the prompt information and answer information in the text string into the prompt template according to the pre-determined prompt template; The prediction module is used to input the filled prompt template into a pre-trained reinforcement learning agent, output the answer information in the prompt template, and convert the answer information into the prediction result of the reinforcement learning agent; if the prompt template is a continuous prompt template and the prediction result of the reinforcement learning agent does not match the answer information in the prompt template, then the network parameters of the reinforcement learning agent are frozen, and the weight matrix parameters of the prompt template are updated until the prediction result of the reinforcement learning agent matches the answer information in the prompt template; The training process of the reinforcement learning agent includes: Multiple samples are acquired; each sample includes text data and labels that annotate the answer information in the text data; the multiple samples are input into the reinforcement learning agent, and the prediction result is output. Based on the difference between the prediction result and the label of the sample, the reward information of the prediction result output by the reinforcement learning agent is determined; the network parameters of the reinforcement learning agent are updated according to the reward information and the preset loss function to complete one training cycle; the training process is iterated multiple times, and when the loss function meets the target value, the training is stopped to obtain the trained reinforcement learning agent; The process of inputting multiple samples into a reinforcement learning agent, outputting prediction results, and determining reward information for the prediction results output by the reinforcement learning agent based on the difference between the prediction results and the labels of the samples includes: Multiple samples are input into the reinforcement learning agent. A reward error value is determined between a first reward value (for each sample in the current environment) and a second reward value (for each sample in the next environment). A reward design function is determined based on the first potential function in the current environment and the second potential function in the next environment. The first potential function is used to evaluate the quality of the current environment, and the second potential function is used to evaluate the quality of the next environment. The reward information for the prediction results output by the reinforcement learning agent is determined based on the reward error value and the reward design function.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images