Training method, device, equipment and storage medium for sustainable learning model

Abstract

The present application discloses a training method, device, equipment and storage medium for a sustainable learning model, which belongs to the field of artificial intelligence technology. The method includes: obtaining unlearned new task data based on the sustainable learning model; obtaining a first loss of the sustainable learning model based on the new task data; obtaining a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data; obtaining a third loss of the sustainable learning model based on the sustainable learning model, the old task data and the new task data; and training the sustainable learning model based on the first loss, the second loss and the third loss. In the present application, the model is trained by the first loss, the second loss and the third loss to improve the accuracy and generalization of the sustainable learning model, reduce the forgetting of old knowledge of the sustainable learning model, and thus improve the practicality of the sustainable learning model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device and storage medium for a sustainable learning model. Background Art

[0002] Currently, a dialogue generation model is obtained through model training for application in automatic dialogue.

[0003] In related technologies, in order to ensure the effectiveness of the dialogue generation model, before training the dialogue generation model, it is necessary to obtain a large amount of different types of task data, and then use the task data to train the dialogue generation model so that the dialogue generation model after training can be applicable to various types of automatic dialogue scenarios.

[0004] However, in the above-mentioned related technologies, due to the limitations of data collection, it is difficult to collect enough task data before model training, resulting in poor practicality of the dialogue generation model after training. Summary of the Invention

[0005] The embodiments of the present application provide a training method, apparatus, device, and storage medium for a sustainable learning model, which improves the accuracy, generalization, and practicality of the sustainable learning model and reduces the forgetting of old knowledge. The technical solution is as follows.

[0006] According to one aspect of an embodiment of the present application, a method for training a sustainable learning model is provided, the method comprising the following steps:

[0007] Acquire unlearned new task data based on the sustainable learning model;

[0008] Obtaining a first loss of the sustainable learning model according to the new task data, where the first loss is used to measure the difference between a prediction result of the sustainable learning model for the new task data and a standard result;

[0009] Obtaining a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data, where the second loss is used to measure the degree of forgetting of the old task data by the sustainable learning model;

[0010] Obtaining a third loss of the sustainable learning model based on the sustainable learning model, the old task data, and the new task data, wherein the third loss is used to measure the migration ability of the sustainable learning model from the old task data to the new task data;

[0011] The sustainable learning model is trained according to the first loss, the second loss and the third loss.

[0012] According to one aspect of an embodiment of the present application, a training device for a sustainable learning model is provided, the device comprising the following modules:

[0013] A data acquisition module, configured to acquire unlearned new task data based on the sustainable learning model;

[0014] A first acquisition module is configured to acquire a first loss of the sustainable learning model according to the new task data, wherein the first loss is used to measure the difference between a prediction result of the sustainable learning model for the new task data and a standard result;

[0015] A second acquisition module is configured to acquire a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data, where the second loss is used to measure the degree of forgetting of the sustainable learning model with respect to the old task data;

[0016] a third acquisition module, configured to acquire a third loss of the sustainable learning model based on the sustainable learning model, the old task data, and the new task data, wherein the third loss is used to measure the migration ability of the sustainable learning model from the old task data to the new task data;

[0017] A model training module is used to train the sustainable learning model according to the first loss, the second loss and the third loss.

[0018] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the above-mentioned training method of the sustainable learning model.

[0019] According to one aspect of an embodiment of the present application, an embodiment of the present application provides a computer-readable storage medium, in which at least one program is stored, and the at least one program is loaded and executed by a processor to implement the above-mentioned training method of the sustainable learning model.

[0020] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described method for training a sustainable learning model.

[0021] The technical solutions provided in the embodiments of the present application can bring the following beneficial effects:

[0022] The third loss of the sustainable model is determined by the new task data and the old task data, and the third loss is used to measure the generalization and accuracy of the sustainable learning model for the old task data and the new task data. The third loss is subsequently used to train the sustainable learning model to improve the generalization and accuracy of the sustainable learning model for the new task data; moreover, the first loss is determined by the new task data, and the second loss is determined by the old task data. The first loss is used to measure the accuracy of the output result of the sustainable learning model for the new task data, and the second loss is used to measure the degree of forgetting of the sustainable learning model for the old task data. The first loss and the second loss are subsequently used to train the sustainable learning model, so that the sustainable learning model reduces the forgetting of the old task data while learning the new task data; in short, the present application performs model training through the first loss, the second loss and the third loss, improves the accuracy and generalization of the sustainable learning model, reduces the forgetting of old knowledge of the sustainable learning model, and thereby improves the practicality of the sustainable learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of a model architecture of a sustainable learning model provided by one embodiment of the present application;

[0024] Figure 2 A schematic diagram of an automatic interaction system is shown as an example;

[0025] Figure 3 This is a flowchart of a training method for a sustainable learning model provided by one embodiment of the present application;

[0026] Figure 4 A schematic diagram exemplarily showing new task data and old task data;

[0027] Figure 5 This is a flowchart of a training method for a sustainable learning model in the medical field provided by one embodiment of the present application;

[0028] Figure 6 A schematic diagram exemplarily shows a user interface of an automatic dialogue system in the medical field;

[0029] Figure 7 This is a block diagram of a training device for a sustainable learning model provided by one embodiment of the present application;

[0030] Figure 8 is a block diagram of a training device for a sustainable learning model provided by another embodiment of the present application;

[0031] Figure 9 This is a structural block diagram of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0033] The training method of the sustainable learning model in this application involves the following technologies:

[0034] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0035] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0036] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0037] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0038] The solution provided in the embodiments of this application involves technologies such as machine learning of artificial intelligence. Below, the technical solution of this application is introduced and explained in conjunction with several embodiments.

[0039] Please refer to Figure 1 , which shows a schematic diagram of a model architecture of a sustainable learning model provided by an embodiment of the present application. The sustainable learning model may include: a learning network 10, a forgetting preventer 20, and a new task adapter 30.

[0040] The learning network 10 is used to learn and acquire new task data. Optionally, in the embodiment of the present application, the learning network 10 processes the state information s to generate the action information a corresponding to the state information s, and then obtains the reward r of the action information and the next state information s' corresponding to the action information. Further, the state transition information is composed of a four-tuple (s, a, r, s'), and the new task data is composed of multiple state transition information. The new task data is stored in the new task storage area D. n Optionally, in an embodiment of the present application, the learning network 10 includes a policy network and a target network. The policy network generates new task data and the prediction results of the sustainable learning model based on state information. The target network generates a standard result of the sustainable learning model based on the new task data, and then generates a first loss of the sustainable learning model based on the prediction results and the standard result. The first loss is used to measure the accuracy of the output results of the sustainable learning model for the new task data.

[0041] The forgetfulness preventer 20 is used to reduce the degree of forgetfulness of old task data when the sustainable learning model learns new task data. Optionally, in this embodiment of the present application, the forgetfulness preventer 20 obtains old task data (task 1 data and task 2 data) from the old task storage area D0 and uses the old task data to obtain a second loss for the sustainable learning model. The second loss is used to measure the degree of forgetfulness of the sustainable learning model for the old task data.

[0042] The new task adapter 30 is used to improve the generalization of the sustainable learning model. Optionally, in the embodiment of the present application, the new task adapter 30 is used to store the new task in the new task storage area D n Obtain new task data, obtain old task data from the old task storage area D0, and obtain the third loss based on the new task data and the old task data. The third loss is used to measure the generalization and accuracy of the sustainable learning model for the new task data and the old task data.

[0043] Optionally, in an embodiment of the present application, a total loss of the sustainable learning model is generated from the first loss, the second loss, and the third loss, and the sustainable learning model is trained based on the total loss. Optionally, during training, network parameters of the policy network in the sustainable learning model are adjusted based on the total loss.

[0044] Optionally, the above-mentioned sustainable learning model can be applied to an automatic interactive system. For example, Figure 2 As shown, the automatic interactive system includes a terminal 40 and a server 50.

[0045] The terminal 40 may be an electronic device such as a mobile phone, a tablet computer, a game console, an e-book reader, a multimedia player, a wearable device, a PC (Personal Computer), an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, and an aircraft, and the present embodiment of the application does not limit this. Optionally, the terminal 40 includes an application client. The application may be any application with automatic interaction capabilities, such as a social application, a query application, etc. Optionally, the above application may be an application that needs to be downloaded and installed, or an application that can be used immediately upon clicking, and the present embodiment of the application does not limit this.

[0046] The server 50 is used to provide background services for the terminals 40. The server 50 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Alternatively, the server 50 can be a background server for the client of the above-mentioned application. In an exemplary embodiment, the server 50 provides background services for multiple terminals 40.

[0047] Data is transmitted between the terminal 40 and the server 50 via a network. Optionally, the terminal 40 senses a query text input by the user and sends the query text to the server 50 via the network. Furthermore, the server 50 obtains corresponding state information based on the query text and, based on a continuous learning model, obtains action information corresponding to the state information. The server 50 then determines a response strategy corresponding to the query text and sends the response strategy corresponding to the query text to the terminal 40 via the network. The terminal 40 determines a response text corresponding to the query text based on the response strategy, then continues to sense the next query text corresponding to the response text and sends the next query text as a new query text to the server 50. The above steps are repeated until the interaction ends.

[0048] One thing that needs to be explained is that the above Figure 2 The description is merely exemplary and explanatory. In exemplary embodiments, the functions of terminal 40 and server 50 can be flexibly configured and adjusted, and the present application does not limit this. For example, terminal 40 obtains status information corresponding to the query text, and server 50 generates a response text based on the response strategy.

[0049] It should also be noted that the training model of the above-mentioned sustainable learning model can be the above-mentioned server 50 or other computer devices, and this embodiment of the present application does not limit this.

[0050] Optionally, the automatic interaction system may be an automatic dialogue system.

[0051] Please refer to Figure 3 , which shows a flow chart of a method for training a sustainable learning model provided by one embodiment of the present application. The execution entity of each step in the method can be the server 50 described above, or other computer devices other than the server 50 (hereinafter collectively referred to as "computer devices"). The method can include at least one of the following steps (301-305):

[0052] Step 301: Acquire unlearned new task data based on the sustainable learning model.

[0053] A sustainable learning model is one that has the ability to continuously learn. Continuous learning is a form of incremental learning, also known as lifelong learning. It breaks the gap between the training and testing phases in traditional machine learning and aims to develop lifelong learning algorithms that continuously update model parameters using knowledge accumulated over time, enabling the model to achieve increasingly higher accuracy and better coverage.

[0054] Optionally, in the embodiment of the present application, the sustainable learning model corresponds to new task and old task data. The new task data refers to the task data that the sustainable learning model has not learned, and the old task data refers to the task data that the sustainable learning model has learned. For example, Figure 4 As shown, during the initial training, the new task data is the first task data, and the old task data is empty; during the second training, the new task data is the second task data, and the old task data is the first task data; during the third training, the new task data is the third task data, and the old task data is the first task data and the second task data, and so on. During the M-th training, the new task data is the M-th task data, and the old task data is the first task data, the second task data...the M-1-th task data.

[0055] In one possible implementation, the new task data and the old task data are distinguished based on the collection time, that is, the new task data refers to the task data collected during the operation of the sustainable learning model. Optionally, during the operation of the sustainable learning model, the computer device obtains new task data that has not been learned based on the sustainable learning model. It should be noted that in this case, there is no type division between the new task data and the old task data; for example, taking the medical field as an example, the new task data is the unlearned task data, which may include task data in the fields of orthopedics, gynecology, and burns. Similarly, the old task data is the learned task data, which may also include task data in the fields of orthopedics, gynecology, and burns. There is no type division between the new task data and the old task data.

[0056] In another possible implementation, new task data and old task data are distinguished based on data type. Optionally, the computer device determines unlearned new task data based on the data type of the learned old task data. For example, in the medical field, if the old task data includes orthopedic task data and gynecological task data, the new task data includes burn task data, but does not include orthopedic task data and gynecological task data.

[0057] Of course, in an exemplary embodiment, new task data and old task data can also be distinguished based on both collection time and data type. The distinction can be flexibly adjusted according to actual conditions, and the embodiments of the present application do not limit this.

[0058] One thing that needs to be explained is that since the above-mentioned new task data is the unlearned data of the sustainable learning model, the sustainable learning model does not have good accuracy and generalization for the new task data, and subsequent learning is needed to improve the accuracy and generalization of the sustainable learning model for the new task data.

[0059] Step 302: Obtain the first loss of the sustainable learning model according to the new task data.

[0060] In an embodiment of the present application, after acquiring the new task data, the computer device acquires a first loss of the sustainable learning model based on the new task data, wherein the first loss is used to measure the accuracy of the output result of the sustainable learning model for the new task data.

[0061] Optionally, the sustainable learning model includes a policy network and a target network. Optionally, in the embodiment of the present application, the first loss can also be understood as a measure of the difference between the prediction result of the sustainable learning model for the new task data and the standard result. In an exemplary embodiment, the above step 302 includes at least one of the following steps:

[0062] 1. Determine the prediction results of the sustainable learning model based on the new task data and the network parameters of the policy network in the sustainable learning model;

[0063] 2. Input the state information in the new task data into the target network in the sustainable learning model to obtain the output result of the target network for the state information, wherein the output result is the action information output by the target network for the state information;

[0064] 3. Based on the output results of the target network for state information and the network parameters of the target network, the standard results of the sustainable learning model are obtained;

[0065] 4. Determine the first loss based on the standard results and the predicted results.

[0066] For example, assuming that the state transition information (s, a, r, s') in the new task data, the network parameters of the policy network are θ 1：k+1 , the prediction result of the sustainable learning model is Q(s, a; θ 1：k+1 ), the network parameters of the target network are θ' 1：k+1 , the action information corresponding to s' is a', then the first loss L Q (θ 1：k+1 )for:

[0067]

[0068] Where γ∈[0,1] is a discount factor used to balance short-term and long-term effects; y 1：k+1 Refers to the standard outcomes of the sustainable learning model.

[0069] Optionally, the network parameters of the target network are copied from the network parameters of the historical policy network. In an embodiment of the present application, during the policy network update process, the computer device obtains the number of network updates between the historical policy network and the policy network. Further, if the number of network updates meets the update period, the network parameters of the target network are updated based on the policy network. That is, during the policy network update process, the target network is updated every update period (Z). The update period can be any value and can be flexibly set and adjusted according to actual conditions. This embodiment of the present application does not limit this.

[0070] Optionally, in an embodiment of the present application, the above-mentioned state information may also be referred to as a state vector, and the above-mentioned action information may also be referred to as an action vector, which is not limited in the embodiment of the present application.

[0071] Step 303: Obtain a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data.

[0072] In an embodiment of the present application, after the computer device obtains the learned old task data, it obtains a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data. The second loss is used to measure the degree of forgetting of the old task data by the sustainable learning model.

[0073] Optionally, the above-mentioned sustainable learning model includes a forgetting preventer, which is used to obtain a second loss based on the sustainable learning model and the old task data; that is, after the computer device obtains the above-mentioned old task data, the forgetting preventer obtains the second loss of the sustainable learning model based on the sustainable learning model and the learned old task data.

[0074] In an exemplary embodiment, step 303 includes at least one of the following steps:

[0075] 1. Input the state information in the old task data into the policy network of the sustainable learning model to obtain the current action information corresponding to the state information.

[0076] In an embodiment of the present application, after obtaining the above-mentioned old task data, the computer device uses the policy network in the sustainable learning model to process the state information in the old task data, and then obtains the current action information corresponding to the state information output by the policy network.

[0077] 2. Determine a second loss based on the current action information corresponding to the state information and the historical action information corresponding to the state information contained in the old task data.

[0078] In this embodiment of the present application, after obtaining the current action information, the computer device determines the second loss based on the current action information corresponding to the state information and the historical action information corresponding to the state information contained in the old task data. The historical action information refers to the action information corresponding to the state information obtained during the old task data collection process. It should be noted that during the old task data collection process, the old task data is considered new task data.

[0079] Optionally, the old task data includes multiple data subsets, with different data subsets corresponding to different collection times. Furthermore, a data subset includes multiple tasks, and each task corresponds to multiple state transition information. A state transition information includes a state, the action information corresponding to the state information, the reward for the action information, and the next state information corresponding to the action information. Taking an automated dialogue system as an example, the old task data includes first task data and second task data. The first task data refers to the data learned by the continuous learning model two days ago, and the second task data refers to the data learned by the continuous learning model one day ago. The first task data includes multiple tasks, each task being a completed dialogue, and a dialogue includes multiple state transition information. The target state transition information in the multiple state transition information includes the state information of the query text, the action information corresponding to the state information, the reward for the action information, and the state information corresponding to the next query text. The next query text refers to the next query information entered by the user under the influence of the action information.

[0080] In short, as can be seen from the above, the old task data includes multiple pieces of state information. When obtaining the current action information, the computer device inputs each piece of state information into the policy network to obtain the current action information corresponding to each piece of state information. Furthermore, for the target state information among the multiple pieces of state information, the policy network determines the forgetting rate of the target state information based on the current action information corresponding to the target state information and the historical action information corresponding to the target state information. Furthermore, based on the forgetting rate of the policy network for each piece of state information in the old task data, the second loss is determined.

[0081] For example, the old task data (T1, T2...T k ) in the i-th data subset T i A historical task τ i Upper strategy π i The forgetting rate is defined as the action change rate of the task state following the strategy, that is, the forgetting rate f π (τ i ):

[0082]

[0083] Among them, I a＝π(.|s) is an indicator function that maps state-action pairs (s, a) that satisfy the condition a = π(.|s) to 1, which filters out those state-action pairs that are remembered; |x| A Is an operator that indicates the number of elements x conditioned on A. In theory, a sufficiently large buffer can be used to retain and store all old task data. However, as the number of tasks increases, it is not easy to maintain such a large buffer. Therefore, the initial old task data is processed to obtain the actual stored old task data. Considering the old task data Therefore, T i Upper strategy π i The forgetting rate f π (τ i ) can be estimated as:

[0084]

[0085] Among them, N i It's T i The amount of data included.

[0086] Will Applied to indicator functions to train value functions To minimize the following formula:

[0087]

[0088] In the embodiment of this application, in order to prevent the old task data from being forgotten, the cross entropy loss is used for training to minimize the action a of the old task data. t i and the loss between the corresponding current action, that is, the second loss L CQL (θ 1：k+1 )for:

[0089]

[0090] Optionally, the above-mentioned old task data includes data subsets collected at different times, that is, the task data is determined based on the collection time, and there is no need to divide the old task data and new tasks according to the type of task data, which reduces the data preprocessing process and improves the training efficiency of the sustainable learning model.

[0091] Step 304 : Obtain a third loss of the sustainable learning model based on the sustainable learning model, the old task data, and the new task data.

[0092] In an embodiment of the present application, after obtaining the aforementioned old task data and the aforementioned new task data, the computer device obtains a third loss for the sustainable learning model based on the sustainable learning model, the old task data, and the new task data. The third loss is used to measure the sustainable learning model's ability to migrate from the old task data to the new task data. It is understood that migration ability can also be referred to as adaptability. That is, in this embodiment of the present application, the third loss is used to measure the generalization and accuracy of the sustainable learning model with respect to the old task data and the new task data. In short, using the third loss to train the sustainable learning model can improve the coverage of the sustainable learning model, enabling the sustainable learning model to better process new task data in addition to its original ability to better process the old task data.

[0093] Optionally, the above-mentioned sustainable learning model includes a new task adapter, which is used to obtain a third loss based on the sustainable learning model, old task data and new task data; that is, after the computer device obtains the above-mentioned new task data and old task data, the new task adapter obtains the third loss of the sustainable learning model based on the sustainable learning model, old task data and new task data.

[0094] In an exemplary embodiment, step 304 includes at least one of the following steps:

[0095] 1. Input the new task data into the policy network of the sustainable learning model to obtain the first empirical risk and the first gradient norm penalty.

[0096] The first empirical risk is used to indicate the accuracy of the output results of the policy network for new task data, and the first gradient norm penalty is used to indicate the generalization of the policy network for new task data.

[0097] Optionally, in an embodiment of the present application, for new task data, when obtaining the first empirical risk, the above-mentioned first loss is determined as the first empirical risk; when obtaining the first gradient norm penalty, the state information in the new task data is input into the policy network, and the parameters of the policy network are regularized to obtain the first gradient norm penalty.

[0098] 2. Input the old task data into the policy network to obtain the second empirical risk and the second gradient norm penalty.

[0099] The second empirical risk is used to indicate the accuracy of the output results of the policy network for the old task data, and the second gradient norm penalty is used to indicate the generalization of the policy network for the old task data.

[0100] Optionally, in an embodiment of the present application, the old task data includes at least one data subset. When obtaining the second empirical risk and the second gradient norm penalty, for a target data subset in the at least one data subset, state information in the target data subset is input into the policy network to obtain the empirical risk and gradient norm penalty corresponding to the target data subset. Furthermore, a second empirical wind direction is determined based on the empirical risks corresponding to each data subset, and a second gradient norm penalty is determined based on the gradient norm penalty corresponding to each data subset.

[0101] 3. Determine the third loss based on the first empirical risk, the first gradient norm penalty, the second empirical risk, and the second gradient norm penalty.

[0102] In the embodiment of the present application, the computer device determines the third loss based on the first empirical risk, the first gradient norm penalty, the second empirical risk and the second gradient norm penalty. For example, the third loss L AA (θ 1：k+1 ) can be expressed as:

[0103]

[0104] Step 305 : Training the sustainable learning model based on the first loss, the second loss, and the third loss.

[0105] In this embodiment of the present application, after obtaining the first, second, and third losses, the computer device trains the sustainable learning model based on the first, second, and third losses. Optionally, in this embodiment of the present application, the computer device determines a total loss of the sustainable learning model based on the first, second, and third losses, and then adjusts network parameters of the policy network of the sustainable learning model based on the total loss.

[0106] For example, the total loss L of the continuous learning model is:

[0107]

[0108] in, According to the above L Q (θ 1：k+1 ) in dataset D n Obtained on According to the above L AA (θ 1：k+1 ) in dataset D n and D 0 Obtained on According to the above L CQL (θ 1：k+1 ) in dataset D 0 Obtained from ;D n Used to represent new task data, D 0 Used to represent old task data.

[0109] To sum up, in the technical solution provided by the embodiment of the present application, the third loss of the sustainable model is determined by the new task data and the old task data, and the third loss is used to measure the generalization and accuracy of the sustainable learning model for the old task data and the new task data. The third loss is subsequently used to train the sustainable learning model to improve the generalization and accuracy of the sustainable learning model for the new task data; moreover, the first loss is determined by the new task data, and the second loss is determined by the old task data. The first loss is used to measure the accuracy of the output result of the sustainable learning model for the new task data, and the second loss is used to measure the degree of forgetting of the sustainable learning model for the old task data. The first loss and the second loss are subsequently used to train the sustainable learning model, so that the sustainable learning model can reduce the forgetting of the old task data while learning the new task data; in short, the present application uses the first loss, the second loss and the third loss to perform model training, improve the accuracy and generalization of the sustainable learning model, reduce the forgetting of old knowledge of the sustainable learning model, and thereby improve the practicality of the sustainable learning model.

[0110] Next, the method for obtaining the above new task data is introduced.

[0111] In an exemplary embodiment, step 301 includes at least one of the following steps:

[0112] 1. Get current status information.

[0113] The status information is used to describe the status of the target object. Optionally, different target objects correspond to different status information. For example, if the target object is text, the status information is used to indicate entity words extracted from the text that can characterize the text features; if the target object is a plant, the status information is used to indicate the plant growth, ambient temperature, ambient humidity, etc.; if the target object is an animal, the status information is used to indicate weight gain data, lifestyle change data, etc.

[0114] In this embodiment of the present application, a computer device obtains a first target object and, based on the first target object, obtains current state information. Taking an automated dialogue system as an example, the first target object is a query text entered by a user; the current state information indicates key entity words in the query text, as well as relevant content of the user's account information (e.g., gender, age, place of residence, etc.).

[0115] 2. Input the current state information into the strategy network of the sustainable learning model to obtain the action information corresponding to the current state information.

[0116] In an embodiment of the present application, after obtaining the above-mentioned current state information, the computer device uses the policy network of the sustainable learning model to process the current state information, and then obtains the action information corresponding to the current state information output by the policy network. The action information is used to indicate the response that should be made based on the current state information. For example, if the above-mentioned first target object is a text (such as a query text in an automatic dialogue system), the above-mentioned action information is used to indicate the response strategy corresponding to the text (such as the response text corresponding to the query text in the automatic dialogue system); if the above-mentioned first target object is a plant, the above-mentioned action information is used to indicate the irrigation strategy for the plant; if the above-mentioned first target object is an animal, the above-mentioned action information is used to indicate the feeding strategy for the animal.

[0117] 3. Obtain the reward for the action information corresponding to the current state information, as well as the next state information obtained for the action information corresponding to the current state information.

[0118] In an embodiment of the present application, after obtaining the action information corresponding to the above-mentioned current state information, the computer device obtains a reward for the action information and next state information obtained for the action information.

[0119] The reward is used to characterize the quality of a strategy based on the action information. Optionally, the computer device evaluates the action information based on an evaluation mechanism to obtain the reward. Exemplarily, the evaluation mechanism includes an evaluation network, which evaluates the action information to determine the reward; or the evaluation mechanism includes evaluation rules, which evaluate the action information based on the evaluation rules to determine the reward.

[0120] The above-mentioned next-state information refers to the state information collected under the influence of the action information. In one possible implementation, the next-state information is obtained based on the response information corresponding to the action information; taking the automatic dialogue system as an example, after obtaining the new query text output by the user in response to the above-mentioned response text, the next-state information is obtained based on the new query text. In another possible implementation, the next-state information is obtained based on the response information corresponding to the action information and the historical target object before this task; taking the automatic dialogue system as an example, after obtaining the new query text output by the user in response to the above-mentioned response text, the next-state information is obtained based on the new query text and the historical query texts during this dialogue.

[0121] 4. Determine the next state information as the new current state information, and start again from the step of inputting the current state information into the policy network of the sustainable learning model to obtain the action information corresponding to the current state information, until the current task is completed.

[0122] In an embodiment of the present application, after obtaining the above-mentioned next state information, the computer device determines the next state information as the new current state information, and then repeats the above-mentioned steps again until the current task is completed.

[0123] 5. Based on the current task, generate new task data corresponding to the current task.

[0124] In an embodiment of the present application, after the current task is completed, new task data corresponding to the current task is generated based on the current task. The new task data corresponding to the current task includes multiple state transition information; for target state transition information among the multiple state transition information, the target state transition information includes the current target state information, the target action information corresponding to the current target state information, the reward for the target action information, and the next target state information corresponding to the target action information.

[0125] It should be noted that, in the embodiment of the present application, the above-mentioned new task data includes new task data corresponding to multiple tasks, and different tasks correspond to different state transition information.

[0126] To sum up, in the technical solution provided by the embodiment of the present application, the sustainable learning model collects new task data through current status information during operation, continuously adds new task data, and subsequently uses the new task data to train the sustainable learning model, thereby improving the coverage of the sustainable learning model for different knowledge, thereby improving the practicality of the sustainable learning model.

[0127] Next, the method for obtaining the above-mentioned learned old task data is introduced.

[0128] 1. Obtain the initial old task data;

[0129] 2. Perform linear attenuation processing on the initial old task data to obtain the processed old task data;

[0130] 3. Based on the correlation between the new task data and the processed old task data, sample the processed old task data to obtain the learned old task data.

[0131] Linear decay processing refers to the operation of filtering the initial old task data based on the collection time. Optionally, in an embodiment of the present application, after obtaining the initial old task data, the computer device removes part of the data from the initial old task data based on the collection time of the initial old task data to obtain the processed old task data. The collection time is used to indicate the collection time of the initial old task data. Optionally, there is a positive correlation between the collection time and the partial data, that is, the earlier the collection time, the greater the amount of partial data removed.

[0132] Optionally, in an embodiment of the present application, after obtaining the processed old task data, the computer device stores the processed old task data in an old task storage area. Subsequently, when determining to use the new task data to train the sustainable learning model, the computer device samples the learned old task data from the old task storage area. The learned old task data is the old task data required for training the sustainable learning model.

[0133] Optionally, in an embodiment of the present application, when acquiring learned old task data, the computer device obtains a correlation between the processed old task data and the new task data; further, data with a correlation less than or equal to a threshold value is sampled from the processed old task data to obtain the old task data. The threshold value can be any value and can be flexibly set and adjusted according to actual circumstances, and is not limited in this embodiment of the present application.

[0134] To sum up, in the technical solution provided in the embodiment of the present application, by processing the initial old task data, part of the task data is obtained from the initial old task data as the learned old task data, thereby reducing the amount of old task data used by the sustainable learning model during the training process and improving the training efficiency of the sustainable learning model.

[0135] Moreover, when processing the initial old task data, on the one hand, linear attenuation processing is used to reduce the amount of processed task data that needs to be stored, avoiding energy consumption caused by excessive data storage space. On the other hand, the old task data that the model needs to use is determined based on the correlation between the new and old task data, thereby improving the training efficiency of the sustainable learning model.

[0136] In addition, if Figure 5 As shown in the figure, taking the automatic dialogue system in the medical field as an example, the training process of the sustainable learning model is introduced. The specific steps are as follows:

[0137] Step 501: Acquire current status information corresponding to a first query text input by a user. Optionally, the current status information is used to indicate symptom description information acquired from the first query text.

[0138] Step 502 : Based on the policy network of the sustainable learning model, the current state information is processed to obtain action information corresponding to the current state information.

[0139] Step 503: Obtain the reward for the action information corresponding to the current state information, as well as the next state information obtained for the action information corresponding to the current state information. The next state information is determined as the new current state information, and the process repeats from step 502 until the current task is completed.

[0140] Step 504: Generate new task data and store the new task data in a new task storage area.

[0141] Step 505: Obtain old task data from the old task storage area.

[0142] Step 506 : Obtain a first loss of the sustainable learning model based on the new task data and the target network of the sustainable learning model.

[0143] Step 507 : Using the forgetting preventer of the sustainable learning model, based on the sustainable learning model and the old task data, obtain a second loss of the sustainable learning model.

[0144] Step 508 : Using the new task adapter of the sustainable learning model, based on the sustainable learning model, the old task data, and the new task data, obtain a third loss of the sustainable learning model.

[0145] Step 509 : training the sustainable learning model according to the first loss, the second loss, and the third loss.

[0146] For example, in the application process of the sustainable learning model, the user interface of the automatic dialogue system in the medical field is as follows: Figure 6 As shown, the user inputs a query text 61, and the automatic dialogue system automatically outputs a response text 62 corresponding to the query text 61.

[0147] It should be noted that the above introduction to the medical field is only exemplary and explanatory. The sustainable learning model can also be applied to other fields, such as agriculture, animal husbandry, etc.

[0148] It is understandable that in the specific implementation of this application, user information and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and maps. For example, the collection of new task data and the collection of old task data involved in this application are all collected with the permission of the user and in compliance with relevant laws, regulations and standards; illustratively, when collecting data, the subsequent data collection is introduced in the form of a pop-up window in the user interface, and then when the user's confirmation operation on the pop-up window is detected, it is determined that the user agrees to the data collection, and then the data collection process is started.

[0149] The following are device embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0150] Please refer to Figure 7 , which shows a block diagram of a training device for a sustainable learning model provided by one embodiment of the present application. The device has the function of implementing the above-mentioned training method for a sustainable learning model. The function can be implemented by hardware or by hardware executing corresponding software. The device can be a computer device or can be set in a computer device. The device 700 can include: a data acquisition module 710, a first acquisition module 720, a second acquisition module 730, a third acquisition module 740 and a model training module 750.

[0151] The data acquisition module 710 is configured to acquire unlearned new task data based on the sustainable learning model.

[0152] The first acquisition module 720 is configured to acquire a first loss of the sustainable learning model according to the new task data, where the first loss is used to measure the difference between a prediction result of the sustainable learning model for the new task data and a standard result.

[0153] The second acquisition module 730 is configured to acquire a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data, where the second loss is used to measure the degree of forgetting of the old task data by the sustainable learning model.

[0154] The third acquisition module 740 is used to obtain a third loss of the sustainable learning model based on the sustainable learning model, the old task data and the new task data. The third loss is used to measure the migration ability of the sustainable learning model from the old task data to the new task data.

[0155] The model training module 750 is used to train the sustainable learning model according to the first loss, the second loss and the third loss.

[0156] In an exemplary embodiment, the third acquisition module 740 includes: a new task processing unit, an old task processing unit, and a third acquisition unit.

[0157] A new task processing unit is used to input the new task data into the policy network of the sustainable learning model to obtain a first empirical risk and a first gradient norm penalty; wherein the first empirical risk is used to indicate the accuracy of the output result of the policy network for the new task data, and the first gradient norm penalty is used to indicate the generalization of the policy network for the new task data.

[0158] An old task processing unit is used to input the old task data into the policy network to obtain a second empirical risk and a second gradient norm penalty; wherein the second empirical risk is used to indicate the accuracy of the output result of the policy network for the old task data, and the second gradient norm penalty is used to indicate the generalization of the policy network for the old task data.

[0159] A third acquisition unit is used to determine the third loss according to the first empirical risk, the first gradient norm penalty, the second empirical risk and the second gradient norm penalty.

[0160] In an exemplary embodiment, the new task processing unit is used to: determine the first loss as the first empirical risk; input the state information in the new task data into the policy network, and regularize the parameters of the policy network to obtain the first gradient norm penalty.

[0161] In an exemplary embodiment, the old task data includes at least one data subset; the old task processing unit is used to input the state information of the target data subset into the strategy network for the target data subset in the at least one data subset, and obtain the empirical risk and gradient norm penalty corresponding to the target data subset; determine the second empirical wind direction according to the empirical risks corresponding to each of the data subsets; and determine the second gradient norm penalty according to the gradient norm penalty corresponding to each of the data subsets.

[0162] In an exemplary embodiment, the second acquisition module 730 includes: an information acquisition unit and a second acquisition unit.

[0163] The information acquisition unit is used to input the state information in the old task data into the strategy network of the sustainable learning model to obtain current action information corresponding to the state information.

[0164] The second acquiring unit is configured to determine the second loss based on current action information corresponding to the state information and historical action information corresponding to the state information contained in the old task data.

[0165] In an exemplary embodiment, the old task data includes multiple state information; the second acquisition unit is used to determine the forgetting rate of the policy network for the target state information among the multiple state information based on the current action information corresponding to the target state information and the historical action information corresponding to the target state information; and determine the second loss according to the forgetting rate of the policy network for each state information in the old task data.

[0166] In an exemplary embodiment, the first acquisition module 720 is used to determine the prediction result of the sustainable learning model based on the new task data and the network parameters of the policy network in the sustainable learning model; input the state information in the new task data into the target network in the sustainable learning model to obtain the output result of the target network for the state information; obtain the standard result of the sustainable learning model based on the output result of the target network for the state information and the network parameters of the target network; and determine the first loss based on the standard result and the prediction result.

[0167] In an exemplary embodiment, the data acquisition module 710 is used to obtain current state information; input the current state information into the strategy network of the sustainable learning model to obtain action information corresponding to the current state information; obtain the reward of the action information corresponding to the current state information, and the next state information obtained for the action information corresponding to the current state information; determine the next state information as the new current state information, and start again from the step of inputting the current state information into the strategy network of the sustainable learning model to obtain the action information corresponding to the current state information until the current task is completed; based on the current task, generate new task data corresponding to the current task, and multiple state transition information in the new task data corresponding to the current task; wherein the target state transition information includes first target state information, target action information corresponding to the first target state information, the reward of the target action information and second target state information corresponding to the target action information.

[0168] In an exemplary embodiment, as Figure 8 As shown, the device 700 further includes: a task acquisition module 760.

[0169] The task acquisition module 760 is used to obtain initial old task data; perform linear attenuation processing on the initial old task data to obtain processed old task data; and sample the processed old task data based on the correlation between the new task data and the processed old task data to obtain the learned old task data.

[0170] In an exemplary embodiment, the task acquisition module 760 is further used to remove part of the data from the initial old task data based on the collection time of the initial old task data to obtain the processed old task data; wherein the collection time is used to indicate the collection moment of the initial old task data.

[0171] In an exemplary embodiment, the old task data includes data subsets collected at different times.

[0172] To sum up, in the technical solution provided by the embodiment of the present application, the third loss of the sustainable model is determined by the new task data and the old task data, and the third loss is used to measure the generalization and accuracy of the sustainable learning model for the old task data and the new task data. The third loss is subsequently used to train the sustainable learning model to improve the generalization and accuracy of the sustainable learning model for the new task data; moreover, the first loss is determined by the new task data, and the second loss is determined by the old task data. The first loss is used to measure the accuracy of the output result of the sustainable learning model for the new task data, and the second loss is used to measure the degree of forgetting of the sustainable learning model for the old task data. The first loss and the second loss are subsequently used to train the sustainable learning model, so that the sustainable learning model can reduce the forgetting of the old task data while learning the new task data; in short, the present application uses the first loss, the second loss and the third loss to perform model training, improve the accuracy and generalization of the sustainable learning model, reduce the forgetting of old knowledge of the sustainable learning model, and thereby improve the practicality of the sustainable learning model.

[0173] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0174] Please refer to Figure 9 , which shows a block diagram of the structure of a computer device provided by an embodiment of the present application. The computer device can be used to implement the functions of the training method of the above-mentioned sustainable learning model. Specifically:

[0175] Computer device 900 includes a central processing unit (CPU) 901, a system memory 904 including a random access memory (RAM) 902 and a read-only memory (ROM) 903, and a system bus 905 connecting system memory 904 and CPU 901. Computer device 900 also includes a basic input / output (I / O) system 906 that facilitates information transfer between various components within the computer, and a mass storage device 907 for storing an operating system 913, application programs 914, and other program modules 915.

[0176] The basic input / output system 906 includes a display 908 for displaying information and an input device 909, such as a mouse and keyboard, for user input. Both the display 908 and the input device 909 are connected to the central processing unit 901 via an input / output controller 910 connected to the system bus 905. The basic input / output system 906 may also include an input / output controller 910 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 910 also provides output to a display screen, printer, or other types of output devices.

[0177] The mass storage device 907 is connected to the central processing unit 901 via a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. In other words, the mass storage device 907 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0178] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage devices, CD-ROM, DVD (Digital Video Disc) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 904 and mass storage device 907 can be collectively referred to as memory.

[0179] According to various embodiments of the present application, the computer device 900 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 900 may be connected to a network 912 via a network interface unit 911 connected to the system bus 905, or the network interface unit 911 may be used to connect to other types of networks or remote computer systems (not shown).

[0180] The memory further includes a computer program, which is stored in the memory and configured to be executed by one or more processors to implement the above-mentioned training method of the sustainable learning model.

[0181] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. When the at least one instruction, the at least one program, the code set or the instruction set is executed by a processor, the training method of the above-mentioned sustainable learning model is implemented.

[0182] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or an optical disk, etc. Among them, the random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0183] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described method for training a sustainable learning model.

[0184] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application do not limit this.

[0185] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for a sustainable learning model, characterized in that: The method comprises: Acquiring new, unlearned task data based on the sustainable learning model, the new task data including a plurality of state transition information, the state transition information including state information, action information corresponding to the state information, reward information for the action information, and next state information corresponding to the action information; wherein the state information is determined based on a query text in the automated dialogue system, and the action information corresponding to the state information is a response text corresponding to the query text output by a policy network of the sustainable learning model based on the state information; Obtaining a first loss of the sustainable learning model according to the new task data, where the first loss is used to measure the difference between a prediction result of the sustainable learning model for the new task data and a standard result; Obtaining a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data, where the second loss is used to measure the degree of forgetting of the old task data by the sustainable learning model; Inputting the new task data into the policy network of the sustainable learning model to obtain a first empirical risk and a first gradient norm penalty; wherein the first empirical risk is used to indicate the accuracy of the output result of the policy network for the new task data, and the first gradient norm penalty is used to indicate the generalization of the policy network for the new task data; Inputting the old task data into the policy network to obtain a second empirical risk and a second gradient norm penalty; wherein the second empirical risk is used to indicate the accuracy of the output result of the policy network for the old task data, and the second gradient norm penalty is used to indicate the generalization of the policy network for the old task data; Determining a third loss based on the first empirical risk, the first gradient norm penalty, the second empirical risk, and the second gradient norm penalty, wherein the third loss is used to measure the migration ability of the sustainable learning model from the old task data to the new task data; The sustainable learning model is trained according to the first loss, the second loss and the third loss.

2. The method according to claim 1, characterized in that The step of inputting the new task data into the policy network of the sustainable learning model to obtain a first empirical risk and a first gradient norm penalty includes: determining the first loss as the first empirical risk; Inputting state information in the new task data into the policy network, and performing regularization processing on parameters of the policy network to obtain the first gradient norm penalty.

3. The method according to claim 1, characterized in that The old task data includes at least one data subset; Inputting the old task data into the policy network to obtain a second empirical risk and a second gradient norm penalty includes: For a target data subset in the at least one data subset, inputting state information in the target data subset into the policy network to obtain an empirical risk and a gradient norm penalty corresponding to the target data subset; determining the second empirical wind direction according to the empirical risks corresponding to each of the data subsets; The second gradient norm penalty is determined according to the gradient norm penalties corresponding to each of the data subsets.

4. The method according to claim 1, wherein The obtaining, based on the sustainable learning model and the learned old task data, a second loss of the sustainable learning model includes: Inputting the state information in the old task data into the strategy network of the sustainable learning model to obtain current action information corresponding to the state information; The second loss is determined based on current action information corresponding to the state information and historical action information corresponding to the state information included in the old task data.

5. The method according to claim 4, characterized in that The old task data includes a plurality of status information; The determining the second loss based on the current action information corresponding to the state information and the historical action information corresponding to the state information contained in the old task data includes: For target state information among the multiple state information, determining a forgetting rate of the policy network for the target state information based on current action information corresponding to the target state information and historical action information corresponding to the target state information; The second loss is determined according to a forgetting rate of the policy network for each piece of state information in the old task data.

6. The method according to claim 1, characterized in that The obtaining the first loss of the sustainable learning model according to the new task data includes: Determining a prediction result of the sustainable learning model based on the new task data and network parameters of the policy network in the sustainable learning model; Inputting the state information in the new task data into the target network in the sustainable learning model to obtain an output result of the target network for the state information; Obtaining a standard result of the sustainable learning model according to an output result of the target network for the state information and a network parameter of the target network; The first loss is determined based on the standard result and the prediction result.

7. The method according to claim 1, characterized in that The acquiring of unlearned new task data based on the sustainable learning model includes: Get current status information; Inputting the current state information into the strategy network of the sustainable learning model to obtain action information corresponding to the current state information; Obtaining a reward for the action information corresponding to the current state information, and next state information obtained for the action information corresponding to the current state information; Determining the next state information as the new current state information, and again starting from the step of inputting the current state information into the policy network of the sustainable learning model to obtain action information corresponding to the current state information, until the current task is completed; Based on the current task, new task data corresponding to the current task is generated, and multiple state transition information is included in the new task data corresponding to the current task; wherein the target state transition information includes first target state information, target action information corresponding to the first target state information, a reward for the target action information, and second target state information corresponding to the target action information.

8. The method according to claim 1, characterized in that The method further comprises: Get the initial old task data; Performing linear attenuation processing on the initial old task data to obtain processed old task data; Based on the correlation between the new task data and the processed old task data, the processed old task data is sampled to obtain the learned old task data.

9. The method according to claim 8, characterized in that The performing linear attenuation processing on the initial old task data to obtain processed old task data includes: Based on the collection time of the initial old task data, removing part of the data from the initial old task data to obtain the processed old task data; The collection time is used to indicate the collection time of the initial old task data.

10. The method according to any one of claims 1 to 9, characterized in that The old task data includes data subsets collected at different times.

11. A training device for a sustainable learning model, characterized in that: The device comprises: a data acquisition module configured to acquire unlearned new task data based on the sustainable learning model, the new task data comprising a plurality of state transition information, the state transition information comprising state information, action information corresponding to the state information, reward information for the action information, and next state information corresponding to the action information; wherein the state information is determined based on a query text in the automated dialogue system, and the action information corresponding to the state information is a response text corresponding to the query text output by the policy network of the sustainable learning model based on the state information; A first acquisition module is configured to acquire a first loss of the sustainable learning model according to the new task data, wherein the first loss is used to measure the difference between a prediction result of the sustainable learning model for the new task data and a standard result; A second acquisition module is configured to acquire a second loss of the sustainable learning model based on the sustainable learning model and the learned old task data, where the second loss is used to measure the degree of forgetting of the sustainable learning model with respect to the old task data; a third acquisition module, configured to input the new task data into the policy network of the sustainable learning model to obtain a first empirical risk and a first gradient norm penalty; wherein the first empirical risk is used to indicate the accuracy of the output result of the policy network for the new task data, and the first gradient norm penalty is used to indicate the generalization of the policy network for the new task data; input the old task data into the policy network to obtain a second empirical risk and a second gradient norm penalty; wherein the second empirical risk is used to indicate the accuracy of the output result of the policy network for the old task data, and the second gradient norm penalty is used to indicate the generalization of the policy network for the old task data; determine a third loss based on the first empirical risk, the first gradient norm penalty, the second empirical risk, and the second gradient norm penalty, and the third loss is used to measure the migration ability of the sustainable learning model from the old task data to the new task data; A model training module is used to train the sustainable learning model according to the first loss, the second loss and the third loss.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the training method of the sustainable learning model according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the training method of the sustainable learning model according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the training method of the sustainable learning model according to any one of claims 1 to 10.

Images