A data processing method, device, apparatus, storage medium, and program product
By constructing an emotional state pool and calculating similarity, and combining basic rewards and emotional rewards to update the agent's strategy, the problem of inaccurate user emotion recognition in existing technologies is solved, and a more efficient emotional interaction and dialogue experience is achieved.
Patent Information
- Application Number
- CN202610300689.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-12
- Publication Date
- 2026-07-31
AI Technical Summary
Existing reinforcement learning algorithms struggle to accurately identify user emotions, resulting in poor output strategies in human-computer interaction and impacting user experience.
An emotional state pool is constructed. The emotional state pool is updated by calculating the similarity between the emotional feature vector and the existing emotional state and the expected reward. The total reward is calculated by combining the basic reward and the emotional reward, and the policy network parameters of the agent are updated.
It improves the agent's ability to recognize emotions in human-computer interaction, outputs more satisfactory answers to users, and enhances the dialogue experience.
Smart Images
Figure CN122491327A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of reinforcement learning technology, specifically relating to a data processing method, apparatus, device, storage medium, and program product. Background Technology
[0002] In recent years, with the rapid development of artificial intelligence (AI) technology, reinforcement learning, as a major branch of AI, has achieved remarkable results in many complex scenarios by guiding intelligent agents to optimize strategies through task-based reward and punishment mechanisms. Especially in human-computer interaction (HCI) scenarios, the ability of users to engage in efficient and natural multi-turn dialogues with intelligent agents has become a core element in improving service quality. In HCI scenarios, the ability of intelligent agents to perceive and respond to user emotions is particularly important. The emotions expressed by users in dialogue reflect their needs; only by accurately recognizing these emotions and making appropriate responses and actions can true HCI be achieved.
[0003] Intelligent agents continuously interact with their environment through trial and error, unlike traditional machine learning which requires a large number of samples provided by experts. Instead, they learn effectively by perceiving their environment, making reinforcement learning more suitable for scenarios where agents need to make decisions. User emotions are context-dependent, ambiguous, and dynamic. Existing reinforcement learning algorithms struggle to capture emotional differences and their strength, leading to poor output strategies and impacting the user's experience in dialogue with the agent. Summary of the Invention
[0004] This application provides a data processing method, apparatus, device, storage medium, and program product to solve the problem in the prior art that intelligent agents cannot accurately recognize user emotions and output satisfactory answers to users.
[0005] Firstly, a data processing method is provided, including:
[0006] In each time step of training, based on the current state, the agent selects the current action and executes the current action to obtain the basic reward and transition state;
[0007] Based on the current state, an emotional feature vector is obtained, and the emotional state pool is updated based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, as well as the expected reward corresponding to the emotional feature vector; the emotional state pool is used to store emotional states and the expected rewards corresponding to the emotional states.
[0008] The total reward is obtained based on the basic reward and the emotional reward, and the network parameters of the agent's policy are updated based on the total reward; the emotional reward is obtained based on the current state.
[0009] Optionally, the data processing method, wherein updating the emotional state pool based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, and the expected reward corresponding to the emotional feature vector, includes:
[0010] If the similarity between the emotional feature vector and the feature vector of an existing emotional state in the emotional state pool is less than a set threshold, and the expected reward corresponding to the emotional feature vector is greater than the feature vector of the existing emotional state, then the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool, and the existing emotional state and the expected reward corresponding to the existing emotional state are deleted from the emotional state pool.
[0011] If the similarity is greater than or equal to the set threshold, the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool.
[0012] Optionally, the data processing method further includes:
[0013] When the capacity of the emotional state pool reaches a set upper limit, at least one first expected reward and the emotional state corresponding to the first expected reward are deleted from the emotional state pool. The emotional state pool includes at least one first expected reward and at least one second expected reward, wherein the first expected reward is less than the second expected reward.
[0014] Optionally, the data processing method further includes:
[0015] Based on the similarity between the emotional feature vector and the feature vector of an existing emotional state in the emotional state pool, the target existing emotional state corresponding to the feature vector most similar to the emotional feature vector is obtained.
[0016] The emotional reward is obtained based on the Euclidean distance between the target's existing emotional state and the current emotional state extracted from the current state.
[0017] Optionally, the data processing method, wherein obtaining the sentiment feature vector based on the current state includes:
[0018] Extract the current emotional state from the current state;
[0019] Word vectors are extracted from the current emotional state to obtain multiple word vectors;
[0020] The sentiment feature vector is obtained by weighted averaging of multiple word vectors.
[0021] Optionally, the data processing method further includes:
[0022] In the nth time step training, the transition state obtained in the (n-1)th time step training is taken as the current state, and the network parameters of the emotional state pool and the agent's policy obtained in the (n-1)th time step are updated. The training steps of each time step are repeated, where n is an integer greater than 1 and less than or equal to N, and N is the preset number of time steps.
[0023] Secondly, a data processing apparatus is also provided, comprising:
[0024] The execution module is used to select the agent's current action based on the current state at each training time step, and execute the current action to obtain basic rewards and transition states;
[0025] The first update module is used to obtain an emotion feature vector based on the current state, and update the emotion state pool based on the similarity between the emotion feature vector and the feature vectors of existing emotion states in the emotion state pool, as well as the expected reward corresponding to the emotion feature vector; the emotion state pool is used to store emotion states and the expected rewards corresponding to the emotion states.
[0026] The second update module is used to obtain a total reward based on the basic reward and the emotional reward, and to update the network parameters of the agent's policy based on the total reward; the emotional reward is obtained based on the current state.
[0027] Thirdly, a data processing apparatus is also provided, comprising: a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor executes the program or instructions to implement the data processing method as described in the first aspect.
[0028] Fourthly, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the data processing method as described in the first aspect.
[0029] Fifthly, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the data processing method as described in the first aspect.
[0030] Compared with existing technologies, embodiments of this application provide a data processing method, apparatus, device, storage medium, and program product. In each time step of training, based on the current state, the agent selects its current action and executes it to obtain a basic reward and a transition state. Based on the current state, an emotional feature vector is obtained, and the emotional state pool is updated based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, as well as the expected reward corresponding to the emotional feature vector. The emotional state pool stores emotional states and their corresponding expected rewards. Based on the basic reward and the emotional reward, a total reward is obtained, and the network parameters of the agent's policy are updated based on the total reward. The emotional reward is obtained based on the current state. Thus, the agent can acquire the user's emotional state and output a satisfactory answer to the user. Attached Figure Description
[0031] Figure 1 This is a schematic flowchart of the data processing method described in the embodiments of this application;
[0032] Figure 2 This is a flowchart illustrating one embodiment of the data processing method described in the embodiments of this application;
[0033] Figure 3 This is a schematic diagram of the module of the device described in the embodiments of this application;
[0034] Figure 4 This is a hardware block diagram of the device described in the embodiments of this application. Detailed Implementation
[0035] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0036] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0037] like Figure 1 As shown in the figure, this application provides a data processing method, including:
[0038] Step 101: In each time step of training, select the agent's current action based on the current state, and execute the current action to obtain basic rewards and transition states;
[0039] It should be noted that the data processing method provided in this application embodiment is essentially an agent training method. The agent training process can be defined as a Markov Decision Process (MDP), formally defined as a quintuple: ,in, For state space; For action space; To perform the action ,state Transition to state The probability of; For state Execute action The rewards received; This is a discount factor used to control the weight of expected rewards.
[0040] At the start of training the agent, parameters need to be initialized, including: environmental state. Training time steps Experience Pool Emotional State Pool ,Strategy Network parameters Among them, strategy for Time step in state Next action probability distribution function .
[0041] In this embodiment of the application, the intelligent agent performs the current action. And obtain the basic reward according to the preset business rules of the intelligent agent. and transition to the current state To the next state That is, to obtain the transition state .
[0042] In one implementation, optionally, during training at each time step, the agent's current action is selected based on the current state, including:
[0043] In each time step of training, a greedy strategy is adopted, based on the current state. In the action space Select the action with the highest expected reward. The current action of the intelligent agent As shown in the following formula (1):
[0044] (1).
[0045] in, Represents any action in the action space; Indicates the expected reward.
[0046] Step 102: Based on the current state, obtain the emotional feature vector, and update the emotional state pool based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, as well as the expected reward corresponding to the emotional feature vector; the emotional state pool is used to store emotional states and the expected rewards corresponding to the emotional states.
[0047] In this embodiment of the application, the emotional state pool Used to record the emotional state of the intelligent agent This allows the agent to reuse emotional experiences, enabling it to efficiently reuse strategies in similar scenarios.
[0048] The emotional state From the environmental state Extracted from this, it is used to represent emotional information and is an emotional state pool. The basic components of the emotional state-expected reward pair.
[0049] Specifically, the emotional state pool The emotional state pool is used to store the emotional state and the expected reward corresponding to the emotional state. Used for storing emotion state-expected reward pairs An emotional state-expected reward pair Including an emotional state and this emotional state A corresponding expected reward .
[0050] In one implementation, optionally, obtaining an emotion feature vector based on the current state includes:
[0051] Extract the current emotional state from the current state;
[0052] Word vectors are extracted from the current emotional state to obtain multiple word vectors;
[0053] The sentiment feature vector is obtained by weighted averaging of multiple word vectors.
[0054] It should be noted that, since the current emotional state is determined by the current state... The set of words used to indicate the emotional semantic information of the agent. Therefore, in this embodiment of the application, it can first be determined from the current state. Extract the current emotional state Then, regarding the current emotional state Perform word vector extraction to obtain multiple word vectors. Then, for multiple of the aforementioned word vectors A weighted average is performed to obtain a sentiment feature vector with uniform dimensions. ,in, Indicates the weight.
[0055] Similarly, since the emotional state pool already contains emotional states... It is determined by the environmental state The set of words used to indicate the emotional semantic information of the agent. Therefore, in this embodiment of the application, it can first be determined from the environmental state. Extract the existing emotional state Then, regarding the existing emotional state... Perform word vector extraction to obtain multiple word vectors. Then, for multiple of the aforementioned word vectors By performing a weighted average, a feature vector of the existing emotional states with a unified dimension is obtained. ,in, Indicates the weight.
[0056] In this embodiment, the intelligent agent can learn autonomously by interacting with the environmental state, accurately capture the semantic information related to emotions in the environmental state, realize emotional responses, and obtain emotional states. This solves the problem that intelligent agents in the prior art cannot accurately identify user emotions, and can output a satisfactory answer to the user based on the identified emotional state.
[0057] In one implementation, optionally, obtaining the similarity between the emotion feature vector and the feature vectors of existing emotion states in the emotion state pool includes:
[0058] The emotion feature vector is obtained by using cosine distance. Feature vectors of existing emotional states in the emotional state pool The similarity between them is equivalent to obtaining the current emotional state. The existing emotional states in the emotional state pool The similarity between them is shown in the following formula (2):
[0059] (2);
[0060] Wherein, the similarity The smaller the value, the stronger the emotional feature vector. The feature vectors of the existing emotional states in the emotional state pool The more similar, i.e., the more similar the current emotional state The existing emotional states in the emotional state pool The more you think about the meaning, the more similar it seems.
[0061] In this embodiment of the application, the emotion feature vector is obtained. Feature vectors of existing emotional states in the emotional state pool similarity between By using the similarity, the current emotional state can reuse the existing emotional states in the emotional state pool, which can solve the semantic ambiguity problem caused by the diversity of natural language expression in emotional states.
[0062] In one implementation, optionally, the emotional state pool is updated based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, and the expected reward corresponding to the emotional feature vector, including:
[0063] If the similarity between the emotional feature vector and the feature vector of an existing emotional state in the emotional state pool is less than a set threshold, and the expected reward corresponding to the emotional feature vector is greater than the feature vector of the existing emotional state, then the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool, and the existing emotional state and the expected reward corresponding to the existing emotional state are deleted from the emotional state pool.
[0064] If the similarity is greater than or equal to the set threshold, the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool.
[0065] It should be noted that the emotional state pool is used to store high-value emotional state-expected reward pairs. The update of the emotional state pool is shown in the following formula (3):
[0066] (3);
[0067] in, Used to indicate updating the emotional state pool ; In the current strategy Current status The action to be performed; The set threshold is defined.
[0068] In this embodiment of the application, the emotional state pool is traversed. The existing emotional states described above are used to filter whether they meet the criteria. If satisfied Then the emotional state pool There is a memory related to the current emotional state. Similar to the existing emotional states That is, there exists a relationship with the emotional feature vector. Similar feature vectors of existing emotional states .
[0069] Furthermore, the emotional feature vectors are compared. The corresponding expected reward and the feature vector of the existing emotional state The corresponding expected reward is used to store the emotional state with a high expected reward and its corresponding expected reward into the emotional state pool. Inside.
[0070] Specifically, if the emotional feature vector The corresponding expected reward is greater than the feature vector of the existing emotional state. The corresponding expected reward will be the emotional feature vector. The corresponding emotional state and the emotional feature vector The corresponding expected reward is stored in the emotional state pool. Within the emotional state pool, the existing emotional states and their feature vectors are deleted. The corresponding expected reward.
[0071] Conversely, if the emotional feature vector The corresponding expected reward is less than or equal to the feature vector of the existing emotional state. The corresponding expected reward is then retained in the emotional state pool. The existing emotional state and the feature vector of the existing emotional state are described within. The corresponding expected reward remains unchanged.
[0072] If not satisfied Then the emotional state pool There is no content related to the current emotional state. Similar to the existing emotional states That is, there is no such relationship with the emotional feature vector. Similar feature vectors of existing emotional states Then the emotional feature vector The corresponding current emotional state and the emotional feature vector The corresponding expected reward is stored in the emotional state pool. Inside.
[0073] Therefore, by calculating similarity and clustering to compress redundant states, the emotional state pool retains only high-value emotional state-expected reward pairs, effectively solving the problem of excessively large state space in human-computer interaction scenarios, effectively reducing the state space complexity of the agent, and improving exploration efficiency.
[0074] In one embodiment, optionally, the method further includes:
[0075] When the capacity of the emotional state pool reaches a set upper limit, at least one first expected reward and the emotional state corresponding to the first expected reward are deleted from the emotional state pool. The emotional state pool includes at least one first expected reward and at least one second expected reward, wherein the first expected reward is less than the second expected reward.
[0076] In this embodiment of the application, when the emotional state pool When the capacity reaches the set limit, the emotional state pool is deleted. The emotional state corresponding to the lower expected reward (i.e., the first expected reward) ensures the emotional state pool. It always retains a high-value emotional state and corresponding expected reward.
[0077] Step 103: Obtain the total reward based on the basic reward and the emotional reward, and update the network parameters of the agent's policy based on the total reward; the emotional reward is obtained based on the current state.
[0078] It should be noted that the total reward includes basic business rewards and emotional rewards, ensuring that the agent completes business tasks, guiding it to learn emotionally adapted interaction strategies, and alleviating the problem of sparse reward signals.
[0079] In this embodiment of the application, a basic reward is obtained according to preset business rules. And, based on the current state Receive emotional rewards Furthermore, based on the aforementioned basic reward... With the aforementioned emotional reward The weighted sum is used to obtain the total reward. As shown in the following formula (4):
[0080] (4);
[0081] in, This refers to the basic reward; For the emotional reward The weights are used to control the emotional rewards. In the total reward The intensity of the impact.
[0082] Therefore, the total reward of the agent is constructed as a weighted sum of the basic reward and the emotional reward, which breaks through the single reward mode of existing reinforcement learning and can obtain effective feedback during the interaction process, thereby alleviating the problem of sparse rewards in human-computer interaction scenarios.
[0083] In one embodiment, optionally, the method further includes:
[0084] Based on the similarity between the emotional feature vector and the feature vector of an existing emotional state in the emotional state pool, the target existing emotional state corresponding to the feature vector most similar to the emotional feature vector is obtained.
[0085] The emotional reward is obtained based on the Euclidean distance between the target's existing emotional state and the current emotional state extracted from the current state.
[0086] In this embodiment of the application, the target existing emotional state is obtained by searching the emotional state pool and finding the feature vector in the emotional state pool that is most similar to the emotional feature vector. The emotional reward is obtained based on the Euclidean distance between the current emotional state and the target existing emotional state.
[0087] Alternatively, since the current emotional state is derived from the current state The target's existing emotional state is obtained from the corresponding environmental state. Therefore, the emotional reward can be approximately equal to the current state. With the environmental state The Euclidean distance between them is shown in the following formula (5):
[0088] (5);
[0089] in, It is the forward network of the intelligent agent. Regarding the current state Feature transformation The predicted estimate; For the feedforward network The parameters.
[0090] It is understandable that the smaller the Euclidean distance, the greater the emotional reward. The larger the value, the greater the positive emotional incentive it can provide to the intelligent agent.
[0091] Therefore, the data processing method described in this application constructs the emotional state pool to store emotional state-expected reward pairs with high expected rewards. Through the update and retrieval mechanism of the emotional state pool, it can quickly locate emotional states with high historical value, and then calculate the emotional reward based on the similarity between the current emotional state and the most similar existing emotional state, as a supplement to the agent's reward.
[0092] In one implementation, optionally, updating the network parameters of the agent's policy based on the total reward includes:
[0093] The network parameters of the agent's policy are updated using the temporal difference method based on the total reward. loss function As shown in the following formula (6):
[0094] (6);
[0095] Among them, sequence Stored in the experience pool middle; This represents the total reward; Discount factor; Indicates the current state; Indicates the current action; Indicates the transition state; Indicates a transition state All possible actions.
[0096] In this embodiment, by combining the network parameters of the agent's strategy to update the total reward in reverse, the agent can learn the emotional interaction strategy corresponding to "high total reward".
[0097] In one embodiment, optionally, the method further includes:
[0098] In the nth time step training, the transition state obtained in the (n-1)th time step training is taken as the current state, and the network parameters of the emotional state pool and the agent's policy obtained in the (n-1)th time step are updated. The training steps of each time step are repeated, where n is an integer greater than 1 and less than or equal to N, and N is the preset number of time steps.
[0099] In this embodiment, the preset time step number N is the training termination condition for the agent. When the preset time step number N is reached, the training of the agent is terminated, and the network parameters of the agent's policy are output. The preset time step number N is related to the aforementioned training time step. The corresponding number of time steps.
[0100] It should be noted that in the first time step of training, the current state, action space, sentiment state pool, policy, and network parameters are obtained based on the initialization parameters mentioned above.
[0101] It should also be noted that the intelligent agent described in this application embodiment can be applied to human-computer interaction scenarios, and the method further includes:
[0102] Based on the user's input data and the trained agent, the output data of the agent is obtained.
[0103] Figure 2 This is a flowchart illustrating one embodiment of the data processing method described in this application. Figure 2 As shown, the method includes:
[0104] Step 201: Start training and initialize the parameters of the agent;
[0105] Step 202: Using a greedy strategy, select the agent's current action;
[0106] Step 203: Execute the current action and obtain the transition state;
[0107] Step 204: Obtain the similarity between the current emotional state's emotional feature vector and the feature vectors of existing emotional states in the emotional state pool;
[0108] Step 205: Determine whether to update the sentiment state pool based on the similarity.
[0109] If the sentiment state pool is updated, proceed to step 206 to update the sentiment state pool;
[0110] Step 207: Calculate the total reward based on the basic reward and the emotional reward;
[0111] If the sentiment pool is not updated, proceed to step 207;
[0112] Step 208: Update the agent's environmental state, emotional state, network parameters of the policy, etc.
[0113] Step 209: Determine whether the training termination condition is met;
[0114] If the training termination condition is not met, return to step 202;
[0115] If the training termination conditions are met, then the training ends.
[0116] In summary, the data processing method described in the embodiments of this application can be understood as a model training method for reinforcement learning algorithms based on emotion modeling. It constructs an emotion state pool, calculates similarity, clusters and compresses redundant states, and retains only high-value emotion state-expected reward pairs. This effectively solves the problem of excessively large state space in human-computer interaction scenarios, effectively reduces the state space complexity of the agent, and improves exploration efficiency. Furthermore, it enables the reuse of similar experiences through targeted retrieval.
[0117] Moreover, by calculating similarity, the semantic association between two states can be accurately quantified, thereby realizing the identification of similar states. At the same time, a method for updating the sentiment state pool is proposed. When there are similar states, the expected rewards are compared and the sentiment state-expected reward pair with the higher expected reward is retained. When there are no similar states, new sentiment states are inserted to expand the exploration scope, balancing the agent's use of known experience and exploration of unknown states.
[0118] The reward structure of the intelligent agent is a weighted sum of basic reward and emotional reward, which breaks through the single reward mode of existing reinforcement learning. It can also obtain effective feedback during the interaction process, thereby alleviating the problem of reward sparsity in human-computer interaction scenarios. The calculation method of emotional reward can reflect the actual emotional interaction situation. The smaller the distance between two states, the higher the emotional reward value, realizing real-time feedback on the quality of emotional interaction and targeted learning of high-quality experiences.
[0119] Furthermore, by constructing reward signals closely tied to the state and integrating them with the emotional state pool, the agent can obtain phased rewards before completing the task, while guiding it to learn from high-quality emotional experiences. This enables the agent to accurately capture and efficiently reuse emotional experiences. This collaborative mechanism allows the agent to more accurately identify user emotions in emotional interaction environments, output more satisfactory answers to users, and significantly improve the dialogue experience between users and the agent.
[0120] like Figure 3 As shown in the illustration, this application also provides a data processing apparatus, including:
[0121] The execution module 301 is used to select the current action of the agent according to the current state in each time step of training, and execute the current action to obtain the basic reward and the transition state.
[0122] The first update module 302 is used to obtain an emotion feature vector based on the current state, and update the emotion state pool based on the similarity between the emotion feature vector and the feature vectors of existing emotion states in the emotion state pool, as well as the expected reward corresponding to the emotion feature vector; the emotion state pool is used to store emotion states and the expected rewards corresponding to the emotion states.
[0123] The second update module 303 is used to obtain a total reward based on the basic reward and the emotional reward, and to update the network parameters of the agent's policy based on the total reward; the emotional reward is obtained based on the current state.
[0124] Optionally, in the data processing apparatus, the first update module 302 is specifically used for:
[0125] If the similarity between the emotional feature vector and the feature vector of an existing emotional state in the emotional state pool is less than a set threshold, and the expected reward corresponding to the emotional feature vector is greater than the feature vector of the existing emotional state, then the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool, and the existing emotional state and the expected reward corresponding to the existing emotional state are deleted from the emotional state pool.
[0126] If the similarity is greater than or equal to the set threshold, the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool.
[0127] Optionally, the data processing apparatus further includes:
[0128] The deletion module is used to delete at least one first expected reward and the emotional state corresponding to the first expected reward from the emotional state pool when the capacity of the emotional state pool reaches a set upper limit. The emotional state pool includes at least one first expected reward and at least one second expected reward, wherein the first expected reward is less than the second expected reward.
[0129] Optionally, the data processing apparatus further includes:
[0130] The first obtaining module is used to obtain the target existing emotional state corresponding to the feature vector most similar to the emotional feature vector based on the similarity between the emotional feature vector and the feature vector of the existing emotional state in the emotional state pool.
[0131] The second obtaining module is used to obtain the emotional reward based on the Euclidean distance between the target's existing emotional state and the current emotional state extracted from the current state.
[0132] Optionally, in the data processing apparatus, the first update module 302 is specifically used for:
[0133] Extract the current emotional state from the current state;
[0134] Word vectors are extracted from the current emotional state to obtain multiple word vectors;
[0135] The sentiment feature vector is obtained by weighted averaging of multiple word vectors.
[0136] Optionally, the data processing apparatus further includes:
[0137] The training module is used to take the transition state obtained in the (n-1)th time step training as the current state in the nth time step training, and repeat the training steps of each time step according to the network parameters of the emotional state pool and the agent's policy obtained in the (n-1)th time step. n is an integer greater than 1 and less than or equal to N, and N is a preset number of time steps.
[0138] It should be noted that the apparatus provided in this application embodiment can implement all the method steps implemented in the above data processing method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.
[0139] This application also provides a data processing device, such as... Figure 4 As shown, it includes:
[0140] The processor 401, memory 402, transceiver 403, and a program or instructions stored in the memory 402 and executable on the processor 401; when the processor 401 executes the program or instructions, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effect. To avoid repetition, these will not be described again here.
[0141] The transceiver 403 is used to receive and send data under the control of the processor 401.
[0142] Among them, Figure 4 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically connecting various circuits of one or more processors represented by processor 401 and memory represented by memory 402. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. Transceiver 403 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. For different user equipment, the user interface 404 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.
[0143] The processor 401 is responsible for managing the bus architecture and general processing, while the memory 402 can store the data used by the processor 401 when performing operations.
[0144] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0145] This application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the various processes of the above-described data processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.
[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0148] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A data processing method, characterized in that, include: In each time step of training, based on the current state, the agent selects the current action and executes the current action to obtain the basic reward and transition state; Based on the current state, obtain an emotional feature vector, and update the emotional state pool based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, as well as the expected reward corresponding to the emotional feature vector. The emotional state pool is used to store emotional states and the expected rewards corresponding to those emotional states; Based on the basic reward and the emotional reward, a total reward is obtained, and the network parameters of the agent's policy are updated based on the total reward; The emotional reward is obtained based on the current state.
2. The method according to claim 1, characterized in that, The emotional state pool is updated based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, and the expected reward corresponding to the emotional feature vector, including: If the similarity between the emotional feature vector and the feature vector of an existing emotional state in the emotional state pool is less than a set threshold, and the expected reward corresponding to the emotional feature vector is greater than the feature vector of the existing emotional state, then the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool, and the existing emotional state and the expected reward corresponding to the existing emotional state are deleted from the emotional state pool. If the similarity is greater than or equal to the set threshold, the current emotional state and expected reward corresponding to the emotional feature vector are stored in the emotional state pool.
3. The method according to claim 1 or 2, characterized in that, The method further includes: When the capacity of the emotional state pool reaches a set upper limit, at least one first expected reward and the emotional state corresponding to the first expected reward are deleted from the emotional state pool. The emotional state pool includes at least one first expected reward and at least one second expected reward, wherein the first expected reward is less than the second expected reward.
4. The method of claim 1, wherein, The method further includes: Based on the similarity between the emotional feature vector and the feature vector of an existing emotional state in the emotional state pool, the target existing emotional state corresponding to the feature vector most similar to the emotional feature vector is obtained. The emotional reward is obtained based on the Euclidean distance between the target's existing emotional state and the current emotional state extracted from the current state.
5. The method of claim 1, wherein, Based on the current state, an emotion feature vector is obtained, including: Extract the current emotional state from the current state; Word vectors are extracted from the current emotional state to obtain multiple word vectors; The sentiment feature vector is obtained by weighted averaging of multiple word vectors.
6. The method of claim 1, wherein, The method further includes: In the nth time step training, the transition state obtained in the (n-1)th time step training is taken as the current state, and the network parameters of the emotional state pool and the agent's policy obtained in the (n-1)th time step are updated. The training steps of each time step are repeated, where n is an integer greater than 1 and less than or equal to N, and N is the preset number of time steps.
7. A data processing apparatus, characterized by include: The execution module is used to select the agent's current action based on the current state at each training time step, and execute the current action to obtain basic rewards and transition states; The first update module is used to obtain an emotional feature vector based on the current state, and update the emotional state pool based on the similarity between the emotional feature vector and the feature vectors of existing emotional states in the emotional state pool, as well as the expected reward corresponding to the emotional feature vector. The emotional state pool is used to store emotional states and the expected rewards corresponding to those emotional states; The second update module is used to obtain the total reward based on the basic reward and the emotional reward, and to update the network parameters of the agent's policy based on the total reward; The emotional reward is obtained based on the current state.
8. A data processing device, characterized by include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the processor, when executing the program or instructions, implements the data processing method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1 to 6.
10. A computer program product, characterised in that, It includes computer instructions that, when executed by a processor, implement the data processing method as described in any one of claims 1 to 6.