Agent control model training method and device, computer device, and storage medium

Abstract

The application relates to the field of artificial intelligence, and provides an agent control model training method and device, computer equipment and a storage medium. The method comprises the following steps: an initial control model is used to control an agent to interact with a preset virtual environment, so that the agent outputs an interaction action according to a self state and an environment state of the virtual environment; a first execution frequency of each sub operation in the interaction action is counted; a target sub operation to be adjusted is determined from a plurality of sub operations according to the first execution frequency; first parameter information to be adjusted in the initial control model is determined according to the target sub operation, and at least the first parameter information is adjusted until a target control model is obtained, wherein the target control model controls a second execution frequency of the target sub operation in the process of interaction between the agent and the virtual environment to be the first execution frequency. In the process of training the agent control model, each sub operation is balancedly sampled, and the efficiency and training effect of model training are improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, computer device, and storage medium for training an intelligent agent control model. Background Technology

[0002] Reinforcement learning (RL) is a machine learning approach. It primarily guides an agent in making decisions at each step, determining which actions to take to achieve a specific objective or maximize rewards. By sampling actions from a predefined action space based on their probabilities, the agent uses the reward for performing the corresponding action as the basis for deciding which action to take in the current state.

[0003] However, existing reinforcement learning-based agent training methods often struggle to sample actions with low sampling probabilities. This means that these low-probability actions can only be sampled with an extremely large number of sampling attempts, resulting in low training efficiency, long training periods, and impacting the agent's exploration and learning abilities. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, computer device, and storage medium for training intelligent agent control models, aiming to solve the problems of low efficiency and long training time of existing intelligent agent control model training methods.

[0005] In a first aspect, this application provides a method for training an intelligent agent control model, the method comprising:

[0006] An initial control model is used to control the interaction between the intelligent agent and a preset virtual environment, so that the intelligent agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations.

[0007] Calculate the first execution frequency of each sub-operation in the aforementioned interactive action;

[0008] The target sub-operation to be adjusted is determined from the plurality of sub-operations based on the first execution frequency;

[0009] The first parameter information to be adjusted in the initial control model is determined according to the target sub-operation, and the first parameter information is adjusted at least until the target control model is obtained. In the process of the target control model controlling the interaction between the agent and the virtual environment, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency.

[0010] Secondly, this application also provides an intelligent agent control model training device, comprising:

[0011] An interactive action output model is used to control an agent to interact with a preset virtual environment using an initial control model, so that the agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations.

[0012] An execution frequency statistical model is used to count the first execution frequency of each sub-operation in the interactive action;

[0013] The target sub-operation determination module is used to determine the target sub-operation to be adjusted from the plurality of sub-operations according to the first execution frequency;

[0014] The parameter information adjustment module is used to determine the first parameter information to be adjusted in the initial control model according to the target sub-operation, and to adjust at least the first parameter information until the target control model is obtained. In the process of the target control model controlling the agent to interact with the virtual environment, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency.

[0015] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the intelligent agent control model training method as described above.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the intelligent agent control model training method described above.

[0017] This application provides a method, apparatus, computer device, and storage medium for training an intelligent agent control model. The method includes: controlling an intelligent agent to interact with a preset virtual environment using an initial control model, so that the intelligent agent outputs interactive actions according to its own state and the environmental state of the virtual environment; calculating the first execution frequency of each sub-operation in the interactive actions; determining the target sub-operation to be adjusted from multiple sub-operations based on the first execution frequency; determining the first parameter information to be adjusted in the initial control model based on the target sub-operation, and adjusting at least the first parameter information until a target control model is obtained. The target control model controls the second execution frequency and the first execution frequency of the target sub-operation during the interaction between the intelligent agent and the virtual environment. Balanced sampling of each sub-operation is performed during the training of the intelligent agent control model, improving the efficiency and effectiveness of model training and shortening the model training cycle. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a method for training an intelligent agent control model according to an embodiment of this application;

[0020] Figure 2 A flowchart illustrating the sub-steps of a training method for an intelligent agent control model provided in an embodiment of this application;

[0021] Figure 3 A schematic block diagram illustrating the implementation process of an agent control model training method provided in an embodiment of this application;

[0022] Figure 4 A schematic block diagram of an intelligent agent control model training device provided in one embodiment of this application;

[0023] Figure 5 This is a schematic block diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0026] This application provides a method, apparatus, computer device, and storage medium for training an intelligent agent control model.

[0027] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0028] Please refer to Figure 1 , Figure 1This is a flowchart illustrating a method for training an intelligent agent control model, provided as an embodiment of this application. The intelligent agent control model training method can be used in a terminal or server to train the intelligent agent control model. The terminal can be an electronic device such as a mobile phone, tablet, laptop, desktop computer, personal digital assistant, or wearable device; the server can be a standalone server, a server cluster, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0029] An intelligent agent is a game character that can interact with the game environment. For example, an intelligent agent can be controlled by a pre-set control model. In a specific virtual environment of the game, based on its own perception of the virtual environment, it can interact with the virtual environment according to existing instructions or through autonomous learning, or communicate, cooperate, or fight with other intelligent agents, and autonomously complete the goals set by the game in its virtual environment.

[0030] In related technologies, the interactive actions that an agent can output and execute typically include multiple sub-operations in the action space. However, since the probability of different sub-operations being output by the agent control model is not the same, there may be sub-operations that are difficult for the agent to output due to their extremely low probability. During the training process of the agent control model, it is necessary to control the agent to output interactive actions more frequently in order to make the agent output sub-operations with extremely low probabilities and obtain feedback information from the agent outputting the sub-operation. This leads to problems such as low training efficiency, poor training effect, and long training time for the agent control model.

[0031] like Figure 1 As shown, the training method for the intelligent agent control model includes steps S101 to S104.

[0032] Step S101: Use the initial control model to control the agent to interact with the preset virtual environment, so that the agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations.

[0033] For example, the agent control model includes: an initial control model and a target control model. The initial control model represents the model's initial state when training the agent control model begins; the target control model represents the model's state after adjustments to the initial control model, or the model's state after training the initial control model to convergence, and is not limited thereto.

[0034] For example, by using an initial control model to control an agent to interact with the virtual environment, the agent outputs interactive actions to train the initial control model and obtain a target control model. It is understood that the interactive actions include multiple sub-operations. The agent determines the sub-operations to output when interacting with the virtual environment based on its own state and the environmental state of the virtual environment, according to the value information of each sub-operation or the control strategy preset by the initial control model.

[0035] For example, the intelligent agent control model can be a deep learning model, but it is not limited to this, and no limitation is made here.

[0036] In some implementations, step S101 uses an initial control model to control the agent's interaction with a preset virtual environment, so that the agent outputs an interactive action based on its own state and the environmental state of the virtual environment. This includes: using the initial control model to control the agent's interaction with the virtual environment, so that the agent outputs an interactive action based on the value information of each sub-operation in the initial control model, according to its own state and the environmental state of the virtual environment; or using the initial control model to control the agent's interaction with the virtual environment, so that the agent outputs an interactive action based on the control strategy corresponding to the initial control model, according to its own state and the environmental state of the virtual environment.

[0037] For example, the intelligent agent control model can determine the sampled sub-operations based on the value of each sub-operation in the action space. The value of each sub-operation can be determined by a preset value function. During the training of the initial control model, the values ​​of the sub-operations are adjusted based on the feedback information obtained from the output sub-operations to obtain the target control model.

[0038] For example, the intelligent agent control model can also determine sub-operations from the action space according to a preset control policy. Different control policies can be represented by different dominance evaluation functions. When training the initial control model, the parameters in the dominance evaluation function are adjusted based on the feedback information obtained from the output sub-operations in order to obtain the target control model.

[0039] Step S102: Calculate the first execution frequency of each sub-operation in the interactive action.

[0040] For example, to identify sub-operations that are difficult to sample during the training of the agent control model, it is necessary to statistically analyze the frequency of the agent's execution of these sub-operations. For instance, suppose the agent performs 100 interactive actions, and the action types of these 100 actions are statistically analyzed. If these 100 actions include 50 first sub-operations, 30 second sub-operations, and 20 third sub-operations, then the first execution frequency of the first sub-operation is 50%, the first execution frequency of the second sub-operation is 30%, and the first execution frequency of the third sub-operation is 20%.

[0041] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the sub-steps of a training method for an intelligent agent control model provided in an embodiment of this application.

[0042] like Figure 2 As shown, in some embodiments, step S102, which counts the first execution frequency of each sub-operation in the interactive action, includes: step S1021, obtaining the historical number of interactions output by the agent; step S1022, identifying the action information of the interactive action output by the agent, and determining the number of operations executed for each sub-operation; and step S1023, determining the first execution frequency of each sub-operation in the interactive action based on the number of operations executed and the historical number of interactions.

[0043] For example, to count the first execution frequency of each sub-operation, it is necessary to accumulate the number of times the agent outputs interactive actions to obtain the historical interaction count. Furthermore, when the agent outputs an interactive action, the action type of the interactive action is identified. Specifically, this can be determined by identifying the action name of the interactive action, or by identifying the action characteristics of the interactive action; however, it is not limited to these methods and is not specified here.

[0044] For example, the first execution frequency of each sub-operation in the interaction action is determined based on the ratio of the number of operation executions to the number of historical interactions.

[0045] In some embodiments, before step S102, which counts the first execution frequency of each sub-operation in the interactive action, the method further includes: obtaining action information corresponding to each sub-operation in the interactive action, and determining the action space of the interactive action.

[0046] For example, in the action space of an interactive action, there may be sub-operations with a first execution frequency of 0, that is, sub-operations that have not been sampled even after the agent outputs a certain number of interactive actions. In this case, if the first execution frequency of each sub-operation is directly counted based on the output of the interactive action, sub-operations that have not been executed at all will be missed.

[0047] Therefore, before calculating the first execution frequency of each sub-operation in the interactive action, the action information corresponding to each sub-operation is obtained in advance to determine the action space of the interactive action. In this way, even for sub-operations that have not been executed, their first execution frequency can be directly determined to be 0 based on the action information, preventing omissions when calculating the first execution frequency.

[0048] Step S103: Determine the target sub-operation to be adjusted from the plurality of sub-operations according to the first execution frequency.

[0049] For example, in order to ensure that each sub-operation in the action space of the interactive action has a chance to be sampled during the training of the agent control model, it is necessary to adjust the target sub-operation with the lowest execution frequency, increase the execution frequency of the target sub-operation, and ensure that different types of sub-operations are executed during the training of the agent control model.

[0050] In some implementations, determining the target sub-operation to be adjusted from the plurality of sub-operations based on the first execution frequency includes: after the number of times the agent outputs the interactive action reaches a preset number, determining the target sub-operation to be adjusted from the plurality of sub-operations based on the first execution frequency.

[0051] For example, when the number of times the agent outputs the interactive action is small, the first execution frequency usually has a large error. For instance, if the agent outputs the interactive action once, and the first sub-operation output by the agent is the first sub-operation, then the first execution frequency of the first sub-operation is 100%, while the first execution frequencies of other sub-operations are all 0. In this case, it is obviously unreasonable to adjust the target sub-operation based on the first execution frequency.

[0052] Therefore, after the agent outputs the interactive action a preset number of times, and the first execution frequency of each sub-operation tends to stabilize, the target sub-operation to be adjusted is determined from the multiple sub-operations based on the first execution frequency, ensuring the rationality of the determined target sub-operation. The preset number of times can be set according to actual needs and is not limited here.

[0053] In some implementations, determining the target sub-operation to be adjusted from the plurality of sub-operations based on the first execution frequency includes: determining the sub-operations in the interaction action whose first execution frequency is less than a preset minimum frequency as the target sub-operations to be adjusted.

[0054] For example, in order to balance the execution frequency of different sub-operations, the sub-operation with the first execution frequency less than the preset minimum frequency in the interaction action is identified as the target sub-operation to be adjusted, so as to increase the probability of the target sub-operation being sampled when adjusting the target sub-operation in the future, thereby increasing the execution frequency of the target sub-operation.

[0055] Step S104: Determine the first parameter information to be adjusted in the initial control model according to the target sub-operation, and adjust at least the first parameter information until the target control model is obtained. In the process of the target control model controlling the interaction between the agent and the virtual environment, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency.

[0056] For example, the target control model can be an intermediate model obtained by adjusting the original control model during the training of the agent control model. In this case, the agent outputs interactive actions through the target control model. Because the target control model can output various sub-operations in a balanced manner, sub-operations with low historical execution frequency also have the opportunity to be sampled during the interaction between the agent and the virtual environment, which improves the training efficiency of the agent control model. Furthermore, it enables the agent to adaptively adjust the value of sub-operations or control strategies based on the feedback information corresponding to each sub-operation, thereby obtaining a final agent control model that conforms to actual needs.

[0057] The first parameter information can be parameters related to the output of interactive actions during the training of the intelligent agent control model.

[0058] For example, the target control model can also be the agent control model obtained by training the original control model until convergence. In this case, the agent control model needs to be adjusted based on the feedback information obtained after the output interaction action.

[0059] In some implementations, before adjusting at least the first parameter information, the method further includes: acquiring feedback information obtained by the agent interacting with the virtual environment based on the interactive action; determining second parameter information to be adjusted in the initial control model based on the feedback information; the step of adjusting at least the first parameter information until a target control model is obtained includes: adjusting the first parameter information and the second parameter information until the target control model is obtained.

[0060] For example, to ensure the trained target control model performs well in a virtual environment, the second parameter information of the original control model is adjusted based on feedback information to obtain the target control model. The feedback information can be the reward value of the deep model; for sub-operations with high reward values, their output probability is increased; for sub-operations with low reward values, their output probability is decreased. However, this is not a limitation and is not specified here.

[0061] The second parameter information can be parameters related to the output interaction action after the model has been trained.

[0062] Please refer to Figure 3 , Figure 3 This is a schematic block diagram illustrating the implementation process of an intelligent agent control model training method provided in an embodiment of this application.

[0063] like Figure 3 As shown, the agent control model can be either an initial control model or a target control model, without limitation. The agent control model samples the action space of interactive actions, causing the agent to output interactive actions. The action space of interactive actions includes multiple sub-operations: first sub-operation, second sub-operation, ..., Nth sub-operation, each with its own sampling probability. The agent control model samples multiple sub-operations based on the sampling probabilities and outputs the sampled interactive actions in the virtual environment, interacting with the virtual environment through these actions. After outputting the interactive actions, the first execution frequency of each sub-operation is calculated based on the output sampled actions, and the sampling probability of each sub-operation in the interactive actions output by the agent control model is adjusted according to the first execution frequency. Furthermore, after outputting the interactive actions, feedback information obtained by the agent based on the interaction with the virtual environment is acquired, and the agent control model is adjusted according to the feedback information.

[0064] The intelligent agent control model training method provided in this application utilizes an initial control model to control an intelligent agent to interact with a preset virtual environment, so that the intelligent agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations. The method involves: statistically analyzing the first execution frequency of each sub-operation in the interactive actions; determining a target sub-operation to be adjusted from the multiple sub-operations based on the first execution frequency; determining the first parameter information to be adjusted in the initial control model based on the target sub-operation; and adjusting at least the first parameter information until a target control model is obtained. During the interaction between the intelligent agent and the virtual environment controlled by the target control model, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency. Balanced sampling of each sub-operation is performed during the training of the intelligent agent control model, improving the efficiency and effectiveness of model training and shortening the model training cycle.

[0065] Please see Figure 4 , Figure 4 This is a schematic diagram of an intelligent agent control model training device provided in an embodiment of this application. The intelligent agent control model training device can be configured in a server or terminal to execute the aforementioned intelligent agent control model training method.

[0066] like Figure 4 As shown, the intelligent agent control model training device includes: an interactive action output module 110, an execution frequency statistics module 120, a target sub-operation determination module 130, and a parameter information adjustment module 140.

[0067] An interactive action output module is used to control the agent to interact with a preset virtual environment using an initial control model, so that the agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations.

[0068] The execution frequency statistics module is used to count the first execution frequency of each sub-operation in the interactive action;

[0069] The target sub-operation determination module is used to determine the target sub-operation to be adjusted from the plurality of sub-operations according to the first execution frequency;

[0070] The parameter information adjustment module is used to determine the first parameter information to be adjusted in the initial control model according to the target sub-operation, and to adjust at least the first parameter information until the target control model is obtained. In the process of the target control model controlling the agent to interact with the virtual environment, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency.

[0071] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0072] The methods and apparatus of this application can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0073] For example, the above-described method and apparatus can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0074] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device provided in an embodiment of this application. The computer device may be a server or a terminal.

[0075] like Figure 5 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and internal memory.

[0076] The storage medium can store the operating system and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any method for training an intelligent agent control model.

[0077] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0078] Internal memory provides an environment for the execution of computer programs stored in the storage medium. When the computer program is executed by the processor, it enables the processor to execute any intelligent agent control model training method.

[0079] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0080] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0081] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0082] An initial control model is used to control the interaction between the intelligent agent and a preset virtual environment, so that the intelligent agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations.

[0083] Calculate the first execution frequency of each sub-operation in the aforementioned interactive action;

[0084] The target sub-operation to be adjusted is determined from the plurality of sub-operations based on the first execution frequency;

[0085] The first parameter information to be adjusted in the initial control model is determined according to the target sub-operation, and the first parameter information is adjusted at least until the target control model is obtained. In the process of the target control model controlling the interaction between the agent and the virtual environment, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency.

[0086] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of training the intelligent agent control model described above can be referred to the corresponding process in the aforementioned embodiments of the intelligent agent control model training method, and will not be repeated here.

[0087] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can refer to various embodiments of the intelligent agent control model training method of this application.

[0088] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0089] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0090] It should also be understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. It should be noted that, herein, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0091] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for training an intelligent agent control model, characterized in that, The method includes: An initial control model is used to control the interaction between the intelligent agent and a preset virtual environment, so that the intelligent agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations. Calculate the first execution frequency of each sub-operation in the aforementioned interactive action; The target sub-operation to be adjusted is determined from the plurality of sub-operations based on the first execution frequency; The first parameter information to be adjusted in the initial control model is determined according to the target sub-operation, and the first parameter information is adjusted at least until the target control model is obtained. In the process of the target control model controlling the interaction between the agent and the virtual environment, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency. The method further includes, before at least adjusting the first parameter information: Obtain feedback information obtained by the intelligent agent based on the interactive action and its interaction with the virtual environment; Based on the feedback information, determine the second parameter information to be adjusted in the initial control model; The step of adjusting at least the first parameter information until the target control model is obtained includes: The first parameter information and the second parameter information are adjusted until the target control model is obtained.

2. The intelligent agent control model training method according to claim 1, characterized in that, The step of determining the target sub-operation to be adjusted from the plurality of sub-operations based on the first execution frequency includes: After the number of times the agent outputs the interactive action reaches a preset number, the target sub-operation to be adjusted is determined from the plurality of sub-operations based on the first execution frequency.

3. The intelligent agent control model training method according to claim 1, characterized in that, The step of determining the target sub-operation to be adjusted from the plurality of sub-operations based on the first execution frequency includes: The sub-operations in the interactive action whose first execution frequency is less than the preset minimum frequency are identified as the target sub-operations to be adjusted.

4. The method for training an intelligent agent control model according to claim 1, characterized in that, The first execution frequency of each sub-operation in the interactive action is statistically analyzed, including: Obtain the historical number of interactions in which the agent outputs the interactive action; The action type of the interactive action output by the intelligent agent is identified, and the number of times each sub-operation is executed is determined. Based on the number of times the operation is executed and the number of historical interactions, the first execution frequency of each sub-operation in the interaction action is determined.

5. The method for training an intelligent agent control model according to claim 1, characterized in that, Before calculating the first execution frequency of each sub-operation in the interactive action, the method further includes: Obtain the action information corresponding to each sub-operation in the interactive action, and determine the action space of the interactive action.

6. The method for training an intelligent agent control model according to any one of claims 1-5, characterized in that, The method of controlling the agent to interact with a preset virtual environment using an initial control model, so that the agent outputs interactive actions based on its own state and the environmental state of the virtual environment, includes: The initial control model is used to control the interaction between the agent and the virtual environment, so that the agent outputs interactive actions based on the value information of each sub-operation in the initial control model, according to its own state and the environmental state of the virtual environment; or The initial control model is used to control the interaction between the agent and the virtual environment, so that the agent outputs interactive actions based on its own state and the environmental state of the virtual environment according to the control strategy corresponding to the initial control model.

7. A training device for an intelligent agent control model, characterized in that, The intelligent agent control model training device includes: An interactive action output module is used to control the agent to interact with a preset virtual environment using an initial control model, so that the agent outputs interactive actions according to its own state and the environmental state of the virtual environment. The interactive actions include multiple sub-operations. The execution frequency statistics module is used to count the first execution frequency of each sub-operation in the interactive action; The target sub-operation determination module is used to determine the target sub-operation to be adjusted from the plurality of sub-operations according to the first execution frequency; The parameter information adjustment module is used to determine the first parameter information to be adjusted in the initial control model according to the target sub-operation, and adjust the first parameter information at least until the target control model is obtained. In the process of the target control model controlling the agent to interact with the virtual environment, the execution frequency of the target sub-operation is a second execution frequency, and the second execution frequency is greater than the first execution frequency. Prior to adjusting at least the first parameter information, the method further includes: Obtain feedback information obtained by the intelligent agent based on the interactive action and its interaction with the virtual environment; Based on the feedback information, determine the second parameter information to be adjusted in the initial control model; The step of adjusting at least the first parameter information until the target control model is obtained includes: The first parameter information and the second parameter information are adjusted until the target control model is obtained.

8. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the intelligent agent control model training method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the intelligent agent control model training method as described in any one of claims 1 to 6.

Images