Simulation system training method, environment simulation method and device, electronic equipment, computer readable storage medium and program product

Abstract

The invention provides a training method of a simulation system, an environment simulation method and device, electronic equipment, a computer readable storage medium and a program product. The method comprises the steps of triggering an execution model to execute first state updating through a regulation and control model to obtain a first prediction state of the execution model; triggering the execution model to apply a sample strategy through the regulation and control model, and triggering the execution model to execute second state updating after the sample strategy is applied, so as to obtain a second prediction state of the execution model under the influence of the sample strategy; and on the basis of a first difference between the first prediction state and a first preset state updated by the first state and a second difference between the second prediction state and a second preset state updated by the second state, parameters of the regulation and control model and parameters of the execution model are updated, and an updated regulation and control model and an updated execution model are obtained. According to the invention, the training efficiency and prediction accuracy of the simulation system can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a simulation system training method, an environment simulation method, a device, an electronic device, a computer-readable storage medium, and a program product. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, large language models (LLMs) have shown great potential and value in simulating complex systems and predicting behaviors. Large models not only have excellent natural language processing capabilities, but can also demonstrate human-like intelligence in simulation systems.

[0003] When building and training a simulation system, related technologies often manually adjust the model's parameters to trigger the model to update its status. At the same time, the model training process relies on multiple sets of pre-acquired data sets. By setting a validation set and a test set, the model is trained to obtain a trained simulation system, which consumes a lot of time and is difficult to provide real-time simulation results for comparison with real-world data, resulting in low accuracy of the simulation system and low training efficiency. Summary of the invention

[0004] The embodiments of the present application provide a simulation system training method, an environment simulation method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the training efficiency and prediction accuracy of the simulation system.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The embodiment of the present application provides a training method for a simulation system, wherein the simulation system includes at least one control model and multiple execution models, wherein the execution model is obtained by modeling an object in a target environment, and one of the control models is used to control at least one of the execution models; the method includes:

[0007] Triggering the execution model to execute a first state update through the control model to obtain a first predicted state of the execution model;

[0008] triggering the execution model to apply the sample strategy through the control model, and,

[0009] After applying the sample strategy, triggering the execution model to perform a second state update, and obtaining a second predicted state of the execution model under the influence of the sample strategy;

[0010] Based on a first difference between the first predicted state and a first preset state updated by the first state, and a second difference between the second predicted state and a second preset state updated by the second state, the parameters of the control model and the parameters of the execution model are updated to obtain an updated control model and an updated execution model.

[0011] The embodiment of the present application provides an environment simulation method for a simulation system, wherein the simulation system is trained by the above-mentioned training method for the simulation system; the environment simulation method comprises:

[0012] In the case where the target policy is not applied, the execution model is triggered by the control model to perform a first state update to obtain a first observed state of the execution model.

[0013] The present application provides a training device for a simulation system, the device comprising:

[0014] A first updating module, configured to trigger the execution model to execute a first state update through the control model to obtain a first predicted state of the execution model;

[0015] A second updating module is configured to trigger the execution model to apply the sample strategy through the control model, and, after applying the sample strategy, trigger the execution model to perform a second state update to obtain a second predicted state of the execution model under the influence of the sample strategy;

[0016] A model updating module is used to update the parameters of the control model and the parameters of the execution model based on a first difference between the first predicted state and a first preset state updated by the first state, and a second difference between the second predicted state and a second preset state updated by the second state, so as to obtain an updated control model and an updated execution model.

[0017] The present application provides an environment simulation device for a simulation system, the device comprising:

[0018] The third updating module is used to trigger the execution model to perform a first state update through the control model when the target strategy is not applied, so as to obtain a first observed state of the execution model.

[0019] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0020] A memory for storing computer executable instructions or computer programs;

[0021] The processor is used to implement the training method of the simulation system provided in the embodiment of the present application, or the environment simulation method of the simulation system when executing the computer executable instructions or computer programs stored in the memory.

[0022] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the training method of the simulation system provided in the embodiment of the present application, or the environment simulation method of the simulation system when executed by a processor.

[0023] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the training method of the simulation system provided in the embodiment of the present application, or the environment simulation method of the simulation system is implemented.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] The execution model is regulated by the regulation model to obtain the predicted states of the execution model before and after the sample strategy is applied, and based on the difference with the preset state, the parameters of the regulation model and the execution model are updated to obtain an updated regulation model and an updated execution model. The regulation model and the execution model are automatically regulated and updated, so that the model in the simulation system can adjust parameters according to the difference in states before and after the sample strategy is applied, so that the trained simulation system meets the expected states before and after the target strategy is applied, thereby improving the accuracy of the simulation system in state prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a schematic diagram of the architecture of a training system 100 of a simulation system provided in an embodiment of the present application;

[0027] Figure 2A is a schematic diagram of the structure of the server 200-1 provided in an embodiment of the present application;

[0028] Figure 2B is a schematic diagram of the structure of the server 200-2 provided in an embodiment of the present application;

[0029] Figure 3A It is a first flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0030] Figure 3B It is a second flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0031] Figure 3C is a third flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0032] Figure 3D is a fourth flow chart of the training method of the simulation system provided in an embodiment of the present application;

[0033] Figure 3E is a fifth flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0034] Figure 3F It is a sixth flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0035] Figure 3G is a seventh flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0036] Figure 3H is an eighth flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0037] Fig. 3I is a ninth flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0038] Figure 3J is a tenth flow chart of a training method for a simulation system provided in an embodiment of the present application;

[0039] Figure 3K This is an eleventh flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0040] Figure 3L It is a twelfth flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0041] Figure 3M is a thirteenth flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0042] Figure 3N It is a fourteenth flow chart of the training method of the simulation system provided in the embodiment of the present application;

[0043] Figure 4A It is a schematic diagram of a game scene provided by an embodiment of the present application in which a control model corresponds to multiple execution models;

[0044] Figure 4B It is a schematic diagram of a game scene provided by an embodiment of the present application, in which multiple control models each correspond to at least one execution model;

[0045] Figure 5A It is a schematic diagram of the principle of training a first machine learning model provided in an embodiment of the present application;

[0046] Figure 5B It is a schematic diagram of the principle of training the second machine learning model provided in an embodiment of the present application;

[0047] Figure 6 It is a schematic diagram of the architecture of the simulation system training and application provided in the embodiment of the present application.

[0048] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0050] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0051] In the following description, the terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0052] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0053] Unless otherwise specified, at least one mentioned below refers to one or more situations, and "plurality" may refer to two or more situations.

[0054] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0055] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed may be executed.

[0058] 2) Human-computer interaction interface, which is used to provide an interface for human-computer interaction functions / an interface for displaying the predicted status of the execution model.

[0059] For example, graphical user interface (GUI) display, such as augmented reality (AR) interface, virtual reality (VR) interface, voice user interface (VUI), interactive projection interface (using projection technology to display information on a plane), eye movement detection interface (interface controlled by detecting the user's line of sight), holographic interface (three-dimensional hologram formed by projecting images through holographic projection technology, and stereoscopic images can be seen without wearing special glasses), multimodal interface (interface that combines multiple interaction methods, such as touch, vision, hearing, etc.), brain-machine interface (Brain-Machine Interface, BMI) interface, etc.

[0060] 3) Simulation system is a computer software system that simulates processes or phenomena in the real world to facilitate analysis, prediction and optimization of the performance of actual systems. The simulation system includes at least one control model and multiple execution models, which are used to simulate the behavior and decision-making process of objects in a virtual environment.

[0061] 4) Prediction state refers to the state at a certain point in the future after the execution model performs a state update in the simulation system. The prediction state usually includes estimates of the future behavior of the object and environmental changes. The first prediction state is the prediction state of the execution model after the first state update is triggered by the control model, which is determined by the initialization state of the execution model. The second prediction state is the prediction state of the execution model after applying the sample strategy and performing the second state update, reflecting the future state of the execution model under the influence of the sample strategy, including possible behaviors and decisions.

[0062] 5) Sample strategy refers to a strategy or action plan used to guide the behavior of the execution model in the simulation system. This strategy is usually generated based on historical data, empirical rules or learning algorithms, and aims to optimize the performance of the execution model in the target environment. Taking the game scenario as an example, the sample strategy can be "increase the attack power of the red camp by 5%"; taking the information recommendation scenario as an example, the sample strategy can be "increase the number of sports news recommendations by 6%."

[0063] 6) Prompt word refers to a message or signal sent by the control model to the execution model, which contains instructions for performing specific operations or state changes on the execution model. Prompt words are usually used to trigger a series of behaviors of the execution model, including but not limited to initialization, state update, policy application, etc. The first sample prompt word is used to instruct the execution model to perform initialization operations. The first prompt word is usually sent at the beginning of the simulation process to ensure that the execution model is in an initialized state, such as "Currently is the first moment, all execution models begin to initialize." The second sample prompt word is used to instruct the execution model to apply a specific sample strategy and update its status according to the strategy, such as "Currently is the second moment, all execution models begin to apply the sample strategy and update the status according to the sample strategy."

[0064] 7) Target behavior refers to a specific behavior instruction or action determined according to the attributes of the object. The first target behavior is to guide the object to take actions in the simulation environment to achieve a predetermined goal or respond to a specific situation.

[0065] 8) Observed state refers to the current state of the execution model recorded during the simulation process, which reflects the behavior and environmental conditions of the model at a specific point in time. The first observed state is the state of the execution model obtained after the first state update is triggered by the control model without applying the target strategy. The second observed state is the state of the execution model after the second state update is triggered by the control model after applying the target strategy, which reflects the impact of the target strategy on the behavior of the execution model.

[0066] 9) Target strategy refers to a strategy or plan designed in the simulation system to guide the execution model to achieve a specific goal or optimize its behavior. This strategy is usually based on the analysis of the system environment and the understanding of the model's capabilities.

[0067] 10) Intervention effect indicators are quantitative indicators used to measure the impact of the application of the target strategy on the execution model behavior. Intervention effect indicators can help evaluate the effectiveness and applicability of the strategy. The first intervention effect indicator refers to the indicator of the intervention effect of the target strategy determined based on the comparison between the first observation state and the second observation state, which reflects the change in the execution model behavior after the target strategy is applied. The second intervention effect indicator is the indicator of the intervention effect of the target strategy determined based on the comparison between the first observation state and the third observation state, which reflects the change in the execution model behavior after the target strategy is canceled.

[0068] 11) Collaboration mode refers to a more efficient and optimized operation mode in which two or more execution models communicate and coordinate with each other under the common goal or task. In a multi-execution model system, collaboration mode is a working state in which the first execution model instructs the second execution model to collaborate with it by sending a first prompt word. Collaboration mode allows two execution models to share information, resources and tasks.

[0069] 12) Competition mode refers to a working state in which two or more execution models compete with each other through mutual communication and strategy adjustment when pursuing the same goal or resource in a multi-execution model system. In a multi-execution model system, competition mode is a working state in which the first execution model instructs the second execution model to compete with it by sending a second prompt word. In this mode, the two execution models will try to obtain limited resources or achieve their goals according to their respective target strategies.

[0070] When building and training a simulation system, related technologies often manually adjust the model's parameters to trigger the model to update its status. At the same time, the model training process relies on multiple sets of pre-acquired data sets. By setting a validation set and a test set, the model is trained to obtain a trained simulation system, which consumes a lot of time and is difficult to provide real-time simulation results for comparison with real-world data, resulting in low accuracy of the simulation system and low training efficiency.

[0071] Based on the above analysis, the applicant found that the training method of the simulation system in the related art cannot quickly and accurately train the simulation system. To address the above problem, an embodiment of the present application provides a training method for a simulation system that can improve the training efficiency and prediction accuracy of the simulation system.

[0072] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the electronic device is implemented as a server.

[0073] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the training system 100 of the simulation system provided in the embodiment of the present application. To support a training simulation system application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0074] The server 200 includes a control model 210 and an execution model 220. The control model 210 is used to trigger the execution model 220 to perform a first state update to obtain a first predicted state of the execution model 220; the control model 210 is used to trigger the execution model 220 to apply a sample strategy, and, after applying the sample strategy, the execution model 220 is triggered to perform a second state update to obtain a second predicted state of the execution model 220 under the influence of the sample strategy; based on a first difference between the first predicted state and a first preset state of the first state update, and a second difference between the second predicted state and a second preset state of the second state update, the parameters of the control model 210 and the parameters of the execution model 220 are updated to obtain an updated control model 210 and an updated execution model 220.

[0075] Taking the game scene as an example, the server 200 runs a highly complex simulation system, which includes a control model 210 and multiple execution models for game characters, such as execution model 220 and execution model 230. These execution models are modeled based on the behavior and decision-making of players in the game environment. When the server 200 detects that the players in the game (such as player 1 and player 2) are at the node of strategic selection, the server 200 sends strategy instructions to the execution model 220 corresponding to the player 1 and the execution model 230 corresponding to the player 2 through the control model 210, such as changing the resource allocation in the game or adjusting the player's tactical arrangement. The server 200 records the game performance of players 1 and 2 before and after the application of the strategy, including key indicators such as winning rate and resource utilization efficiency, and feeds these data back to the control model 210 for optimization. Player 1 is controlled by the user who logs into the game account on terminal 400-1, and player 2 is controlled by the user who logs into the game account on terminal 400-2. When reaching the strategic selection node, the game will adjust the game content according to the strategy instructions sent by server 200, such as increasing the attack power of player 1 or enhancing its defense ability, and increasing the attack power of player 2 or enhancing its defense ability. After the user observes these strategy changes on the human-computer interaction interface 410-1 of terminal 400-1 or the human-computer interaction interface 410-2 of terminal 400-2, his game performance and decision-making may change. These changes are collected by terminal 400-1 and terminal 400-2 and transmitted back to server 200 for analysis by control model 210.

[0076] Taking the information recommendation scenario as an example, the server 200 runs an information recommendation simulation system, which includes a control model 210 and multiple execution models, such as execution model 220 and execution model 230, which model the behavior of objects on the social media platform. When the server 200 detects that the online behavior pattern of users (such as user 1 and user 2) has changed, the server 200 formulates a new recommendation strategy through the control model 210 and sends it to the execution model 220 corresponding to user 1 and the execution model 230 corresponding to user 2, such as adjusting the recommended content according to the latest points of interest of user 1 and user 2. The server 200 records the interaction data of user 1 and user 2 before and after applying the new recommendation strategy, including click-through rate, number of shares and stay time, to evaluate the effectiveness of the recommendation strategy. User 1 uses the social media application on terminal 400-1, and user 2 uses the social media application on terminal 400-2. When the server 200 sends a new recommendation strategy, the applications on terminal 400-1 and terminal 400-2 will adjust the recommended content according to these strategies. When user 1 browses recommended content on the human-computer interaction interface 410-1 of terminal 400-1, or when user 2 browses recommended content on the human-computer interaction interface 410-2 of terminal 400-2, their interactive behaviors (such as likes, comments, and shares) will be recorded and sent to the server 200 regularly for the control model 210 to analyze and optimize the recommendation strategy.

[0077] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0078] See also Figure 2A , Figure 2A 2 is a schematic diagram of the structure of the server 200-1 provided in the embodiment of the present application. The server 200-1 is an implementation method of the server 200 used for training the simulation system. Figure 2A The server 200-1 shown includes: at least one processor 210, a memory 230 and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2AVarious buses are labeled as bus system 240 .

[0079] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0080] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 230 may optionally include one or more storage devices that are physically remote from the processor 210.

[0081] The memory 230 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 230 described in the embodiments of the present application is intended to include any suitable type of memory.

[0082] In some embodiments, memory 230 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0083] Operating system 231, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0084] A network communication module 232, used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0085] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2A The training device 233 of the simulation system stored in the memory 230 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a first update module 2331, a second update module 2332 and a model update module 2333. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0086] See also Figure 2B , Figure 2B 2 is a schematic diagram of the structure of the server 200-2 provided in the embodiment of the present application. The server 200-2 is an implementation method of the above-mentioned server 200 for performing environment simulation in the simulation system. Figure 2B The server 200-2 shown includes: at least one processor 250, a memory 270 and at least one network interface 260. The various components in the server 200-2 are coupled together via a bus system 280. It is understood that the bus system 280 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 280 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 280 is not described in detail. Figure 2B Various buses are labeled as bus system 280 .

[0087] The processor 250 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, wherein the general-purpose processor may be a microprocessor or any conventional processor.

[0088] Memory 270 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives. Memory 270 may optionally include one or more storage devices physically located away from processor 250.

[0089] The memory 270 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 270 described in the embodiment of the present application is intended to include any suitable type of memory.

[0090] In some embodiments, memory 270 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0091] Operating system 271, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, and a driver layer, for implementing various basic services and processing hardware-based tasks;

[0092] A network communication module 272, for reaching other electronic devices via one or more (wired or wireless) network interfaces 260, exemplary network interfaces 260 include: Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB);

[0093] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2B The environment simulation device 273 of the simulation system stored in the memory 270 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a third update module 2731, which is logical and can be further split according to the functions implemented. The functions of this module will be explained below.

[0094] In some embodiments, the terminal or server can implement the training method of the simulation system or the environmental simulation method of the simulation system provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, the computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a game APP or a shopping APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0095] The training method of the simulation system provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0096] See also Figure 3A , Figure 3A This is a first flow chart of the training method of the simulation system provided in the embodiment of the present application, which is based on the server and combines Figure 3A The steps shown are explained.

[0097] In some embodiments, the simulation system includes at least one control model and multiple execution models, the execution model is obtained by modeling objects in the target environment, and one control model is used to control at least one execution model. When the target environment is an environment in a virtual world (such as a game or a metaverse), the object can be a player character or a non-player character in the virtual world; when the target environment is an environment in the real world, the object can be a user in the real world who has a social relationship with the information recommendation.

[0098] For example, when the simulation system includes one control model and multiple execution models, one control model is used to control all execution models; when the simulation system includes multiple control models, each control model is used to control different execution models in the multiple execution models. Figure 4A , Figure 4A Schematic diagram of a game scene provided by an embodiment of the present application in which a control model corresponds to multiple execution models. Figure 4A In the game, there are three execution models, namely, execution model A, execution model B and execution model C. The control model is used to control execution model A, execution model B and execution model C. There are three player characters in the game, namely, object 801, object 802 and object 803. Among them, execution model A corresponds to object 801, execution model B corresponds to object 802, and execution model C corresponds to object 803. When the control model sends policies to execution model A, execution model B and execution model C respectively, execution model A drives object 801 to execute the policy, execution model B drives object 802 to execute the policy, and execution model C drives object 803 to execute the policy. See Figure 4B , Figure 4B is a schematic diagram of a game scene provided by an embodiment of the present application in which multiple control models each correspond to at least one execution model. Figure 4B In the game, there are three control models, namely control model A, control model B and control model C, and three execution models, namely execution model A, execution model B and execution model C. Control model A is used to control execution model A, control model B is used to control execution model B, and control model C is used to control execution model C. There are three player characters in the game, namely object 801, object 802 and object 803, among which execution model A corresponds to object 801, execution model B corresponds to object 802, and execution model C corresponds to object 803. When control model A sends a policy to execution model A, control model B sends a policy to execution model B, and control model C sends a policy to execution model C, execution model A drives object 801 to execute the policy, execution model B drives object 802 to execute the policy, and execution model C drives object 803 to execute the policy. Taking the game scene as an example, the object can be a virtual object in the game, one execution model corresponds to one virtual object, and the target environment is the environment in which the virtual object is located in the game, such as virtual land, virtual battlefield, virtual ocean, etc.; taking the shopping scene as an example, the object can be a real object for shopping on the shopping platform, one execution model corresponds to one real object, and the target environment is the shopping environment of the real object, where you can, for example, purchase items, return goods, modify orders, etc.

[0099] In step 101, the execution model is triggered to perform a first state update through the control model to obtain a first predicted state of the execution model.

[0100] In some embodiments, see Figure 3B , Figure 3B It is a second flow chart of the training method of the simulation system provided in an embodiment of the present application. Figure 3A Step 101 can be Figure 3B Steps 1011 to 1013 are implemented as described in detail below.

[0101] In step 1011, a first sample prompt word is sent to the execution model through the control model, wherein the first sample prompt word is used to instruct the execution model to initialize.

[0102] In some embodiments, when the simulation system includes one control model and multiple execution models, the first sample prompt word is used to indicate that all execution models are initialized; when the simulation system includes multiple control models and multiple execution models, and each control model is used to control different execution models among the multiple execution models, for each control model, the first sample prompt word is used to indicate that the execution models that can be controlled by the control model are initialized.

[0103] For example, taking a control model controlling all execution models as an example, the first sample prompt word may be "It is the first moment now, and all execution models begin to be initialized.".

[0104] In some embodiments, see Figure 3C , Figure 3C 3 is a schematic diagram of a third flow chart of a training method for a simulation system provided in an embodiment of the present application. Figure 3C Steps 201 to 202 are implemented as described in detail below.

[0105] In step 201 , a first target behavior of an object is determined based on a property of the object.

[0106] For example, taking a game scene as an example, the object is a virtual object in the game, such as a non-player character (NPC) or a player character controlled by a real player. Different objects can have the same attributes or different attributes. If there are three camps in the game, namely camp A, camp B and camp C, the attribute of the object in camp A is wind attribute, the attribute of the object in camp B is fire attribute, and the attribute of the object in camp C is water attribute, if the three camps are in combat at this time, then the first target behavior of the object in camp A can be to launch an attack with the wind element, the first target behavior of the object in camp B can be to launch an attack with the fire element, and the first target behavior of the object in camp C can be to launch an attack with the water element.

[0107] For example, taking the shopping scenario as an example, the object is a real object on the shopping platform, and different objects can have the same attributes or different attributes. The attributes of the object include at least one of the following: shopping preferences, price sensitivity, purchase history, browsing behavior, geographic location, payment habits, and shopping frequency. If the shopping preference of object A is to buy fruit, the first target behavior of object A can be to buy apples; if the price sensitivity of object B is higher than the average, the first target behavior of object B can be to buy discounted goods.

[0108] In some embodiments, see Figure 3D , Figure 3D This is a fourth flow chart of the training method for the simulation system provided in an embodiment of the present application. Figure 3C Step 201 of "determining a first target behavior of an object based on the attributes of the object" can be performed by Figure 3D Steps 2011 to 2013 are implemented as described in detail below.

[0109] In step 2011, the attributes of the object are encoded to obtain an attribute encoding result.

[0110] In some embodiments, the attributes of the object may be encoded using One-Hot encoding, Bag-of-Words model, Term Frequency-Inverse Document Frequency (TF-IDF) or N-Gram model to obtain an attribute encoding result.

[0111] For example, taking the bag of words model as an example, for the attributes of an object, the attributes are decomposed into words or phrases, the frequency of occurrence of each word is counted, and the frequency of occurrence of each word is combined into the attribute encoding result. Taking the word frequency-inverse document frequency as an example, for the attributes of an object, the number of occurrences of each word in the attribute is calculated, the inverse document frequency of each word is determined by the inverse proportional function, the product of the number of occurrences of each word and the inverse document frequency is calculated, and the product corresponding to each word is combined into the attribute encoding result. Taking embedded coding as an example, the dimension of the embedded vector, that is, the length of the vector, is determined; the embedded matrix is ​​randomly initialized, and each classification label corresponds to a vector; the embedded vector is trained using unsupervised learning (such as Word2Vec, GloVe) or supervised learning methods. In unsupervised learning, the vector is learned based on the co-occurrence relationship of the attribute; in supervised learning, the vector is adjusted based on the relationship between the attribute and the target variable; regularization techniques are applied to avoid overfitting. Combining the embedded vectors of all attributes into a single representation can be achieved by directly concatenating the vectors of all attributes together to form a long vector, or by using aggregation functions (such as average, maximum, minimum) to combine multiple vectors into one vector. The combined vectors are normalized to ensure that all vectors are on the same scale to obtain the attribute encoding result.

[0112] The embodiment of the present application encodes the number of objects and converts the attributes of the objects into a unified form for subsequent processing. Different types of attributes (such as numerical, categorical, and textual) can be converted into standardized numerical or vector representations through encoding, so that these attributes can be processed and analyzed under the same framework. The encoding process can simplify the complexity of the data. For example, categorical attributes are converted into binary vectors through unique hot encoding, and numerical attributes are processed through normalization or standardization, making the data easier for the model to understand and process. Through encoding, the model can be better generalized to unseen data. The encoded attribute representation can capture the potential patterns and relationships in the data, which helps the model to make accurate predictions on new data.

[0113] In step 2012, the attribute encoding result is mapped to a probability distribution of a plurality of first candidate behaviors, wherein the probability distribution includes probabilities corresponding to the plurality of first candidate behaviors respectively.

[0114] In some embodiments, the pre-trained first machine learning model is called based on the attribute encoding result, and the output result is converted into the attribute encoding result through the softmax function, and the mapping is respectively corresponding to the probabilities of the plurality of first candidate behaviors. Figure 5A , Figure 5A It is a schematic diagram of the principle of training the first machine learning model provided in the embodiment of the present application. The training process of the pre-trained first machine learning model is as follows: obtain the initialized first machine learning model; obtain the sample attribute encoding result and the first true label, wherein the first true label represents the corresponding true behavior of the sample attribute encoding result; based on the sample attribute encoding result, call the convolution layer and the fully connected layer of the initialized first machine learning model to obtain the first predicted label; determine the first loss between the first true label and the first predicted label through the first loss function, update the parameters of the initialized first machine learning model based on the first loss, and obtain the pre-trained first machine learning model.

[0115] For example, the first machine learning model can be a decision tree, a random forest, a support vector machine, a neural network, etc. The first loss function can be a mean square error loss function, a cross entropy loss function, a multi-label classification loss function, and a triplet loss function.

[0116] In some embodiments, the first loss is back-propagated to update the parameters of the initialized first machine learning model, the first loss is calculated and the parameters are updated multiple times until the first loss converges, the iterative process is stopped, and a pre-trained first machine learning model is formed based on the updated parameters.

[0117] For example, back propagation is implemented through the back propagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer, and updates the weight and bias of the neuron according to the gradient, and continuously updates the parameters by gradient descent to reduce the loss value. Gradient descent can use various gradient descent algorithms, such as batch gradient descent algorithm, stochastic gradient descent algorithm, adaptive gradient descent algorithm and momentum gradient descent algorithm.

[0118] In step 2013, the first candidate behavior corresponding to the maximum probability is used as the first target behavior.

[0119] For example, the first candidate behavior includes a first candidate behavior A, a first candidate behavior B, and a first candidate behavior C. The probability corresponding to the first candidate behavior A is 0.5, the probability corresponding to the first candidate behavior B is 0.2, and the probability corresponding to the first candidate behavior C is 0.8, and the first candidate behavior C is used as the first target behavior.

[0120] The embodiment of the present application encodes the attributes of the object and obtains the attribute encoding result, which can make personalized decisions for each object. Using probability distribution instead of a single decision path allows more flexible and robust decisions to be made in an uncertain environment. Probability distribution provides an intuitive explanation for decision-making, making it easier for system users to understand the decision-making process. By considering multiple candidate behaviors, the robustness of the simulation system is improved. Coding and probability mapping methods can help more accurately predict the behavior and reaction of objects, thereby improving decision accuracy.

[0121] Continue to see Figure 3C In step 202, a first target behavior is executed by executing the model, wherein the execution model is in an initialized state after executing the first target behavior.

[0122] In some embodiments, the execution model can control the virtual object to perform the first target behavior, or simulate the real object to perform the first target behavior. The execution model is in an initialization state, which indicates that the parameters of the execution model are all at default values.

[0123] Following the example of step 201 above, taking the game scene as an example, the execution model is used to control the virtual object to execute the first target behavior. For example, the execution model corresponding to camp A controls the object in camp A to launch an attack with the wind element, the execution model corresponding to camp B controls the object in camp B to launch an attack with the fire element, and the execution model corresponding to camp C controls the object in camp C to launch an attack with the water element. The parameters of the execution model include any one of the following: attack power parameters, resource acquisition parameters, technology tree parameters, unit parameters, building parameters, special ability parameters, map and terrain parameters. When the execution model is in the initialization state, the parameters of the execution model are all at the default values.

[0124] Following the example of step 201 above, taking the shopping scenario as an example, the execution model is used to simulate the real object to perform the first target behavior. For example, the execution model corresponding to object A simulates the behavior of object A buying apples, and the execution model corresponding to object B simulates the behavior of object B buying discounted goods. The parameters of the execution model include any of the following: commodity purchase completion parameter, return rate parameter, praise rate parameter, and evaluation quantity parameter. When the execution model is in the initialization state, the parameters of the execution model are all at the default values.

[0125] The embodiment of the present application can adjust the behavior strategy according to the different attributes of the object, thereby enhancing the adaptability of the system to diverse environments. By initializing the execution model by executing a specific target behavior, the initialization process is simplified, making initialization faster and more direct. After the execution model enters the initialization state, the behavior strategy can be dynamically adjusted according to the actual operation situation.

[0126] Continue to see Figure 3B In step 1012, the response information sent by the execution model is received through the control model, wherein the response information represents confirmation of receiving the first sample prompt word.

[0127] For example, the response information sent by the execution model is "Has received the prompt word sent by the control model, started to initialize..." The execution model will be initialized while sending the response information to the control model.

[0128] In step 1013, an initialization state of the execution model sent by the control model is received, and a first prediction state is determined based on the initialization state.

[0129] Here, the first predicted state is the current parameter state of the execution model after the execution model controls the virtual object to perform the first target behavior, or after the execution model simulates the real object to perform the first target behavior.

[0130] The embodiment of the present application can verify whether the execution model is successfully initialized by sending prompt words and receiving response information, thereby improving the reliability of initialization. The use of sample prompt words simplifies the initialization process of the execution model and improves the efficiency of initialization. The first predicted state determined based on the initialization state can make adaptive decisions and operations according to the current state of the execution model.

[0131] In some embodiments, see Figure 3E , Figure 3E This is a fifth flow chart of the training method for the simulation system provided in an embodiment of the present application. Figure 3B Step 1013 of "determining a first predicted state based on the initialization state" can be performed by Figure 3E Steps 10131 to 10132 are implemented as described in detail below.

[0132] In step 10131, when the number of execution models is one, the initialization state of the execution model is used as the first prediction state.

[0133] For example, taking a game scenario as an example, if the game includes a red camp, the red camp corresponds to an execution model, and the initialization state of the execution is when the parameters of the red camp are at default parameter values, the state of the red camp will take the initialization state of the execution model as the first predicted state. For example, the first predicted state may be "the current winning rate of the red camp is 70%."

[0134] For example, taking a shopping scenario as an example, if the execution model corresponds to a real object, the initialization state of the execution model is that the attributes of the real user are in a default state, and the initialization state of the execution model is used as the first prediction state. For example, the first prediction state may be "the user will buy apples."

[0135] In step 10132, when there are multiple execution models, the initialization states of the multiple execution models are counted according to preset statistical rules to obtain a first predicted state, wherein different execution models correspond to different objects in the target environment.

[0136] For example, taking a game scenario as an example, if there are three camps in the game, namely, camp A, camp B, and camp C, the initialization states of multiple execution models are: the parameters of camp A are all default parameter values, the parameters of camp B are all default parameter values, and the parameters of camp C are all default parameter values. The preset statistical rule can be to count the winning rate ratios of the three camps, for example, the first prediction state can be "the winning rate ratios of camp A, camp B, and camp C are 2:2:6".

[0137] For example, taking a shopping scenario as an example, the initialization states of multiple execution models are that each real object is in a default state. For example, multiple real objects buy apples at the same time, and the first prediction state may be "the purchase rate of apples among all fruits is 20%."

[0138] Continue to see Figure 3A In step 102, the sample strategy is applied by triggering the execution model through the regulation model, and after applying the sample strategy, the execution model is triggered to perform a second state update to obtain a second predicted state of the execution model under the influence of the sample strategy.

[0139] In some embodiments, a sample strategy is a pre-designed set of instructions or parameter configurations that guide the execution model on how to take actions or make decisions in a specific scenario. The sample strategy is based on the analysis of the target environment, the learning of historical data, and the prediction of future behavior.

[0140] For example, the sample policy includes at least one of the following:

[0141] Goal setting: What is the goal of the strategy, such as improving customer satisfaction, increasing sales, optimizing resource allocation, etc.

[0142] Execution instructions: Specific action guidelines or rules that instruct the execution model on how to act, such as recommending specific products, adjusting the sorting algorithm of the recommendation list, modifying the layout of the user interface, etc.

[0143] Parameter configuration: Parameters that may be included in the strategy. These parameters can be adjusted according to actual conditions to optimize the strategy effect, such as the weight parameters, discount rate, inventory threshold, etc. of the recommendation system.

[0144] Constraints: The constraints that a policy must adhere to when it is implemented, such as budget limitations, legal compliance, user experience standards, etc.

[0145] Feedback mechanism: After the strategy is executed, it is used to collect feedback and evaluate the effectiveness of the strategy. This feedback can be used for strategy iteration and optimization.

[0146] The embodiment of the present application processes the initialization state differently according to the number of execution models, which can more effectively adapt to different application requirements and resource conditions, thereby improving the quality of prediction and decision-making. When processing a single execution model, the computing and storage requirements are reduced, the computational complexity is reduced, and the state update can be responded to more quickly. The updated state directly reflects the response of the execution model to the target strategy, providing more direct feedback. Multiple execution models can cover different objects in the target environment, provide more comprehensive information and processing capabilities, reduce the risk of single point failures, and improve the robustness and stability of the simulation system. Through preset statistical rules, the results of multiple execution models can be combined, and statistical methods can be used to improve the accuracy of predictions.

[0147] In some embodiments, see Figure 3F , Figure 3F This is the sixth flow chart of the training method of the simulation system provided in the embodiment of the present application. Figure 3A Step 102 can be Figure 3F Steps 1021 to 1022 are implemented as described in detail below.

[0148] In step 1021, a second sample prompt word is sent to the execution model through the control model, wherein the second sample prompt word is used to instruct the execution model to apply the sample strategy and update the state of the execution model according to the sample strategy.

[0149] In some embodiments, when the simulation system includes one control model and multiple execution models, the second sample prompt word is used to indicate that all execution models apply the sample strategy and update the status; when the simulation system includes multiple control models and multiple execution models, and each control model is used to control different execution models among the multiple execution models, for each control model, the second sample prompt word is used to indicate that the execution models that can be controlled by the control model apply the sample strategy and update the status.

[0150] For example, taking a control model controlling all execution models as an example, the second sample prompt word can be "It is the second moment now, all execution models begin to apply the sample strategy and update the status according to the sample strategy." In the game scenario, if the winning rate of camp A and camp B is less than that of camp C, the second sample prompt word can be "The attack power of camp A and camp B is increased by 5%, and camp C remains unchanged."

[0151] In some embodiments, see Figure 3G , Figure 3G is a seventh flow chart of the training method of the simulation system provided in the embodiment of the present application. The above-mentioned "updating the state of the execution model according to the sample strategy" can be achieved by Figure 3G Steps 301 to 302 are implemented as described below.

[0152] In step 301 , a second target behavior is determined based on attributes of an object, context information of the object in a target environment, and a sample strategy.

[0153] For example, taking the game scene as an example, the object is a virtual object in the game, such as a non-player character or a player character controlled by a real player, and different objects can have the same attributes or different attributes. If there are three camps in the game, namely camp A, camp B and camp C, the attribute of the object in camp A is wind attribute, the attribute of the object in camp B is fire attribute, and the attribute of the object in camp C is water attribute, if the target environment of the objects in camp A, camp B and camp C is a battlefield, and the sample strategy is "the attack power of camp A and camp B is increased by 5%, and camp C remains unchanged", then the second target behavior of the object in camp A can be to launch an attack with an attack power increased by 5% and a wind element in the battlefield, the second target behavior of the object in camp B can be to launch an attack with an attack power increased by 5% and a fire element in the battlefield, and the second target behavior of the object in camp C can be to launch an attack with an attack power unchanged and a water element in the battlefield.

[0154] For example, taking the shopping scene as an example, the object is a real object on the shopping platform, and different objects can have the same attributes or different attributes. The attributes of the object include at least one of the following: shopping preferences, price sensitivity, purchase history, browsing behavior, geographic location, payment habits, and shopping frequency. If the target scenarios of objects A and B are both to purchase goods, and the sample strategy is "the price of apples is reduced by 10%", and the shopping preference of object A is to buy fruits, then the second target behavior of object A can be to buy apples at 90% of the price; if the price sensitivity of object B is higher than the mean, then the second target behavior of object B can be to buy apples at 90% of the price.

[0155] In some embodiments, see Figure 3H , Figure 3H This is the eighth flow chart of the training method of the simulation system provided in the embodiment of the present application. Figure 3G Step 301 can be Figure 3H Steps 3011 to 3014 are implemented as described in detail below.

[0156] In step 3011, the attributes of the object, the context information of the object in the target environment, and the sample strategy are encoded respectively to obtain an attribute encoding result, an information encoding result, and a strategy encoding result.

[0157] In some embodiments, one-hot encoding, bag-of-words model, term frequency-inverse document frequency or N-gram model can be used to encode the attributes of the object, the contextual information of the object in the target environment and the sample strategy respectively to obtain attribute encoding results, information encoding results and strategy encoding results.

[0158] For example, taking the bag-of-words model as an example, for the attributes of the object, the attributes are decomposed into words or phrases, the frequency of occurrence of each word is counted, and the frequency of occurrence of each word is combined into the attribute encoding result; for the context information of the target environment, the context information is decomposed into words or phrases, the frequency of occurrence of each word is counted, and the frequency of occurrence of each word is combined into the information encoding result; for the sample strategy, the sample strategy is decomposed into words or phrases, the frequency of occurrence of each word is counted, and the frequency of occurrence of each word is combined into the strategy encoding result.

[0159] In step 3012, the attribute coding result, the information coding result and the strategy coding result are fused to obtain a fused coding result.

[0160] In some embodiments, the fusion method includes any one of the following: Feature Concatenation, Feature Weighting, Feature Mapping, Interaction Features, Deep Neural Networks, Ensemble Learning, Attention Mechanism, etc. Among them, deep neural networks include multi-layer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN), etc., which can automatically learn the relationship between features and fuse them.

[0161] For example, taking fusion through feature concatenation as an example, if encoding result A is the attribute encoding result, encoding result B is the information encoding result, and encoding result C is the strategy encoding result, then the fused encoding feature is (encoding result A, encoding result B, encoding result C). Taking fusion through weighted summation as an example, the weight corresponding to the attribute encoding result is w1, the weight corresponding to the information encoding result is w2, and the weight corresponding to the strategy encoding result is w3, then the fused encoding feature is A·w1+B·w2+C·w3.

[0162] In step 3013, the fusion coding result is mapped to a probability distribution of multiple second candidate behaviors, wherein the probability distribution includes probabilities corresponding to the multiple second candidate behaviors respectively.

[0163] In some embodiments, based on the fusion coding result, a pre-trained second machine learning model is called, and the output result is converted into the attribute coding result through the softmax function, and the mapping is performed into the probabilities corresponding to the plurality of second candidate behaviors. Figure 5B , Figure 5B It is a schematic diagram of the principle of training the second machine learning model provided by the embodiment of the present application. The training process of the pre-trained second machine learning model is as follows: obtain the initialized second machine learning model; obtain the sample fusion coding result and the second real label, wherein the sample fusion coding result is obtained by fusing the sample attribute coding result, the sample information coding result and the sample strategy coding result, and the second real label represents the corresponding real behavior of the sample fusion coding result; based on the sample fusion coding result, call the convolution layer and the fully connected layer of the initialized second machine learning model to obtain the second predicted label; determine the second loss between the second real label and the second predicted label through the second loss function, update the parameters of the initialized second machine learning model based on the second loss, and obtain the pre-trained second machine learning model.

[0164] For example, the second machine learning model can be a decision tree, a random forest, a support vector machine, a neural network, etc. The second loss function can be a mean square error loss function, a cross entropy loss function, a multi-label classification loss function, and a triplet loss function.

[0165] In some embodiments, the second loss is back-propagated to update the parameters of the initialized second machine learning model, and the second loss is calculated and the parameters are updated multiple times until the second loss converges. The iterative process is stopped and a pre-trained second machine learning model is formed based on the updated parameters.

[0166] For example, back propagation is implemented through the back propagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer, and updates the weight and bias of the neuron according to the gradient, and continuously updates the parameters by gradient descent to reduce the loss value. Gradient descent can use various gradient descent algorithms, such as batch gradient descent algorithm, stochastic gradient descent algorithm, adaptive gradient descent algorithm and momentum gradient descent algorithm.

[0167] In step 3014, the second candidate behavior corresponding to the maximum probability is used as the second target behavior.

[0168] For example, the second candidate behaviors include second candidate behaviors A, second candidate behaviors B, and second candidate behaviors C. The probability corresponding to second candidate behaviors A is 0.5, the probability corresponding to second candidate behaviors B is 0.2, and the probability corresponding to second candidate behaviors C is 0.8, and second candidate behaviors C is used as the second target behaviors.

[0169] The embodiment of the present application considers information of multiple dimensions by fusing the coding results, which helps to generate higher quality decisions and reduce blindness and errors. Candidate behaviors based on probability distribution can dynamically select the most appropriate behavior in the target environment, enhancing flexibility and responsiveness. Probability distribution provides an intuitive way to explain the decision-making process of the model and increases the transparency of the decision.

[0170] Continue to see Figure 3G In step 302, a second target behavior is executed by executing the model, wherein the execution model is in an updated state after executing the second target behavior.

[0171] In some embodiments, the execution model can control the virtual object to perform the second target behavior, or simulate the real object to perform the second target behavior. The execution model is in an updated state, which indicates that all parameters of the execution model are updated.

[0172] Following the example of step 301 above, taking the game scene as an example, the execution model is used to control the virtual object to execute the second target behavior. For example, the execution model corresponding to camp A controls the objects in camp A to launch an attack with an attack power increased by 5% and with wind elements on the battlefield, the execution model corresponding to camp B controls the objects in camp B to launch an attack with an attack power increased by 5% and with fire elements on the battlefield, and the execution model corresponding to camp C controls the objects in camp C to launch an attack with an unchanged attack power and with water elements on the battlefield. The parameters of the execution model include any of the following: attack power parameters, resource acquisition parameters, technology tree parameters, unit parameters, building parameters, special ability parameters, map and terrain parameters. When the execution model is in an updated state, the attack power parameter of the execution model is increased by 5%.

[0173] For example, taking the shopping scenario as an example, the execution model simulates the real object to execute the second target behavior. For example, the execution model corresponding to object A simulates the behavior of object A buying apples at 90% of the price, and the execution model corresponding to object B simulates the behavior of object B buying apples at 90% of the price. The parameters of the execution model include any of the following: product purchase completion parameters, return rate parameters, praise rate parameters, and evaluation quantity parameters. When the execution model is in an updated state, the product purchase completion parameters of the execution model are adjusted up compared to the default values.

[0174] The embodiment of the present application integrates the attributes of the object, the context information of the target environment and the target strategy, which helps to improve the accuracy of decision-making. Decisions can be made quickly in the target environment, and by optimizing and personalizing the behavior, the occurrence of wrong decisions can be reduced, thereby improving reliability. Decisions based on comprehensive information help to use resources more effectively and avoid unnecessary waste.

[0175] Continuing to refer to 3F, in step 1022, an updated state sent by the control model is received, and a second predicted state is determined based on the updated state.

[0176] Here, the second predicted state is the current parameter state of the execution model after the execution model controls the virtual object to perform the second target behavior, or after the execution model simulates the real object to perform the second target behavior.

[0177] In some embodiments, see Fig. 3I , Fig. 3I This is the ninth flow chart of the training method of the simulation system provided in the embodiment of the present application. Figure 3F Step 1022 can be achieved by Fig. 3I Steps 10221 to 10222 are implemented as described in detail below.

[0178] In step 10221, when the number of execution models is one, the updated state of the execution model is used as the second prediction state.

[0179] For example, taking the game scenario as an example, if the game includes a red camp, the red camp corresponds to the execution model, and the updated state after execution can be that the construction speed of the red camp is reduced by 2%. The updated state of the execution model is used as the second predicted state. For example, the second predicted state can be "the construction speed of the red camp is reduced by 2%, and the current winning rate is updated to 60%."

[0180] For example, taking a shopping scenario, if the execution model corresponds to a real object, the updated state of the execution model is the state after the attributes of the real user are updated. The updated state of the execution model is used as the second predicted state. For example, the second predicted state may be "the frequency of users buying apples is updated from once a week to twice a week."

[0181] The embodiment of the present application processes a single execution model, reduces computing and storage requirements, reduces computing complexity, and can respond to status updates more quickly. The updated status directly reflects the response of the execution model to the target policy, providing more direct feedback.

[0182] In step 10222, when there are multiple execution models, statistics are performed on the updated states of the multiple execution models according to preset statistical rules to obtain a second predicted state, wherein different execution models correspond to different objects in the target environment.

[0183] For example, taking a game scenario as an example, if there are three camps in the game, namely, camp A, camp B, and camp C, the updated states of multiple execution models are: the attack power of camp A increases by 5%, the attack power of camp B increases by 5%, and the construction speed of camp C decreases by 2%. The preset statistical rule can be to calculate the winning rate ratio of the three camps, for example, the second prediction state can be "the winning rate ratio of camp A, camp B, and camp C is 3:3:4".

[0184] Taking the shopping scenario as an example, the updated states of multiple execution models are the states after the parameter values ​​of each real object are adjusted. For example, multiple real objects all buy apples at a price of 90%, and the second prediction state may be "the purchase rate of apples has increased by 10%."

[0185] The multiple execution models of the embodiment of the present application can cover different objects in the target environment, provide more comprehensive information and processing capabilities, reduce the risk of single point failures, and improve the robustness and stability of the simulation system. Through preset statistical rules, the results of multiple execution models can be combined to improve the accuracy of prediction using statistical methods.

[0186] Continue to see Figure 3A In step 103, based on a first difference between the first predicted state and a first preset state updated by the first state, and a second difference between the second predicted state and a second preset state updated by the second state, the parameters of the control model and the parameters of the execution model are updated to obtain an updated control model and an updated execution model.

[0187] In some embodiments, first, the predicted state and the preset state are quantified, and then the difference between the quantized predicted state and the preset state is calculated, and the parameters of the control model and the parameters of the execution model are updated based on the first difference and the second difference to obtain an updated control model and an updated execution model.

[0188] For example, state quantization can be achieved by performing the following processing: define a set of variables to characterize the state, which can be continuous (such as speed, temperature) or discrete (such as state code, category label); convert the state variables to a uniform scale or range for easy calculation and comparison. For continuous variables, minimum-maximum standardization or Z-score standardization can be used. For discrete variables, one-hot encoding or other encoding methods can be used. Combine the state variables into a feature vector, which can be mathematically represented as a point in a multidimensional space as a quantized state. The first difference or the second difference can be calculated by any of the following methods: Euclidean distance, cosine similarity, Hamming distance.

[0189] Taking the first difference determined in the game scene as an example, the first predicted state is: player position (x=10, y=20), health value 75%, and resource quantity 30. The first preset state is: player position (x=15, y=25), attack power 80%, and resource quantity 35. The first difference includes position difference, health value difference, and resource quantity difference, where the position difference is The difference in attack power is |75%-80%|=5%, and the difference in resource quantity is |30-35|=5.

[0190] Taking the information recommendation scenario as an example, the first predicted state: user interest vector [0.3, 0.2, 0.5], historical interaction [1, 0, 1, 0], recommended item features [0.6, 0.1, 0.3]. The first preset state: user interest vector [0.4, 0.1, 0.5], historical interaction [1, 0, 0, 1], recommended item features [0.5, 0.2, 0.4]. The first difference includes the interest vector difference, the historical interaction difference and the item feature difference, where the interest vector difference is ([0.3, 0.2, 0.5] · [0.4, 0.1, 0.5]) / (‖[0.3, 0.2, 0.5]‖ · ‖[0.4, 0.1, 0.5]‖) = 0.98, the historical interaction difference is |[1, 0, 1, 0] - [1, 0, 0, 1]| = 1, and the item feature difference is

[0191] Here, when the number of execution models is one, the first preset state updated by the first state is the state preset by the execution model after executing the first target behavior, and the second preset state updated by the second state is the state preset by the execution model after executing the second target behavior; when the number of execution models is multiple, the first preset state updated by the first state is the overall preset state of multiple execution models after the execution model executes the first target behavior, and the second preset state updated by the second state is the overall preset state of multiple execution models after the execution model executes the second target behavior.

[0192] For example, taking the game scene as an example, the parameters of the control model include at least one of the following: resource growth rate, unit production speed and cost, technology research and development speed, building construction speed and cost, and special ability unlocking conditions. The parameters of the execution model include at least one of the following: resource acquisition parameters (such as resource growth rate), technology tree parameters (such as technology research and development speed), unit parameters (such as unit production speed and unit cost), building parameters (such as building construction speed and building cost), special ability parameters (special ability unlocking conditions), map and terrain parameters (such as resource distribution).

[0193] In some embodiments, see Figure 3J , Figure 3J This is the tenth flow chart of the training method of the simulation system provided in the embodiment of the present application. Figure 3A Step 103 can be Figure 3J Steps 1031 to 1033 are implemented as described in detail below.

[0194] In step 1031, a first loss is determined based on a first difference between the first predicted state and a first preset state updated by the first state, and a preset first loss function, wherein the first preset state is a preset state reached by the execution model after initialization.

[0195] In some embodiments, the first predicted state and the first preset state are both numerically processed, and the processed first predicted state and the first preset state are substituted into a preset first loss function to obtain a first loss.

[0196] For example, the first loss function may be a mean square error (MSE) loss function, a cross entropy loss function, a logarithmic loss function, etc. Taking the mean square error loss function as an example, if the first preset state after numerical processing is ([1, 2, 3]), the first predicted state is ([1.1, 1.9, 3.2]), the square of the difference between each corresponding element in the first preset state and the first predicted state is determined, the sum of the squares of the differences corresponding to multiple elements is determined, and the ratio of the sum to the number of elements is taken as the first loss, then the first loss is ((1.1-1)^2+(1.9-2)^2+(3.2-3)^2) / 3=0.02.

[0197] In step 1032, a second loss is determined based on a second difference between the second predicted state and a second preset state updated by the second state, and a preset second loss function, wherein the second preset state is a preset state in which the execution model complies with the sample strategy.

[0198] In some embodiments, the second preset state is a preset state in which the execution model complies with the sample policy, the attributes of the object, and the context information in the target environment.

[0199] For example, the second loss function may be a mean square error (MSE) loss function, a cross entropy loss function, a logarithmic loss function, etc. The calculation process of the second loss is similar to the first loss, and will not be repeated here.

[0200] In step 1033, the total loss is determined based on the first loss and the second loss, and the parameters of the control model and the parameters of the execution model are updated based on the total loss to obtain an updated control model and an updated execution model.

[0201] In some embodiments, the total loss may be the sum or the square root of the first loss and the second loss.

[0202] For example, the first loss is x and the second loss is y. If the sum is taken as the total loss, the total loss is x+y; if the square root of the sum is taken as the total loss, the total loss is (x^2+y^2)^(1 / 2).

[0203] By considering both the preset state after initialization and the preset state that complies with the sample strategy, the execution model can better learn diverse features and patterns, thereby improving its generalization ability when faced with new data. Calculating the first loss and the second loss separately can more accurately evaluate the performance of the execution model and make adjustments based on specific differences, which helps to improve the accuracy of the model. Parameters can be dynamically adjusted during the operation of the model to better adapt it to the current task requirements. The method of automatically adjusting model parameters can reduce manual intervention, improve the level of automation, and reduce long-term operating costs.

[0204] The environment simulation method of the simulation system provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0205] In some embodiments, the simulation system is trained by the training method of the above-mentioned simulation system, and the environment simulation method includes: without applying the target strategy, triggering the execution model to execute a first state update through the control model to obtain a first observation state of the execution model.

[0206] Here, without applying the target strategy, a first update prompt word is sent to the execution model through the control model, wherein the first update prompt word is used to instruct the execution model to perform a first state update; the first update information sent by the execution model is received through the control model, wherein the first update information represents that the execution model has determined that the first update prompt word has been received; the execution model performs a first state update to obtain a first observation state.

[0207] For example, the first update prompt sent by the control model may be "It is the third moment now, start the status update.", and the first update information may be "Received, start the status update."

[0208] In some embodiments, see Figure 3K , Figure 3K is a first flow chart of the environment simulation method of the simulation system provided by the embodiment of the present application. After the above-mentioned "triggering the execution model to execute the first state update through the control model to obtain the first observation state of the execution model", execute Figure 3K Steps 401 to 402 are described in detail below.

[0209] In step 401, the control model is used to trigger the execution model to apply the target policy, and after applying the target policy, the execution model is triggered to perform a second state update to obtain a second observed state of the execution model under the influence of the target policy.

[0210] In some embodiments, a second update prompt word is sent to the execution model through the control model, wherein the second update prompt word is used to instruct the execution model to apply the target strategy and update the state of the execution model according to the target strategy; second update information sent by the execution model is received through the control model, wherein the second update information represents that the execution model determines that the second update prompt word is received and starts a second state update; the updated state of the execution model sent by the control model is received, and the second observation state is determined based on the updated state.

[0211] For example, the second update prompt sent by the control model may be "It is the fourth moment now, please apply the target policy "XXX" and start the status update." The second update information may be "Received, the target policy is being applied, and the status update based on the target policy is started."

[0212] In some embodiments, see Figure 3L , Figure 3L is a second flow chart of the environment simulation method of the simulation system provided by the embodiment of the present application. After step 401 "triggering the execution model to execute the second state update, obtaining the second observation state of the execution model under the influence of the target strategy", execute Figure 3L Steps 501 to 502 are described in detail below.

[0213] In step 501, after the target policy is canceled, the execution model is triggered to perform a third state update, and a third observation state of the execution model is obtained when the target policy is canceled.

[0214] In some embodiments, a third update prompt word is sent to the execution model through the control model, wherein the third update prompt word is used to instruct the execution model to cancel the application of the target policy and update the state of the execution model; third update information sent by the execution model is received through the control model, wherein the third update information represents that the execution model determines that the third update prompt word is received and starts a third state update; the updated state of the execution model sent by the control model is received, and the third observation state is determined based on the updated state.

[0215] For example, the third update prompt sent by the control model may be "It is the fifth moment now, please cancel the application of the target policy "XXX" and start the status update." The third update information may be "Received, ready to cancel the application of the target policy and start the status update."

[0216] In step 502, a second intervention effect indicator of the target strategy is determined based on the first observation state and the third observation state.

[0217] In some embodiments, the difference between the first observation state and the third observation state may be used as a second intervention effect indicator of the target strategy.

[0218] For example, taking the game as an example, if there are three camps in the game, namely camp A, camp B and camp C, the first observation state can be "the winning rate ratio of camp A, camp B, camp C is 2:2:6", and the third observation state can be "the winning rate ratio of camp A, camp B, camp C is 2.1:2.1:5.8". The second intervention effect indicator of the target strategy is the difference in the winning rate ratio of camp A, camp B, camp C, that is, 0.1:0.1:0.2.

[0219] Here, the second intervention effect indicator reflects whether the impact of the target strategy will persist after the strategy is canceled. If the third observation state is significantly different from the first observation state, it indicates that the target strategy has a long-term sustained effect. The second intervention effect indicator can quantify the sustained effect of the target strategy.

[0220] By comparing the first and third observation states, the embodiment of the present application can evaluate the long-term effectiveness of the strategy after implementation. The second intervention effect indicator provides detailed information about the state changes after the strategy is implemented and cancelled, so as to evaluate the overall effect of the strategy and make corresponding decisions accordingly.

[0221] Continue to see Figure 3K In step 402, based on the first observation state and the second observation state, a first intervention effect indicator of the target strategy is determined.

[0222] In some embodiments, the difference between the first observed state and the second observed state may be used as a first intervention effect indicator of the target strategy.

[0223] For example, taking the game as an example, if there are three camps in the game, namely camp A, camp B and camp C, the first observation state can be "the winning rate ratio of camp A, camp B, and camp C is 2:2:6". When applying the target strategy, such as enhancing the attack power of camp A and camp B, the second observation state can be "the winning rate ratio of camp A, camp B, and camp C is 3:3:4". The first intervention effect indicator of the target strategy is the difference in the winning rate ratio of camp A, camp B, and camp C, that is, 1:1:2.

[0224] The embodiment of the present application quantifies the difference between the first observation state and the second observation state through the first intervention effect indicator to evaluate the direct impact of the target strategy on the execution model state. By clarifying the changes before and after the implementation of the target strategy, the effectiveness of the target strategy can be verified. Feedback is provided for strategy adjustment and optimization. If the intervention effect indicator does not meet expectations, it can indicate the need for further strategy adjustment.

[0225] Here, the first observation state, the second observation state, and the third observation state may be individual states for a single object, or may be overall states obtained by statistics for multiple objects.

[0226] In some embodiments, see Figure 3M , Figure 3M 3 is a schematic diagram of a third flow chart of an environment simulation method of a simulation system provided in an embodiment of the present application. Before applying a target strategy, or after applying a target strategy, executing Figure 3M Steps 601 to 602 are described in detail below.

[0227] In step 601, a first prompt word is sent to a second execution model via a first execution model, wherein the first prompt word instructs the second execution model to collaborate with the first execution model.

[0228] For example, the first prompt word sent by the first execution model to the second execution model may be "In order to achieve a common goal, the following cooperation strategy is recommended: xxx", which is used to instruct the second execution model to collaborate with the first execution model.

[0229] In step 602, a first response message sent by the second execution model is received, wherein the first response message indicates that the second execution model and the first execution model enter a collaborative mode, or the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute a target policy.

[0230] In some embodiments, before sending the first response information, the second execution model determines the state of the second execution model after collaborating with the first execution model to execute the target policy. If the state of the second execution model after the cooperation is improved compared to before the cooperation, the first response information is sent, wherein the first response information indicates that the second execution model and the first execution model enter a collaborative mode; if the state of the second execution model after the cooperation is reduced compared to before the cooperation, the first response information is sent, wherein the first response information indicates that the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute the target policy.

[0231] For example, if the status of the second execution model after the cooperation is improved compared to before the cooperation, the first response information is "Agree, I will cooperate in implementing the target strategy"; if the status of the second execution model after the cooperation is reduced compared to before the cooperation, the first response information is "Reject, I still implement the target strategy alone."

[0232] The embodiments of the present application can promote resource sharing between execution models through the cooperation mode, such as computing power, data, etc., thereby improving resource utilization efficiency. The cooperation mode allows the models to allocate tasks according to their respective advantages and optimize the overall workflow. Through collaboration, the execution models can share the workload, reduce duplication of work, and improve the execution efficiency of the entire simulation system.

[0233] In some embodiments, see Figure 3N , Figure 3N 4 is a schematic diagram of a fourth flow chart of an environment simulation method of a simulation system provided in an embodiment of the present application. Before applying a target strategy, or after applying a target strategy, executing Figure 3N Steps 701 to 702 are described in detail below.

[0234] In step 701, a second prompt word is sent to a second execution model through a first execution model, wherein the second prompt word indicates that the second execution model competes with the first execution model.

[0235] For example, the second prompt word sent by the first execution model to the second execution model may be "Based on the current situation, I think we should adopt a competition strategy to obtain more resources.", which is used to instruct the second execution model to compete with the first execution model.

[0236] In step 702, a second response message sent by the second execution model is received, wherein the second response message indicates that the second execution model and the first execution model enter a competition mode, or the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute a target policy.

[0237] In some embodiments, before sending the second response information, the second execution model determines the state of the second execution model after competing with the first execution model to execute the target strategy. If the state of the second execution model after the competition is improved compared to before the cooperation, the second response information is sent, wherein the second response information indicates that the second execution model and the first execution model enter a competition mode; if the state of the second execution model after the competition is decreased compared to before the cooperation, the second response information is sent, wherein the second response information indicates that the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute their target strategy.

[0238] For example, if the status of the second execution model after the competition is improved compared to before the cooperation, the first response information is "Agree, I will compete with you to execute the target strategy"; if the status of the second execution model after the competition is reduced compared to before the cooperation, the first response information is "Reject, I still execute the target strategy alone."

[0239] In the embodiment of the present application, the execution model will continuously optimize its own algorithms and strategies through the competition mode, thereby improving performance. This can prompt the model to use resources more efficiently to gain a competitive advantage. In a competitive environment, the execution model needs to adapt to changes quickly, which helps the simulation system to respond quickly to changes in external conditions. Competition enables the execution model to develop a variety of strategic solutions, increasing the overall adaptability and robustness of the simulation system.

[0240] Below, an exemplary application of an embodiment of the present application in an application scenario of a game will be described.

[0241] In recent years, with the rapid development of artificial intelligence technology, large models have shown great potential and value in simulating complex systems and predicting behaviors. Large models not only have excellent natural language processing capabilities, but can also demonstrate human-like intelligence in simulation systems.

[0242] When building and training a simulation system, related technologies often manually adjust the model's parameters to trigger the model to update its status. At the same time, the model training process relies on multiple sets of pre-acquired data sets. By setting a validation set and a test set, the model is trained to obtain a trained simulation system, which consumes a lot of time and is difficult to provide real-time simulation results for comparison with real-world data, resulting in low accuracy of the simulation system and low training efficiency.

[0243] The embodiment of the present application controls the execution model through the control model to obtain the predicted state of the execution model before and after the application of the sample strategy, and based on the difference between the preset state, updates the parameters of the control model and the parameters of the execution model to obtain an updated control model and an updated execution model. The control model and the execution model are automatically regulated and updated, so that the model in the simulation system can adjust the parameters according to the difference in the state before and after the application of the sample strategy, so that the trained simulation system meets the expected state before and after the application of the target strategy, thereby improving the accuracy of the simulation system in state prediction.

[0244] Based on the large-model agent simulation technology, a simulation environment (i.e., the above-mentioned simulation system) including a control agent (i.e., the above-mentioned control model) and an execution agent (i.e., the above-mentioned execution model) is constructed to simulate the behavior changes of real objects or virtual objects under strategies or marketing interventions (i.e., the above-mentioned target strategies). The simulation system observes the macro results in the real world and uses the simulation capabilities of the large-model agent to predict and evaluate the effects of intervention measures, while realizing counterfactual construction, providing comprehensive analysis and support for decision makers.

[0245] See also Figure 6 , Figure 6The simulation system includes multiple services, such as real environment observation service, large model intelligent agent micro-individual simulation service, result alignment service, counterfactual construction service, and result evaluation and output service.

[0246] The real environment observation service is used to observe the changes in macro results before and after strategy or marketing intervention in the real world. Input the time point t0 before the intervention and the time point t1 after the intervention in the real environment observation service, and the real environment observation service can output the macro results real_t0 (i.e. the first preset state mentioned above) and real_t1 (i.e. the second preset state mentioned above) corresponding to the real world at time points t0 and t1 respectively.

[0247] The large-model intelligent agent micro-individual simulation service is used to build a simulation system that includes a control agent and an execution agent, simulating the behavior changes of real objects or virtual objects under strategic or marketing interventions. The large-model intelligent agent micro-individual simulation service includes a control agent and an execution agent. The control agent is responsible for setting the global rules of the simulation system and constraining the behavior of the execution agent. The control agent is implemented through a large model and can understand and respond to complex instructions, and guide and adjust the execution agent according to the global rules. The execution agent is heterogeneous and is used to simulate the reactions of different individuals under strategic or marketing interventions. The execution agent is also built based on the large model. Each agent has unique attributes and characteristics and can make decisions and adjust behaviors according to its own situation and the guidance of the control agent. By inputting the time point t0 before the intervention and the time point t1 after the intervention, as well as the real-world macro results real_t0 and real_t1 into the large-model intelligent agent micro-individual simulation service, the large-model intelligent agent micro-individual simulation service can output the emergent results sim_t0 (that is, the first predicted state mentioned above) and sim_t1 (that is, the second predicted state mentioned above) corresponding to the simulation system at the time points t0 and t1 respectively.

[0248] The result alignment service is used to ensure that the emergent results of the simulation system are consistent with the macro results of the real world, that is, real_t0=sim_t0, real_t1=sim_t1. The simulation system is iteratively optimized by adjusting the parameters of the control agent and the behavior model of the execution agent. The specific methods include: adjusting the decision logic of the control agent, optimizing the distribution of the attributes of the execution agent (that is, the parameters of the execution agent mentioned above), adjusting the environmental parameters of the simulation system, etc., until the simulation results are aligned with the macro results of the real world, where attributes or parameters are parameters inside the agent. In the game scene, they are equivalent to the configuration data of the game itself. Each camp in the game corresponds to a configuration table to record these parameters and decision logic. When updating the parameters, simply change the data recorded in the configuration table.

[0249] Take the game scenario as an example. There are three camps in the game: red, blue, and green. The game developer found that the number of players in the red camp was significantly greater than that of the other two camps, causing the game to lose balance. In order to adjust the balance of the camps, the developer plans to balance the winning rate of each camp in the game by adjusting the game parameters. Among them, the decision logic of the regulating agent includes at least one of the following:

[0250] 1) Resource growth rate: For the blue and green camps, increase the resource growth rate to attract new players.

[0251] 2) Unit production speed and cost: For the Red Faction, reduce unit production speed and increase unit cost to reduce its advantage.

[0252] 3) Technology research and development speed: For the blue camp and the green camp, increase the speed of technology research and development to quickly gain strategic advantages.

[0253] 4) Building construction speed and cost: For the blue and green camps, increase the building construction speed and reduce the construction cost to promote rapid development.

[0254] 5) Special ability unlocking conditions: For the Blue and Green camps, lower the special ability unlocking conditions to increase their attractiveness.

[0255] The properties of an executive agent include at least one of the following:

[0256] 1) Resource acquisition parameters, such as resource growth rates. Adjust the resources (such as virtual props, wood, stone) of different camps to adjust the speed. For example, enhance the resource growth rate of the blue camp and the green camp to match or slightly higher than the red camp to promote new players to join, or reduce the resource growth rate of the red camp to reduce its advantage.

[0257] 2) Technology tree parameters, such as technology development speed. Adjust the technology development speed of different camps, for example, provide faster technology development speed for the blue camp and the green camp to quickly gain strategic advantages, or reduce the technology development speed of the red camp to maintain a balance between the camps.

[0258] 3) Unit parameters, such as unit production speed and unit cost. If the unit production speed is adjusted, the production speed of the blue and green camps can be increased so that they can respond quickly to the pressure of the red camp, or the production speed of the red camp can be adjusted to reduce its advantage. If the unit production cost is adjusted, the production cost of the blue and green camp units can be reduced to make it easier to expand the camp, or the production cost of the red camp units can be increased to reduce its advantage.

[0259] 4) Building parameters, such as building speed and building cost. If you adjust the building speed, you can increase the building speed of the blue and green camps to quickly build defenses, or reduce the building speed of the red camp to reduce its advantage in buildings. If you adjust the building cost, you can reduce the cost of blue and green camp buildings to promote their rapid development, or increase the cost of red camp buildings to balance its economic advantage.

[0260] 5) Special ability parameters, such as the conditions for unlocking special abilities. Adjust the conditions for unlocking special abilities, for example, providing easier unlocking special abilities for the Blue and Green camps to increase their appeal, or making the conditions for unlocking special abilities for the Red camp more difficult to maintain a balance between the camps.

[0261] 6) Map and terrain parameters, such as resource distribution. Adjust the distribution of resources on the map, for example, add resource-rich areas to the blue and green camps to attract players to choose these camps, or reduce the resource distribution of the red camp to reduce its resource advantage.

[0262] The counterfactual construction service is used to analyze the changes in the attributes / traits of the executing agent based on the aligned simulation system, and obtain the counterfactual results (i.e., the third observed state) in the absence of intervention (i.e., after the target strategy is cancelled). The counterfactual results are used to evaluate the effectiveness of strategies or marketing interventions. The counterfactual results include the attribute changes and behavior patterns of the executing agent in the absence of intervention. By comparing them with the results after intervention, the effectiveness of the intervention can be quantitatively evaluated.

[0263] The control agent controls the execution agent by sending prompts, so that the execution agent adjusts parameters according to the prompts. The interaction between the control agent and the execution agent through prompts includes the following situations: simulation system initialization, intervention implementation, and status update.

[0264] When the simulation system is initialized, the control agent sends a prompt word (that is, the first sample prompt word mentioned above), such as "The system has been initialized, and the current time is t0. Please guide the execution agent to set the initial state according to the global rules.", where the control agent sets the initial state of all execution agents (such as multiple camps in the game), such as resource acquisition parameters, unit parameters, and technology tree parameters. After receiving the prompt word, the execution agent sends a response message, such as "Received, setting the initial state according to the global rules..." At this time, Prompt is used to start the simulation system and guide the execution agent to set the initial state according to the global rules. The control agent ensures that the execution agent has the correct initial conditions before the simulation starts by sending clear instructions.

[0265] When implementing the intervention, the control agent sends a prompt word (that is, the second sample prompt word mentioned above), such as "The current time is t1, and the strategy / marketing intervention is about to be implemented. Please ask the execution agents to adjust their behavior according to the intervention measures." After receiving the prompt word, the execution agent sends a response message, such as "Received, adjusting behavior according to the intervention measures..." At this time, Prompt is used to introduce strategy or marketing intervention measures in the simulation process. The control agent guides the execution agent to adjust its behavior according to the intervention measures by sending intervention instructions. The execution agent adjusts its behavior to adapt to the new environment based on its own attributes and characteristics, as well as the specific content of the intervention measures.

[0266] When the status is updated, the control agent sends a prompt, such as "The current time has advanced to t2, please update the status of the execution agents and report the changes." Among them, t2 is the time point after t1, and the status changes of the execution agents within a period of time after the intervention can be viewed at t2. After receiving the prompt, the execution agent sends a response message, such as "Received, updating the status and reporting changes..." At this time, Prompt is used to collect the status information of the execution agent after the simulation is completed. The control agent sends a status update instruction to ask the execution agent to report the status changes during the simulation process. The execution agent generates a status report based on its own behavior and decision-making process and submits it to the control agent for analysis.

[0267] Taking the game scene as an example, the simulation system includes at least one control model and multiple execution models, and the objects include the player characters (PC) or non-player characters (NPC) in the game. The control model is responsible for formulating and adjusting the game strategy. The execution model models the player characters and NPCs respectively to simulate their behaviors in the game. At this time, two strategies are set, strategy A is to increase the attack power of the player character, and strategy B is to increase the movement speed of the NPC. First, set the initial state, the player character: basic attack power is 50, basic defense power is 30, and basic movement speed is 5. NPC: basic attack power is 40, basic defense power is 20, and basic movement speed is 4. Secondly, the control model applies strategies, for example, strategy A is to increase the attack power of the player character by 20%, that is, the new attack power of the player character is 60. Strategy B is to increase the movement speed of the NPC by 25%, that is, the new movement speed of the NPC is 5. Then, the execution model performs simulation, and the execution model corresponding to the player character simulates the performance of the player character in battle after applying strategy A. The execution model corresponding to the NPC simulates the movement and interaction of the NPC in the game after applying strategy B. Next, observe the status difference. For example, before applying strategy A, the player character can defeat 3 enemies on average in a battle. After applying strategy A, due to the increased attack power, the player character can defeat 5 enemies on average. Before applying strategy B, the NPC Figure 1 It takes 10 game rounds to move from one end to the other. After applying strategy B, the NPC moves from the ground to the other end due to the increased movement speed. Figure 1 It only takes 8 game rounds to move from one end to the other. Finally, the intervention effect index is analyzed. The effect index of strategy A, that is, the increase rate of the number of defeated enemies, is ((5-3) / 3)*100%=66.67%, and the effect index of strategy B, that is, the reduction rate of movement time, is ((10-8) / 10)*100%=20%.

[0268] Taking the information recommendation scenario as an example, the simulation system includes at least one control model and multiple execution models. The execution model is obtained by modeling the objects in the target environment. When the target environment is a social group in the real world and the objects are members of the social group, some strategies are applied to the execution model through the control model. The difference in the reaction state of the execution model to the recommended information before and after the strategy is applied can be observed, so as to analyze the intervention effect indicators of these strategies on information recommendation. Among them, the control model is responsible for formulating and adjusting the strategy of information recommendation. The execution model models the behavior and reaction of social group members. At this time, two strategies are set. Strategy C recommends personalized content based on the interests and activity history of members, and strategy D recommends the dynamics of other members in the group to increase social interaction. First, set the initial state. For example, member A usually checks social media 10 times a day, and the proportion of interest in recommended information is 30%. Member B usually checks social media 15 times a day, and the proportion of interest in recommended information is 40%. Secondly, the regulation model applies strategies. Strategy C is to personalize the recommended information of members A and B to ensure that the recommended content matches their interests and activity history. Strategy D is to increase the social interaction recommendations of members A and B, such as posts, comments, and event invitations of other members. Then, the execution model is simulated. The execution model corresponding to member A simulates the behavior changes of member A after receiving personalized recommended content, and the execution model corresponding to member B simulates the behavior changes of member B after receiving more social interaction recommendations. Next, observe the state difference. Before applying strategy C, member A checks social media 10 times a day and is interested in 3 recommended information. After applying strategy C, he checks social media 12 times a day and is interested in 5 recommended information. Before applying strategy D, member B checks social media 15 times a day and is interested in 6 recommended information. After applying strategy D, he checks social media 18 times a day and is interested in 8 recommended information. Finally, the intervention effect indicators were analyzed. The effect indicators of strategy C, that is, the interest ratio increased by ((5-3) / 3)*100%=66.67%, and the number of views increased by ((12-10) / 10)*100%=20%. The effect indicators of strategy D, that is, the interest ratio increased by ((8-6) / 6)*100%=33.33%, and the number of views increased by ((18-15) / 15)*100%=20%.

[0269] In the actual simulation process, there may be complex interactions between executing agents, such as cooperation or competition. In order to simulate these interactions, corresponding interactive prompts can be designed.

[0270] When the execution agents collaborate, the execution agent A sends a prompt word (that is, the first prompt word mentioned above) to the execution agent B, such as "In order to achieve a common goal, I suggest that we adopt the following cooperation strategy..." After receiving the prompt word, the execution agent B sends a response message (that is, the first response message mentioned above), such as "Agree, I will cooperate in implementing this strategy..." At this time, Prompt is used to simulate the cooperation behavior between the execution agents. The execution agent A negotiates and cooperates with the execution agent B by sending a cooperation strategy suggestion (that is, the transition to the cooperation mode mentioned above). The execution agent B decides whether to accept the cooperation suggestion based on its own interests and the feasibility of the cooperation strategy.

[0271] When the executive agents compete with each other, the executive agent A sends a prompt word (that is, the second prompt word mentioned above) to the executive agent B, such as "Based on the current situation, I think we should adopt a competitive strategy to obtain more resources." After receiving the prompt word, the executive agent B sends a response message (that is, the second response message mentioned above), such as "I disagree, I think cooperation is a better choice..." At this time, Prompt is used to simulate the competitive behavior between the executive agents. The executive agent A competes and bargains with the executive agent B by sending competitive strategy suggestions (that is, the above-mentioned transition to the competitive mode). The executive agent B decides whether to accept the competitive suggestion or propose an anti-competitive strategy based on its own interests and the feasibility of the competitive strategy.

[0272] Applying large model agent technology to the simulation analysis of the effects of strategy or marketing intervention has realized the simulation of complex systems from top to bottom. By building a simulation environment that includes control agents and execution agents, the behavior changes of real objects or virtual objects under strategy or marketing intervention are simulated, which improves the authenticity and accuracy of the simulation.

[0273] A result alignment mechanism is proposed to ensure that the simulation results are consistent with the macro results in the real world by iteratively optimizing the simulation system. This mechanism achieves the alignment of the simulation system with the real world by continuously adjusting the parameters of the control agent and the behavior model of the execution agent, providing a reliable basis for counterfactual construction.

[0274] Based on the aligned simulation system, the counterfactual construction capability is realized. By analyzing the changes in the attributes or characteristics of the executing agent, the counterfactual results without intervention are obtained, providing a scientific basis for strategic or marketing decisions. The counterfactual construction capability enables decision makers to quantitatively evaluate the effects of intervention measures, optimize intervention strategies, and improve decision-making results.

[0275] In marketing application scenarios, the impact of different marketing strategies on consumer behavior and market response can be simulated. By setting different marketing intervention measures, the changes in the purchasing behavior and brand preferences of the agents in the simulation system can be observed to evaluate the effectiveness and potential benefits of marketing strategies. Marketers can optimize marketing strategies based on simulation results to improve market competitiveness and sales performance.

[0276] In some embodiments, the computational cost of large model agents can be reduced through technologies such as model compression and knowledge distillation. At the same time, the computational efficiency of the simulation system can be improved by using technologies such as distributed computing and parallel computing. Dynamic adjustment and load balancing mechanisms can be introduced into the simulation system to adjust the allocation of computing resources according to the actual situation of the simulation task to ensure that the simulation system can maintain efficient and stable operation when facing large-scale simulation tasks. The accuracy of the result alignment mechanism can be improved by applying data-driven and machine learning algorithms. The simulation system is trained and optimized using real-world data to improve the prediction ability and robustness of the simulation system. At the same time, the simulation results are analyzed and evaluated in combination with machine learning algorithms to ensure the accuracy of the result alignment mechanism.

[0277] The embodiment of the present application realizes top-down simulation of complex systems by constructing a simulation environment including a control agent and an execution agent to simulate the behavior changes of real objects or virtual objects under strategy or marketing intervention. The simulation system has a result alignment mechanism and counterfactual construction capabilities, which can provide scientific decision support.

[0278] The following is a description of an exemplary structure of the training device 233 of the simulation system provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the training device 233 of the simulation system of the memory 230 may include:

[0279] The first updating module 2331 is used to trigger the execution model to perform a first state update through the regulation model to obtain a first predicted state of the execution model.

[0280] The second updating module 2332 is used to trigger the execution model to apply the sample strategy through the regulation model, and, after applying the sample strategy, trigger the execution model to perform a second state update to obtain a second predicted state of the execution model under the influence of the sample strategy.

[0281] The model update module 2333 is used to update the parameters of the control model and the parameters of the execution model based on the first difference between the first predicted state and the first preset state updated by the first state, and the second difference between the second predicted state and the second preset state updated by the second state, so as to obtain an updated control model and an updated execution model.

[0282] In some embodiments, the first update module 2331 is also used to send a first sample prompt word to the execution model through the control model, wherein the first sample prompt word is used to instruct the execution model to initialize; receive response information sent by the execution model through the control model, wherein the response information represents confirmation of receipt of the first sample prompt word; receive the initialization state of the execution model sent by the control model, and determine the first prediction state based on the initialization state.

[0283] In some embodiments, the first update module 2331 is further used to determine a first target behavior of the object based on the attributes of the object; and execute the first target behavior through the execution model, wherein the execution model is in an initialized state after executing the first target behavior.

[0284] In some embodiments, the first update module 2331 is also used to encode the attributes of the object to obtain an attribute encoding result; map the attribute encoding result to a probability distribution of multiple first candidate behaviors, wherein the probability distribution includes the probabilities corresponding to the multiple first candidate behaviors; and take the first candidate behavior corresponding to the largest probability as the first target behavior.

[0285] In some embodiments, the first update module 2331 is also used to, when the number of execution models is one, use the initialization state of the execution model as the first prediction state; when the number of execution models is multiple, the initialization states of multiple execution models are counted according to preset statistical rules to obtain the first prediction state, wherein different execution models correspond to different objects in the target environment.

[0286] In some embodiments, the second update module 2332 is also used to send a second sample prompt word to the execution model through the control model, wherein the second sample prompt word is used to instruct the execution model to apply the sample strategy and update the state of the execution model according to the sample strategy; receive the updated state sent by the control model, and determine the second predicted state based on the updated state.

[0287] In some embodiments, the second update module 2332 is also used to determine a second target behavior based on the attributes of the object, the contextual information of the object in the target environment, and the sample strategy; and execute the second target behavior through the execution model, wherein the execution model is in an updated state after executing the second target behavior.

[0288] In some embodiments, the second update module 2332 is also used to encode the attributes of the object, the contextual information of the object in the target environment, and the sample strategy, respectively, to obtain attribute coding results, information coding results, and strategy coding results; to fuse the attribute coding results, information coding results, and strategy coding results to obtain a fused coding result; to map the fused coding result to a probability distribution of multiple second candidate behaviors, wherein the probability distribution includes the probabilities corresponding to the multiple second candidate behaviors; and to use the second candidate behavior corresponding to the largest probability as the second target behavior.

[0289] In some embodiments, the second update module 2332 is also used to, when the number of execution models is one, use the updated state of the execution model as the second predicted state; when the number of execution models is multiple, perform statistics on the updated states of multiple execution models according to preset statistical rules to obtain the second predicted state, wherein different execution models correspond to different objects in the target environment.

[0290] In some embodiments, the model update module 2333 is also used to determine a first loss based on a first difference between a first predicted state and a first preset state updated from the first state, and a preset first loss function, wherein the first preset state is a preset state reached by the execution model after initialization; determine a second loss based on a second difference between a second predicted state and a second preset state updated from the second state, and a preset second loss function, wherein the second preset state is a preset state in which the execution model complies with the sample strategy; determine a total loss based on the first loss and the second loss, and update the parameters of the control model and the parameters of the execution model based on the total loss to obtain an updated control model and an updated execution model.

[0291] Next, the exemplary structure of the environment simulation device 273 of the simulation system provided by the embodiment of the present application is implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the environment simulation device 273 of the simulation system of the memory 270 may include:

[0292] The third updating module 2731 is used to trigger the execution model to perform a first state update through the regulation model when the target policy is not applied, so as to obtain a first observed state of the execution model.

[0293] In some embodiments, the third update module 2731 is also used to trigger the execution model to apply the target strategy through the regulation model, and, after applying the target strategy, trigger the execution model to perform a second state update to obtain a second observation state of the execution model under the influence of the target strategy; based on the first observation state and the second observation state, determine the first intervention effect indicator of the target strategy.

[0294] In some embodiments, the third update module 2731 is also used to trigger the execution model to perform a third state update after the target policy is canceled, so as to obtain the third observation state of the execution model when the target policy is canceled; and determine the second intervention effect indicator of the target policy based on the first observation state and the third observation state.

[0295] In some embodiments, the third update module 2731 is also used to send a first prompt word to the second execution model through the first execution model, wherein the first prompt word indicates that the second execution model collaborates with the first execution model; receive a first response message sent by the second execution model, wherein the first response message indicates that the second execution model and the first execution model enter a collaboration mode, or the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute the target policy.

[0296] In some embodiments, the third update module 2731 is also used to send a second prompt word to the second execution model through the first execution model, wherein the second prompt word indicates that the second execution model competes with the first execution model; receive a second response message sent by the second execution model, wherein the second response message indicates that the second execution model and the first execution model enter a competition mode, or the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute the target strategy.

[0297] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the training method of the simulation system described in the embodiment of the present application, or the environment simulation method of the simulation system.

[0298] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the training method of the simulation system provided in the embodiment of the present application, or the environment simulation method of the simulation system, for example, Figure 3A The training method of the simulation system is shown.

[0299] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0300] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0301] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0302] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0303] To summarize, through the embodiments of the present application, the execution model is regulated by the control model to obtain the predicted state of the execution model before and after the application of the sample strategy, and based on the difference between the preset state, the parameters of the control model and the parameters of the execution model are updated to obtain an updated control model and an updated execution model. The control model and the execution model are automatically regulated and updated, so that the model in the simulation system can adjust the parameters according to the difference in the state before and after the application of the sample strategy, so that the trained simulation system meets the expected state before and after the application of the target strategy, thereby improving the accuracy of the simulation system in state prediction.

[0304] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A training method for a simulation system, characterized in that: The simulation system includes at least one control model and multiple execution models, wherein the execution model is obtained by modeling objects in a target environment, and one of the control models is used to control at least one of the execution models; The method comprises: Triggering the execution model to execute a first state update through the control model to obtain a first predicted state of the execution model; triggering the execution model to apply the sample strategy through the control model, and, After applying the sample strategy, triggering the execution model to perform a second state update, and obtaining a second predicted state of the execution model under the influence of the sample strategy; Based on a first difference between the first predicted state and a first preset state updated by the first state, and a second difference between the second predicted state and a second preset state updated by the second state, the parameters of the control model and the parameters of the execution model are updated to obtain an updated control model and an updated execution model.

2. The method according to claim 1, characterized in that The step of triggering the execution model to execute a first state update through the control model to obtain a first predicted state of the execution model includes: Sending a first sample prompt word to the execution model through the control model, wherein the first sample prompt word is used to instruct the execution model to initialize; Receiving, through the control model, a response message sent by the execution model, wherein the response message indicates confirmation of receipt of the first sample prompt word, and An initialization state of the execution model sent by the control model is received, and a first prediction state is determined based on the initialization state.

3. The method according to claim 2, characterized in that The execution model is initialized in the following manner: determining a first target behavior of the object based on the attribute of the object; The first target behavior is executed by the execution model, wherein the execution model is in the initialization state after executing the first target behavior.

4. The method according to claim 3, characterized in that The determining a first target behavior of the object based on the attribute of the object comprises: Encoding the attributes of the object to obtain an attribute encoding result; Mapping the attribute encoding result into a probability distribution of multiple first candidate behaviors, wherein the probability distribution includes probabilities corresponding to the multiple first candidate behaviors respectively; The first candidate behavior corresponding to the maximum probability is used as the first target behavior.

5. The method according to claim 2, characterized in that: The determining a first predicted state based on the initialization state comprises: In a case where the number of the execution model is one, taking the initialization state of the execution model as a first prediction state; In the case where there are multiple execution models, the initialization states of the multiple execution models are counted according to a preset statistical rule to obtain the first predicted state, wherein different execution models correspond to different objects in the target environment.

6. The method according to claim 1, characterized in that The step of triggering the execution model to apply the sample strategy through the control model, and, after applying the sample strategy, triggering the execution model to perform a second state update to obtain a second predicted state of the execution model under the influence of the sample strategy, includes: Sending a second sample prompt word to the execution model through the control model, wherein the second sample prompt word is used to instruct the execution model to apply the sample strategy and update the state of the execution model according to the sample strategy; An updated state sent by the control model is received, and a second predicted state is determined based on the updated state.

7. The method according to claim 6, characterized in that The method of adjusting the state of the execution model according to the sample strategy includes: determining a second target behavior based on the attribute of the object, context information of the object in the target environment, and the sample strategy; The second target behavior is executed by the execution model, wherein the execution model is in the updated state after executing the second target behavior.

8. The method according to claim 7, characterized in that The determining of the second target behavior based on the attribute of the object, the context information of the object in the target environment, and the sample strategy comprises: Encoding the attributes of the object, the context information of the object in the target environment, and the sample strategy respectively to obtain an attribute encoding result, an information encoding result, and a strategy encoding result; Fusing the attribute coding result, the information coding result and the strategy coding result to obtain a fused coding result; Mapping the fusion coding result into a probability distribution of multiple second candidate behaviors, wherein the probability distribution includes probabilities corresponding to the multiple second candidate behaviors respectively; The second candidate behavior corresponding to the maximum probability is used as the second target behavior.

9. The method according to claim 6, characterized in that The determining a second predicted state based on the updated state comprises: In a case where the number of the execution model is one, taking the updated state of the execution model as a second predicted state; In the case where there are multiple execution models, the updated states of the multiple execution models are counted according to preset statistical rules to obtain the second predicted state, wherein different execution models correspond to different objects in the target environment.

10. The method according to any one of claims 1 to 9, characterized in that: The updating of the parameters of the control model and the parameters of the execution model based on a first difference between the first predicted state and a first preset state updated from the first state, and a second difference between the second predicted state and a second preset state updated from the second state, to obtain the updated control model and the updated execution model, comprises: determining a first loss based on a first difference between the first predicted state and a first preset state updated by the first state, and a preset first loss function, wherein the first preset state is a preset state reached by the execution model after initialization; Determine a second loss based on a second difference between the second predicted state and a second preset state updated by the second state, and a preset second loss function, wherein the second preset state is a preset state in which the execution model complies with the sample strategy; A total loss is determined based on the first loss and the second loss, and parameters of the control model and parameters of the execution model are updated based on the total loss to obtain an updated control model and an updated execution model.

11. An environment simulation method for a simulation system, characterized in that: The simulation system is obtained by training through the simulation system training method according to any one of claims 1 to 10; The environmental simulation method comprises: In the case where the target policy is not applied, the execution model is triggered by the control model to perform a first state update to obtain a first observed state of the execution model.

12. The method according to claim 11, characterized in that After triggering the execution model to execute the first state update through the control model to obtain the first observed state of the execution model, the method further includes: triggering the execution model to apply a target strategy through the control model, and, After applying the target policy, triggering the execution model to perform a second state update to obtain a second observed state of the execution model under the influence of the target policy; Based on the first observation state and the second observation state, a first intervention effect indicator of the target strategy is determined.

13. The method according to claim 12, characterized in that After triggering the execution model to execute the second state update and obtaining the second observed state of the execution model under the influence of the target strategy, the method further includes: After cancelling the application of the target policy, triggering the execution model to perform a third state update, to obtain a third observed state of the execution model when cancelling the application of the target policy; Based on the first observation state and the third observation state, a second intervention effect indicator of the target strategy is determined.

14. The method according to any one of claims 11 to 13, characterized in that: Before applying the target policy, or after applying the target policy, the method further includes: Sending a first prompt word to a second execution model through a first execution model, wherein the first prompt word indicates that the second execution model cooperates with the first execution model; Receive a first response message sent by the second execution model, wherein the first response message indicates that the second execution model and the first execution model enter a collaborative mode, or the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute the target policy.

15. The method according to any one of claims 11 to 13, characterized in that: Before applying the target policy, or after applying the target policy, the method further includes: Sending a second prompt word to the second execution model through the first execution model, wherein the second prompt word indicates that the second execution model competes with the first execution model; Receive a second response message sent by the second execution model, wherein the second response message indicates that the second execution model and the first execution model enter a competition mode, or the second execution model and the first execution model maintain a default mode, wherein the default mode is a mode in which the second execution model and the first execution model each execute the target policy.

16. A training device for a simulation system, characterized in that: The device comprises: A first updating module, configured to trigger the execution model to execute a first state update through the control model to obtain a first predicted state of the execution model; A second updating module is used to trigger the execution model to apply the sample strategy through the control model, and, after applying the sample strategy, trigger the execution model to perform a second state update to obtain a second predicted state of the execution model under the influence of the sample strategy; A model updating module is used to update the parameters of the control model and the parameters of the execution model based on a first difference between the first predicted state and a first preset state updated by the first state, and a second difference between the second predicted state and a second preset state updated by the second state, so as to obtain an updated control model and an updated execution model.

17. An environment simulation device for a simulation system, characterized in that: The device comprises: The third updating module is used to trigger the execution model to perform a first state update through the control model when the target strategy is not applied, so as to obtain a first observed state of the execution model.

18. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the training method of the simulation system described in any one of claims 1 to 10, or to implement the environment simulation method of the simulation system described in any one of claims 11 to 15 when executing the computer executable instructions or computer programs stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the training method for a simulation system described in any one of claims 1 to 10 is implemented, or the environment simulation method for a simulation system described in any one of claims 11 to 15 is implemented.

20. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the training method for a simulation system described in any one of claims 1 to 10 is implemented, or the environment simulation method for a simulation system described in any one of claims 11 to 15 is implemented.

Images