Human-machine intelligent confrontation method and system based on cloud architecture

Abstract

The invention belongs to the technical field of computers, and aims to provide a man-machine intelligent confrontation method and system based on a cloud architecture. On one hand, multiple terminal devices are used for collecting user behavior data in a distributed mode and generating local confrontation samples, strategy training is conducted through the multiple terminal devices on the basis of the local confrontation samples of the terminal devices, and a global model is formed through aggregation optimization of a cloud server; the generalization ability and flexibility of the model under diversified human behavior data are remarkably improved, and the problems of single strategy and poor adaptability in the prior art are solved; and on the other hand, after the cloud server completes global model training, a strategy revenue matrix is generated through large-scale man-machine confrontation simulation, and a game optimization method is further introduced to solve an equilibrium strategy, so that the confrontation adaptability and the intelligent decision-making capability of the AI strategy in a complex human behavior mode are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and specifically relates to a human-machine intelligent confrontation method and system based on a cloud architecture. Background Art

[0002] With the rapid development of artificial intelligence (AI), human-machine intelligent confrontation technology has been widely used in scenarios such as military simulation training, strategy games, educational games, and security simulations. Existing human-machine confrontation systems are typically based on locally deployed agent models, trained locally using user behavior data, and execute confrontation tasks using static strategies. Some systems rely on expert experience to construct rules or finite state machines, while others employ deep reinforcement learning (DRL) techniques to learn strategies and enhance confrontation capabilities.

[0003] However, in the process of using the existing technology, the inventors found that the traditional human-machine confrontation system generally has the following problems: a. Locally trained models have poor generalization capabilities. When faced with new adversarial environments or unknown human behavior patterns, the models struggle to adapt, exhibiting problems such as a single strategy and insufficient flexibility. b. Resource silos are severe, preventing effective sharing of data and models between adversarial terminals. This leads to low model iteration efficiency, limiting the rapid optimization and capability transfer of intelligent agents. Summary of the Invention

[0004] The present invention aims to solve the above technical problems at least to a certain extent, and provides a human-machine intelligent confrontation method and system based on cloud architecture.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a human-machine intelligent confrontation method based on a cloud architecture, which is executed by a cloud server and multiple terminal devices, and the method includes: Each terminal device collects user behavior data and constructs the user behavior data into local adversarial sample data; The cloud server initializes the parameters of the global model and sets the obtained initial global model parameters as the current global model parameters; The cloud server broadcasts the current global model parameters to each terminal device; Each terminal device trains a local model based on its own local adversarial sample data and the current global model parameters, and uploads the trained local model parameters to the cloud server; The cloud server aggregates the local model parameters sent by each terminal device to obtain new global model parameters; The cloud server determines whether a preset number of training rounds has been reached. If so, it proceeds to the next step. If not, it updates the current global model parameters to the new global model parameters, and then broadcasts the current global model parameters to each terminal device so that each terminal device trains its local model based on its own local adversarial sample data and the new global model parameters. The cloud server uses the final global model parameters to update the parameters of the global model to obtain a trained global model; The cloud server performs a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructs a human-machine strategy benefit matrix based on the simulated adversarial sample data; The cloud server performs game theory to solve the human-machine strategy payoff matrix to obtain an equilibrium strategy.

[0006] In one possible design, the user behavior data is constructed as local adversarial sample data, including: Constructing the user behavior data as initial local adversarial sample data; Normalizing the state vector in the initial local adversarial sample data to obtain a normalized state vector; Performing behavior encoding processing on the actions in the initial local adversarial sample data to obtain encoded actions; Based on the normalized state vector and the encoded action, final local adversarial sample data is constructed.

[0007] In one possible design, the local model of each terminal device adopts the DQN model; correspondingly, the target optimization function of the terminal device adopts the DQN loss function.

[0008] In a possible design, the cloud server aggregates the local model parameters sent by each terminal device using a weighted average method; correspondingly, each terminal device performs a first t After a round of training, the cloud server aggregates the local model parameters sent by each terminal device to obtain the new global model parameters: ; Where, N is the number of terminal devices participating in the training; D i ∣ is the i The amount of local data on each terminal device; D j ∣ is the j The amount of local data on each terminal device; For the i The terminal device performs the firstt The local model parameters obtained after the round.

[0009] In one possible design, the cloud server performs a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructs a human-machine strategy benefit matrix based on the simulated adversarial sample data; The cloud server presets multiple types of user policies; The cloud server simulates human-machine confrontation based on the AI ​​strategy defined by the trained global model and various user strategies, and obtains multiple sets of simulated confrontation sample data; The human-machine strategy profit matrix is ​​constructed based on multiple sets of simulated adversarial sample data.

[0010] In one possible design, the cloud server performs a game solution on the human-machine strategy payoff matrix to obtain an equilibrium strategy, including: The cloud server regards the human-machine strategy payoff matrix as the payoff matrix of the game, and uses the minimum-maximum optimization method to solve the game on the human-machine strategy payoff matrix to obtain the equilibrium strategy of the AI ​​and the user; wherein, the human-machine strategy payoff matrix is ​​expressed as: M =[ U kl ]∈R K×L ; in, K is the number of AI strategies, L is the number of user policies, U kl AI Strategy and user policies The expected benefit of the AI ​​during the confrontation; The balancing strategy is: ; Where, π k The first step for AI k The probability of an AI strategy, is the probability simplex corresponding to the AI ​​strategy, argmax() is the maximum value function, and min() is the minimum value function.

[0011] In a second aspect, the present invention provides a human-machine intelligent confrontation method based on a cloud architecture, which is executed by a cloud server, and the method includes: Initialize the parameters of the global model and set the obtained initial global model parameters as the current global model parameters; Broadcasting the current global model parameters to each terminal device so that each terminal device trains a local model based on its own local adversarial sample data and the current global model parameters, and uploading the trained local model parameters to the cloud server; Aggregate the local model parameters sent by each terminal device to obtain new global model parameters; Determine whether a preset number of training rounds has been reached. If so, proceed to the next step. If not, update the current global model parameters to the new global model parameters, and then broadcast the current global model parameters to each terminal device so that each terminal device can train its local model based on its own local adversarial sample data and the new global model parameters. Using the final global model parameters to update the parameters of the global model to obtain a trained global model; Performing a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructing a human-machine strategy benefit matrix based on the simulated adversarial sample data; The human-machine strategy payoff matrix is ​​solved by game theory to obtain an equilibrium strategy.

[0012] In a third aspect, the present invention provides a human-machine intelligent confrontation system based on a cloud architecture, comprising: The model initialization module is used to initialize the parameters of the global model and set the obtained initial global model parameters as the current global model parameters; A model training module, in communication with the model initialization module, configured to broadcast the current global model parameters to each terminal device so that each terminal device trains a local model based on its own local adversarial sample data and the current global model parameters, and upload the trained local model parameters to the cloud server; The model training module is further configured to aggregate the local model parameters sent by each terminal device to obtain new global model parameters, and determine whether a preset training round has been reached. If so, the final global model parameters are used to update the parameters of the global model to obtain a trained global model. If not, the current global model parameters are updated to the new global model parameters, and then the current global model parameters are broadcast to each terminal device so that each terminal device can train the local model according to its own local adversarial sample data and the new global model parameters. The adversarial simulation module is communicatively connected to the model training module and is used to perform human-machine adversarial simulation operations based on the trained global model to obtain simulated adversarial sample data, and to construct a human-machine strategy benefit matrix based on the simulated adversarial sample data; it is also used to perform game solving on the human-machine strategy benefit matrix to obtain an equilibrium strategy.

[0013] In a fourth aspect, the present invention provides an electronic device, comprising: a memory for storing computer program instructions; and A processor is used to execute the computer program instructions to complete the operation of a human-machine intelligent confrontation method based on a cloud architecture as described in any one of the above.

[0014] In a fifth aspect, the present invention provides a computer program product, including a computer program or instructions, which, when executed by a computer, implements a human-computer intelligent confrontation method based on a cloud architecture as described in any one of the above.

[0015] The beneficial effects of the present invention are: The present invention discloses a human-machine intelligent confrontation method and system based on cloud architecture. By combining federated learning with cloud game optimization, it effectively breaks the phenomenon of resource islands between terminals, significantly improves model sharing efficiency and strategy migration capabilities, and realizes rapid optimization and collaborative evolution of intelligent bodies. Specifically, on the one hand, the present invention utilizes multiple terminal devices to distribute and collect user behavior data and generate local adversarial samples, and multiple terminal devices perform strategy training based on their respective local adversarial samples, and the cloud server aggregates and optimizes to form a global model, which significantly improves the generalization ability and flexibility of the model under diverse human behavior data, and solves the problems of single strategy and poor adaptability in the existing technology; on the other hand, after completing the global model training, the cloud server generates a strategy benefit matrix through large-scale human-machine confrontation simulation, and further introduces game optimization methods to solve the equilibrium strategy, thereby effectively improving the confrontation adaptability and intelligent decision-making ability of AI strategies under complex human behavior patterns.

[0016] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of a human-machine intelligent confrontation method based on cloud architecture in an embodiment; Figure 2 This is a module block diagram of a human-machine intelligent confrontation system based on a cloud architecture in an embodiment; Figure 3 It is a module block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0019] Example 1: This embodiment discloses a human-machine intelligent confrontation method based on a cloud architecture, which is executed by a cloud server and multiple terminal devices.

[0020] like Figure 1 As shown, a human-machine intelligent confrontation method based on cloud architecture may include but is not limited to the following steps: S1. Each terminal device collects user behavior data and constructs the user behavior data into local adversarial sample data. It should be understood that in this embodiment, each terminal device pre-collects multiple sets of local adversarial sample data, and each terminal device uses different local adversarial sample data for training when performing different rounds of training on the local model.

[0021] In step S1, the user behavior data is constructed as local adversarial sample data, including: S101. Construct the user behavior data into initial local adversarial sample data; wherein, the initial local adversarial sample data includes state vector, action and immediate reward, i The initial local adversarial sample data of each terminal device is: ; Where, For the i The terminal device is t The state vector of the wheel; For the i The terminal device has a state vector The action to be performed next; For the i Terminal devices are performing actions Immediate rewards after T For the i The number of interactive steps for collecting user behavior data on each terminal device, T is a natural number greater than 1; S102. Normalize the state vector in the initial local adversarial sample data to obtain a normalized state vector; wherein, i The terminal device is tThe normalized state vector of the wheel is: ; Where, μ s and σ s Respectively i The expected value and standard deviation of the state vector of each terminal device at all times; S103. Perform behavioral coding on the actions in the initial local adversarial sample data to obtain coded actions; wherein, i The terminal device is t The encoded action of the wheel is: ; Where, OneHot( ) represents the one-hot encoding function; S104. Based on the normalized state vector and the encoded action, construct the final local adversarial sample data.

[0022] It should be noted that the user behavior data includes state transition, action selection and result feedback, etc., which are used to construct local adversarial sample data; in this embodiment, by standardizing the state vector in the initial local adversarial sample data and performing behavior encoding processing on the actions therein, it is beneficial to improve the versatility of the final local adversarial sample data.

[0023] S2. The cloud server initializes the parameters of the global model and sets the obtained initial global model parameters as the current global model parameters. In this embodiment, the initial global model parameters can be expressed as θ (0) .

[0024] S3. The cloud server broadcasts the current global model parameters to each terminal device.

[0025] S4. Each terminal device trains its local model based on its own local adversarial sample data and the current global model parameters, and uploads the trained local model parameters to the cloud server. It should be noted that the local training method of each terminal device is conducive to better capturing local data features.

[0026] In step S4, the local model of each terminal device adopts the DQN (Deep Q-Network) model; correspondingly, the target optimization function of the terminal device adopts the DQN loss function.

[0027] It should be noted that the local model of each terminal device uses the DQN model and is optimized using the corresponding DQN loss function. This has the following technical advantages: DQN combines the powerful nonlinear fitting capabilities of deep neural networks with the policy optimization mechanism of reinforcement learning, effectively handling high-dimensional and complex state space problems, and is suitable for dynamic environment modeling and decision learning in human-machine competition. Training the action-value function Q(s, a) using the DQN loss function enables accurate estimation of long-term rewards, thereby guiding the agent to adopt a more optimal strategy. Furthermore, within the federated learning framework, each terminal device can train the Q network based on local human-machine competition experience, improving the model's personalized adaptability. The DQN structure is relatively lightweight, facilitating on-terminal deployment and iterative training, balancing policy learning accuracy and computational efficiency.

[0028] S5. The cloud server aggregates the local model parameters sent by each terminal device to obtain new global model parameters.

[0029] In step S5, the cloud server aggregates the local model parameters sent by each terminal device using a weighted average method; correspondingly, in step S5, each terminal device performs the first weighted average on the local model. t After a round of training, the cloud server aggregates the local model parameters sent by each terminal device to obtain the new global model parameters: ; Where, N is the number of terminal devices participating in the training; D i ∣ is the i The amount of local data on each terminal device; D j ∣ is the j The amount of local data on each terminal device; For the i The terminal device performs the first t The local model parameters obtained after the round.

[0030] In this embodiment, the preset training rounds are set as T round, the final global model parameters can be expressed as θ (T) .

[0031] It should be noted that the use of weighted averaging to aggregate local model parameters is conducive to balancing the deviations of different data distributions, and is particularly suitable for distributed scenarios such as mobile devices or the Internet of Things.

[0032] S6. The cloud server determines whether the preset training rounds are reached. If so, it proceeds to the next step. If not, it updates the current global model parameters to the new global model parameters, and then broadcasts the current global model parameters to each terminal device so that each terminal device can train the local model according to its own local adversarial sample data and the new global model parameters.

[0033] S7. The cloud server updates the global model using the final global model parameters to obtain a trained global model. It should be understood that the final global model parameters are the new global model parameters obtained by aggregating the local model parameters obtained by the cloud server after the last iteration of model training on each terminal device, and the trained global model is the global model trained by each terminal device based on the local adversarial sample data.

[0034] It should be noted that, based on the above steps S1 to S7, this embodiment introduces a federated reinforcement learning mechanism to achieve distributed collection and fusion training of user behavior data in a heterogeneous environment with multiple terminal devices. It can cope with changing strategies and environments, significantly improves the generalization ability of intelligent agent strategies under different human behavior patterns and environmental conditions, and avoids the disadvantage of overfitting a single scenario.

[0035] S8. The cloud server performs a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructs a human-machine strategy benefit matrix based on the simulated adversarial sample data.

[0036] In step S8, the cloud server performs a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructs a human-machine strategy benefit matrix based on the simulated adversarial sample data; S801. The cloud server presets multiple types of user policies; specifically, the types of user policies include conservative, aggressive, random, etc., which are not limited here, wherein the first The class user policy can be expressed as .

[0037] S802. The cloud server simulates human-machine confrontation based on the AI ​​(Artificial Intelligence) strategy defined by the trained global model and various user strategies, and obtains multiple sets of simulated confrontation sample data; it should be understood that the AI ​​strategy is the basic strategy function of the AI ​​defined by the trained global model, and the trained global model can output the corresponding action strategy according to the state s, that is, the probability distribution of taking each action under a given environmental state. In this embodiment, k The AI-like strategy is expressed as ; Correspondingly, the simulated adversarial sample data is the interaction results obtained by the cloud server in a virtual simulation environment by simulating the confrontation between AI and various user strategies, such as cumulative rewards, winning rate and income.

[0038] S803. Construct a human-machine strategy benefit matrix based on multiple sets of simulated adversarial sample data. In this embodiment, the human-machine strategy benefit matrix can be expressed as: M =[ U kl ]∈R K×L ; in, K is the number of AI strategies, L is the number of user policies, U kl AI Strategy and user policies The expected gain of the AI ​​during the confrontation.

[0039] It should be noted that in this human-computer confrontation simulation process, it does not rely on the real-time participation or feedback of end users. User strategies are abstracted and simulated through historical confrontation data, behavior modeling or manual settings.

[0040] S9. The cloud server performs game theory to solve the human-machine strategy payoff matrix to obtain an equilibrium strategy.

[0041] In step S9, the cloud server performs a game solution on the human-machine strategy payoff matrix to obtain an equilibrium strategy, including: The cloud server regards the human-machine strategy payoff matrix as the payoff matrix of the game, forms a static finite game model, and uses the minimum-maximum optimization method to solve the human-machine strategy payoff matrix to obtain the equilibrium strategy between the AI ​​and the user; wherein, the equilibrium strategy is: ; Where, π k The first step for AI k The probability of an AI strategy, is the probability simplex corresponding to the AI ​​strategy, argmax() is the maximum value function, and min() is the minimum value function.

[0042] It should be noted that this embodiment further enhances the adversarial robustness and strategic balance of AI by utilizing game theory methods, thereby preventing the strategy from being deliberately induced or single-point breached by human users.

[0043] In addition, in order to make the AI ​​strategy adapt to the behavioral preferences and local environment differences of different human users, in this embodiment, the cloud server also sends the policy model parameters corresponding to the balancing strategy to each terminal device, so that each terminal device can perform online transfer learning of the balancing strategy in the local environment, and use the policy gradient method to optimize the policy parameters, thereby obtaining the local balancing strategy of each terminal device.

[0044] Based on this, each terminal device can perform lightweight adaptation based on local individual user behavior, device resources, etc. while retaining the global policy structure, thereby greatly improving the interaction effect and human-computer matching of each terminal device in local tasks while maintaining the overall confrontationalness.

[0045] This embodiment is based on the centralized processing power and high-concurrency data processing capabilities of cloud computing resources. By constructing a human-computer intelligent confrontation method based on cloud architecture, it can realize the fusion training of multi-terminal confrontation data, effectively improve the strategic adaptability, game coordination ability and human-computer interaction performance of the intelligent body, thereby demonstrating a higher level of intelligence and response efficiency in complex and dynamic actual application scenarios.

[0046] This embodiment combines federated learning with cloud-based game optimization to effectively break down resource silos between terminals, significantly improving model sharing efficiency and strategy migration capabilities, and enabling rapid optimization and collaborative evolution of intelligent agents. Specifically, on the one hand, this embodiment utilizes multiple terminal devices to distribute user behavior data and generate local adversarial samples. Multiple terminal devices then perform strategy training based on their respective local adversarial samples, and the cloud server aggregates and optimizes the model to form a global model. This significantly improves the model's generalization and flexibility under diverse human behavior data, resolving the issues of single strategies and poor adaptability in existing technologies. On the other hand, after completing global model training, the cloud server generates a strategy payoff matrix through large-scale human-machine adversarial simulations, and further introduces game optimization methods to solve equilibrium strategies, effectively improving the AI ​​strategy's adaptability and intelligent decision-making capabilities under complex human behavior patterns. Based on this, this embodiment not only achieves data decentralization for model training, but also enhances the global optimality of strategy deployment through unified cloud-based game optimization. It has excellent scalability, security, and intelligent performance, making it particularly suitable for human-computer interaction systems in various scenarios such as security confrontation, game competition, and intelligent question-and-answering.

[0047] As an implementation method, this embodiment is applied to a military command simulation training platform, in which terminal devices are distributedly deployed in different training camps to collect behavioral data of real human commanders and perform global model training and strategy optimization. The cloud server is used to coordinate the training of the global model and strategy optimization.

[0048] Example 2: This embodiment discloses a human-machine intelligent confrontation method based on a cloud architecture, characterized in that it is executed by a cloud server and includes: Q1. Initialize the parameters of the global model and set the obtained initial global model parameters as the current global model parameters; in this embodiment, the initial global model parameters can be expressed as θ (0) .

[0049] Q2. Broadcast the current global model parameters to each terminal device so that each terminal device can train the local model according to its own local adversarial sample data and the current global model parameters, and upload the local model parameters obtained after training to the cloud server; it should be noted that the way in which each terminal device performs local training is conducive to better capturing local data features.

[0050] Q3. Aggregate the local model parameters sent by each terminal device to obtain new global model parameters; Q4. Determine whether the preset number of training rounds has been reached. If so, proceed to the next step. If not, update the current global model parameters to the new global model parameters, and then broadcast the current global model parameters to each terminal device so that each terminal device can train its local model based on its own local adversarial sample data and the new global model parameters. Q5. Use the final global model parameters to update the parameters of the global model to obtain the trained global model; it should be understood that the final global model parameters, that is, the new global model parameters formed by aggregating the local model parameters obtained after the cloud server performs the last model training iteration on each terminal device, and the trained global model is also the global model completed by each terminal device based on the local adversarial sample data.

[0051] It should be noted that, based on the above steps Q1 to Q5, this embodiment introduces a federated reinforcement learning mechanism to achieve distributed collection and fusion training of user behavior data in a heterogeneous environment with multiple terminal devices. It can cope with changing strategies and environments, significantly improves the generalization ability of intelligent agent strategies under different human behavior patterns and environmental conditions, and avoids the disadvantage of overfitting a single scenario.

[0052] Q6. Perform a human-machine adversarial simulation based on the trained global model to obtain simulated adversarial sample data, and construct a human-machine strategy payoff matrix based on the simulated adversarial sample data. Q7. Perform game theory to solve the human-machine strategy payoff matrix and obtain the equilibrium strategy.

[0053] It should be noted that the working process, working details and technical effects of the human-machine intelligent confrontation method based on cloud architecture provided in this embodiment 2 can be found in embodiment 1 and will not be repeated here.

[0054] Example 3: This embodiment discloses a cloud-based human-machine intelligent confrontation system for implementing the cloud-based human-machine intelligent confrontation method in Example 2; Figure 2 As shown, the human-machine intelligent confrontation system based on cloud architecture includes: The model initialization module is used to initialize the parameters of the global model and set the obtained initial global model parameters as the current global model parameters; A model training module, in communication with the model initialization module, configured to broadcast the current global model parameters to each terminal device so that each terminal device trains a local model based on its own local adversarial sample data and the current global model parameters, and upload the trained local model parameters to the cloud server; The model training module is further configured to aggregate the local model parameters sent by each terminal device to obtain new global model parameters, and determine whether a preset training round has been reached. If so, the final global model parameters are used to update the parameters of the global model to obtain a trained global model. If not, the current global model parameters are updated to the new global model parameters, and then the current global model parameters are broadcast to each terminal device so that each terminal device can train the local model according to its own local adversarial sample data and the new global model parameters. The adversarial simulation module is communicatively connected to the model training module and is used to perform human-machine adversarial simulation operations based on the trained global model to obtain simulated adversarial sample data, and to construct a human-machine strategy benefit matrix based on the simulated adversarial sample data; it is also used to perform game solving on the human-machine strategy benefit matrix to obtain an equilibrium strategy.

[0055] It should be noted that the working process, working details and technical effects of the human-machine intelligent confrontation system based on cloud architecture provided in this embodiment 3 can be found in embodiments 1 and 2 and will not be repeated here.

[0056] Example 4: Based on the second or third embodiment, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a laptop computer, or a desktop computer. The electronic device may be called a user terminal, a portable terminal, a desktop terminal, etc. Figure 3 As shown, the electronic equipment includes: a memory for storing computer program instructions; and A processor is used to execute the computer program instructions to complete the operation of the human-machine intelligent confrontation method based on cloud architecture as in Example 2.

[0057] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen.

[0058] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the human-machine intelligent confrontation method based on the cloud architecture provided in Example 1 of the present application.

[0059] In some embodiments, the terminal may optionally include a communication interface 303 and at least one peripheral device. The processor 301, memory 302, and communication interface 303 may be connected via a bus or signal lines. Each peripheral device may be connected to the communication interface 303 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 304, a display screen 305, and a power supply 306.

[0060] The communication interface 303 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board. In other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0061] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices via electromagnetic signals.

[0062] The display screen 305 is used to display a UI (User Interface). The UI may include any combination of graphics, text, icons, and videos.

[0063] The power supply 306 is used to supply power to various components in the electronic device.

[0064] Example 5: Based on any one of Embodiments 2 to 4, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements the human-machine intelligent confrontation method based on a cloud architecture as described in Embodiment 2. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0065] Obviously, those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0066] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art will appreciate that modifications may be made to the technical solutions described in the above embodiments, or that some of the technical features may be replaced with equivalents. Such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A human-machine intelligent confrontation method based on cloud architecture, characterized in that: Executed by a cloud server and multiple terminal devices, the method includes: Each terminal device collects user behavior data and constructs the user behavior data into local adversarial sample data; The cloud server initializes the parameters of the global model and sets the obtained initial global model parameters as the current global model parameters; The cloud server broadcasts the current global model parameters to each terminal device; Each terminal device trains a local model based on its own local adversarial sample data and the current global model parameters, and uploads the trained local model parameters to the cloud server; The cloud server aggregates the local model parameters sent by each terminal device to obtain new global model parameters; The cloud server determines whether a preset number of training rounds has been reached. If so, it proceeds to the next step. If not, it updates the current global model parameters to the new global model parameters, and then broadcasts the current global model parameters to each terminal device so that each terminal device trains its local model based on its own local adversarial sample data and the new global model parameters. The cloud server uses the final global model parameters to update the parameters of the global model to obtain a trained global model; The cloud server performs a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructs a human-machine strategy benefit matrix based on the simulated adversarial sample data; The cloud server performs game theory to solve the human-machine strategy payoff matrix to obtain an equilibrium strategy.

2. The human-machine intelligent confrontation method based on cloud architecture according to claim 1 is characterized in that: Constructing the user behavior data into local adversarial sample data includes: Constructing the user behavior data as initial local adversarial sample data; Normalizing the state vector in the initial local adversarial sample data to obtain a normalized state vector; Performing behavior encoding processing on the actions in the initial local adversarial sample data to obtain encoded actions; Based on the normalized state vector and the encoded action, final local adversarial sample data is constructed.

3. The human-machine intelligent confrontation method based on cloud architecture according to claim 1 is characterized in that: The local model of each terminal device adopts the DQN model; correspondingly, the target optimization function of the terminal device adopts the DQN loss function.

4. The human-machine intelligent confrontation method based on cloud architecture according to claim 1 is characterized in that: The cloud server aggregates the local model parameters sent by each terminal device using a weighted average method; correspondingly, each terminal device performs the first t After a round of training, the cloud server aggregates the local model parameters sent by each terminal device to obtain the new global model parameters: ； Where, N is the number of terminal devices participating in the training; D i ∣ is the i The amount of local data on each terminal device; D j ∣ is the j The amount of local data on each terminal device; For the i The terminal device performs the first t The local model parameters obtained after the round.

5. The human-machine intelligent confrontation method based on cloud architecture according to claim 1 is characterized in that: The cloud server performs a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructs a human-machine strategy benefit matrix based on the simulated adversarial sample data; The cloud server presets multiple types of user policies; The cloud server simulates human-machine confrontation based on the AI ​​strategy defined by the trained global model and various user strategies, and obtains multiple sets of simulated confrontation sample data; The human-machine strategy profit matrix is ​​constructed based on multiple sets of simulated adversarial sample data.

6. The human-machine intelligent confrontation method based on cloud architecture according to claim 1 is characterized in that: The cloud server performs game solving on the human-machine strategy payoff matrix to obtain an equilibrium strategy, including: The cloud server regards the human-machine strategy payoff matrix as the payoff matrix of the game, and uses the minimum-maximum optimization method to solve the game on the human-machine strategy payoff matrix to obtain the equilibrium strategy of the AI ​​and the user; wherein, the human-machine strategy payoff matrix is ​​expressed as: M =[ U kl ]∈R K×L ； in, K is the number of AI strategies, L is the number of user policies, U kl AI Strategy and user policies The expected benefit of the AI ​​during the confrontation; The balancing strategy is: ； Where, π k The first step for AI k The probability of an AI strategy, is the probability simplex corresponding to the AI ​​strategy, argmax() is the maximum value function, and min() is the minimum value function.

7. A human-machine intelligent confrontation method based on cloud architecture, characterized in that: Executed by a cloud server, the method includes: Initialize the parameters of the global model and set the obtained initial global model parameters as the current global model parameters; Broadcasting the current global model parameters to each terminal device so that each terminal device trains a local model based on its own local adversarial sample data and the current global model parameters, and uploading the trained local model parameters to the cloud server; Aggregate the local model parameters sent by each terminal device to obtain new global model parameters; Determine whether a preset number of training rounds has been reached. If so, proceed to the next step. If not, update the current global model parameters to the new global model parameters, and then broadcast the current global model parameters to each terminal device so that each terminal device can train its local model based on its own local adversarial sample data and the new global model parameters. Using the final global model parameters to update the parameters of the global model to obtain a trained global model; Performing a human-machine adversarial simulation operation based on the trained global model to obtain simulated adversarial sample data, and constructing a human-machine strategy benefit matrix based on the simulated adversarial sample data; The human-machine strategy payoff matrix is ​​solved by game theory to obtain an equilibrium strategy.

8. A human-machine intelligent confrontation system based on cloud architecture, characterized in that: include: The model initialization module is used to initialize the parameters of the global model and set the obtained initial global model parameters as the current global model parameters; A model training module, in communication with the model initialization module, configured to broadcast the current global model parameters to each terminal device so that each terminal device trains a local model based on its own local adversarial sample data and the current global model parameters, and upload the trained local model parameters to the cloud server; The model training module is further configured to aggregate the local model parameters sent by each terminal device to obtain new global model parameters, and determine whether a preset training round has been reached. If so, the final global model parameters are used to update the parameters of the global model to obtain a trained global model. If not, the current global model parameters are updated to the new global model parameters, and then the current global model parameters are broadcast to each terminal device so that each terminal device can train the local model according to its own local adversarial sample data and the new global model parameters. The adversarial simulation module is communicatively connected to the model training module and is used to perform human-machine adversarial simulation operations based on the trained global model to obtain simulated adversarial sample data, and to construct a human-machine strategy benefit matrix based on the simulated adversarial sample data; it is also used to perform game solving on the human-machine strategy benefit matrix to obtain an equilibrium strategy.

9. An electronic device, characterized in that: include: a memory for storing computer program instructions; as well as, A processor is used to execute the computer program instructions to complete the operation of the human-machine intelligent confrontation method based on cloud architecture as described in claim 7.

10. A computer program product comprising a computer program or instructions, characterized in that When executed by a computer, the computer program or the instruction implements the human-machine intelligent confrontation method based on cloud architecture as described in claim 7.

Images