Game model-based AI man-machine collaborative decision optimization method and system

Abstract

The invention provides an AI man-machine collaborative decision optimization method and system based on a game model, and the method comprises the steps: constructing a game model comprising AI participants and human participants, and designing a dual-utility function and a joint utility function to balance the interests of the two parties; a deep neural network is adopted to realize an AI strategy function, supervised learning training is carried out based on human expert data, temperature parameter adjustment and a strategy backspacing mechanism are introduced to execute online game training, and dynamic adaptability adjustment is carried out based on task type similarity. The technical problems of man-machine utility imbalance, unstable training, poor generalization ability and the like in an existing AI man-machine cooperation system are solved, balance optimization of the AI decision speed and the human intervention cost is achieved, the rapid adaptive capacity of the model in a new task scene is improved, the stability of the training process is guaranteed, and the training efficiency is improved. And the overall efficiency of man-machine collaborative decision making is obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an AI human-machine collaborative decision-making optimization method and system based on a game model. Background Art

[0002] With the rapid development of artificial intelligence technology, the application scenarios of AI and human collaborative decision-making are increasing. In the fields of financial risk control, medical diagnosis, intelligent manufacturing, smart cities, etc., AI systems need to continuously interact with human experts to jointly complete complex decision-making tasks. This collaborative decision-making model can not only give full play to the advantages of AI systems in efficient data processing and rapid reasoning, but also combine the experience, judgment and creative thinking of human experts to achieve complementary advantages.

[0003] At present, human-machine collaborative decision-making systems mainly use supervised learning, deep reinforcement learning, multi-agent learning and other methods to build AI decision-making models, combining human feedback and expert knowledge to optimize model performance. Some systems also introduce active learning and online learning mechanisms to adapt to dynamically changing decision-making environments. These methods have shown good potential in practical applications, but still face many challenges.

[0004] However, the prior art still has the following outstanding problems:

[0005] First, most systems only focus on the decision-making performance of AI models, ignoring the cost of human intervention, and have the problem of single utility evaluation. The system often relies too much on human intervention to improve decision-making accuracy, resulting in increased fatigue of human experts and low collaboration efficiency. Especially in scenarios that require rapid response, frequent human intervention will significantly prolong the decision-making time and affect the actual application effect of the system.

[0006] Secondly, existing training methods generally lack effective stability guarantee mechanisms. During long-term training, model performance is prone to fluctuation or degradation, especially when dealing with complex and changeable decision-making tasks, the instability of training is more prominent. Although some methods use simple checkpoint saving mechanisms, they lack systematic policy evaluation and fallback solutions.

[0007] Third, the ability to adapt to new task types is insufficient. Most existing systems use fixed model structures and training methods. When faced with new task types, a lot of retraining is required to adapt, which cannot meet the needs of quickly responding to new scenarios in practical applications. The lack of effective modeling and rapid adaptation mechanisms for task similarity seriously limits the generalization ability of the system.

[0008] These technical problems seriously restrict the actual application effect of AI human-machine collaborative decision-making systems. In practice, the trade-off between decision-making efficiency and labor costs, the guarantee of training stability, and the rapid adaptation to new tasks need to be solved urgently. Therefore, there is an urgent need for a human-machine collaborative decision-making optimization method that can balance AI decision-making efficiency and human intervention costs, ensure training stability, have the ability to adapt to tasks quickly, and have good practicality. Summary of the invention

[0009] In order to solve the above technical problems, the present invention proposes an AI human-machine collaborative decision-making optimization method and system based on a game model. By designing a dual utility function to balance the AI ​​decision-making efficiency and the cost of human intervention, a dynamic adaptation mechanism is introduced to improve the generalization ability of the model, and a strategy fallback mechanism is adopted to ensure training stability, so as to achieve efficient human-machine collaborative decision-making.

[0010] The technical solution adopted by the present invention is as follows:

[0011] An AI human-machine collaborative decision optimization method based on a game model comprises the following steps:

[0012] Constructing a human-machine game model, the human-machine game model includes an AI participant and a human participant, constructing a task state and a historical interaction record into a state space, wherein the task state T includes input feature information and type semantic information of the task, defining the action space of the AI ​​participant and the action space of the human participant, and constructing a bilateral utility function and a joint utility function for realizing collaborative decision-making of the AI ​​participant and the human participant;

[0013] A deep neural network is used to implement the strategy function of the AI ​​participant. The strategy function is trained by supervised learning based on historical decision data of human experts, the state space is mapped to the behavior probability distribution in the action space of the AI ​​participant, and verification and evaluation are performed to ensure that the strategy function has reliable decision-making capabilities, so as to construct the initial strategy of the AI ​​participant.

[0014] Execute online game training, which includes using temperature parameters to adjust the probability distribution of the neural network output corresponding to the policy function and sampling decision-making behaviors, recording interactive data including states, behaviors, and utility values ​​calculated based on the utility functions of the AI ​​participants, constructing a value evaluation network and using a policy gradient method to update the parameters of the neural network, ensuring training stability through training process monitoring and a policy fallback mechanism, and optimizing the decision-making effect of the AI ​​participants;

[0015] Based on the type semantic information in the task state T, similar tasks are determined and the policy network is dynamically adaptively adjusted, including saving checkpoint data and performing fast adaptation training, wherein the checkpoint data includes the task type identifier, the utility value calculated by the joint utility function and the parameters of the policy network, so as to improve the adaptability of the AI ​​participant in new task scenarios.

[0016] Furthermore, the human-computer game model is constructed as follows:

[0017] The state space is represented by a binary pair S=(T, H), where the task state T includes a digital input feature vector, a semantic embedding vector of the task type, a normalized task completion progress value, and a normalized time limit value, and the historical interaction record H includes a discretized decision timestamp sequence, a binary identification sequence of the decision subject, and a vector sequence of the decision content;

[0018] The action space A1 of the AI ​​participant includes an m-dimensional decision action vector, an execution action identifier with a value of {0, 1}, and a confirmation request identifier with a value of {0, 1}, and the action space A2 of the human participant includes a response action identifier with a value of {0, 1, 2, 3}, which respectively correspond to accepting suggestions, modifying suggestions, making decisions directly, and terminating tasks;

[0019] The utility function of the AI ​​participant is:

[0020]

[0021] Among them, C is the task completion mark, when the task time does not exceed the maximum time limit t max The value is 1 when the AI ​​participant is in the task, otherwise it is 0. P is the task goal achievement score, which is calculated by dividing the number of task goals achieved by the total number of task goals. Its value range is [0, 1]. t is the decision time of the AI ​​participant. The maximum time limit t max It is used to limit the maximum time to complete the task. All variables are normalized using the maximum and minimum method so that their value range is limited to the interval [0, 1];

[0022] The utility function of the human participant is:

[0023]

[0024] Where n is the number of interventions by the human participant, n max is the preset maximum number of interventions, all variables are normalized using the maximum and minimum method so that their value range is limited to the interval [0, 1]; the joint utility function:

[0025] U=min(U1,U2)

[0026] As the optimization goal of the strategy network training, the utility of the AI ​​participants and the human participants can be balanced.

[0027] Furthermore, the deep neural network and supervised learning process are implemented as follows:

[0028] The deep neural network includes three fully connected layers, which receive processed state space inputs, wherein the task state T maintains the complete input history interaction record H and only retains the decision timestamp sequence and decision content sequence of the corresponding moment identified as a human participant in the binary identification sequence of the decision subject. The number of neurons in each layer is set to decrease layer by layer according to the dimension of the state space S and the dimension of the AI ​​participant action space A1 according to a preset contraction ratio. Each layer uses a ReLU activation function to achieve nonlinear transformation. The last layer uses a Softmax function to map the output to a probability distribution, which is used to characterize the selection probability of different decision behaviors of the AI ​​participant in the action space A1.

[0029] Extract expert decision data from the historical decision data of the human participants, wherein the screening of the expert decision data satisfies that the task completion time is less than a preset time threshold, the decision result meets the quality assessment standard, and has a high historical decision success rate, and the decision data that meets the conditions are constructed as a data set D = {(T, H, a)}, wherein a is the expert decision behavior identifier corresponding to the action space A2 of the human participants;

[0030] Dividing the expert decision data into a training set and a validation set according to a preset ratio, wherein the training set is used for parameter optimization of the strategy function, and the validation set is used for evaluating the generalization performance of the strategy function;

[0031] The cross entropy loss function is used to optimize the parameters of the strategy function, and the decision accuracy is evaluated on the validation set. When the accuracy reaches a preset threshold, the training is stopped, so that the AI ​​participant can learn the decision-making mode of human experts and provide a good initial strategy for subsequent online game training.

[0032] Furthermore, the online game training is performed as follows:

[0033] The output probability distribution of the policy network is adjusted by using a temperature parameter τ that is dynamically adjusted based on training rounds, wherein the temperature parameter decreases as the number of training rounds increases, and decision-making behavior sampling is performed based on the adjusted probability distribution to balance policy exploration and utilization;

[0034] Collect interaction data(s t , a t , r t ), where s tis the state representation of the state space at time t, a t is the decision behavior obtained by the AI ​​participant based on the sampling of the strategy network, r t is the utility value calculated based on the AI ​​participant utility function U1;

[0035] Construct a value network V(s) with the same input layer structure as the policy network. The value network adopts a three-layer fully connected structure, and the last layer outputs a single value to evaluate the state value. Update the parameter θ of the policy network based on the following policy gradient formula:

[0036]

[0037] Among them, α is the learning rate used to control the parameter update step size, and γ is the discount factor used to balance immediate utility and long-term utility;

[0038] Calculate and record the average utility value during the training process, where the average utility value includes the single-step utility mean calculated based on the AI ​​participant utility function and the synergistic utility mean calculated based on the joint utility function. When the single-step utility mean of the new strategy continues to be lower than a preset ratio threshold of the average utility value after multiple consecutive rounds of training, execute strategy rollback. The strategy rollback includes loading the historical parameters with the highest synergistic utility mean, resetting the temperature parameters, reducing the learning rate, and keeping the shrinkage ratio unchanged to maintain the stability of the network structure, so as to ensure the stability of the training and the continuous improvement of the utility, and finally obtain the optimized strategy network parameters;

[0039] The optimized policy network parameters are periodically saved as checkpoint data for the dynamic adaptive adjustment. At the same time, the optimized policy network parameters are applied to the actual human-machine collaborative decision-making process, and decision suggestions are output based on the input state information for reference by the human participants.

[0040] Furthermore, the dynamic adaptive adjustment is achieved as follows:

[0041] The semantic embedding vector of the task type in the task state T is used to determine similar tasks by calculating the cosine similarity, so as to identify historical tasks similar to the current task. When there are historical tasks with a similarity higher than a preset threshold, the checkpoint with the highest similarity is selected for parameter loading. Otherwise, the current parameters of the policy network are maintained.

[0042] The checkpoint data is saved at a preset time interval, and the checkpoint data includes a semantic embedding vector of the task type, a utility value calculated based on the joint utility function, and parameters of the policy network; the rapid adaptive training includes calculating the similarity between the current task and the historical tasks in the checkpoint data, selecting the historical checkpoint with the highest similarity for parameter loading, and collecting the interaction data (s t , a t , r t ) trains the policy network, wherein the number of training rounds is inversely proportional to the task similarity, for achieving rapid adaptation while maintaining the stability of the policy network.

[0043] The present invention also discloses an AI human-machine collaborative decision optimization system based on a game model, comprising: a training data management unit for storing and managing the historical decision data of the human experts and the interaction data in the online game training process;

[0044] A policy network unit, used to construct the deep neural network and output a decision suggestion through the policy function based on the input state information;

[0045] A training execution unit, used to implement the supervised learning training and the online game training, and perform parameter optimization of the strategy network;

[0046] A utility evaluation unit, configured to calculate the utility values ​​of the AI ​​participant utility function and the joint utility function, and to guide the optimization process of the training execution unit based on the utility evaluation result;

[0047] A task adaptation unit, used to perform the similar task determination and the rapid adaptation training, so as to achieve rapid adaptation of the policy network to new tasks;

[0048] The training data management unit provides training data to the training execution unit, the evaluation result of the utility evaluation unit is used to guide the parameter optimization process of the training execution unit, and the task adaptation unit selects the optimal checkpoint for rapid adaptation training based on the evaluation result of the utility evaluation unit.

[0049] The beneficial effects of the present invention are as follows:

[0050] (1) By designing a dual utility function and a joint utility function, the AI ​​decision-making speed and the cost of human intervention are effectively balanced. The AI ​​utility function takes into account the completion time, and the human utility function takes into account the number of interventions. The joint utility function is used to achieve a balanced improvement in the utility of both parties. This design enables the AI ​​system to consider reducing the disturbance to humans while pursuing decision-making efficiency, avoiding the problem of over-reliance on human intervention.

[0051] (2) The stability of the training process is ensured by adopting dynamic temperature parameter adjustment and policy fallback mechanism. The probability distribution of the policy network output is dynamically adjusted through the temperature parameter, and a complete policy fallback mechanism is designed based on the utility value, including loading the optimal historical parameters, resetting the temperature parameters, and adjusting the learning rate. This mechanism effectively prevents the performance degradation problem in long-term training and ensures the monotonic improvement of the model performance.

[0052] (3) A dynamic adaptation mechanism based on task type similarity is introduced to enable the model to quickly adapt to new tasks. Similar tasks are determined by calculating the cosine similarity of task types, and a rapid adaptation strategy is adopted in which the number of training rounds is inversely proportional to the task similarity, so that the model can quickly adapt to new task types while maintaining the stability of historical experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of the AI ​​human-machine collaborative decision-making optimization method.

[0054] Figure 2 This is a flowchart for online game training.

[0055] Figure 3 Adjust the flow chart for dynamic adaptability.

[0056] Figure 4 Optimizing system architecture for AI human-machine collaborative decision-making. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0058] like Figure 1 As shown, the main steps of the method provided by the present invention are as follows:

[0059] (1) Constructing a human-machine game model: Constructing the task status and historical interaction records into a state space, defining the action space of AI participants and human participants, and constructing a utility function to achieve collaborative decision-making;

[0060] (2) Implement and train AI policy functions: Use deep neural networks to implement the policy functions of AI participants, and train them through supervised learning based on historical decision data of human experts;

[0061] (3) Perform online game training: optimize the decision-making effect of AI participants through temperature parameter adjustment, interaction data collection, policy gradient update and fallback mechanism;

[0062] (4) Perform dynamic adaptive adjustments: Determine similar tasks based on task types and improve the adaptability of AI participants through checkpoint data and rapid adaptation training.

[0063] As a preferred embodiment of the present invention, in step (1), the construction process of the human-computer game model is as follows: First, construct a state space. The present invention uses a tuple S = (T, H) to represent the state space, where T is the task state and H is the historical interaction record. Specifically, the task state T contains an input feature vector in digital form, a semantic embedding vector of the task type, a normalized task completion progress value, and a normalized time limit value. The historical interaction record H contains a discretized decision timestamp sequence, a binary identification sequence of the decision subject, and a vector sequence of the decision content. Preferably, the dimension of the input feature vector is 64 dimensions, and the dimension of the semantic embedding vector of the task type is 32 dimensions.

[0064] In order to better understand the construction method of the above state space, the software development task scenario is used as an example. In this scenario, the task state T contains multiple features related to software development: the input feature vector reflects the various indicators of the code, such as the number of lines of code, cyclomatic complexity, number of functions, etc. These indicators are converted into normalized values ​​between 0 and 1; the semantic embedding vector of the task type, which converts the task description into a 32-dimensional semantic vector through the pre-trained language model. For example, "fix the login failure bug of the user authentication module" may be encoded as [-0.15, 0.42, ..., 0.28], and "develop new data visualization functions" may be encoded as [-0.08, 0.35, ..., 0.45]. This representation method can capture the semantic association between task types, and similar task types will have similar vector representations; the task progress is directly represented by a normalized value, such as 0.5 means 50% completed; the time limit also uses a normalized value, such as 0.8 means 80% of the required completion time is left. This design ensures that all information is stored in a standardized numerical form, while maintaining semantic association information between task types, making it easier for the system to process and learn. For the historical interaction record H, the system saves all interaction information in chronological order, including converting the time point of each decision into a timestamp, using 0 and 1 to mark whether the decision-making subject is AI or human, and converting the specific content of each decision (such as code modification suggestions) into a standard numerical vector form for storage.

[0065] Secondly, define the action space. The action space A1 of the AI ​​participant in the present invention includes an m-dimensional decision action vector, an execution action identifier with a value of {0, 1}, and a confirmation request identifier with a value of {0, 1}. The action space A2 of the human participant includes a response action identifier with a value of {0, 1, 2, 3}, which corresponds to accepting suggestions, modifying suggestions, making direct decisions, and terminating tasks, respectively. Preferably, the dimension m of the decision action vector is set to 16 dimensions. To further illustrate the specific application of the above action space, the software development task is still taken as an example. In this scenario, in the action space A1 of the AI ​​participant, the 16-dimensional decision action vector can encode specific code modification suggestions, the execution action identifier 1 indicates direct execution of the modification, 0 indicates confirmation is required, the confirmation request identifier 1 indicates requesting human confirmation, and 0 indicates no confirmation is required. For the response action of the human participant, 0 indicates full acceptance of the modification suggestion of AI, 1 indicates adoption after adjustment of the suggestion, 2 indicates ignoring the suggestion and modifying it by itself, and 3 indicates termination of the current task.

[0066] Finally, a utility function is constructed. The present invention designs utility functions for AI participants and human participants respectively, and constructs a joint utility function based on the utility functions of both parties. Among them, the utility function of the AI ​​participant is:

[0067]

[0068] Among them, C is the task completion mark, when the task time does not exceed the maximum time limit t max The value is 1 when the task is completed, otherwise it is 0; P is the task goal achievement score, which is calculated by dividing the number of task goals achieved by the total number of task goals, and its value range is [0, 1]; t is the decision time of the AI ​​participant; the maximum time limit t max It is used to limit the maximum time to complete the task. All variables are normalized using the maximum and minimum method so that their value range is limited to the interval [0, 1].

[0069] The utility function of human participants is:

[0070]

[0071] Where n is the number of human participant interventions, n max is the preset maximum number of interventions. Similarly, all variables are normalized using the maximum and minimum method so that their value range is limited to the interval [0, 1].

[0072] Based on the above two utility functions, the present invention constructs a joint utility function:

[0073] U=min(U1,U2)

[0074] The joint utility function is used as the optimization target of the policy network training to achieve a balanced improvement in the utility of AI participants and human participants. Preferably, the preset maximum number of interventions n max Set to 20 times, the maximum time limit t max Set to 60 minutes. To illustrate the calculation process of the utility function, we will now use an actual collaborative development task as an example. In this task, if the task is completed within 40 minutes (less than t max =60 minutes), then C = 1; 7 out of 8 goals were achieved, then P = 0.875; AI decision-making took 15 minutes, then Substituting the AI ​​participant utility function, we get U1 to be 0.625. At the same time, if humans intervene 5 times (n=5, n max =20), then the utility U2 of the human participant is 0.625, and the final joint utility U=min(0.625, 0.625)=0.625.

[0075] Through the design and example illustration of the above state space, action space and utility function, the present invention constructs a complete human-computer game model with practical application value, which provides a solid foundation for the subsequent implementation and training of strategy functions.

[0076] As a preferred embodiment of the present invention, in step (2), the implementation and training process of the AI ​​strategy function includes: first, constructing a deep neural network. The present invention adopts a three-layer fully connected layer structure to implement the strategy function. The network receives processed state space input, wherein the task state T maintains the complete input, and the historical interaction record H only retains the decision timestamp sequence and decision content sequence of the corresponding moment identified as a human participant in the binary identification sequence of the decision subject. The number of neurons in each layer is set to decrease layer by layer according to the dimension of the state space S and the dimension of the AI ​​participant action space A1 according to a preset shrinkage ratio. Preferably, the shrinkage ratio is set to 0.7. Each layer uses the ReLU activation function to implement nonlinear transformation, and the last layer uses the Softmax function to map the output to a probability distribution, which is used to characterize the selection probability of different decision-making behaviors of the AI ​​participant in the action space A1.

[0077] Secondly, the training data is processed. The present invention extracts expert decision data from the historical decision data of human participants, wherein the screening of expert decision data must meet three conditions: the task completion time is less than the preset time threshold, the decision result meets the quality assessment standard, and has a high historical decision success rate. Preferably, the preset time threshold is 80% of the standard completion time, the quality assessment standard score is not less than 85 points (out of 100 points), and the historical decision success rate is required to be not less than 90%. The decision data that meets the conditions is constructed as a data set D = {(T, H, a)}, where a is the expert decision behavior identifier corresponding to the action space A2 of the human participant. Finally, supervised learning training is performed. The present invention divides the expert decision data into a training set and a validation set according to a preset ratio, wherein the training set is used for parameter optimization of the strategy function, and the validation set is used to evaluate the generalization performance of the strategy function. Preferably, the division ratio of the training set to the validation set is 8:2. The cross entropy loss function is used to optimize the parameters of the strategy function, and the decision accuracy is evaluated on the validation set. When the accuracy reaches the preset threshold, the training is stopped, so that the AI ​​participants learn the decision-making mode of human experts and provide a good initial strategy for subsequent online game training. Preferably, the preset accuracy threshold is 85%.

[0078] In order to better understand the above implementation process, the specific operation process of the neural network is now explained. The input state space dimension is 100, and the preset shrinkage ratio of 0.7 is used. The number of neurons in the three fully connected layers is 100, 70 and 49 respectively. For an input state vector [0.1, 0.2, ..., 0.8], the first layer obtains a 70-dimensional intermediate feature vector through linear transformation and ReLU activation function processing, and the second layer continues to reduce the dimension to obtain a 49-dimensional feature vector. Finally, the feature is mapped to a probability distribution through the Softmax function, such as [0.15, 0.25, 0.60], which respectively represents the probabilities of three possible decision behaviors in the action space A1 of the AI ​​participant. In the training stage, if the corresponding expert decision label is [0, 0, 1], the loss value calculated by the cross entropy loss function is 0.51. The network adjusts the parameters accordingly so that the output distribution gradually approaches the standard distribution of expert decisions.

[0079] Through the above network structure design, data processing and supervised learning training, the present invention realizes an AI strategy function that can imitate the decision-making mode of human experts, laying a foundation for subsequent online game training. As a preferred embodiment of the present invention, in step (3), refer to Figure 2 ,The execution process of online game training includes:

[0080] First, the temperature parameter is adjusted. The present invention uses a temperature parameter τ dynamically adjusted based on the training rounds to adjust the output probability distribution of the policy network, where the temperature parameter decreases as the training rounds increase, and the decision behavior is sampled based on the adjusted probability distribution to balance the strategy exploration and utilization. Preferably, the initial value of the temperature parameter τ is set to 1.0, and the original value is multiplied by 0.95 every 100 training rounds until the minimum value is 0.1.

[0081] Secondly, collect interaction data. The present invention collects interaction data (s t , a t , r t ), where s t is the state representation of the state space at time t, a t is the decision behavior obtained by the AI ​​participant based on the sampling of the strategy network, r y is the utility value calculated based on the AI ​​participant utility function U1. Preferably, 100 sets of interaction data are collected in each round of training for parameter updating.

[0082] Next, perform policy gradient update. The present invention constructs a value network V(s) with the same input layer structure as the policy network. The value network adopts a three-layer fully connected structure, and the last layer outputs a single value to evaluate the state value. The parameter θ of the policy network is updated based on the following policy gradient formula:

[0083]

[0084] Among them, α is the learning rate used to control the parameter update step size, and γ is the discount factor used to balance immediate utility and long-term utility. Preferably, the initial value of the learning rate α is set to 0.001, and the discount factor γ is set to 0.99. Finally, a strategy fallback mechanism is implemented. The present invention calculates and records the average utility value during the training process, wherein the average utility value includes the single-step utility mean calculated based on the AI ​​participant utility function and the synergistic utility mean calculated based on the joint utility function. When the single-step utility mean of the new strategy continues to be lower than the preset ratio threshold of the average utility value after multiple consecutive rounds of training, a strategy fallback is performed. The strategy fallback includes loading historical parameters with the highest synergistic utility mean, resetting temperature parameters, reducing the learning rate, and keeping the contraction ratio unchanged to maintain the stability of the network structure. Preferably, the preset ratio threshold is 0.9, and fallback is triggered if it is lower than this threshold for 5 consecutive rounds. After fallback, the learning rate is multiplied by 0.5 on the basis of the original value.

[0085] In order to better understand the above training process, we will now explain it with a specific parameter update process. At a certain time t, the input state s t = [0.3, 0.5, 0.7], the policy network outputs decision behavior a after adjusting the temperature parameter τ = 0.8 tThe probability distribution is [0.2, 0.3, 0.5], and the sampling is a t =2. After executing this decision, the utility value r is obtained t =0.6, the value network's evaluation of the current state V(s t )=0.55, the evaluation of the next state V(s t+1 )=0.65. Substitute into the policy gradient formula, where γ=0.99, then [r t +γ·V(s t+1 )-V(s t )] is 0.6935. Based on this result and the learning rate α=0.001, the parameter update amount is calculated to complete a parameter iteration. If the average single-step utility after 5 consecutive rounds of training is 0.3, which is lower than the 90% threshold of the historical average utility of 0.5, the policy fallback mechanism is triggered.

[0086] Through the design of the above-mentioned temperature parameter adjustment, interactive data collection, strategy gradient update and fallback mechanism, the present invention realizes a stable and efficient online game training process, and effectively improves the decision-making effect of AI participants. Figure 3 In step (4), the implementation process of dynamic adaptive adjustment includes:

[0087] First, similar tasks are judged. The present invention uses cosine similarity calculation to determine similar tasks using the semantic embedding vector of the task type in the task state T, which is used to identify historical tasks that are similar to the current task. Preferably, the similarity threshold is set to 0.8. When the calculated cosine similarity is greater than the threshold, the two tasks are considered to be similar. At this time, the historical checkpoint with the highest similarity is selected for parameter loading; when the similarity of all historical tasks is lower than the threshold, the current parameters of the policy network are kept unchanged to avoid performance interference caused by the introduction of irrelevant historical experience.

[0088] Secondly, manage the checkpoint data. The present invention saves the checkpoint data at a preset time interval. The checkpoint data includes the semantic embedding vector of the task type, the utility value calculated based on the joint utility function, and the parameters of the policy network. Preferably, the checkpoint data is saved once every 100 rounds of training, and a maximum of nearly 10 checkpoints are saved.

[0089] Finally, the rapid adaptation training is performed. The rapid adaptation training of the present invention includes calculating the similarity between the current task and the historical tasks in the checkpoint data, selecting the historical checkpoint with the highest similarity for parameter loading, and collecting the interaction data (s t , a t , r t) to train the policy network, wherein the number of training rounds is inversely proportional to the task similarity, for achieving rapid adaptation while maintaining the stability of the policy network. Preferably, the base training rounds are set to 100 rounds, and the actual training rounds are the base rounds multiplied by (1-similarity).

[0090] In order to better understand the above adjustment process, the specific calculation process is now used for explanation. Assuming that the current task is in order to better understand the above adjustment process, the specific calculation process is now used for explanation. Assume that the semantic embedding vector of the current task "Optimize database query performance" is [-0.13, 0.42, 0.25, -0.18, ..., 0.31], there are three historical tasks and their corresponding semantic embedding vectors in the checkpoint database: "Improve interface response time" is [-0.15, 0.45, 0.28, -0.16, ..., 0.33], "Add user management function" is [-0.32, 0.15, -0.21, 0.35, ..., -0.12], and "Fix user authentication bug" is [0.28, -0.19, -0.25, 0.12, ..., -0.28]. The cosine similarity calculation shows that the similarity of the first historical task is 0.92 (both are performance optimization tasks), the similarity of the second historical task is 0.25 (there are large differences between function development and performance optimization), and the similarity of the third historical task is 0.18 (the characteristics of bug repair and performance optimization tasks are significantly different). The first historical checkpoint with the highest similarity (similarity 0.92) is selected for parameter loading, and the actual number of training rounds is calculated as: 100×(1-0.92)=8 rounds. This means that only 8 rounds of fast adaptation training are needed to fine-tune the policy network parameters to adapt to the characteristics of the new task, reflecting the improvement in training efficiency brought about by task similarity. If the similarity of all historical tasks is lower than the threshold of 0.8, the current parameters of the policy network are kept unchanged, and only necessary training optimization is performed based on the current parameters. This mechanism ensures full utilization of similar task experience and avoids interference of irrelevant task parameters on the policy network.

[0091] Through the above-mentioned designs of similarity calculation, checkpoint management and fast adaptive training, the present invention realizes the fast adaptation of the policy network to new tasks and improves the versatility and efficiency of the system.

[0092] like Figure 4 As shown, the present invention also provides a specific implementation scheme of an AI human-machine collaborative decision-making optimization system based on a game model, and the system includes the following functional modules:

[0093] Training data management unit: This unit is used to store and manage the historical decision data of human experts and the interaction data during the online game training process. The historical decision data includes the expert's decision record and the corresponding task status information, and the interaction data includes information such as status, behavior and utility value.

[0094] Policy network unit: This unit is used to build a deep neural network and output decision suggestions through a policy function based on the input state information. This unit adopts a three-layer fully connected layer structure and realizes the processing of state information and the output of decision probability through the ReLU activation function and the Softmax function.

[0095] Training execution unit: This unit is used to implement supervised learning training and online game training, and perform parameter optimization of the policy network. This unit performs supervised learning through the cross entropy loss function, performs online game training through the policy gradient method, and implements mechanisms such as temperature parameter adjustment and policy fallback.

[0096] Utility evaluation unit: This unit is used to calculate the utility value of the AI ​​participant utility function and the joint utility function, and guide the optimization process of the training execution unit based on the utility evaluation results. This unit maintains the calculation rules of the utility function and evaluates the decision effect in real time.

[0097] Task Adaptation Unit: This unit is used to perform similar task determination and fast adaptation training, so that the policy network can quickly adapt to new tasks. This unit determines task similarity through cosine similarity calculation and manages checkpoint data.

[0098] The collaborative relationship between the units is as follows: the training data management unit provides training data to the training execution unit, the evaluation results of the utility evaluation unit are used to guide the parameter optimization process of the training execution unit, and the task adaptation unit selects the optimal checkpoint for rapid adaptation training based on the evaluation results of the utility evaluation unit. Through the organic cooperation of the functional units, the present invention realizes the continuous optimization of AI human-machine collaborative decision-making.

Claims

1. An AI human-machine collaborative decision-making optimization method based on a game model, characterized in that: The method comprises the following steps: constructing a human-machine game model, the human-machine game model including an AI participant and a human participant, constructing a task state and a historical interaction record into a state space, wherein the task state T includes input feature information and type semantic information of the task, defining the action space of the AI ​​participant and the action space of the human participant, and constructing a bilateral utility function and a joint utility function for realizing collaborative decision-making of the AI ​​participant and the human participant; A deep neural network is used to implement the strategy function of the AI ​​participant. The strategy function is trained by supervised learning based on historical decision data of human experts, the state space is mapped to the behavior probability distribution in the action space of the AI ​​participant, and verification and evaluation are performed to ensure that the strategy function has reliable decision-making capabilities, so as to construct the initial strategy of the AI ​​participant. Execute online game training, which includes using temperature parameters to adjust the probability distribution of the neural network output corresponding to the policy function and sampling decision-making behaviors, recording interactive data including states, behaviors, and utility values ​​calculated based on the utility functions of the AI ​​participants, constructing a value evaluation network and using a policy gradient method to update the parameters of the neural network, ensuring training stability through training process monitoring and a policy fallback mechanism, and optimizing the decision-making effect of the AI ​​participants; Based on the type semantic information in the task state T, similar tasks are determined and the policy network is dynamically adaptively adjusted, including saving checkpoint data and performing fast adaptation training, wherein the checkpoint data includes the task type identifier, the utility value calculated by the joint utility function and the parameters of the policy network, so as to enhance the adaptability of the AI ​​participant in new task scenarios.

2. The AI ​​human-machine collaborative decision-making optimization method based on a game model according to claim 1 is characterized in that: The human-computer game model is constructed as follows: The state space is represented by a binary pair S=(T, H), where the task state T includes a digital input feature vector, a semantic embedding vector of the task type, a normalized task completion progress value, and a normalized time limit value, and the historical interaction record H includes a discretized decision timestamp sequence, a binary identification sequence of the decision subject, and a vector sequence of the decision content; The action space A1 of the AI ​​participant includes an m-dimensional decision action vector, an execution action identifier with a value of {0, 1}, and a confirmation request identifier with a value of {0, 1}, and the action space A2 of the human participant includes a response action identifier with a value of {0, 1, 2, 3}, which respectively correspond to accepting suggestions, modifying suggestions, making decisions directly, and terminating tasks; The utility function of the AI ​​participant is: Among them, C is the task completion mark, when the task time does not exceed the maximum time limit t max The value is 1 when the AI ​​participant is in the task, otherwise it is 0. P is the task goal achievement score, which is calculated by dividing the number of task goals achieved by the total number of task goals. Its value range is [0, 1]. t is the decision time of the AI ​​participant. The maximum time limit t max It is used to limit the maximum time to complete the task. All variables are normalized using the maximum and minimum method so that their value range is limited to the interval [0, 1]; The utility function of the human participant is: Where n is the number of interventions by the human participant, n max is the preset maximum number of interventions, all variables are normalized using the maximum and minimum method so that their value range is limited to the interval [0, 1]; the joint utility function: U=min(U1,U2) As the optimization goal of the strategy network training, the utility of the AI ​​participants and the human participants can be balanced.

3. The AI ​​human-machine collaborative decision-making optimization method based on a game model according to claim 1 is characterized in that: The deep neural network and supervised learning process are implemented as follows: The deep neural network includes three fully connected layers, which receive processed state space inputs, wherein the task state T maintains the complete input history interaction record H and only retains the decision timestamp sequence and decision content sequence of the corresponding moment identified as a human participant in the binary identification sequence of the decision subject. The number of neurons in each layer is set to decrease layer by layer according to the dimension of the state space S and the dimension of the AI ​​participant action space A1 according to a preset contraction ratio. Each layer uses a ReLU activation function to achieve nonlinear transformation. The last layer uses a Softmax function to map the output to a probability distribution, which is used to characterize the selection probability of different decision behaviors of the AI ​​participant in the action space A1. Extract expert decision data from the historical decision data of the human participants, wherein the screening of the expert decision data satisfies that the task completion time is less than a preset time threshold, the decision result meets the quality assessment standard, and has a high historical decision success rate, and the decision data that meets the conditions are constructed as a data set D = {(T, H, a)}, wherein a is the expert decision behavior identifier corresponding to the action space A2 of the human participants; Dividing the expert decision data into a training set and a validation set according to a preset ratio, wherein the training set is used for parameter optimization of the strategy function, and the validation set is used for evaluating the generalization performance of the strategy function; The cross entropy loss function is used to optimize the parameters of the strategy function, and the decision accuracy is evaluated on the validation set. When the accuracy reaches a preset threshold, the training is stopped, so that the AI ​​participant can learn the decision-making mode of human experts and provide a good initial strategy for subsequent online game training.

4. The AI ​​human-machine collaborative decision-making optimization method based on a game model according to claim 1 is characterized in that: The online game training is performed as follows: The output probability distribution of the policy network is adjusted by using a temperature parameter τ that is dynamically adjusted based on training rounds, wherein the temperature parameter decreases as the number of training rounds increases, and decision-making behavior sampling is performed based on the adjusted probability distribution to balance policy exploration and utilization; Collect interaction data(s t , a t , r t ), where s t is the state representation of the state space at time t, a t is the decision behavior obtained by the AI ​​participant based on the sampling of the strategy network, r t is the utility value calculated based on the AI ​​participant utility function U1; Construct a value network V(s) with the same input layer structure as the policy network. The value network adopts a three-layer fully connected structure, and the last layer outputs a single value to evaluate the state value. Update the parameter θ of the policy network based on the following policy gradient formula: Among them, α is the learning rate used to control the parameter update step size, and γ is the discount factor used to balance immediate utility and long-term utility; Calculate and record the average utility value during the training process, where the average utility value includes the single-step utility mean calculated based on the AI ​​participant utility function and the synergistic utility mean calculated based on the joint utility function. When the single-step utility mean of the new strategy continues to be lower than a preset ratio threshold of the average utility value after multiple consecutive rounds of training, execute strategy rollback. The strategy rollback includes loading the historical parameters with the highest synergistic utility mean, resetting the temperature parameters, reducing the learning rate, and keeping the shrinkage ratio unchanged to maintain the stability of the network structure, so as to ensure the stability of the training and the continuous improvement of the utility, and finally obtain the optimized strategy network parameters; The optimized policy network parameters are periodically saved as checkpoint data for the dynamic adaptive adjustment. At the same time, the optimized policy network parameters are applied to the actual human-machine collaborative decision-making process, and decision suggestions are output based on the input state information for reference by the human participants.

5. The AI ​​human-machine collaborative decision-making optimization method based on a game model according to claim 1 is characterized in that: The dynamic adaptive adjustment is achieved as follows: The semantic embedding vector of the task type in the task state T is used to determine similar tasks by calculating the cosine similarity, so as to identify historical tasks similar to the current task. When there are historical tasks with a similarity higher than a preset threshold, the checkpoint with the highest similarity is selected for parameter loading. Otherwise, the current parameters of the policy network are maintained. The checkpoint data is saved at a preset time interval, and the checkpoint data includes a semantic embedding vector of the task type, a utility value calculated based on the joint utility function, and parameters of the policy network; the rapid adaptive training includes calculating the similarity between the current task and the historical tasks in the checkpoint data, selecting the historical checkpoint with the highest similarity for parameter loading, and collecting the interaction data (s t , a t , r t ) trains the policy network, wherein the number of training rounds is inversely proportional to the task similarity, for achieving rapid adaptation while maintaining the stability of the policy network.

6. An AI human-machine collaborative decision-making optimization system based on a game model, characterized in that: include: A training data management unit, used to store and manage the historical decision data of the human experts and the interaction data during the online game training process; A policy network unit, used to construct the deep neural network and output a decision suggestion through the policy function based on the input state information; A training execution unit, used to implement the supervised learning training and the online game training, and perform parameter optimization of the strategy network; A utility evaluation unit, configured to calculate the utility values ​​of the AI ​​participant utility function and the joint utility function, and to guide the optimization process of the training execution unit based on the utility evaluation result; A task adaptation unit, used to perform the similar task determination and the rapid adaptation training, so as to achieve rapid adaptation of the policy network to new tasks; The training data management unit provides training data to the training execution unit, the evaluation result of the utility evaluation unit is used to guide the parameter optimization process of the training execution unit, and the task adaptation unit selects the optimal checkpoint for rapid adaptation training based on the evaluation result of the utility evaluation unit.

Images