Method and system for predicting belief state based on attention bidirectional LSTM (Long Short Term Memory) model
By applying the attention bidirectional LSTM model in the adversarial scenario, predicting the belief state during the adversarial process, the problem of inaccurate prediction in complex systems and dynamic environments is solved, and the accuracy of prediction and the long-term dependency capture ability of the model are improved.
Patent Information
- Application Number
- CN202411941085.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-26
- Publication Date
- 2025-04-22
AI Technical Summary
Traditional belief state prediction algorithms are difficult to accurately predict the other party's belief state in an adversarial scenario when dealing with complex systems and dynamic environments.
The belief state prediction method based on the attention bidirectional LSTM model is adopted. By obtaining the action trajectory data of the target object, pre-processing, the observation sequence is processed using the pre-trained attention bidirectional LSTM model to predict the belief state of the target object's next time step.
The accuracy of belief state prediction is improved, and the model's ability to capture long-term dependencies of sequence data is enhanced by simultaneously utilizing the forward and backward information of the observation sequence and dynamically weighting the information of different parts.
Smart Images
Figure CN119862908A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of data processing technology, and specifically relates to a belief state prediction method and system based on an attention bidirectional LSTM model. Background Art
[0002] Adversarial game theory is an important theoretical framework in the adversarial field. It studies how to formulate optimal strategies under uncertainty and conflict environments. As game confrontation becomes increasingly complex and intelligent, the adversarial field begins to introduce computer simulation technology to simulate and analyze adversarial data, predict the opponent's action intentions to form a belief state, and provide strong support for adversarial decision-making. Based on this, a state prediction algorithm is required to predict the belief state.
[0003] Traditional belief state prediction algorithms usually rely on explicit rules and hand-crafted features, which often face great challenges when dealing with complex systems and dynamic environments. Therefore, it is crucial to accurately predict the opponent's belief state in adversarial scenarios. Summary of the invention
[0004] In view of this, it is necessary to provide a belief state prediction method and system based on an attention bidirectional LSTM model to address the above technical problems, so as to accurately predict the belief state during the game confrontation process.
[0005] A belief state prediction method based on an attention bidirectional LSTM model, the method comprising:
[0006] Obtain the action trajectory data of the target object in the most recent time step during the confrontation process;
[0007] Preprocessing the action trajectory data to obtain an observation sequence;
[0008] The observation sequence is processed using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object at the next time step. The attention bidirectional LSTM model is predicted using the action trajectory sample data of object samples collected in historical adversarial scenarios and the corresponding actual belief state.
[0009] In one embodiment, the attention bidirectional LSTM model includes a trained first bidirectional LSTM layer, a second bidirectional LSTM layer, an attention mechanism Attention layer and a fully connected layer;
[0010] The observation sequence is processed using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object at the next time step, including:
[0011] Processing the observation sequence using a first bidirectional LSTM layer to obtain a first forward feature sequence and a first reverse feature sequence of the observation sequence;
[0012] Obtaining a first global feature sequence of the target object according to the first forward feature sequence and the first reverse feature sequence;
[0013] Using the Attention layer to perform weighted processing on the first global feature sequence to obtain a weighted feature sequence, wherein the weight assigned to each time step is positively correlated with the importance of the time step;
[0014] Processing the weighted feature sequence using a second bidirectional LSTM layer to obtain a second forward feature sequence and a second reverse feature sequence of the weighted feature sequence;
[0015] Obtaining a second global feature sequence of the target object according to the second forward feature sequence and the second reverse feature sequence;
[0016] The second global feature sequence is input into a fully connected layer, and the fully connected layer outputs the belief state of the target object at the next time step.
[0017] In one embodiment, the attention bidirectional LSTM model further includes a first random inactivation Dropout layer between the first bidirectional LSTM layer and the Attention layer, a second Dropout layer between the Attention layer and the second bidirectional LSTM layer, and a third Dropout layer between the second bidirectional LSTM layer and the fully connected layer;
[0018] Among them, each Dropout layer is used to prevent overfitting of the model by randomly discarding neurons.
[0019] In one embodiment, the attention bidirectional LSTM model includes a first bidirectional LSTM model, an attention mechanism Attention layer, a second bidirectional LSTM model, and a fully connected layer, wherein the first bidirectional LSTM model is connected to the attention mechanism Attention layer, the Attention layer is connected to the second bidirectional LSTM model, and the second bidirectional LSTM model is connected to the fully connected layer; the training process of the attention bidirectional LSTM model is:
[0020] Acquire a training sample set, wherein the training sample set includes action trajectory sample data collected in a confrontation scenario;
[0021] Normalizing the training sample set to obtain observation sequence samples;
[0022] Input the observation sequence samples into the first bidirectional LSTM model, use the output of the first bidirectional LSTM model as the input of the Attention layer, use the output of the Attention layer as the input of the second bidirectional LSTM model, and use the output of the second bidirectional LSTM model as the input of the fully connected layer;
[0023] Inputting the belief state prediction result output by the fully connected layer and the actual belief state of the observation sequence sample into a minimization loss function, and calculating the gap between the belief state prediction result and the actual belief state;
[0024] The parameters in the attention bidirectional LSTM model are adjusted according to the gap, and when the adjusted model meets the training termination condition, the trained attention bidirectional LSTM model is obtained.
[0025] In one embodiment, the belief state prediction method based on the attention bidirectional LSTM model provided in this embodiment further includes:
[0026] The Adam optimizer is used to adjust the learning rate to accelerate the training process of the attention bidirectional LSTM model;
[0027] When the gap is within the set gap threshold range, the ReduceLROnPlateau callback function is used to reduce the learning rate to perform fine-grained parameter adjustment on the attention bidirectional LSTM model;
[0028] When the gap obtained by the most recent set number of times does not change, the EarlyStopping callback function is used to restore the attention bidirectional LSTM model corresponding to the optimal parameter value; and the training is terminated, and the attention bidirectional LSTM model corresponding to the optimal parameter value is determined as the trained attention bidirectional LSTM model.
[0029] In a second aspect, a belief state prediction system based on an attention bidirectional LSTM model is provided, the system comprising:
[0030] An acquisition module is used to obtain the action trajectory data of the target object in the most recent time step during the confrontation process;
[0031] A preprocessing module, used for preprocessing the action trajectory data to obtain an observation sequence;
[0032] The prediction module is used to process the observation sequence using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object at the next time step. The attention bidirectional LSTM model is predicted using the action trajectory sample data of the object samples collected in the historical confrontation scenario and the corresponding actual belief state.
[0033] In one embodiment, the attention bidirectional LSTM model includes a trained first bidirectional LSTM layer, a second bidirectional LSTM layer, an attention mechanism Attention layer and a fully connected layer; the prediction module is specifically used for:
[0034] Processing the observation sequence using a first bidirectional LSTM layer to obtain a first forward feature sequence and a first reverse feature sequence of the observation sequence;
[0035] Obtaining a first global feature sequence of the target object according to the first forward feature sequence and the first reverse feature sequence;
[0036] Using the Attention layer to perform weighted processing on the first global feature sequence to obtain a weighted feature sequence, wherein the weight assigned to each time step is positively correlated with the importance of the time step;
[0037] Processing the weighted feature sequence using a second bidirectional LSTM layer to obtain a second forward feature sequence and a second reverse feature sequence of the weighted feature sequence;
[0038] Obtaining a second global feature sequence of the target object according to the second forward feature sequence and the second reverse feature sequence;
[0039] The second global feature sequence is input into a fully connected layer, and the fully connected layer outputs the belief state of the target object at the next time step.
[0040] In one embodiment, the attention bidirectional LSTM model further includes a first random inactivation Dropout layer between the first bidirectional LSTM layer and the Attention layer, a second Dropout layer between the Attention layer and the second bidirectional LSTM layer, and a third Dropout layer between the second bidirectional LSTM layer and the fully connected layer;
[0041] The prediction module is also used to utilize each Dropout layer to randomly discard neurons to prevent overfitting of the model.
[0042] In one embodiment, the attention bidirectional LSTM model includes a first bidirectional LSTM model, an attention mechanism Attention layer, a second bidirectional LSTM model, and a fully connected layer, wherein the first bidirectional LSTM model is connected to the attention mechanism Attention layer, the Attention layer is connected to the second bidirectional LSTM model, and the second bidirectional LSTM model is connected to the fully connected layer; the system further includes:
[0043] A training module is used to: obtain a training sample set, wherein the training sample set includes action trajectory sample data collected in a confrontation scenario;
[0044] Normalizing the training sample set to obtain observation sequence samples;
[0045] Input the observation sequence samples into the first bidirectional LSTM model, use the output of the first bidirectional LSTM model as the input of the Attention layer, use the output of the Attention layer as the input of the second bidirectional LSTM model, and use the output of the second bidirectional LSTM model as the input of the fully connected layer;
[0046] Inputting the belief state prediction result output by the fully connected layer and the actual belief state of the observation sequence sample into a minimization loss function, and calculating the gap between the belief state prediction result and the actual belief state;
[0047] The parameters in the attention bidirectional LSTM model are adjusted according to the gap, and when the adjusted model meets the training termination condition, the trained attention bidirectional LSTM model is obtained.
[0048] In one embodiment, the training module is further used to:
[0049] The Adam optimizer is used to adjust the learning rate to accelerate the training process of the attention bidirectional LSTM model;
[0050] When the gap is within the set gap threshold range, the ReduceLROnPlateau callback function is used to reduce the learning rate to perform fine-grained parameter adjustment on the attention bidirectional LSTM model;
[0051] When the gap obtained by the most recent set number of times does not change, the EarlyStopping callback function is used to restore the attention bidirectional LSTM model corresponding to the optimal parameter value; and the training is terminated, and the attention bidirectional LSTM model corresponding to the optimal parameter value is determined as the trained attention bidirectional LSTM model.
[0052] In the above-mentioned belief state prediction method and system based on the attention bidirectional LSTM model, the action trajectory data of the target object in the most recent time step during the confrontation process is obtained; the action trajectory data is preprocessed to obtain an observation sequence; the observation sequence is processed using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object in the next time step. Using the above-mentioned method, since the above-mentioned attention bidirectional LSTM model can simultaneously utilize the forward information and backward information of the observation sequence, and the attention mechanism can dynamically weight different parts of the above-mentioned forward information and backward information, when these two are combined for belief state prediction, the model's ability to capture long-term dependencies of sequence data can be improved, thereby improving the accuracy of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 A flowchart of a belief state prediction method based on an attention bidirectional LSTM model in one embodiment;
[0054] Figure 2 A schematic diagram of a confrontation logic in which the red and blue sides use different AI strategies to confront each other on a MicroRTS platform map in an embodiment;
[0055] Figure 3 is a structural diagram of an attention bidirectional LSTM model in one embodiment;
[0056] Figure 4 It is a structural diagram of a belief state prediction system based on an attention bidirectional LSTM model in one embodiment;
[0057] Figure 5 FIG. 4 is a diagram showing the internal structure of a computing device in one embodiment. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0059] In one embodiment, Figure 1 As shown, a belief state prediction method based on an attention bidirectional LSTM model is provided, and the method is applied to an electronic device as an example for explanation. The electronic device may be, but not limited to, a server and other devices. In addition, the above method can be applied to any scenario where there is a confrontation situation, which may be, but not limited to, game confrontation, etc. When the above electronic device implements the above method, the following steps may be included:
[0060] S11. Obtain the action trajectory data of the target object in the most recent time step during the confrontation process.
[0061] The belief state in this application can be understood as: by predicting the action intention of the target object, a set of enemy action intentions with a time sequence after prediction is formed, which is the belief state.
[0062] In this step, the target object may be any object in the confrontation process, and the embodiment does not limit the specific number of target objects in the confrontation process. For example, in a confrontation scene including a red team and a blue team, the target object may be the red team or the blue team.
[0063] Based on this, in order to predict the belief state of the target object, the action trajectory data of the target object in the most recent time step during the confrontation process can be obtained, so as to predict the belief state of the target object in the next time step based on the obtained action trajectory data.
[0064] To better understand this step, we can take the simulation of the red and blue confrontation on the MicroRTS open source platform as an example. The platform is set as follows: the platform map frame size is 100*100 units, the blue AI strategy is set to RandomBiasedAI, and the red AI strategy is set to LightRush. The red and blue sides use these two AI strategies to compete. The starting point is in the upper left corner with the coordinates (0,0). Based on this, the action trajectory data of the red or blue side is obtained.
[0065] It should be noted that the above-mentioned AI strategy body can be understood as the strategy adopted at each time step. In addition to the above-mentioned random bias and light raid, the above-mentioned strategy body can also include but is not limited to worker raid (WorkerRush), ranged raid (RangedRush), heavy raid (HeavyRush), economic raid (EconomyRush), early raid one (Crush_V1), early raid two (Crush_V2), unplanned puppet (Puppet), etc. Different strategy bodies have different corresponding meanings. For example, the above-mentioned random bias strategy means that random selection can be made in different decision cycles. For example, in each decision cycle, possible actions can be randomly selected according to a preset probability distribution, and the selection probability of different actions can be adjusted through the bias item. Similarly, the above-mentioned light raid strategy focuses on the mobility of light units, and gains advantages in confrontation scenarios through high speed and flexibility. The red and blue sides use these two AI strategy bodies to compete. Figure 2 shown.
[0066] S12: pre-processing the action trajectory data to obtain an observation sequence.
[0067] Specifically, in order to predict the belief state, after obtaining the action trajectory data, the action trajectory data needs to be preprocessed. Specifically, feature extraction is performed on the action trajectory data to extract the action trajectory features; then the action trajectory features are normalized to ensure that the value range of each feature is within a unified range. After normalization, a normalized action feature sequence can be obtained, and then the feature sequence is subjected to data conversion processing to obtain the above observation sequence that meets the input requirements of the pre-trained attention bidirectional LSTM model.
[0068] S13. Use the pre-trained attention bidirectional LSTM model to process the observation sequence and predict the belief state of the target object in the next time step.
[0069] Among them, the above-mentioned attention bidirectional LSTM model is obtained by using the action trajectory sample data of object samples collected in historical confrontation scenarios and the corresponding actual belief state prediction.
[0070] In this step, after obtaining the observation sequence, the observation sequence can be input into the above-mentioned trained attention bidirectional LSTM model, so that the model can process the observation sequence and output the belief state of the target object in the next time step.
[0071] Since the above-mentioned attention bidirectional LSTM model is trained using the above-mentioned object samples, in addition, the attention bidirectional LSTM model itself can capture the time dependency in the observation sequence data and consider the uncertainty and complexity in the game confrontation process, thereby ensuring the accuracy of the prediction results of the belief state. Given the action trajectory data of the red party or the blue party, the above-mentioned model can be used to predict the belief state of the red party or the blue party at the next time step, such as the position at the next time step, which has a very important reference value for the analysis of the confrontation situation and decision-making.
[0072] In the implementation of the above-mentioned belief state prediction method, the action trajectory data of the target object in the most recent time step during the confrontation process is obtained; the action trajectory data is preprocessed to obtain an observation sequence; the observation sequence is processed using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object in the next time step. Since the above-mentioned attention bidirectional LSTM model can simultaneously utilize the forward information and backward information of the observation sequence, and the attention mechanism can dynamically weight different parts of the forward information and backward information, when these two are combined for belief state prediction, the model's ability to capture long-term dependencies of sequence data can be improved, so that not only can the belief state in the confrontation process be accurately predicted, that is, the accuracy of the prediction result is improved, but also the predicted belief state can be output, so that it is convenient to execute corresponding defense measures based on the belief state of the target object.
[0073] Optionally, the attention bidirectional LSTM model includes a trained first bidirectional LSTM layer, a second bidirectional LSTM layer, an attention mechanism Attention layer, and a fully connected layer. Figure 3 As shown, on this basis, this embodiment proposes that step S13 can be performed according to the following process: using the first bidirectional LSTM layer to process the observation sequence to obtain the first forward feature sequence and the first reverse feature sequence of the observation sequence; according to the first forward feature sequence and the first reverse feature sequence, obtaining the first global feature sequence of the target object; using the Attention layer to weight the first global feature sequence to obtain a weighted feature sequence, wherein the weight assigned to each time step is positively correlated with the importance of the time step; using the second bidirectional LSTM layer to process the weighted feature sequence to obtain the second forward feature sequence and the second reverse feature sequence of the weighted feature sequence; according to the second forward feature sequence and the second reverse feature sequence, obtaining the second global feature sequence of the target object; inputting the second global feature sequence into the fully connected layer, and the fully connected layer outputs the belief state of the target object at the next time step.
[0074] Specifically, the above-mentioned bidirectional LSTM expands the traditional LSTM structure and can capture information not only from past time steps, but also from future time steps, thereby obtaining long-term dependencies with temporal relationships in the observation sequence, thereby enhancing the model's ability to understand the context of the observation sequence, thereby better coping with the prediction of belief states in adversarial scenarios.
[0075] In the above attention bidirectional LSTM model, there are two bidirectional LSTM layers, refer to Figure 3As shown, they are the first bidirectional LSTM layer and the second bidirectional LSTM layer, respectively, which process different levels of temporal dependencies. Among them, each bidirectional LSTM layer can read the forward information and reverse information in the input sequence, and then capture the global dependency based on the forward information and reverse information. The forward information can be understood as the information obtained by traversing the sequence from front to back; similarly, the reverse information can be understood as the information obtained by traversing the sequence from back to front. In order to better understand this embodiment, the first bidirectional LSTM layer is used as an example for explanation. After the observation sequence is input into the first bidirectional LSTM layer, the first bidirectional LSTM layer can obtain the forward information, which is recorded as the first forward feature sequence, and can also obtain the reverse information, which is recorded as the first reverse feature sequence. The processing process of the second bidirectional LSTM layer is similar, and will not be described in detail here.
[0076] In addition, the attention layer of the above attention mechanism is mainly used to enhance the performance of the model, and an attention layer is introduced after each bidirectional LSTM model to weight the sequence output by the bidirectional LSTM layer, thereby enhancing the attention to important features in the sequence and weakening the attention to unimportant features, so that the model can accurately predict the belief state based on important features. Figure 3 In the above Attention layer, Q, K, and V represent query, key, and value, respectively. In addition, MatMul in the above Attention layer represents multiplication, Scale represents division by square root dk, Mask represents mask, and Softmax represents regression function.
[0077] The above fully connected layer, also recorded as the Dense layer, is the last layer of the attention bidirectional LSTM model and is used to output the belief state prediction results of the model.
[0078] Furthermore, the above-mentioned attention bidirectional LSTM model also includes a first random inactivation Dropout layer between the first bidirectional LSTM layer and the Attention layer, a second Dropout layer between the Attention layer and the second bidirectional LSTM layer, and a third Dropout layer between the second bidirectional LSTM layer and the fully connected layer. Please also refer to Figure 3 As shown in the figure, each Dropout layer is used to prevent overfitting of the model by randomly dropping neurons.
[0079] Specifically, the Dropout layer is a regularization technology that can prevent overfitting of the attention bidirectional LSTM model by randomly discarding some neurons, thereby improving the stability and reliability of the model; in addition, the Dropout layer also helps the model to reduce its dependence on training samples during training, improves the generalization ability of the model on unseen data, and further ensures the accuracy of the model's prediction results.
[0080] By providing the above Figure 3 The attention bidirectional LSTM model with the structure shown can more accurately capture the key parts of the observation sequence of the target object to more effectively predict the future belief state.
[0081] Optionally, the above attention bidirectional LSTM model is Figure 3 When the structure is shown, the training process of the attention bidirectional LSTM model is as follows: obtain a training sample set, wherein the training sample set includes action trajectory sample data collected in the confrontation scene; normalize the training sample set to obtain observation sequence samples; input the observation sequence samples into the first bidirectional LSTM model, the output of the first bidirectional LSTM model is used as the input of the Attention layer, the output of the Attention layer is used as the input of the second bidirectional LSTM model, and the output of the second bidirectional LSTM model is used as the input of the fully connected layer; input the belief state prediction result output by the fully connected layer and the actual belief state of the observation sequence sample into the minimization loss function, and calculate the gap between the belief state prediction result and the actual belief state; adjust the parameters in the attention bidirectional LSTM model according to the gap, and when the adjusted model meets the training termination condition, obtain the trained attention bidirectional LSTM model.
[0082] Specifically, the method for obtaining the above-mentioned training sample set can be as follows: Taking several rounds of red-blue confrontation on the game confrontation platform (MicroRTS) as an example, the action trajectory data in this scenario is recorded as time series data; each round of confrontation is carried out step by step until the winner is determined. Based on this, the time series data of all entity trajectory movements of the red side in the previous T steps are recorded as the data set for the red side's belief state prediction. In terms of data set division, most of the red side's time series data are selected as the training sample set, and the remaining time series data are used as the test set. The specific value of the above T can be set according to the actual situation. This embodiment does not limit its value. For example, T can be 200, etc.
[0083] In order to facilitate the training of the attention bidirectional LSTM model and the unification of subsequent data and improve the training effect of the model, the data set can be normalized, that is, the time series data of the red team's first T steps of trajectory movement can be normalized by selecting an appropriate method to ensure that the value range of each feature is within a unified range, and the normalized data is used as the data set for the prediction of the red team's trajectory information, including the training set and the test set. The above normalization method can be: maximum value normalization, find the maximum absolute value of the column data, and divide each value in the column data by the maximum value. This normalization preprocessing step helps to accelerate the training process of the subsequent model and improve its convergence stability.
[0084] After obtaining the normalized training set, it is divided into a series of observation sequences and corresponding labels, where the observation sequence samples are used as the input of the attention bidirectional LSTM model, and the label is the belief state at the next time point that is expected to be predicted. Each input observation sequence sample consists of a fixed length of time steps. At the same time, for each observation sequence sample, a corresponding prediction target (actual belief state) of fixed length is constructed to represent the future belief state.
[0085] The above attention bidirectional LSTM model contains two bidirectional LSTM layers, one Attention layer, three Dropout layers and one Dense layer, as shown in Figure 3 As shown in the figure, this architecture combines the attention mechanism with the bidirectional LSTM to more accurately capture the key parts of the observed sequence samples and effectively predict future states.
[0086] In the above attention bidirectional LSTM model, there are two bidirectional LSTM layers. Bidirectional LSTM can capture long-term dependencies in time series data and has advantages in processing complex tasks that require information from the front and back. Each bidirectional LSTM layer is used to process different levels of temporal dependencies and learn temporal patterns in input samples. Each bidirectional LSTM layer reads both the forward information of the sequence (traversing the observed sequence samples from front to back) and the reverse information of the sequence (traversing the observed sequence samples from back to front), thereby capturing global dependencies and being able to model more accurately when processing long-term dependencies. The output of this layer is the hidden state of each time step, which contains information from the past to the future and from the future to the past.
[0087] In order to further enhance the performance of the model, an attention layer is introduced between the two bidirectional LSTM layers. Figure 3 As shown in Figure 1, the attention mechanism is used to weight the output of the bidirectional LSTM layer, increasing the focus on important time steps and weakening the impact on unimportant time steps. In this way, the network can dynamically adjust its attention based on the context.
[0088] The above-mentioned Attention layer is a technology that dynamically weights different parts of the input information. The core idea is to dynamically assign different weights to each time step in the observation sequence sample when processing the observation sequence sample. In the time series prediction task, the importance of historical information may change over time, so the model needs to be able to "focus" on important time steps. The self-attention mechanism (Self-Attention) is introduced based on the output of the bidirectional LSTM layer, so that different weights are dynamically assigned according to the current task, reducing the reliance on redundant information, thereby improving the prediction accuracy. For example, when using the red square training set for model training, because not every time step has the same impact on the final prediction result of the attention bidirectional LSTM model, therefore, by introducing the Attention layer, the Attention layer can focus on those time steps that are more important to the prediction result, thereby significantly improving the prediction effect of the model.
[0089] A Dropout layer is added after each bidirectional LSTM layer and Attention layer. Dropout is a regularization technique that prevents overfitting of the model by randomly discarding some neurons. The Dropout layer helps the model reduce its dependence on training data and improves its generalization ability on unseen data. Especially in deep networks with a large number of parameters, Dropout can effectively reduce the complexity of the model, thereby improving the stability and reliability of the model.
[0090] Further, after being processed by the bidirectional LSTM layer and the Attention layer, the output of the attention bidirectional LSTM model is passed through a fully connected (Dense) layer to predict the belief state. The role of the Dense layer is to map the high-dimensional features extracted previously to the final output space so that its output dimension matches the dimension of the prediction target. The Dense layer enables the model to accurately output the required prediction results, thereby achieving efficient belief state prediction. The Relu activation function is used in the Dense layer to ensure that the range and format of the output value meet the requirements. The Dense layer enables the model to accurately generate the required prediction results, thereby achieving efficient trajectory prediction.
[0091] Furthermore, the attention bidirectional LSTM model has a large number of parameters that need to be determined through training. The goal of training is to minimize the loss function so that the gap between the predicted belief state and the actual belief state is minimized. Model training is an iterative process, in which the training set is input into the attention bidirectional LSTM model and the model parameters are continuously adjusted through the back propagation algorithm. During the training process, the value of the loss function is monitored to evaluate the model performance until the model performance is no longer significantly improved, thus obtaining a trained attention bidirectional LSTM model.
[0092] In addition, when training the model, we can also consider different lengths of observation sequence historical position information as input, and try the input lengths from the most recent 1 observation sequence sample historical position to the most recent 10 observation sequence sample historical positions. Through experiments, we found that the best effect is to predict the next step based on the historical position information of the previous observation sequence samples, indicating that the historical position information of the two-step observation sequence samples contains enough context information to provide an effective prediction basis for the trajectory of the red party at the next time point.
[0093] Optionally, when performing model training, the following process can also be included: using the Adam optimizer to adjust the learning rate to accelerate the training process of the attentive bidirectional LSTM model; when the gap is within the set gap threshold range, using the ReduceLROnPlateau callback function to reduce the learning rate to perform fine-grained parameter adjustment on the attentive bidirectional LSTM model; when the gap obtained for the most recent set number of times has not changed, using the EarlyStopping callback function to restore the attentive bidirectional LSTM model corresponding to the optimal parameter value; and terminating the training, and determining the attentive bidirectional LSTM model corresponding to the optimal parameter value as the trained attentive bidirectional LSTM model.
[0094] Specifically, in order to ensure that the model can learn efficiently and prevent overfitting, the following training strategies and parameter settings can be adopted: (1) Adopt the Adam optimizer to accelerate the training process with its adaptive learning rate adjustment mechanism; (2) The learning rate scheduler uses the ReduceLROnPlateau callback function. When the loss no longer decreases significantly during model training, the learning rate is automatically reduced to fine-tune the model parameters; (3) Early stopping. To prevent overfitting, the EarlyStopping callback function is used. If the verification loss does not improve for multiple consecutive cycles, the training will stop early and return to the optimal weight state.
[0095] After the attention bidirectional LSTM model is trained, the attention bidirectional LSTM model can be verified using the test set to output the attention bidirectional LSTM model for belief state prediction. The training status of the attention bidirectional LSTM model can be measured by the mean square error and / or mean displacement error.
[0096] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.
[0097] Based on the same technical concept, Figure 4 As shown in the structural diagram, the present application also provides a belief state prediction system based on an attention bidirectional LSTM model, the system comprising:
[0098] An acquisition module 41 is used to acquire the action trajectory data of the target object in the most recent time step during the confrontation process;
[0099] A preprocessing module 42, used for preprocessing the motion trajectory data to obtain an observation sequence;
[0100] The prediction module 43 is used to process the observation sequence using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object at the next time step. The attention bidirectional LSTM model is predicted using the action trajectory sample data of the object samples collected in the historical confrontation scene and the corresponding actual belief state.
[0101] Using the above system, since the above-mentioned attention bidirectional LSTM model can simultaneously utilize the forward information and backward information of the observation sequence, and the attention mechanism can dynamically weight different parts of the above-mentioned forward information and backward information, when these two are combined for belief state prediction, the model's ability to capture long-term dependencies of sequence data can be improved, thereby improving the accuracy of model predictions.
[0102] Optionally, in one embodiment, the above-mentioned attention bidirectional LSTM model includes a trained first bidirectional LSTM layer, a second bidirectional LSTM layer, an attention mechanism Attention layer and a fully connected layer; the prediction module 43 is specifically used for:
[0103] Processing the observation sequence using a first bidirectional LSTM layer to obtain a first forward feature sequence and a first reverse feature sequence of the observation sequence;
[0104] Obtaining a first global feature sequence of the target object according to the first forward feature sequence and the first reverse feature sequence;
[0105] Using the Attention layer to perform weighted processing on the first global feature sequence to obtain a weighted feature sequence, wherein the weight assigned to each time step is positively correlated with the importance of the time step;
[0106] Processing the weighted feature sequence using a second bidirectional LSTM layer to obtain a second forward feature sequence and a second reverse feature sequence of the weighted feature sequence;
[0107] Obtaining a second global feature sequence of the target object according to the second forward feature sequence and the second reverse feature sequence;
[0108] The second global feature sequence is input into a fully connected layer, and the fully connected layer outputs the belief state of the target object at the next time step.
[0109] In one embodiment, the above-mentioned attention bidirectional LSTM model further includes a first random inactivation Dropout layer between the first bidirectional LSTM layer and the Attention layer, a second Dropout layer between the Attention layer and the second bidirectional LSTM layer, and a third Dropout layer between the second bidirectional LSTM layer and the fully connected layer;
[0110] The prediction module 43 is also used to utilize each Dropout layer to randomly discard neurons to prevent overfitting of the model.
[0111] In one embodiment, the above-mentioned attention bidirectional LSTM model includes a first bidirectional LSTM model, an attention mechanism Attention layer, a second bidirectional LSTM model, and a fully connected layer, wherein the first bidirectional LSTM model is connected to the attention mechanism Attention layer, the Attention layer is connected to the second bidirectional LSTM model, and the second bidirectional LSTM model is connected to the fully connected layer; the system also includes:
[0112] A training module (not shown in the figure) is used to: obtain a training sample set, wherein the training sample set includes action trajectory sample data collected in the confrontation scenario;
[0113] Normalizing the training sample set to obtain observation sequence samples;
[0114] Input the observation sequence samples into the first bidirectional LSTM model, use the output of the first bidirectional LSTM model as the input of the Attention layer, use the output of the Attention layer as the input of the second bidirectional LSTM model, and use the output of the second bidirectional LSTM model as the input of the fully connected layer;
[0115] Inputting the belief state prediction result output by the fully connected layer and the actual belief state of the observation sequence sample into a minimization loss function, and calculating the gap between the belief state prediction result and the actual belief state;
[0116] The parameters in the attention bidirectional LSTM model are adjusted according to the gap, and when the adjusted model meets the training termination condition, the trained attention bidirectional LSTM model is obtained.
[0117] Furthermore, the above training module is also used for:
[0118] The Adam optimizer is used to adjust the learning rate to accelerate the training process of the attention bidirectional LSTM model;
[0119] When the gap is within the set gap threshold range, the ReduceLROnPlateau callback function is used to reduce the learning rate to perform fine-grained parameter adjustment on the attention bidirectional LSTM model;
[0120] When the gap obtained by the most recent set number of times does not change, the EarlyStopping callback function is used to restore the attention bidirectional LSTM model corresponding to the optimal parameter value; and the training is terminated, and the attention bidirectional LSTM model corresponding to the optimal parameter value is determined as the trained attention bidirectional LSTM model.
[0121] In one embodiment, a computing device is provided. The computing device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computing device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computing device is used to provide computing and control capabilities. The memory of the computing device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computing device is used to store data. The network interface of the computing device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements the belief state prediction method based on the attention bidirectional LSTM model provided in any of the above embodiments of the present application.
[0122] In one embodiment, a computing device is provided. The computing device may be a terminal or a server. The internal structure diagram thereof may be as follows: Figure 5As shown. The computing device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computing device is used to provide computing and control capabilities. The memory of the computing device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computing device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method and device for scene situation assembly with adaptive demand is implemented. The display screen of the computing device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computing device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the housing of the computing device, or an external keyboard, touchpad or mouse, etc.
[0123] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computing device to which the scheme of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0124] In one embodiment, a computing device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the method described in any of the above embodiments when executing the computer program.
[0125] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.
[0126] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0127] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0128] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A belief state prediction method based on an attention bidirectional LSTM model, characterized in that: The method comprises: Obtain the action trajectory data of the target object in the most recent time step during the confrontation process; Preprocessing the action trajectory data to obtain an observation sequence; The observation sequence is processed using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object at the next time step. The attention bidirectional LSTM model is predicted using the action trajectory sample data of object samples collected in historical adversarial scenarios and the corresponding actual belief state.
2. The method according to claim 1, characterized in that: The attention bidirectional LSTM model includes a trained first bidirectional LSTM layer, a second bidirectional LSTM layer, an attention mechanism Attention layer and a fully connected layer; The observation sequence is processed using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object at the next time step, including: Processing the observation sequence using a first bidirectional LSTM layer to obtain a first forward feature sequence and a first reverse feature sequence of the observation sequence; Obtaining a first global feature sequence of the target object according to the first forward feature sequence and the first reverse feature sequence; Using the Attention layer to perform weighted processing on the first global feature sequence to obtain a weighted feature sequence, wherein the weight assigned to each time step is positively correlated with the importance of the time step; Processing the weighted feature sequence using a second bidirectional LSTM layer to obtain a second forward feature sequence and a second forward feature sequence of the weighted feature sequence; Obtaining a second global feature sequence of the target object according to the second forward feature sequence and the second reverse feature sequence; The second global feature sequence is input into a fully connected layer, and the fully connected layer outputs the belief state of the target object at the next time step.
3. The method according to claim 2, characterized in that The attention bidirectional LSTM model also includes a first random inactivation Dropout layer between the first bidirectional LSTM layer and the Attention layer, a second Dropout layer between the Attention layer and the second bidirectional LSTM layer, and a third Dropout layer between the second bidirectional LSTM layer and the fully connected layer; Among them, each Dropout layer is used to prevent overfitting of the model by randomly discarding neurons.
4. The method according to claim 2 or 3, characterized in that: The attention bidirectional LSTM model includes a first bidirectional LSTM model, an attention mechanism Attention layer, a second bidirectional LSTM model, and a fully connected layer, wherein the first bidirectional LSTM model is connected to the attention mechanism Attention layer, the Attention layer is connected to the second bidirectional LSTM model, and the second bidirectional LSTM model is connected to the fully connected layer; the training process of the attention bidirectional LSTM model is: Acquire a training sample set, wherein the training sample set includes action trajectory sample data collected in a confrontation scenario; Normalizing the training sample set to obtain observation sequence samples; Input the observation sequence samples into the first bidirectional LSTM model, use the output of the first bidirectional LSTM model as the input of the Attention layer, use the output of the Attention layer as the input of the second bidirectional LSTM model, and use the output of the second bidirectional LSTM model as the input of the fully connected layer; Inputting the belief state prediction result output by the fully connected layer and the actual belief state of the observation sequence sample into a minimization loss function, and calculating the gap between the belief state prediction result and the actual belief state; The parameters in the attention bidirectional LSTM model are adjusted according to the gap, and when the adjusted model meets the training termination condition, the trained attention bidirectional LSTM model is obtained.
5. The method according to claim 4, characterized in that Also includes: The Adam optimizer is used to adjust the learning rate to accelerate the training process of the attention bidirectional LSTM model; When the gap is within the set gap threshold range, the ReduceLROnPlateau callback function is used to reduce the learning rate to perform fine-grained parameter adjustment on the attention bidirectional LSTM model; When the gap obtained by the most recent set number of times does not change, the EarlyStopping callback function is used to restore the attention bidirectional LSTM model corresponding to the optimal parameter value; and the training is terminated, and the attention bidirectional LSTM model corresponding to the optimal parameter value is determined as the trained attention bidirectional LSTM model.
6. A belief state prediction system based on an attention bidirectional LSTM model, characterized in that: include: An acquisition module is used to obtain the action trajectory data of the target object in the most recent time step during the confrontation process; A preprocessing module, used for preprocessing the action trajectory data to obtain an observation sequence; The prediction module is used to process the observation sequence using a pre-trained attention bidirectional LSTM model to predict the belief state of the target object at the next time step. The attention bidirectional LSTM model is predicted using the action trajectory sample data of the object samples collected in the historical confrontation scenario and the corresponding actual belief state.
7. The system according to claim 6, characterized in that The attention bidirectional LSTM model includes a trained first bidirectional LSTM layer, a second bidirectional LSTM layer, an attention mechanism Attention layer and a fully connected layer; the prediction module is specifically used for: Processing the observation sequence using a first bidirectional LSTM layer to obtain a first forward feature sequence and a first reverse feature sequence of the observation sequence; Obtaining a first global feature sequence of the target object according to the first forward feature sequence and the first reverse feature sequence; Using the Attention layer to perform weighted processing on the first global feature sequence to obtain a weighted feature sequence, wherein the weight assigned to each time step is positively correlated with the importance of the time step; Processing the weighted feature sequence using a second bidirectional LSTM layer to obtain a second forward feature sequence and a second reverse feature sequence of the weighted feature sequence; Obtaining a second global feature sequence of the target object according to the second forward feature sequence and the second reverse feature sequence; The second global feature sequence is input into a fully connected layer, and the fully connected layer outputs the belief state of the target object at the next time step.
8. The system according to claim 7, characterized in that The attention bidirectional LSTM model also includes a first random inactivation Dropout layer between the first bidirectional LSTM layer and the Attention layer, a second Dropout layer between the Attention layer and the second bidirectional LSTM layer, and a third Dropout layer between the second bidirectional LSTM layer and the fully connected layer; The prediction module is also used to utilize each Dropout layer to randomly discard neurons to prevent overfitting of the model.
9. The system according to claim 7 or 8, characterized in that: The attention bidirectional LSTM model includes a first bidirectional LSTM model, an attention mechanism Attention layer, a second bidirectional LSTM model, and a fully connected layer, wherein the first bidirectional LSTM model is connected to the attention mechanism Attention layer, the Attention layer is connected to the second bidirectional LSTM model, and the second bidirectional LSTM model is connected to the fully connected layer; the system also includes: A training module is used to: obtain a training sample set, wherein the training sample set includes action trajectory sample data collected in a confrontation scenario; Normalizing the training sample set to obtain observation sequence samples; Input the observation sequence samples into the first bidirectional LSTM model, use the output of the first bidirectional LSTM model as the input of the Attention layer, use the output of the Attention layer as the input of the second bidirectional LSTM model, and use the output of the second bidirectional LSTM model as the input of the fully connected layer; Inputting the belief state prediction result output by the fully connected layer and the actual belief state of the observation sequence sample into a minimization loss function, and calculating the gap between the belief state prediction result and the actual belief state; The parameters in the attention bidirectional LSTM model are adjusted according to the gap, and when the adjusted model meets the training termination condition, the trained attention bidirectional LSTM model is obtained.
10. The system according to claim 9, characterized in that The training module is also used for: The Adam optimizer is used to adjust the learning rate to accelerate the training process of the attention bidirectional LSTM model; When the gap is within the set gap threshold range, the ReduceLROnPlateau callback function is used to reduce the learning rate to perform fine-grained parameter adjustment on the attention bidirectional LSTM model; When the gap obtained by the most recent set number of times does not change, the EarlyStopping callback function is used to restore the attention bidirectional LSTM model corresponding to the optimal parameter value; The training is terminated, and the attention bidirectional LSTM model corresponding to the optimal parameter value is determined as the trained attention bidirectional LSTM model.