Game processing method and device for electromagnetic data

Abstract

The invention discloses a game processing method and device for electromagnetic data. The method comprises the following steps: acquiring an electromagnetic game data set; utilizing the electromagnetic game data set to construct an offline data set; performing training processing on the constructed electromagnetic data evaluation model by using the offline data set to obtain an electromagnetic game model; and processing the acquired electromagnetic parameter information by using the electromagnetic game model to obtain electromagnetic game information. According to the method, through dynamically updating and historical electromagnetic interaction data, an action decision is output for an electromagnetic game of an actual environment. According to the method, the technology of accurate estimation of the game parameters in the distribution and conservative estimation of the game parameter values outside the distribution is applied, the failure of the electromagnetic game strategy caused by distribution offset is avoided, and the game effect of the game strategy is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a game processing method and device for electromagnetic data. Background Art

[0002] With the widespread application of various radio frequency devices, electromagnetic signal data has become complex and changeable. At the same time, with the different application scenarios, electromagnetic signals are superimposed on scene effects, making it more difficult to obtain various features of the received electromagnetic data. Combining the above two factors, a lot of the received electromagnetic data is unknown. Therefore, how to generate effective game strategies based only on historical electromagnetic data is an urgent problem to be solved. If the online off-track learning method is directly applied to the offline scenario, it will not be able to adapt to the changes in the actual scenario, resulting in a distribution shift problem, that is, the benefits of game strategies that have not appeared in historical strategies may be overestimated, ignoring the changing images of environmental factors in the real game environment and the historical game environment, resulting in poor game performance. Therefore, how to generate effective game strategies based on historical electromagnetic data while considering the influence of environmental factors is an urgent problem to be solved. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a game processing method and device for electromagnetic data, which can generate effective game strategies based on historical electromagnetic data while taking into account the influence of environmental factors, and can also effectively improve the adaptability to actual scenarios.

[0004] In order to solve the above technical problems, the first aspect of the embodiments of the present invention discloses a game processing method and device for electromagnetic data, the method comprising:

[0005] S1, obtaining an electromagnetic game data set; the electromagnetic game data set includes a plurality of electromagnetic game data;

[0006] S2, using the electromagnetic game data set to construct an offline data set;

[0007] S3, using the offline data set to train the constructed electromagnetic data evaluation model to obtain an electromagnetic game model;

[0008] S4, using the electromagnetic game model, processing the acquired electromagnetic parameter information to obtain electromagnetic game information;

[0009] The electromagnetic game data includes: wavelength, frequency band, pulse width, pulse repetition frequency, power, game action data and game effect data;

[0010] The electromagnetic parameter information includes: wavelength, frequency band, pulse width, pulse repetition frequency, and power;

[0011] The electromagnetic game information includes: game signal carrier frequency, game signal transmission pulse power, and expected value of electromagnetic parameter change after the game.

[0012] The method of using the electromagnetic game data set to construct an offline data set includes:

[0013] S21, encode each electromagnetic game data to obtain a corresponding game quintuple; the game quintuple is represented by (s, a, _s, r, done), where s represents the target electromagnetic parameter information before the game signal is sent, a represents the game action taken on the target, including the game signal carrier frequency and the game signal power, _s represents the target electromagnetic parameter information after the game signal is sent, r represents the real-time profit value of the electromagnetic game process, done represents whether the game is successful for the target, if the game is successful for the target, done takes a value of 1, if the game is unsuccessful for the target, done takes a value of 0;

[0014] S22, merging all game quintuples to obtain an offline data set.

[0015] The electromagnetic data evaluation model includes: an input module, an evaluation module, a decision network and a parameter updating module; the evaluation module includes a first evaluation network, a second evaluation network and a target evaluation network;

[0016] The input module receives the offline data set using the input port and sends the offline data set to the corresponding network using each output port;

[0017] The input module comprises an input end, a first output end, a second output end and a third output end;

[0018] The first input terminal of the second evaluation network is connected to the first output terminal of the input module;

[0019] The first input terminal of the first evaluation network is connected to the second output terminal of the input module;

[0020] The first input terminal of the decision network is connected to the third output terminal of the input module;

[0021] The parameter updating module comprises a first input terminal, a second input terminal, a third input terminal, a fourth input terminal, a first output terminal, a second output terminal, and a third output terminal;

[0022] The second input terminal of the second evaluation network is connected to the first output terminal of the parameter updating module;

[0023] The second input terminal of the first evaluation network is connected to the second output terminal of the parameter updating module;

[0024] The second input terminal of the decision network is connected to the third output terminal of the parameter updating module;

[0025] The output terminal of the second evaluation network is connected to the first input terminal of the parameter updating module;

[0026] The first output terminal of the first evaluation network is connected to the second input terminal of the parameter updating module;

[0027] The first output terminal of the decision network is connected to the third input terminal of the parameter updating module;

[0028] The second output terminal of the first evaluation network is connected to the input terminal of the target evaluation network;

[0029] The third input terminal of the first evaluation network is connected to the second output terminal of the decision network;

[0030] The fourth input terminal of the parameter updating module is used to receive the offline data set.

[0031] The decision network comprises: a first input layer, a first hidden layer, a second hidden layer, a third hidden layer and a first output layer;

[0032] The second evaluation network comprises: a second input layer, a fourth hidden layer, a fifth hidden layer, a sixth hidden layer and a second output layer;

[0033] The first evaluation network comprises: a third input layer, a seventh hidden layer, an eighth hidden layer, a ninth hidden layer and a third output layer;

[0034] The target evaluation network comprises: a fourth input layer, a tenth hidden layer, an eleventh hidden layer, a twelfth hidden layer and a fourth output layer;

[0035] The hidden layers are all implemented using a fully connected network;

[0036] The second evaluation network is used to process the variable s in the input game quintuple to obtain the estimated value V(s) of the variable s; and update its parameters using the parameter update value of the second evaluation network sent by the received parameter update module;

[0037] The first evaluation network processes s and a in the game quintuple to obtain an estimated value Q(s,a) of the state-action pair; updates its parameters using the parameter update value of the first evaluation network sent by the parameter update module; the s variable processed by the first evaluation network is obtained from the game quintuple, and the a variable processed by the first evaluation network is the game action a output by the decision network;

[0038] The decision network is used to process s in the game quintuple to obtain a probability value π(a|s) of taking game action a for s; and update its parameters using the parameter update value of the decision network sent by the parameter update module;

[0039] The target evaluation network processes s and a in the game quintuple to obtain an estimated value of the state-action pair Using the received parameter update values ​​of the target evaluation network sent by the parameter update module, the parameters of the target evaluation network are updated;

[0040] The parameter updating module performs scrambling processing on the sampled data set to obtain a first scrambled data set and a second scrambled data set and outputs them; uses a first parameter updating model to process the first evaluation network, the sampled data set, the first scrambled data set and the second scrambled data set to obtain a parameter update value of the first evaluation network; uses a second parameter updating model to process the target evaluation network and the sampled data set to obtain a parameter update value of the target evaluation network; uses a third parameter updating model to process the second evaluation network and the sampled data set to obtain a parameter update value of the second evaluation network; uses a fourth parameter updating model to process the decision network, the first scrambled data set, the second scrambled data set and the sampled data set to obtain a parameter update value of the decision network; the disturbance amount applied to the second scrambled data set is greater than the disturbance amount applied to the first scrambled data set.

[0041] The offline data set is used to train the constructed electromagnetic data evaluation model to obtain an electromagnetic game model, including:

[0042] S31, randomly initializing the parameters of the evaluation module and decision network of the electromagnetic data evaluation model; initializing the number of training times to 1;

[0043] S32, randomly sampling the offline data set to obtain a sampled data set; inputting the sampled data set into the electromagnetic data evaluation model; the sampled data set includes a plurality of game quintuples;

[0044] S33, using the evaluation module and the decision network to process the input sample data set to obtain a corresponding output;

[0045] S34, using a parameter updating module to update parameters of the first evaluation network using a first parameter updating model;

[0046] S35, using a parameter updating module to update parameters of the target evaluation network using a second parameter updating model;

[0047] S36, using a parameter updating module to update parameters of the second evaluation network using a third parameter updating model;

[0048] S37, using a parameter updating module to update parameters of the decision network using a fourth parameter updating model;

[0049] S38, increase the training times value by 1, and determine whether the training times value is greater than the set training threshold. If it is not greater than the training threshold, execute S32; if it is greater than the training threshold, determine that the decision network is an electromagnetic game model.

[0050] The calculation expression of the first parameter updating model is:

[0051]

[0052] Where θ represents the parameters of the first evaluation network, l 1 represents the update step size of the first evaluation network, Indicates L 1 The gradient of the θ parameter in (θ), L 1 (θ) represents the loss function of the first evaluation network, represents the parameter update value of the first evaluation network, Indicates the use of Updating the parameter θ of the first evaluation network;

[0053] The second parameter updates the model, and its calculation expression is:

[0054]

[0055] Wherein, ω represents the update coefficient value of the target evaluation network, represents the parameter value of the target evaluation network, represents the updated parameter value of the target evaluation network;

[0056] The calculation expression of the third parameter updating model is:

[0057]

[0058] Among them, l 2 represents the update step size of the second evaluation network, φ represents the parameters of the second evaluation network, Indicates L 2 The gradient of the φ parameter in (φ), L 2 (φ) represents the loss function of the second evaluation network, represents the parameter update value of the second evaluation network, Indicates the use of Updating the parameter φ of the second evaluation network;

[0059] The calculation expression of the fourth parameter updating model is:

[0060]

[0061] Among them, ψ represents the parameters of the decision network, l 3 represents the update step size of the decision network, Indicates L 3 The gradient of the ψ parameter in (ψ) is calculated, L 3 (ψ) represents the loss function of the decision network, represents the parameter update value of the decision network, Indicates the use of The parameter ψ of the decision network is updated.

[0062] The loss function L of the first evaluation network 1 (θ), its calculation expression is:

[0063]

[0064] Wherein, α is a weight value, Q(s,a) represents the output of the first evaluation network after s and a in the game quintuple are input into the first evaluation network, represents the operator estimation value of the first evaluation network, γ represents the discount rate, D′ represents the first scrambled data set, D″ represents the second scrambled data set, the first scrambled data set includes a number of first scrambled quintuples, the second scrambled data set includes a number of second scrambled quintuples, the first scrambled quintuple is represented by (s′, a′, _s′, r′, done′), the second scrambled quintuple is represented by (s″, a″, _s″, r″, done″), Q(s′, a′) represents the output of the first evaluation network after s′ and a′ are input to the first evaluation network, Q(s″, a″) represents the output of the first evaluation network after s″ and a″ are input to the first evaluation network, E s′,a′～D′,s″,a″～D″ represents the expected calculation of Q(s′,a′)-Q(s″,a″) corresponding to all s′, a′, s″ and a″ in the first scrambled data set and the second scrambled data set, E s,a～D Indicates the corresponding values ​​of all s and a in the sample data set Perform expectation calculations;

[0065] The loss function L of the second evaluation network is 2 (φ), its calculation expression is:

[0066]

[0067] in, represents the output of the target evaluation network after s and a in the game quintuple are input into the target evaluation network, V(s) represents the output of the second evaluation network after s in the game quintuple is input into the second evaluation network, Indicates the corresponding values ​​of all s and a in the sample data set The second-order norm value of is used to obtain the τ quantile, where τ is the preset quantile value, 0<τ<1;

[0068] The loss function L of the decision network 3 (ψ), its calculation expression is:

[0069]

[0070] Among them, λ is the calculation weight of the loss function of the decision network, α′ is the calculation weight of the loss function of the decision network, represents the variable s in the game quintuple in the second scrambled data set, represents the variable a in the game quintuple in the second scrambled data set, Indicates that After inputting into the decision network, the probability of game action a1 is obtained. Indicates that a1 and After being input into the first evaluation network, the output of the first evaluation network, λ is the proportional coefficient of the loss function of the decision network, Indicates that the After input into the decision network, the game action is obtained The probability of Indicates that all the second scrambled data sets And all when After inputting into the decision network, the game action a1 is obtained, corresponding to all value, perform expected calculation, Indicates that all the second scrambled data sets Corresponding to all Take the value and perform the expected calculation.

[0071] A second aspect of an embodiment of the present invention discloses an offline updated electromagnetic game device, the device comprising:

[0072] A memory storing executable program code;

[0073] a processor coupled to the memory;

[0074] The processor calls the executable program code stored in the memory to execute the offline updated electromagnetic game device.

[0075] A third aspect of an embodiment of the present invention discloses a computer storable medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the offline updated electromagnetic game device.

[0076] A fourth aspect of an embodiment of the present invention discloses an information data processing terminal, which is used to implement the offline updated electromagnetic game device.

[0077] The beneficial effects of the present invention are:

[0078] The method outputs action decisions for electromagnetic games in actual environments by dynamically updating and historical electromagnetic interaction data. The present invention applies the technology of accurate estimation of game parameters within the distribution and conservative estimation of game parameter values ​​outside the distribution, avoiding the failure of electromagnetic game strategies due to distribution offset and improving the game effect of game strategies.

[0079] The present invention uses electromagnetic historical interaction data through a dynamic update method to learn electromagnetic game strategies. This reduces the difficulty of learning electromagnetic games and improves the learning efficiency of electromagnetic game strategies. Secondly, the present invention uses a distribution interpolation method based on conservative value estimation, which reduces the problem of distribution over-estimation caused by distribution offset when the strategy is being learned, improves the efficiency and quality of dynamic updates, and enhances the effectiveness of electromagnetic games. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 1 is a flow chart for implementing the method of the present invention;

[0081] Figure 2 It is a structural diagram of the electromagnetic data evaluation model of the present invention. DETAILED DESCRIPTION

[0082] In order to better understand the content of the present invention, an embodiment is given here. Figure 1 1 is a flow chart for implementing the method of the present invention; Figure 2 It is a structural diagram of the electromagnetic data evaluation model of the present invention.

[0083] A first aspect of an embodiment of the present invention discloses a game processing method and device for electromagnetic data, the method comprising:

[0084] S1, obtaining an electromagnetic game data set; the electromagnetic game data set includes a plurality of electromagnetic game data;

[0085] S2, using the electromagnetic game data set to construct an offline data set;

[0086] S3, using the offline data set to train the constructed electromagnetic data evaluation model to obtain an electromagnetic game model;

[0087] S4, using the electromagnetic game model, processing the acquired electromagnetic parameter information to obtain electromagnetic game information;

[0088] The electromagnetic game data includes: wavelength, frequency band, pulse width, pulse repetition frequency, power, game action data and game effect data;

[0089] The electromagnetic parameter information includes: wavelength, frequency band, pulse width, pulse repetition frequency, and power;

[0090] The electromagnetic game information includes: game signal carrier frequency, game signal transmission pulse power, and expected value of electromagnetic parameter change after the game.

[0091] The electromagnetic game information is used to change the parameters of the target's electromagnetic signal.

[0092] The S1 can be obtained from the historical record data of the electromagnetic game process.

[0093] The method of using the electromagnetic game data set to construct an offline data set includes:

[0094] S21, encode each electromagnetic game data to obtain a corresponding game quintuple; the game quintuple is represented by (s, a, _s, r, done), where s represents the target electromagnetic parameter information before the game signal is sent, a represents the game action information taken on the target, including the game signal carrier frequency and the game signal power, _s represents the target electromagnetic parameter information after the game signal is sent, r represents the real-time profit value of the electromagnetic game process, done represents whether the game is successful for the target, if the game is successful for the target, done takes the value of 1, if the game is unsuccessful for the target, done takes the value of 0;

[0095] The game action data is represented by a; the game effect data is represented by _s, r and done.

[0096] The real-time profit value of the electromagnetic game process can be obtained by using the target electromagnetic parameter information at the current moment and the previous moment through an electromagnetic game effect evaluation method, which can be the sum of the squares of the change values ​​of the target electromagnetic parameter information at the current moment and the previous moment.

[0097] The game effect data can be obtained by using the target electromagnetic parameter information at the current moment and the previous moment through an electromagnetic game effect evaluation method;

[0098] S22, merging all the game quintuples to obtain an offline data set.

[0099] The electromagnetic data evaluation model includes: an input module, an evaluation module, a decision network and a parameter updating module; the evaluation module includes a first evaluation network, a second evaluation network and a target evaluation network;

[0100] The input module receives the offline data set using the input port and sends the offline data set to the corresponding network using each output port;

[0101] The input module comprises an input end, a first output end, a second output end and a third output end;

[0102] The first input terminal of the second evaluation network is connected to the first output terminal of the input module;

[0103] The first input terminal of the first evaluation network is connected to the second output terminal of the input module;

[0104] The first input terminal of the decision network is connected to the third output terminal of the input module;

[0105] The parameter updating module comprises a first input terminal, a second input terminal, a third input terminal, a fourth input terminal, a first output terminal, a second output terminal, and a third output terminal;

[0106] The second input terminal of the second evaluation network is connected to the first output terminal of the parameter updating module;

[0107] The second input terminal of the first evaluation network is connected to the second output terminal of the parameter updating module;

[0108] The second input terminal of the decision network is connected to the third output terminal of the parameter updating module;

[0109] The output terminal of the second evaluation network is connected to the first input terminal of the parameter updating module;

[0110] The first output terminal of the first evaluation network is connected to the second input terminal of the parameter updating module;

[0111] The first output terminal of the decision network is connected to the third input terminal of the parameter updating module;

[0112] The second output terminal of the first evaluation network is connected to the input terminal of the target evaluation network;

[0113] The third input terminal of the first evaluation network is connected to the second output terminal of the decision network;

[0114] The fourth input terminal of the parameter updating module is used to receive the offline data set.

[0115] The decision network comprises: a first input layer, a first hidden layer, a second hidden layer, a third hidden layer and a first output layer;

[0116] The second evaluation network comprises: a second input layer, a fourth hidden layer, a fifth hidden layer, a sixth hidden layer and a second output layer;

[0117] The first evaluation network comprises: a third input layer, a seventh hidden layer, an eighth hidden layer, a ninth hidden layer and a third output layer;

[0118] The target evaluation network includes: a fourth input layer, a tenth hidden layer, an eleventh hidden layer, a twelfth hidden layer and a fourth output layer.

[0119] The network structures of the target evaluation network and the first evaluation network are completely the same.

[0120] The hidden layers are all implemented using a fully connected network.

[0121] ReLU is used as the activation function between each hidden layer to realize parameter transfer.

[0122] The second evaluation network is used to process the variable s in the input game quintuple to obtain the estimated value V(s) of the variable s; and update its parameters using the parameter update value of the second evaluation network sent by the received parameter update module;

[0123] The processing method of the second evaluation network, the first evaluation network, the target evaluation network, and the decision network may be a neural network forward propagation algorithm;

[0124] The first evaluation network processes s and a in the game quintuple to obtain an estimated value Q(s,a) of the state-action pair; updates its parameters using the parameter update value of the first evaluation network sent by the parameter update module; the s variable processed by the first evaluation network is obtained from the game quintuple, and the a variable processed by the first evaluation network is the game action a output by the decision network;

[0125] The decision network is used to process s in the game quintuple to obtain a probability value π(a|s) of taking game action a for s; and update its parameters using the parameter update value of the decision network sent by the parameter update module;

[0126] The target evaluation network processes s and a in the game quintuple to obtain an estimated value of the state-action pair Using the received parameter update values ​​of the target evaluation network sent by the parameter update module, the parameters of the target evaluation network are updated;

[0127] The parameter updating module performs scrambling processing on the sampled data set to obtain a first scrambled data set and a second scrambled data set and outputs them; uses a first parameter updating model to process the first evaluation network, the sampled data set, the first scrambled data set and the second scrambled data set to obtain a parameter update value of the first evaluation network; uses a second parameter updating model to process the target evaluation network and the sampled data set to obtain a parameter update value of the target evaluation network; uses a third parameter updating model to process the second evaluation network and the sampled data set to obtain a parameter update value of the second evaluation network; uses a fourth parameter updating model to process the decision network, the first scrambled data set, the second scrambled data set and the sampled data set to obtain a parameter update value of the decision network; the disturbance amount applied to the second scrambled data set is greater than the disturbance amount applied to the first scrambled data set.

[0128] The scrambling the offline data set to obtain a scrambled data set includes:

[0129] A disturbance amount is randomly generated, and the disturbance amount is added to each element of the offline data set to obtain the scrambled data set.

[0130] The offline data set is used to train the constructed electromagnetic data evaluation model to obtain an electromagnetic game model, including:

[0131] S31, randomly initializing the parameters of the evaluation module and decision network of the electromagnetic data evaluation model; initializing the number of training times to 1;

[0132] S32, randomly sampling the offline data set to obtain a sampled data set; inputting the sampled data set into the electromagnetic data evaluation model; the sampled data set includes a plurality of game quintuples;

[0133] S33, using the evaluation module and the decision network to process the input sample data set to obtain a corresponding output;

[0134] S34, using a parameter updating module to update parameters of the first evaluation network using a first parameter updating model;

[0135] S35, using a parameter updating module to update parameters of the target evaluation network using a second parameter updating model;

[0136] S36, using a parameter updating module to update parameters of the second evaluation network using a third parameter updating model;

[0137] S37, using a parameter updating module to update parameters of the decision network using a fourth parameter updating model;

[0138] S38, increase the training times value by 1, and determine whether the training times value is greater than the set training threshold. If it is not greater than the training threshold, execute S32; if it is greater than the training threshold, determine that the decision network is an electromagnetic game model.

[0139] The calculation expression of the first parameter updating model is:

[0140]

[0141] Where θ represents the parameters of the first evaluation network, l 1 represents the update step size of the first evaluation network, Indicates L 1 The gradient of the θ parameter in (θ), L 1 (θ) represents the loss function of the first evaluation network, represents the parameter update value of the first evaluation network, Indicates the use of Updating the parameter θ of the first evaluation network;

[0142] The second parameter updates the model, and its calculation expression is:

[0143]

[0144] Wherein, ω represents the update coefficient value of the target evaluation network, represents the parameter value of the target evaluation network, represents the updated parameter value of the target evaluation network, It indicates that the parameter values ​​of the target evaluation network are updated by using the updated parameter values ​​of the target evaluation network.

[0145] The calculation expression of the third parameter updating model is:

[0146]

[0147] Among them, l 2 represents the update step size of the second evaluation network, φ represents the parameters of the second evaluation network, Indicates L 2 The gradient of the φ parameter in (φ), L 2 (φ) represents the loss function of the second evaluation network, represents the parameter update value of the second evaluation network, Indicates the use of Updating the parameter φ of the second evaluation network;

[0148] The calculation expression of the fourth parameter updating model is:

[0149]

[0150] Among them, ψ represents the parameters of the decision network, l 3 represents the update step size of the decision network, Indicates L 3 The gradient of the ψ parameter in (ψ) is calculated, L 3 (ψ) represents the loss function of the decision network, represents the parameter update value of the decision network, Indicates the use of Updating the parameter ψ of the decision network;

[0151] The loss function L of the first evaluation network 1 (θ), its calculation expression is:

[0152]

[0153] Wherein, α is a weight value, Q(s,a) represents the output of the first evaluation network after s and a in the game quintuple are input into the first evaluation network, represents the operator estimation value of the first evaluation network, γ represents the discount rate, D′ represents the first scrambled data set, D″ represents the second scrambled data set, the first scrambled data set includes a number of first scrambled quintuples, the second scrambled data set includes a number of second scrambled quintuples, the first scrambled quintuple is represented by (s′, a′, _s′, r′, done′), the second scrambled quintuple is represented by (s″, a″, _s″, r″, done″), Q(s′, a′) represents the output of the first evaluation network after s′ and a′ are input to the first evaluation network, Q(s″, a″) represents the output of the first evaluation network after s″ and a″ are input to the first evaluation network, E s′,a′～D′,s″,a″～D″ represents the expected calculation of Q(s′,a′)-Q(s″,a″) corresponding to all s′, a′, s″ and a″ in the first scrambled data set and the second scrambled data set, E s,a～D Indicates the corresponding values ​​of all s and a in the sample data set Perform expectation calculations.

[0154] The loss function L of the second evaluation network is 2 (φ), its calculation expression is:

[0155]

[0156] in, represents the output of the target evaluation network after s and a in the game quintuple are input into the target evaluation network, V(s) represents the output of the second evaluation network after s in the game quintuple is input into the second evaluation network, Indicates the corresponding values ​​of all s and a in the sample data set The second-order norm of is used to obtain the τ quantile, where τ is the preset quantile value, 0<τ<1. Specifically,

[0157] The loss function L of the decision network 3 (ψ), its calculation expression is:

[0158]

[0159] Among them, α′ is the calculation weight of the loss function of the decision network, represents the variable s in the game quintuple in the second scrambled data set, represents the variable a in the game quintuple in the second scrambled data set, Indicates that After inputting into the decision network, the probability of game action a1 is obtained. Indicates that a1 and After being input into the first evaluation network, the output of the first evaluation network, λ is the proportional coefficient of the loss function of the decision network, Indicates that the After input into the decision network, the game action is obtained The probability of Indicates that all the second scrambled data sets And all when After inputting into the decision network, the game action a1 is obtained, corresponding to all value, perform expected calculation, Indicates that all the second scrambled data sets Corresponding to all Take the value and perform the expected calculation.

[0160] In the above S3, the present invention adopts the method of alternately training the evaluation network and the decision network. The input of this step is the offline data set constructed in the second step. After training using the offline reinforcement learning algorithm, an evaluator network and a decision maker network are output. This step provides a basic electromagnetic game strategy by learning the electromagnetic game strategy on the offline data set.

[0161] The offline reinforcement learning algorithm used in this step is now introduced in detail. In this step, in each alternating training, each network parameter is updated in sequence according to the gradient back propagation algorithm of the loss function in the order of the first evaluation network, the target evaluation network, the second evaluation network, and the π-strategy network.

[0162] When updating the first evaluation network parameters, the following formula is used:

[0163]

[0164] Among them l 1 Represents the update step size of the first evaluation network. represents the operator for finding the gradient of the network parameter θ. During the update process, θ will gradually minimize L 1 The estimated value of (θ). The loss function L 1 The calculation formula of (θ) is:

[0165]

[0166] Where α is the weight, which is usually greater than 0. D′, D″ are scrambled data sets, which are obtained by adding different degrees of perturbations to the offline data set, where D′ corresponds to large perturbations and D″ corresponds to small perturbations. represents the estimate of the Bellman operator, and defines γ is the discount rate. Because the scrambling degree of the D″ dataset is relatively small, it is believed that the scrambled state-action pair (s″, a″) is still within the distribution implied by the offline dataset; the scrambling degree of the D′ dataset is relatively large, it is believed that the scrambled state-action pair (s ′ ,a ′ ) will most likely fall outside the distribution implied by the offline dataset. Q (θ) process, will minimize and E s′,a′～D′ [Q(s′,a′)], maximize E s″,a″～D″ [Q(s″,a″)]. When the first evaluation network estimates the Q value of the state-action pair (s, a) in the offline data set, it will gradually approach the true value; when minimizing E s′,a′～D′ When [Q(s′,a′)], the first evaluation network will reduce the Q value estimate of the state-action pair (s,a) outside the distribution of the offline data set, and achieve a conservative estimate of the state-action pair (s,a) in the entire space; when maximizing E s″,a″～D″[Q(s″,a″)], the first evaluation network will increase the Q value estimate of the state-action pair (s,a) within the offline data set distribution, offsetting the effect of the previous conservative estimate, and achieve accurate estimation of the Q value within the offline data set distribution. In general, during the parameter update process, the first evaluation network will tend to be accurate in estimating the Q value of the state-action pair (s,a) within the offline data set distribution, and will tend to be conservative in estimating the Q value of the state-action pair (s,a) outside the offline data set distribution, thereby avoiding the problem of over-estimation of the Q value caused by distribution shift.

[0167] When the target evaluation network parameters are updated, the target evaluation network is updated according to the parameters of the current first evaluation network:

[0168]

[0169] Usually, ω is selected to be a value between 0 and 1 and close to 0, such as 0.005. Therefore, after each update of the first evaluation network, only a small part of the update effect will be transferred to the target evaluation network, so the iterative update speed of the target evaluation network is much slower than that of the first evaluation network. After sufficient iterations, the target evaluation network can have similar properties to the first evaluation network, that is, the Q value estimation of the state-action pair (s, a) within the distribution of the offline data set will tend to be accurate, and the Q value estimation of the state-action pair (s, a) outside the distribution of the offline data set will tend to be conservative.

[0170] When updating the second evaluation network parameters, use the formula:

[0171]

[0172] Among them l 2 Represents the update step size of the second evaluation network. represents the operator that the network uses to find the gradient of the parameter φ. In the parameter update process, the loss function L 2 The calculation formula of (φ) is:

[0173]

[0174] in is the τ quantile of the second-order norm. After sufficient iterations, the second evaluation network can have similar properties to the target evaluation network, that is, the V value estimation of the state s within the distribution of the offline dataset will tend to be accurate, and the V value estimation of s outside the distribution of the offline dataset will tend to be conservative.

[0175] When deciding network parameter updates, use the formula:

[0176]

[0177] Among them l 3Represents the update step size of the decision network. represents the network's gradient operator for parameter ψ. During the update process, ψ will gradually minimize L 3 The estimated value of (ψ). Among them, the loss function L 3 The calculation formula of (ψ) is:

[0178]

[0179] Where D″ is a scrambled data set with a small disturbance, and the state-action pair obtained after the scramble is considered to be Still within the distribution of the data set; α′ and λ are weights, usually constants greater than 0. π (ψ) process, will be minimized Maximize and In minimization When , the strategy π will be more similar to the strategy contained in the offline data set, which can avoid the strategy π from selecting a strategy that has not appeared in the offline data set to a certain extent; When , strategy π will try to choose actions with larger Q value estimates to achieve the optimization of the game strategy; when maximizing , the strategy π will explore as many different strategies as possible, realizing the continuous exploration of diverse strategies in the strategy learning process.

[0180] This alternating update process will continue until the number of updates reaches a preset value.

[0181] In the actual working environment, decisions are made according to the output strategy of the trained game decision model, the relevant state s of the current game target is input to the decision network, and the decision network outputs the decision action a. The electromagnetic game device generates the corresponding electromagnetic game signal based on the information, and collects the relevant state _s at the next moment. This step is repeated until the electromagnetic game process ends.

[0182] The initialization parameters of the target evaluation network are the same as those of the first evaluation network, but its iterative update speed is much slower than that of the first evaluation network.

[0183] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A game processing method for electromagnetic data, characterized in that: include: S1, obtaining an electromagnetic game data set; the electromagnetic game data set includes a plurality of electromagnetic game data; S2, using the electromagnetic game data set to construct an offline data set; S3, using the offline data set to train the constructed electromagnetic data evaluation model to obtain an electromagnetic game model; S4, using the electromagnetic game model, processing the acquired electromagnetic parameter information to obtain electromagnetic game information; The electromagnetic game information includes: game signal carrier frequency, game signal transmission pulse power, and expected value of electromagnetic parameter change after the game.

2. The game processing method of electromagnetic data according to claim 1, characterized in that: The method of using the electromagnetic game data set to construct an offline data set includes: S21, encode each electromagnetic game data to obtain a corresponding game quintuple; the game quintuple is represented by (s, a, _s, r, done), where s represents the target electromagnetic parameter information before the game signal is sent, a represents the game action information taken on the target, including the game signal carrier frequency and the game signal power, _s represents the target electromagnetic parameter information after the game signal is sent, r represents the real-time profit value of the electromagnetic game process, done represents whether the game is successful for the target, if the game is successful for the target, done takes the value of 1, if the game is unsuccessful for the target, done takes the value of 0; S22, merging all game quintuples to obtain an offline data set.

3. The game processing method of electromagnetic data according to claim 2, characterized in that: The electromagnetic data evaluation model includes: an input module, an evaluation module, a decision network and a parameter updating module; the evaluation module includes a first evaluation network, a second evaluation network and a target evaluation network; The input module receives the offline data set using the input port and sends the offline data set to the corresponding network using each output port; The input module comprises an input end, a first output end, a second output end and a third output end; The first input terminal of the second evaluation network is connected to the first output terminal of the input module; The first input terminal of the first evaluation network is connected to the second output terminal of the input module; The first input terminal of the decision network is connected to the third output terminal of the input module; The parameter updating module comprises a first input terminal, a second input terminal, a third input terminal, a fourth input terminal, a first output terminal, a second output terminal, and a third output terminal; The second input terminal of the second evaluation network is connected to the first output terminal of the parameter updating module; The second input terminal of the first evaluation network is connected to the second output terminal of the parameter updating module; The second input terminal of the decision network is connected to the third output terminal of the parameter updating module; The output terminal of the second evaluation network is connected to the first input terminal of the parameter updating module; The first output terminal of the first evaluation network is connected to the second input terminal of the parameter updating module; The first output terminal of the decision network is connected to the third input terminal of the parameter updating module; The second output terminal of the first evaluation network is connected to the input terminal of the target evaluation network; The third input terminal of the first evaluation network is connected to the second output terminal of the decision network; The fourth input terminal of the parameter updating module is used to receive the offline data set.

4. The game processing method of electromagnetic data according to claim 3, characterized in that: The decision network comprises: a first input layer, a first hidden layer, a second hidden layer, a third hidden layer and a first output layer; The second evaluation network comprises: a second input layer, a fourth hidden layer, a fifth hidden layer, a sixth hidden layer and a second output layer; The first evaluation network comprises: a third input layer, a seventh hidden layer, an eighth hidden layer, a ninth hidden layer and a third output layer; The target evaluation network comprises: a fourth input layer, a tenth hidden layer, an eleventh hidden layer, a twelfth hidden layer and a fourth output layer; The hidden layers are all implemented using a fully connected network; The second evaluation network is used to process the variable s in the input game quintuple to obtain the estimated value V(s) of the variable s; and update its parameters using the parameter update value of the second evaluation network sent by the received parameter update module; The first evaluation network processes s and a in the game quintuple to obtain an estimated value Q(s,a) of the state-action pair; updates its parameters using the parameter update value of the first evaluation network sent by the parameter update module; the s variable processed by the first evaluation network is obtained from the game quintuple, and the a variable processed by the first evaluation network is the game action a output by the decision network; The decision network is used to process s in the game quintuple to obtain a probability value π(a|s) of taking game action a for s; and update its parameters using the parameter update value of the decision network sent by the parameter update module; The target evaluation network processes s and a in the game quintuple to obtain an estimated value of the state-action pair Using the received parameter update values ​​of the target evaluation network sent by the parameter update module, the parameters of the target evaluation network are updated; The parameter updating module is used to scramble the sampled data set to obtain and output the first scrambled data set and the second scrambled data set; use the first parameter updating model to process the first evaluation network, the sampled data set, the first scrambled data set and the second scrambled data set to obtain the parameter update value of the first evaluation network; use the second parameter updating model to process the target evaluation network and the sampled data set to obtain the parameter update value of the target evaluation network; use the third parameter updating model to process the second evaluation network and the sampled data set to obtain the parameter update value of the second evaluation network; use the fourth parameter updating model to process the decision network, the first scrambled data set, the second scrambled data set and the sampled data set to obtain the parameter update value of the decision network.

5. The game processing method of electromagnetic data according to claim 4, characterized in that: The offline data set is used to train the constructed electromagnetic data evaluation model to obtain an electromagnetic game model, including: S31, randomly initializing the parameters of the evaluation module and decision network of the electromagnetic data evaluation model; initializing the number of training times to 1; S32, randomly sampling the offline data set to obtain a sampled data set; inputting the sampled data set into the electromagnetic data evaluation model; the sampled data set includes a plurality of game quintuples; S33, using the evaluation module and the decision network to process the input sample data set to obtain a corresponding output; S34, using a parameter updating module to update parameters of the first evaluation network using a first parameter updating model; S35, using a parameter updating module to update parameters of the target evaluation network using a second parameter updating model; S36, using a parameter updating module to update parameters of the second evaluation network using a third parameter updating model; S37, using a parameter updating module to update parameters of the decision network using a fourth parameter updating model; S38, increase the training times value by 1, and determine whether the training times value is greater than the set training threshold. If it is not greater than the training threshold, execute S32; if it is greater than the training threshold, determine that the decision network is an electromagnetic game model.

6. The game processing method of electromagnetic data according to claim 5, characterized in that: The calculation expression of the first parameter updating model is: Wherein, θ represents the parameters of the first evaluation network, l1 represents the update step size of the first evaluation network, V θ represents the gradient of the θ parameter in L1(θ), L1(θ) represents the loss function of the first evaluation network, represents the parameter update value of the first evaluation network, Indicates the use of Updating the parameter θ of the first evaluation network; The second parameter updates the model, and its calculation expression is: Wherein, ω represents the update coefficient value of the target evaluation network, represents the parameter value of the target evaluation network, represents the updated parameter value of the target evaluation network; The calculation expression of the third parameter updating model is: Wherein, l2 represents the update step size of the second evaluation network, φ represents the parameters of the second evaluation network, represents the gradient of the φ parameter in L2(φ), L2(φ) represents the loss function of the second evaluation network, represents the parameter update value of the second evaluation network, Indicates the use of Updating the parameter φ of the second evaluation network; The calculation expression of the fourth parameter updating model is: Where, ψ represents the parameters of the decision network, l3 represents the update step size of the decision network, represents the gradient of the ψ parameter in L3(ψ), L3(ψ) represents the loss function of the decision network, represents the parameter update value of the decision network, Indicates the use of The parameter ψ of the decision network is updated.

7. The game processing method of electromagnetic data according to claim 6, characterized in that: The loss function L1(θ) of the first evaluation network is calculated as follows: Wherein, α is a weight value, Q(s, a) represents the output of the first evaluation network after s and a in the game quintuple are input into the first evaluation network, represents the operator estimation value of the first evaluation network, γ represents the discount rate, D′ represents the first scrambled data set, D″ represents the second scrambled data set, the first scrambled data set includes a number of first scrambled quintuples, the second scrambled data set includes a number of second scrambled quintuples, the first scrambled quintuple is represented by (s′, a′, _s′, r′, done′), the second scrambled quintuple is represented by (s″, a″, _s″, r″, done″), Q(s′, a′) represents the output of the first evaluation network after s′ and a′ are input to the first evaluation network, Q(s″, a″) represents the output of the first evaluation network after s″ and a″ are input to the first evaluation network, E s′，a′～D′，s″，a″～D″ represents the expected calculation of Q(s′, a′)-Q(s″, a″) corresponding to all s′, a′, s″ and a″ in the first scrambled data set and the second scrambled data set, E s，a～D Indicates the corresponding values ​​of all s and a in the sample data set Perform expectation calculations; The loss function L2(φ) of the second evaluation network is calculated as follows: in, represents the output of the target evaluation network after s and a in the game quintuple are input into the target evaluation network, V(s) represents the output of the second evaluation network after s in the game quintuple is input into the second evaluation network, Indicates the corresponding values ​​of all s and a in the sample data set The second-order norm value of is used to obtain the τ quantile, where τ is the preset quantile value, 0<τ<1; The loss function L3(ψ) of the decision network is calculated as follows: Among them, λ is the calculation weight of the loss function of the decision network, α′ is the calculation weight of the loss function of the decision network, represents the variable s in the game quintuple in the second scrambled data set, represents the variable a in the game quintuple in the second scrambled data set, Indicates that After inputting into the decision network, the probability of game action a1 is obtained. Indicates that a1 and After being input into the first evaluation network, the output of the first evaluation network, λ is the proportional coefficient of the loss function of the decision network, Indicates that the After input into the decision network, the game action is obtained The probability of Indicates that all the second scrambled data sets And all when After inputting into the decision network, the game action a1 is obtained, corresponding to all value, perform expected calculation, Indicates that all the second scrambled data sets Corresponding to all Take the value and perform the expected calculation.

8. An electromagnetic gaming device for offline updating, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the game processing method for electromagnetic data as described in any one of claims 1-7.

9. A computer storable medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the game processing method for electromagnetic data as described in any one of claims 1-7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the game processing method of electromagnetic data described in any one of claims 1-7.

Images