Model training method and device

Abstract

The embodiment of the invention provides a model training method and device, and the method comprises the steps: obtaining a plurality of pieces of base station sample data, the base station sample data comprises first state data, first action data, first reward data, and second state data, the first state data is to-be-audited data obtained by processing basic data, equipment power consumption data and historical electricity charge data; the first action data is action data obtained based on the first state data; repeatedly executing the following steps until a preset training termination condition is met, and obtaining a target neural network model: inputting the first state data into the neural network model, and obtaining a predicted action value based on the first action data; inputting the second state data into the neural network model to obtain a target action value of second action data corresponding to the second state data; and updating the weight of the neural network model according to the predicted action value and the target action value, so as to train the neural network model according to the updated weight.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a model training method and device. Background Art

[0002] At present, the existing technical scheme for auditing base station electricity charges is as follows:

[0003] Comprehensive base station power consumption and attribute data are obtained through smart meters and contract equipment data. Feature vectors are obtained through anomaly detection and processing, data completion and correction, standardization conversion and other steps. The feature vectors are weighted, the average distance of the feature vectors is calculated, and the feature vectors are modeled to form a reference set. The audit records are standardized, and the nearest neighbor algorithm with a variable radius is used to find the reference set. The electricity bill reimbursement is judged based on the threshold whether it is abnormal, the audit results are output, and the records exceeding the standard are sent back for processing.

[0004] The auditing methods of the prior art have the following disadvantages:

[0005] The effective feature combination selected by the existing technology makes the model's prediction accuracy low; and when processing large amounts of electricity consumption data, the existing technology is unable to make predictions and decisions efficiently; and the audit process requires manual participation, which is time-consuming and error-prone, and has low efficiency and accuracy. Summary of the invention

[0006] In view of the above problems, the embodiments of the present application provide a model training method and device that overcome the above problems or at least partially solve the above problems.

[0007] In a first aspect, an embodiment of the present application provides a model training method, the method comprising:

[0008] Acquire a base station sample data set, wherein the base station sample data set includes a plurality of base station sample data, and each base station sample data includes: first state data, first action data, first reward data, and second state data after the first state data; the first state data is the data to be audited obtained after processing the basic data of the base station, the power consumption data of the equipment, and the historical electricity fee data; the first action data is the action data obtained by the neural network model from a plurality of action data based on the first state data; the first reward data is the reward data obtained by the neural network model based on the first action data; and the second state data is the next state data obtained by the neural network model based on the first action data;

[0009] Repeat the following steps until the preset training termination conditions are met to obtain the trained target neural network model:

[0010] Inputting the first state data into a neural network model, and obtaining a predicted action value based on the first action data;

[0011] Inputting the second state data into the neural network model to obtain a target action value of second action data corresponding to the second state data;

[0012] The weights of the neural network model are updated according to the predicted action value and the target action value, so as to train the neural network model according to the updated weights.

[0013] Optionally, the step of inputting the second state data into the neural network model to obtain a target action value of the second action data corresponding to the second state data includes:

[0014] Inputting the second state data into the neural network model to obtain a first random number;

[0015] When the first random number is greater than or equal to the first exploration rate, obtaining action values ​​corresponding to the plurality of action data through the neural network model;

[0016] Determine the action data corresponding to the maximum value among the multiple action values ​​as the second action data;

[0017] The target action value is determined according to the action value corresponding to the second action data.

[0018] Optionally, the method further comprises:

[0019] Calculating an error value between the predicted action value and the target action value;

[0020] When the error value is greater than the second exploration rate, updating the first exploration rate by a decay coefficient;

[0021] The second exploration rate is less than or equal to the first exploration rate.

[0022] Optionally, the target action value is calculated by the following formula:

[0023] Q2=M+γ*Q1

[0024] Among them, Q2 represents the target action value;

[0025] M represents the first reward data;

[0026] γ represents the discount factor;

[0027] Q1 represents the action value corresponding to the second action data.

[0028] Optionally, the method for acquiring the plurality of base station sample data includes:

[0029] Obtain basic data of base stations, equipment user data, and historical electricity bill data;

[0030] Processing the basic data, the device user data and the historical electricity fee data to obtain a plurality of first status data to be audited;

[0031] Input each first state data into the neural network model to obtain first action data corresponding to the first state data, and execute the first action data to obtain first reward data and second state data;

[0032] The first state data, the first action data, the first reward data and the second state data are combined into a base station sample data to obtain a plurality of base station sample data.

[0033] Optionally, inputting each first state data into the neural network model to obtain first action data corresponding to the first state data, and executing the first action data to obtain first reward data and second state data includes:

[0034] Input each first state data into the neural network model to obtain a second random number corresponding to each first state data;

[0035] When the second random number is less than the first exploration rate, determining one of the plurality of action data as the first action data;

[0036] When the second random number is greater than or equal to the first exploration rate, obtaining an action value corresponding to each action data in a plurality of action data through the neural network model, and determining the action data corresponding to the maximum value of the plurality of action values ​​as the first action data;

[0037] The first action data is executed in the neural network model to obtain first reward data and second state data.

[0038] Optionally, the preset training termination condition includes at least one of the following:

[0039] The number of iterations reaches the preset number of iterations;

[0040] A second difference value obtained by subtracting the second error value from the first error value is less than a first preset difference value, the first error value is an error value between the predicted action value and the target action value obtained after the Nth iteration, the second error value is an error value between the predicted action value and the target action value obtained after the Mth iteration, M and N are both values ​​less than or equal to the preset number of iterations, M is greater than N, and the difference between M and N is a preset value;

[0041] When the difference between the error values ​​obtained after two adjacent iterations is less than the second preset difference, the number of the early stop counter is increased by one until the preset number of the early stop counter is met.

[0042] In a second aspect, an embodiment of the present application further provides a model training device, the device comprising:

[0043] An acquisition module is used to acquire a base station sample data set, wherein the base station sample data set includes multiple base station sample data, and each base station sample data includes: first state data, first action data, first reward data, and second state data after the first state data; the first state data is the data to be audited obtained after processing the basic data of the base station, the power consumption data of the equipment, and the historical electricity fee data; the first action data is the action data obtained by the neural network model from multiple action data based on the first state data; the first reward data is the reward data obtained by the neural network model based on the first action data; and the second state data is the next state data obtained by the neural network model based on the first action data;

[0044] The training module is used to repeatedly perform the following steps until the preset training termination conditions are met to obtain the trained target neural network model:

[0045] Inputting the first state data into a neural network model, and obtaining a predicted action value based on the first action data;

[0046] Inputting the second state data into the neural network model to obtain a target action value of second action data corresponding to the second state data;

[0047] The weights of the neural network model are updated according to the predicted action value and the target action value, so as to train the neural network model according to the updated weights.

[0048] In a third aspect, an embodiment of the present application further provides an electronic device, including a memory, a transceiver, and a processor:

[0049] A memory for storing a computer program; a transceiver for sending and receiving data under the control of a processor; and a processor for reading the computer program in the memory and executing the method as described in the first aspect above.

[0050] In a fourth aspect, an embodiment of the present application further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the method described in the first aspect above.

[0051] In the above embodiment of the present application, a plurality of base station sample data are first obtained, and each base station sample data includes: first state data, first action data, first reward data, and second state data after the first state data. Then the first state data is input into the neural network model, and a predicted action value is obtained based on the first action data; the second state data is input into the neural network model, and a target action value of the second action data corresponding to the second state data is obtained; according to the predicted action value and the target action value, the weight of the neural network model is updated, so as to train the neural network model according to the updated weight, and the above training steps are repeated until the preset training termination condition is met, and a trained target neural network model is obtained.

[0052] In the above embodiment, the base station sample data comprehensively considers the basic data of the base station, the power consumption data of the equipment, and the historical electricity bill data. By training the model with various data, it can more accurately identify normal and abnormal electricity bill data, reduce the need for manual review, and reduce the possibility of human error. Moreover, by adding reward data to the base station sample data, it can not only ensure that the model can learn the correct audit strategy, but also consider the actual cost. In addition, the first state data, the first action data, the first reward data, and the second state data after the first state data are packaged into a base station sample data, and each base station sample data is a complete set of interactive information to improve the accuracy of model training. In summary, the target neural network model after the above training can automatically analyze a large amount of data and make decisions quickly to improve the efficiency of the entire audit process. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 One of the flow charts of the model training method provided in the embodiment of the present application;

[0055] Figure 2 The second flowchart of the model training method provided in the embodiment of the present application;

[0056] Figure 3 Flowchart 3 of the model training method provided in the embodiment of the present application;

[0057] Figure 4 Flow chart 4 of the model training method provided in the embodiment of the present application;

[0058] Figure 5 A schematic diagram of the composition of the DQNAgent class provided in the embodiment of the present application;

[0059] Figure 6 A schematic diagram of the composition of the Memory class provided in an embodiment of the present application;

[0060] Figure 7 A flowchart of visualization processing and data report generation provided in an embodiment of the present application;

[0061] Figure 8 A structural block diagram of a model training device provided in an embodiment of the present application;

[0062] Fig. 9 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0064] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0065] The model training method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0066] Specifically, the present application embodiment provides a model training method, such as Figure 1 As shown, the following steps may be specifically included:

[0067] Step 101, obtain a base station sample data set, the base station sample data set includes multiple base station sample data, each base station sample data includes: first state data, first action data, first reward data and second state data after the first state data; the first state data is the data to be audited obtained after processing the basic data of the base station, the equipment power consumption data and the historical electricity fee data, the first action data is the action data obtained by the neural network model based on the first state data from multiple action data, the first reward data is the reward data obtained by the neural network model based on the first action data, and the second state data is the next state data obtained by the neural network model based on the first action data.

[0068] Specifically, the basic data of the base station, the equipment power consumption data and the historical electricity bill data are obtained, and the basic data, the equipment power consumption data and the historical electricity bill data are processed to obtain the data to be audited (i.e., the first state data). There are multiple first state data. For each first state data: the first state data is input into a neural network model (such as a DQN model) to obtain the first action data corresponding to the first state data, and the first action data is executed to obtain the first reward data and the second state data after the first state data. The first state data, the first action data, the first reward data and the second state data are combined into a base station sample data. The first action data, the first reward data, and the second state data corresponding to each first state data are obtained in the above manner and combined into a base station sample data, thereby obtaining multiple base station sample data, and the multiple base station sample data are combined into a base station sample data set.

[0069] Step 102, repeatedly perform the following steps until the preset training termination condition is met, and obtain the trained target neural network model:

[0070] Inputting the first state data into the neural network model, and obtaining a predicted action value based on the first action data;

[0071] Inputting the second state data into the neural network model to obtain a target action value of second action data corresponding to the second state data;

[0072] The weights of the neural network model are updated according to the predicted action value and the target action value, so as to train the neural network model according to the updated weights.

[0073] Specifically, during the training process, the first state data in the base station sample data is input into the neural network model, and the corresponding predicted action value (i.e., predicted Q value) is obtained based on the first action data. The second state data in the base station sample data is input into the neural network model to obtain the second action data corresponding to the second state data, and the corresponding target action value (i.e., target Q value) is obtained based on the second action data. The weights of the neural network model are updated by the predicted action value and the target action value, so that the neural network model is iteratively trained according to the updated weights until the preset training termination condition is met to stop the iterative training and obtain the trained target neural network model.

[0074] Set the preset training termination condition. During the model training process, if the preset training termination condition is met, stop the iterative training and determine whether to save the trained neural network model. If the neural network model is saved, call the save() function of the neural network model to save the weights of the current neural network model through the save() function, and output the final performance indicators of the model on the training set and the validation set to obtain the trained target neural network model, and deploy the target neural network model to practical applications, such as the decision module of the base station electricity fee audit system.

[0075] In the above embodiment of the present application, the base station sample data comprehensively considers various data such as the basic data of the base station, the power consumption data of the equipment, and the historical electricity bill data. By training the model with various data, normal and abnormal electricity bill data can be more accurately identified, the need for manual review is reduced, and the possibility of human error is reduced. Moreover, by adding reward data to the base station sample data, it can not only ensure that the model can learn the correct audit strategy, but also consider the actual cost. Moreover, the first state data, the first action data, the first reward data, and the second state data after the first state data are packaged into a base station sample data, and each base station sample data is a complete set of interactive information to improve the accuracy of model training. In summary, the target neural network model after the above training can automatically analyze a large amount of data and make decisions quickly to improve the efficiency of the entire audit process.

[0076] As an optional specific embodiment, the method of acquiring the sample data of multiple base stations in step 101 specifically includes:

[0077] Obtain basic data of base stations, equipment user data, and historical electricity bill data;

[0078] Processing the basic data, the device user data and the historical electricity fee data to obtain a plurality of first status data to be audited;

[0079] Input each first state data into the neural network model to obtain first action data corresponding to the first state data, and execute the first action data to obtain first reward data and second state data;

[0080] The first state data, the first action data, the first reward data and the second state data are combined into a base station sample data to obtain a plurality of base station sample data.

[0081] Specifically, Figure 2 As shown, step 201: obtain the basic data of the base station, the equipment power consumption data, and the historical electricity fee data, and pre-process the above data, as follows:

[0082] like Figure 3 As shown, step 301: by using cron to set a scheduled task, regularly use the read_csv() of the pandas library to read the basic data of the base station, the equipment power consumption data, and the historical electricity fee data from the historical data CSV file downloaded from the equipment system of the base station, and save them in the database to realize real-time data update.

[0083] Among them, cron is a tool for executing commands at a scheduled time. The importdata function is used to import data from a text or data file. Using the importdata function, data can be loaded into a structure containing data.

[0084] The pandas library is a powerful Python data analysis tool library that provides data structures and operations for data processing and analysis. Python is a cross-platform computer programming language that is interpretable, variable, interactive, and object-oriented. read_csv() is a function in the pandas library that is used to read data from a CSV file. A CSV file is a common data format, a plain text file with commas as delimiters between fields and line breaks as delimiters between records.

[0085] The above basic data, equipment power consumption data, and historical electricity bill data are sorted by time and named Rd1, Rd2, and Rd3 respectively. Then the train_test_split() function is used to classify the data set in a 7:3 ratio, dividing Rd1 into training set TrS1 and validation set VaS1, and Rd2 into training set TrS2 and validation set VaS2, where train_size is set to 0.7, validate_size parameter is set to 0.3, and in order to ensure the reproducibility of the results, the random_state parameter is assigned a value of 42, which is used to determine the random process of dividing the data set to ensure that the results of each division are consistent.

[0086] In Python, train_size is usually used to define the size of the training set. It is a floating point number or integer, which represents the ratio of the number of samples used for training to the size of the entire data set. The validate parameter is mainly used to verify the validity of the input data in the user interface. The random_state parameter is mainly to ensure that the same training set and test set are split each time. The size can be any integer.

[0087] Step 302: Use the isnull function to check missing values, and remove missing longitude and latitude, site name, address, power consumption data, equipment model data, etc. Other non-critical missing data can be retained; and use the drop_duplicates function to remove duplicate information based on site name, longitude and latitude, and time as conditions, laying a good foundation for subsequent data analysis and model training. Among them, the isnull function is a computer language function that is used to determine whether an expression is NULL and return the specified default value. The drop_duplicates function compares all columns and deletes duplicate rows, retaining the first row that appears, and does not change the order of the original data. By removing missing and duplicate data, the data quality is improved, the noise of model training is reduced, and the accuracy of model prediction is improved.

[0088] Extract features highly correlated with energy consumption from the above processed basic data, equipment power consumption data, and historical electricity bill data, such as site longitude and latitude, site name, equipment type, equipment load, time, etc. Step 303: Use the sort_values ​​function of pandas to sort the data according to the timestamp to ensure that the data is arranged in chronological order. In addition, use the season function of pandas to classify by date and month to extract seasonal information (spring, summer, autumn, winter). Use the isoweekday function of pandas combined with holiday data to mark working days and rest days. Step 304: Use the shift function to create time series lag features, such as the energy consumption data of the previous day.

[0089] The following is a description of the initialized DQN model and the model training process:

[0090] like Figure 2 As shown, step 202: define the reinforcement learning environment and construct the DQNAgent class and the Memory class. The details are as follows:

[0091] like Figure 4As shown, step 401: define the three key components of the reinforcement learning environment: state space, action space, and reward function, providing a framework for the neural network model. Step 402: construct the DQNAgent class, which is a deep reinforcement learning agent class, including but not limited to: initialization method, model construction method, target model update method, memory method, DQN model decision method, experience playback method, loop training method, and model saving method, etc. Figure 5 Step 403: Construct the Memory class. The Memory class is used to store experience and support priority experience playback, including but not limited to initializing memory methods, adding experience methods, sampling methods, updating priority methods, etc. Figure 6 shown.

[0092] The following is a detailed description of the contents of the state space:

[0093] The state space contains all key information that affects the audit decision, including: basic data of base stations, equipment power consumption data, historical electricity bill data and other information. The basic data includes but is not limited to: longitude and latitude, site name, address, corresponding tower information, equipment quantity and equipment status, etc.; equipment power consumption data includes but is not limited to: base station equipment, equipment type, equipment load, electric meter, equipment operating power reading, etc.; historical electricity bill data includes but is not limited to: historical electricity bill records, which are used to analyze electricity usage patterns.

[0094] Label encoding is performed on the category features (such as site name and address) in the basic data and equipment power consumption data after the above processing to obtain the encoded category feature data, and the category feature data is converted into numerical feature data; the numerical feature data (such as latitude and longitude, equipment load, and meter reading) is normalized, and the numerical feature data is normalized to the interval [0, 1], and the np.concatenate function is used to combine the encoded category feature data and the normalized numerical feature data into a large feature array, and then the flatten function is used to convert the combined multi-dimensional feature array into a one-dimensional state vector suitable for the input of the Deep Q-Network (DQN) model, that is, the audit data. Among them, the DQN model is mainly used to solve decision-making problems with high-dimensional input space, especially in an environment with large spatial state and action space, by learning a value function to guide the best action that the intelligent agent should take in a given state; the np.concatenate function is used to merge series or matrices; the flatten function is a function for reducing the dimensionality of multi-dimensional data.

[0095] The normalization formula is as follows:

[0096]

[0097] Where μ is the mean of the feature;

[0098] σ is the standard deviation;

[0099] X1 is the numerical feature before normalization;

[0100] X2 is the normalized numerical feature.

[0101] The main purpose of the above normalization formula is to standardize the features so that they have zero mean and unit variance, which helps the training of the DQN model.

[0102] The content of the action space is described in detail below.

[0103] The action space defines the action set, which can be defined as {0, 1, 2, 3}, corresponding to "normal data", "abnormal data", "early warning prompt", and "adjust billing strategy". Among them, for normal data: if the DQN model believes that there is no abnormality in the electricity bill data, it means that no further investigation is required, and the corresponding action data is to take action 0; for data abnormality: if the DQN model detects abnormal electricity bill data, it means that manual review or further inspection is required, and the corresponding action data is to take action 1; for early warning prompts: if the DQN model believes that there is a potential abnormality in the electricity bill data, but is not sure whether it is really abnormal, it is recommended to conduct a manual review, and the corresponding action data is to take action 2; for adjusting the billing strategy: if the DQN model detects an abnormal pattern, it can automatically adjust the billing strategy to adapt to the abnormal situation, and the corresponding action data is to take action 3. These action data are used as the output of the DQN model.

[0104] The following is a detailed description of the reward function:

[0105] The reward function includes immediate rewards and long-term rewards. The following is an explanation of the immediate rewards:

[0106] Instant rewards include basic rewards, efficiency rewards, cost-saving rewards, etc.

[0107] For the basic reward: if the DQN model correctly identifies the normal electricity charges, a positive reward (+1) is given; if the DQN model correctly identifies the abnormal electricity charges, a higher positive reward (+2) is given; if the DQN model incorrectly identifies the abnormal electricity charges or fails to identify the actual anomaly, a negative reward (-1) is given.

[0108] For efficiency rewards: If the DQN model can reduce the need for manual review, an additional positive reward (+0.5) is given.

[0109] For cost saving rewards: If the decision of the DQN model leads to actual cost saving, an additional positive reward (+1) is given.

[0110] In summary, the formula of the immediate reward function is expressed as:

[0111] i reward =α*(b rewards +e rewards )+β*c rewards

[0112] Among them, α and β are weight coefficients, which can be adjusted according to actual needs. For example, α and β are assigned values ​​of 0.2 and 0.8 respectively;

[0113] b rewards It is a basic reward;

[0114] e rewards It is an efficiency reward;

[0115] c rewards It is a cost saving reward.

[0116] The following is an explanation of the long-term rewards:

[0117] The long-term reward is the cumulative effect of actions on future states and final goals, and is expressed as:

[0118]

[0119] Among them, R t is the immediate reward at time t;

[0120] γ t is the discount factor at time t;

[0121] l reward It’s a long-term reward.

[0122] Then, call the initialization method of the DQNAgent class to configure the key parameters of the DQN model, such as: setting the discount factor γ (such as 0.99) to balance the immediate reward and long-term reward; setting the first exploration rate epsilon (such as 1.0) to ensure full exploration of the environment; setting the second exploration rate (i.e., the minimum exploration rate) epsilon_min (such as 0.01) to avoid too low an exploration rate; setting the exploration rate decay coefficient epsilon_decay (such as 0.995) to reduce random exploration. In addition, set the preset number of iterations (i.e., the maximum number of iterations) max_iterations (such as 1000 times), and set the early stopping counter early_stopping to 0; set the batch size batch_size to 64, that is, 64 base station sample data as one batch.

[0123] Call the model construction method of the DQNAgent class to build the DQN model, and use the build_model() function to receive the state space and action space as parameters of the DQN model. Create a Sequential model in the function, add an input layer to the DQN model, which is used to receive a one-dimensional state vector; 2 hidden layers (Long Short-Term Memory (LSTM) layer), set 128 neurons, and use the linear rectified linear unit (ReLU). In addition, add an output layer, whose number of neurons matches the action space, that is, 3 neurons, corresponding to the three audit frequency adjustment decisions, and the output layer uses a linear activation function to output the Q value of each action. Among them, the Sequential model allows users to build a model by adding layers sequentially, and each layer is stacked in the order of addition. The use of the Sequential model is very intuitive. Users only need to add different layers in the order of the hierarchy of the neural network.

[0124] In addition, the initialize memory method of the Memory class is called, which is used to store experience tuples (i.e. base station sample data) and support the playback of high-priority experience tuples. In the initialize memory method, the capacity can be set to 10,000, the priority weight parameter to 0.6, and the parameter for controlling importance sampling to 0.4 to create a memory structure that can store and manage experience tuples.

[0125] Define an Adam optimizer instance, set the learning rate learning_rate (e.g. 0.001), use TD error as the loss function, and use the Adam optimizer to compile the DQN model. TD stands for temporal difference, which is a reinforcement learning algorithm used in scenarios where complete reward information cannot be directly obtained to help agents learn optimal strategies when exploring unknown environments. Adam stands for adaptive moment estimation, which is an optimization algorithm with an adaptive learning rate and is used for gradient descent.

[0126] At the beginning of the training iteration, the environment is reset to obtain the current environment state. Each first state is input into the neural network model constructed above, and the DQN model decision method implemented by the DQNAgent class is called to obtain the first action data corresponding to the first state data.

[0127] Execute the first action data in the above environment, that is, call the step() function of the environment to pass in the first action data, obtain the first reward data and the second state data through the memory method of the Memory class, and store and package the first state data, the first action data, the first reward data and the second state data into a base station sample data, and store the base station sample data as an experience tuple in the memory through the add experience method of the Memory class to accumulate data to provide more sample data for experience playback.

[0128] Among them, for the memory method: call the reward function through the remember() function, pass in the basic reward, efficiency reward, and cost saving reward as parameters, then create an experience tuple experience, call the add() function of the memory to package the first state data, the first action data, the first reward data, and the second state data into an experience tuple (i.e., base station sample data) to achieve the purpose of adding an experience to the memory of the DQN model. This experience tuple contains the complete interaction information of the DQN model in the environment.

[0129] In addition, the sampling method of the Memory class is used to obtain a batch of base station sample data with high priority from the memory, and the experience playback method of the DQNAgent class is used to perform experience playback, calculate the predicted action value and the target action value, and update the weight of the neural network model through the model update method of the DQNAgent class.

[0130] Among them, for the experience replay method: sample base station sample data from the memory, and provide a continuous sequence of base station sample data to the LSTM layer to update the weights of the neural network model.

[0131] For the model update method: regularly call the model update method update_target(), use the weights of the neural network model to update the weights of the target neural network model, the updated target neural network model weights will be used to calculate the target action value, the specific formula for weight update is:

[0132] θ target2 =τ*θ target1 +(1-τ)*θ online

[0133] Among them, θ target2 is the weight of the updated target neural network model;

[0134] θ target1 is the weight of the target neural network model before updating;

[0135] θ online is the weight of the DQN model;

[0136] τ is the update coefficient and can be taken as 0.995.

[0137] Furthermore, the step of inputting each first state data into the neural network model, obtaining the first action data corresponding to the first state data, and executing the first action data to obtain the first reward data and the second state data specifically includes:

[0138] Input each first state data into the neural network model to obtain a second random number corresponding to each first state data;

[0139] When the second random number is less than the first exploration rate, determining one of the plurality of action data as the first action data;

[0140] When the second random number is greater than or equal to the first exploration rate, obtaining an action value corresponding to each action data in a plurality of action data through the neural network model, and determining the action data corresponding to the maximum value of the plurality of action values ​​as the first action data;

[0141] The first action data is executed in the neural network model to obtain first reward data and second state data.

[0142] The following describes the specific process of obtaining the first action data corresponding to the first state data by calling the DQN model decision method implemented by the DQNAgent class:

[0143] Each first state data is input into the neural network model to generate a second random number. If the second random number is less than the first exploration rate epsilon, an action data is randomly selected as the first action data. If the second random number is greater than or equal to the first exploration rate epsilon, the neural network model is used to predict the action value corresponding to each action data, and the action data with the largest action value is used as the first action data. The first action data is executed in the above environment to obtain the first reward data and the second state data.

[0144] As another optional specific embodiment, the step 102 inputs the second state data into the neural network model to obtain the target action value of the second action data corresponding to the second state data, specifically including:

[0145] Inputting the second state data into the neural network model to obtain a first random number;

[0146] When the first random number is greater than or equal to the first exploration rate, obtaining action values ​​corresponding to the plurality of action data through the neural network model;

[0147] Determine the action data corresponding to the maximum value among the multiple action values ​​as the second action data;

[0148] The target action value is determined according to the action value corresponding to the second action data.

[0149] Specifically, the second state data in the base station sample data is input into the neural network model to obtain a first random number. If the first random number is less than the first exploration rate, an action data is randomly used as the second action data, and the second action data is executed to obtain the action value corresponding to the second action data, and the target action value (i.e., target Q value) is determined based on the action value corresponding to the second action data. If the first random number is greater than or equal to the first exploration rate, the action value corresponding to each action data is obtained through the neural network model, and the action data corresponding to the maximum value of the multiple action values ​​is used as the second action data, and the second action data is executed to obtain the action value corresponding to the second action data, and the target action value is determined based on the action value corresponding to the second action data.

[0150] Furthermore, the target action value can be calculated by the following formula:

[0151] Q2=M+γ*Q1

[0152] Among them, Q2 represents the target action value;

[0153] M represents the first reward data;

[0154] γ represents the discount factor;

[0155] Q1 represents the action value corresponding to the second action data.

[0156] Specifically, a discount factor is obtained, the discount factor is multiplied by the action value corresponding to the second action data, and then the first reward data corresponding to the first state data is added to obtain the target action value.

[0157] As another optional specific embodiment, the method may further include:

[0158] Calculating an error value between the predicted action value and the target action value;

[0159] When the error value is greater than the second exploration rate, updating the first exploration rate by a decay coefficient;

[0160] The second exploration rate is less than or equal to the first exploration rate.

[0161] Specifically, the priority of the base station sample data is updated through the update priority method of the Memory class. The specific operation is: for each base station sample data extracted from the memory, its TD error is calculated, and the calculated TD error is used to update the priority of each base station sample data. In the experience playback of the priority, the priority is proportional to the absolute value of the TD error, that is, the larger the TD error, the higher the priority. Among them, the TD error formula is:

[0162] TD error = |predicted Q value - target Q value|

[0163] The priority formula is:

[0164] Priority = TD error + constant

[0165] Then, the Adam optimizer is used to minimize the error between the predicted action value and the target action value. The weight of the neural network model is updated by comparing the error between the predicted action value and the target action value. If the error is greater than the second exploration rate epsilon_min, the first exploration rate is decayed according to the decay coefficient epsilon_decay to automatically update the first exploration rate, which reduces manual intervention and improves efficiency. The update formula of the first exploration rate is as follows:

[0166]

[0167] Among them, ∈2 represents the first exploration rate after the update; ∈1 represents the first exploration rate before the update; Represents the attenuation coefficient.

[0168] As another optional specific embodiment, the preset training termination condition in step 102 includes but is not limited to at least one of the following:

[0169] The first item: The number of iterations reaches the preset number of iterations.

[0170] Specifically, the neural network model is iteratively trained through the cyclic training method of the DQNAgent class until the number of iterations reaches a preset number of iterations and the training is stopped. The trained neural network model and weights are saved through the save model method.

[0171] Second item: The second difference obtained by subtracting the second error value from the first error value is less than the first preset difference, the first error value is the error value between the predicted action value and the target action value obtained after the Nth iteration, the second error value is the error between the predicted action value and the target action value obtained after the Mth iteration, M and N are both values ​​less than or equal to the preset number of iterations, M is greater than N and the difference between M and N is a preset value.

[0172] Specifically, the neural network model is iteratively trained through the cyclic training method of the DQNAgent class. If the decrease value of the error value after multiple consecutive iterations is less than the first preset difference, it means that the error value decreases very slowly (for example, the decrease value of the error value every 100 iterations is less than 0.001), then the training is stopped, and the trained neural network model and weights are saved through the save model method.

[0173] Item 3: When the difference between the error values ​​obtained after two adjacent iterations is less than a second preset difference, the number of the early stop counter is increased by one until the preset number of the early stop counter is met.

[0174] Specifically, the neural network model is iteratively trained through the loop training method of the DQNAgent class. If the error value obtained after this training is less than the error value obtained after the last training, and the difference between the two error values ​​is greater than or equal to the second preset difference, the number of early stop counters is reset. If the error value obtained after this training is less than the error value obtained after the last training, and the difference between the two error values ​​is less than the second preset difference, it means that the error value decreases less, and the number of early stop counters is increased until the number of early stop counters is equal to the preset number (such as 10 times). Stop training and save the trained neural network model and weights through the save model method.

[0175] In addition, after stopping the training, the performance of the trained neural network model is evaluated through the validation set, and the accuracy on the validation set is used as the main performance indicator, which is calculated by the following formula:

[0176] Accuracy = Number of correctly predicted base station sample data / Total number of base station sample data

[0177] In addition, if Figure 3 As shown, the above data is visualized and a data report is generated, as follows:

[0178] like Figure 7 As shown, step 701: extract basic data and equipment power consumption data through the data analysis tool Pandas, and use the visualization tool Matplotlib to draw a bar chart, with the base station logo as the horizontal axis and the basic data and equipment power consumption data as the vertical axis, showing the equipment quantity and equipment type distribution of different base stations; step 702: extract features such as longitude and latitude and equipment status in the basic data, and use Matplotlib to draw a box plot to show the distribution and abnormal values ​​of each feature.

[0179] Step 703: Use Pandas to extract the base station or device identification, electricity bill amount, etc. from the historical electricity bill data, and use Matplotlib to draw a bar chart with the base station identification as the horizontal axis and the electricity bill amount as the vertical axis to compare the electricity usage of different base stations; Step 704: Extract the time and electricity consumption from the historical electricity bill data, and use Matplotlib to draw a heat map with time as the horizontal axis and electricity consumption as the vertical axis to identify abnormal consumption patterns through electricity consumption patterns in different time periods.

[0180] Step 705: Combine Pandas and Matplotlib to automatically generate a statistical analysis report containing multi-dimensional analysis, including but not limited to time series analysis, seasonal changes, and trend forecasts; add a comparative analysis of the model prediction and actual power consumption data to the report, and use a stacked chart or a difference chart to display the power saving data. Use cron() to regularly trigger the report script, input the model output and actual power consumption data file path, automatically load the data, perform cost analysis, and generate a chart of power consumption data changes. Send the regularly generated reports to relevant personnel via email to ensure that key stakeholders are always aware of the latest power consumption and cost savings.

[0181] In summary, in the above-mentioned embodiment of the present application, the base station sample data comprehensively considers the basic data of the base station, the power consumption data of the equipment, and the historical electricity bill data. Through the multi-faceted data training model, the normal and abnormal electricity bill data can be more accurately identified, the need for manual review is reduced, and the possibility of human error is reduced. In addition, by adding reward data to the base station sample data, it can not only ensure that the model can learn the correct audit strategy, but also consider the actual cost. In addition, the first state data, the first action data, the first reward data, and the second state data after the first state data are packaged into a base station sample data, and each base station sample data is a complete set of interactive information to improve the accuracy of model training. In summary, the target neural network model after the above training can automatically analyze a large amount of data and make decisions quickly to improve the efficiency of the entire audit process. In addition, the data is visualized and a report is generated, which provides intuitive and detailed power consumption and cost savings for relevant personnel, which helps to better manage and optimize power costs.

[0182] The above introduces the model training method provided in the embodiment of the present application. The following will introduce the model training device provided in the embodiment of the present application in conjunction with the accompanying drawings.

[0183] like Figure 8 As shown, the embodiment of the present application further provides a model training device 800, the device comprising:

[0184] The acquisition module 801 is used to acquire a base station sample data set, wherein the base station sample data set includes a plurality of base station sample data, and each base station sample data includes: first state data, first action data, first reward data, and second state data after the first state data; the first state data is the audit data obtained after processing the basic data of the base station, the power consumption data of the equipment, and the historical electricity fee data; the first action data is the action data obtained by the neural network model from a plurality of action data based on the first state data; the first reward data is the reward data obtained by the neural network model based on the first action data; and the second state data is the next state data obtained by the neural network model based on the first action data;

[0185] The training module 802 is used to repeatedly perform the following steps until a preset training termination condition is met to obtain a trained target neural network model:

[0186] Inputting the first state data into the neural network model, and obtaining a predicted action value based on the first action data;

[0187] Inputting the second state data into the neural network model to obtain a target action value of second action data corresponding to the second state data;

[0188] The weights of the neural network model are updated according to the predicted action value and the target action value, so as to train the neural network model according to the updated weights.

[0189] Optionally, when the training module 802 inputs the second state data into the neural network model to obtain the target action value of the second action data corresponding to the second state data, it is specifically used to:

[0190] Inputting the second state data into the neural network model to obtain a first random number;

[0191] When the first random number is greater than or equal to the first exploration rate, obtaining action values ​​corresponding to the plurality of action data through the neural network model;

[0192] Determine the action data corresponding to the maximum value among the multiple action values ​​as the second action data;

[0193] The target action value is determined according to the action value corresponding to the second action data.

[0194] Optionally, the training module 802 is further used for:

[0195] Calculating an error value between the predicted action value and the target action value;

[0196] When the error value is greater than the second exploration rate, updating the first exploration rate by a decay coefficient;

[0197] The second exploration rate is less than or equal to the first exploration rate.

[0198] Optionally, the target action value is calculated by the following formula:

[0199] Q2=M+γ*Q1

[0200] Among them, Q2 represents the target action value;

[0201] M represents the first reward data;

[0202] γ represents the discount factor;

[0203] Q1 represents the action value corresponding to the second action data.

[0204] Optionally, the method for acquiring the multiple base station sample data in the acquisition module 801 specifically includes:

[0205] Obtain basic data of base stations, equipment user data, and historical electricity bill data;

[0206] Processing the basic data, the device user data and the historical electricity fee data to obtain a plurality of first status data to be audited;

[0207] Input each first state data into the neural network model to obtain first action data corresponding to the first state data, and execute the first action data to obtain first reward data and second state data;

[0208] The first state data, the first action data, the first reward data and the second state data are combined into a base station sample data to obtain a plurality of base station sample data.

[0209] Optionally, when the acquisition module 801 inputs each first state data into the neural network model, obtains the first action data corresponding to the first state data, and executes the first action data to obtain the first reward data and the second state data, it is specifically used to:

[0210] Input each first state data into the neural network model to obtain a second random number corresponding to each first state data;

[0211] When the second random number is less than the first exploration rate, determining one of the plurality of action data as the first action data;

[0212] When the second random number is greater than or equal to the first exploration rate, obtaining an action value corresponding to each action data in a plurality of action data through the neural network model, and determining the action data corresponding to the maximum value of the plurality of action values ​​as the first action data;

[0213] The first action data is executed in the neural network model to obtain first reward data and second state data.

[0214] Optionally, the preset training termination condition includes at least one of the following:

[0215] The number of iterations reaches the preset number of iterations;

[0216] A second difference value obtained by subtracting the second error value from the first error value is less than a first preset difference value, the first error value is an error value between the predicted action value and the target action value obtained after the Nth iteration, the second error value is an error value between the predicted action value and the target action value obtained after the Mth iteration, M and N are both values ​​less than or equal to the preset number of iterations, M is greater than N, and the difference between M and N is a preset value;

[0217] When the difference between the error values ​​obtained after two adjacent iterations is less than the second preset difference, the number of the early stop counter is increased by one until the preset number of the early stop counter is met.

[0218] In summary, in the above-mentioned embodiment of the present application, the base station sample data comprehensively considers the basic data of the base station, the power consumption data of the equipment, and the historical electricity bill data. Through the multi-faceted data training model, the normal and abnormal electricity bill data can be more accurately identified, the need for manual review is reduced, and the possibility of human error is reduced. In addition, by adding reward data to the base station sample data, it can not only ensure that the model can learn the correct audit strategy, but also consider the actual cost. In addition, the first state data, the first action data, the first reward data, and the second state data after the first state data are packaged into a base station sample data, and each base station sample data is a complete set of interactive information to improve the accuracy of model training. In summary, the target neural network model after the above training can automatically analyze a large amount of data and make decisions quickly to improve the efficiency of the entire audit process. In addition, the data is visualized and a report is generated, which provides intuitive and detailed power consumption and cost savings for relevant personnel, which helps to better manage and optimize power costs.

[0219] It should be noted here that the above-mentioned model training device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned model training method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0220] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0221] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0222] like Fig. 9 As shown, an embodiment of the present application further provides an electronic device, including a memory 920, a transceiver 910, and a processor 900:

[0223] Memory 920, used for storing computer programs;

[0224] A transceiver 910, configured to transmit and receive data under the control of the processor;

[0225] Processor 900 is used to read the computer program in the memory and execute the steps of the model training method described in any of the above embodiments.

[0226] Among them, Fig. 9In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 900 and memory represented by memory 920. The bus architecture may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 910 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, which may include a wireless channel, a wired channel, an optical cable, and other transmission media. The processor 900 is responsible for managing the bus architecture and general processing, and the memory 920 may store data used by the processor 900 when performing operations.

[0227] The processor 900 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.

[0228] The processor calls the computer program stored in the memory to execute the model training method provided in the embodiment of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0229] It should be noted here that the above-mentioned electronic device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned model training method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0230] An embodiment of the present application also provides a processor-readable storage medium, which stores a computer program, and the computer program is used to enable the processor to execute the above-mentioned model training method.

[0231] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0232] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0233] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0234] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0235] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0236] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a base station sample data set, wherein the base station sample data set includes a plurality of base station sample data, and each base station sample data includes: first state data, first action data, first reward data, and second state data after the first state data; the first state data is the data to be audited obtained after processing the basic data of the base station, the power consumption data of the equipment, and the historical electricity fee data; the first action data is the action data obtained by the neural network model from a plurality of action data based on the first state data; the first reward data is the reward data obtained by the neural network model based on the first action data; and the second state data is the next state data obtained by the neural network model based on the first action data; Repeat the following steps until the preset training termination conditions are met to obtain the trained target neural network model: Inputting the first state data into the neural network model, and obtaining a predicted action value based on the first action data; Inputting the second state data into the neural network model to obtain a target action value of second action data corresponding to the second state data; The weights of the neural network model are updated according to the predicted action value and the target action value, so as to train the neural network model according to the updated weights.

2. The method according to claim 1, characterized in that The step of inputting the second state data into the neural network model to obtain a target action value of the second action data corresponding to the second state data includes: Inputting the second state data into the neural network model to obtain a first random number; When the first random number is greater than or equal to the first exploration rate, obtaining action values ​​corresponding to the plurality of action data through the neural network model; Determine the action data corresponding to the maximum value among the multiple action values ​​as the second action data; The target action value is determined according to the action value corresponding to the second action data.

3. The method according to claim 2, characterized in that The method further comprises: Calculating an error value between the predicted action value and the target action value; When the error value is greater than the second exploration rate, updating the first exploration rate by a decay coefficient; The second exploration rate is less than or equal to the first exploration rate.

4. The method according to claim 2, characterized in that: The target action value is specifically calculated by the following formula: Q2=M+γ*Q1 Among them, Q2 represents the target action value; M represents the first reward data; γ represents the discount factor; Q1 represents the action value corresponding to the second action data.

5. The method according to claim 1, characterized in that The method for acquiring the plurality of base station sample data includes: Obtain basic data of base stations, equipment user data, and historical electricity bill data; Processing the basic data, the device user data and the historical electricity fee data to obtain a plurality of first status data to be audited; Input each first state data into the neural network model to obtain first action data corresponding to the first state data, and execute the first action data to obtain first reward data and second state data; The first state data, the first action data, the first reward data and the second state data are combined into a base station sample data to obtain a plurality of base station sample data.

6. The method according to claim 5, characterized in that The step of inputting each first state data into the neural network model, obtaining first action data corresponding to the first state data, and executing the first action data to obtain first reward data and second state data includes: Input each first state data into the neural network model to obtain a second random number corresponding to each first state data; When the second random number is less than the first exploration rate, determining one of the plurality of action data as the first action data; When the second random number is greater than or equal to the first exploration rate, obtaining an action value corresponding to each action data in a plurality of action data through the neural network model, and determining the action data corresponding to the maximum value of the plurality of action values ​​as the first action data; The first action data is executed in the neural network model to obtain first reward data and second state data.

7. The method according to claim 1, characterized in that The preset training termination condition includes at least one of the following: The number of iterations reaches the preset number of iterations; A second difference value obtained by subtracting the second error value from the first error value is less than a first preset difference value, the first error value is an error value between the predicted action value and the target action value obtained after the Nth iteration, the second error value is an error value between the predicted action value and the target action value obtained after the Mth iteration, M and N are both values ​​less than or equal to the preset number of iterations, M is greater than N, and the difference between M and N is a preset value; When the difference between the error values ​​obtained after two adjacent iterations is less than the second preset difference, the number of the early stop counter is increased by one until the preset number of the early stop counter is met.

8. A model training device, characterized in that: The device comprises: An acquisition module is used to acquire a base station sample data set, wherein the base station sample data set includes multiple base station sample data, and each base station sample data includes: first state data, first action data, first reward data, and second state data after the first state data; the first state data is the data to be audited obtained after processing the basic data of the base station, the power consumption data of the equipment, and the historical electricity fee data; the first action data is the action data obtained by the neural network model from multiple action data based on the first state data; the first reward data is the reward data obtained by the neural network model based on the first action data; and the second state data is the next state data obtained by the neural network model based on the first action data; The training module is used to repeatedly perform the following steps until the preset training termination conditions are met to obtain the trained target neural network model: Inputting the first state data into the neural network model, and obtaining a predicted action value based on the first action data; Inputting the second state data into the neural network model to obtain a target action value of second action data corresponding to the second state data; The weights of the neural network model are updated according to the predicted action value and the target action value, so as to train the neural network model according to the updated weights.

9. An electronic device, characterized in that: Including memory, transceiver, processor: Memory for storing computer programs; a transceiver, for transmitting and receiving data under the control of the processor; A processor, configured to read the computer program in the memory and execute the model training method as described in any one of claims 1 to 7.

10. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to enable the processor to execute the model training method as described in any one of claims 1 to 7.

Images