Offline reinforcement learning dimension missing action data filling method based on correlation

Abstract

The invention discloses a correlation-based off-line reinforcement learning dimension missing action data filling method in the technical field of off-line reinforcement learning, and the method comprises the specific steps: obtaining a data set # imgabs0 # after an intelligent agent interacts with an environment, analyzing the data set # imgabs1 #, carrying out the analysis of the data set # imgabs0 #, and carrying out the analysis of the data set # imgabs1 #; the method comprises the following steps: identifying missing dimensions, constructing and training a model, loading a data set containing dimension missing action data, reading the action data in the data set, converting the action data into a PyTorch tensor, then loading a pre-trained model, performing interpolation on missing values in a data set # imgabs2 #, and obtaining a PyTorch tensor; the obtained data set # imgabs3 # is used for training an offline reinforcement learning agent, and the method is beneficial for improving the accuracy and robustness of the offline reinforcement learning model during incomplete data processing, so that the performance of the model in practical application is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of offline reinforcement learning, and specifically to a method for filling missing action data in offline reinforcement learning dimensions based on correlation. Background Art

[0002] Offline reinforcement learning aims to optimize policies by analyzing static datasets collected from the environment. In practical applications, the datasets often have missing action data due to sensor failures, data recording interruptions, or communication problems.

[0003] For conventional data missing problems, the current filling methods mainly include: (1) Data deletion: Removing records with missing data is simple but may result in a significant reduction in the amount of data, affecting the training effect of the model.

[0004] (2) Mean filling: replacing missing values ​​with the mean of existing data. This method is suitable for simple cases but cannot capture complex correlations in the data.

[0005] (3) Interpolation method: interpolation is performed based on adjacent data points. It is suitable for some continuous data, but it does not work well in high-dimensional data.

[0006] (4) Regression filling: Missing data is predicted through regression models. Although it can handle linear relationships, it has limited effect on high-dimensional and complex data.

[0007] In recent years, although some studies have attempted to use the overall correlation of data (such as collaborative filtering and matrix decomposition) for data filling, most of these methods have not deeply explored the specific correlation between action data in different dimensions.

[0008] The existing technologies have the following main problems when processing missing motion data: (1) Insufficient use of inter-dimensional correlation: Although regression filling techniques can be used to predict missing data, traditional regression methods often fail to fully utilize the complex inter-dimensional correlations in the data. Traditional regression methods usually assume that the data relationship is linear and fail to fully explore the complex correlations between dimensions.

[0009] (2) Limited filling effect: In the case of high missing rate or high-dimensional data, the effect of traditional regression filling method may be limited.

[0010] (3) Computational complexity and generalization ability: When dealing with large-scale and high-dimensional data, some filling methods have high computational complexity and insufficient generalization ability, which affects their effectiveness in practical applications.

[0011] It should be noted that in the field of offline reinforcement learning, the problem of missing data in one or more dimensions of action data has not been solved, and the present invention fills this gap. Summary of the invention

[0012] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0013] Therefore, the purpose of the present invention is to provide a correlation-based offline reinforcement learning dimension missing action data filling method, which helps to improve the accuracy and robustness of the offline reinforcement learning model when processing incomplete data, thereby enhancing the performance of the model in practical applications.

[0014] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions: The method for filling missing action data in offline reinforcement learning dimensions based on correlation includes the following steps: S1. After the agent interacts with the environment, the data set is obtained , for the dataset Conduct analysis to identify missing dimensions; S2, build and train the model; S3. Load the dataset containing the missing action data, read the action data, convert it into a PyTorch tensor, and then load the pre-trained model to train the dataset. Interpolate the missing values ​​in to obtain the interpolated data set ; S4. The obtained data set Used for training offline reinforcement learning agents.

[0015] As a preferred solution of the offline reinforcement learning dimension missing action data filling method based on correlation according to the present invention, in step S1, the data set The specific steps to analyze and identify missing dimensions are as follows: The dataset The action dataset in is defined as a matrix ,in represents the total number of samples in the dataset, Represents the dimension size of the action space, matrix Each entity in Corresponding to The sample Dimensions: ; Using a binary mask matrix Identify missing data where express There are missing express is complete: ; Based on this mask matrix, all complete action data samples are defined as , which only contains samples with complete dimensions, It is expressed as: ; in, Representation Matrix No. samples, which have no missing values ​​in any dimension; In addition, using the matrix to classify which dimensions are missing or complete, if at least one sample ,That , then the dimension is considered is missing, otherwise, if all samples of , then the dimension is considered is complete: ; ; As a preferred solution of the method for filling missing action data in offline reinforcement learning dimension based on correlation described in the present invention, the model includes an input layer, a hidden layer and an output layer; The input layer consists of | | units, of which Represents the set of dimensions containing available data, it processes observed features and uses the captured correlations to predict missing values; The hidden layers introduce nonlinearity to capture complex relationships in the data, and Dropout is applied after the ReLU activation function to prevent overfitting by randomly ignoring neurons during training. ; ; in and is the weight matrix, and is the bias vector, is the input action data matrix; The output layer has | | units, of which The set of dimensions representing missing data, the predicted value is given by the following formula: ; in and are the weight matrix and bias vector of the output layer respectively. The function ensures that the output value is within the valid range.

[0016] As a preferred solution of the method for filling missing action data in offline reinforcement learning dimension based on correlation described in the present invention, the specific training steps of the model are as follows: First, load the dataset and filter out samples without missing values ; Secondly, divide the data set into training set, validation set and test set, with the proportions of 70%, 15% and 15% respectively, and use DataLoader to generate an iterable data loader; Then, the model is trained, and the Adam optimizer with a learning rate of 0.0001 is used during the training process to minimize the Huber loss function, which is defined as: ; in is the true value, is the predicted value, is the threshold for switching from quadratic loss to linear loss, In each training round, training is performed first, then validation is performed, and the loss value is calculated. Finally, the best model is saved to the specified path.

[0017] As a preferred solution of the offline reinforcement learning dimension missing action data filling method based on correlation according to the present invention, in step S3, the data set Interpolate the missing values ​​in to obtain the interpolated data set The specific steps are as follows: The input dimension of the model is the number of non-missing columns | |, the output dimension is the number of missing columns| | Then, the code performs data filling processing column by column. For each column with missing values, the code first determines which samples have missing values ​​in the column, and uses the values ​​of non-missing columns as input, and passes them to the model for prediction in batches. The results of each batch of predictions are replaced in turn to the corresponding positions in the cloned dataset. Finally, the filled dataset is obtained. .

[0018] Compared with the prior art, the present invention has the following beneficial effects: the correlation-based dimensional missing action data filling method proposed in the present invention effectively overcomes the limitations of the prior art in processing missing data in offline reinforcement learning, especially in high-dimensional action spaces. Traditional offline reinforcement learning methods are often unable to effectively deal with the problem of missing data caused by sensor failure or communication interruption when faced with static data sets from real environments, thereby affecting the quality of learning strategies. Specifically, in the field of offline reinforcement learning, no research has focused on the problem of missing data in dimensions. The present invention, by utilizing the correlation between dimensions in the action space and within samples, can fully mine and utilize limited available data, accurately fill in missing values, and significantly improve the performance of the model. By optimizing data utilization, not only the training effect of the model is improved, but also the reliability and stability of offline reinforcement learning in practical applications are strongly supported. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them: Figure 1 It is a flow chart of the method for filling missing action data in offline reinforcement learning dimension based on correlation of the present invention; Figure 2 This is a flow chart of the model training provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0021] The present invention provides a correlation-based offline reinforcement learning dimension missing action data filling method, which helps to improve the accuracy and robustness of the offline reinforcement learning model when processing incomplete data, thereby enhancing the performance of the model in practical applications.

[0022] After the intelligent agent interacts with the environment, a data set will be obtained. For this data set, due to sensor failure, data recording interruption or communication problems, the action data therein will have data missing problems in one or more dimensions; first, this data set is analyzed to identify which dimensions have missing problems; secondly, the core of the present invention is to build and train the model. Based on the idea of ​​correlation, the data on the non-missing dimensions are used to predict the data on the missing dimensions to obtain the model hyperparameters; then, the trained model obtained in the previous step is used to interpolate the missing values ​​in the data set to obtain the interpolated data set; finally, it will be applied to downstream tasks. The overall flow chart is as follows: Figure 1 shown.

[0023] Step 1: Identification of missing data The dataset The action dataset in is defined as a matrix ,in represents the total number of samples in the dataset (i.e. data and size), Represents the dimension size of the action space. The matrix Each entity in Corresponding to The sample Dimensions: (1) To identify missing data, a binary mask matrix ,in express There are missing express is complete: (2) Based on this mask matrix, all complete action data samples are defined as , which only contains samples whose dimensions are complete. Specifically, It can be expressed as: (3) in Representation Matrix No. samples, which have no missing values ​​in any dimension In addition, using the matrix to classify which dimensions are missing or complete. If at least one sample ,That , then the dimension is considered is missing. Otherwise, if all samples of , then the dimension is considered is complete: (4) (5) Step 2: Model training (1) Model architecture Inspired by the correlation between joints in moving objects, this paper aims to capture and utilize the correlation between dimensions in motion data. For each motion sample, the data of available dimensions are used to effectively fill in the missing values ​​based on the captured correlation between dimensions.

[0024] ① Input layer: The input layer consists of | | units, of which Represents the set of dimensions that contain the available data. It processes the observed features and uses the captured correlations to predict the missing values.

[0025] ②Hidden layer: ReLU introduces nonlinearity to capture complex relationships in the data. Dropout is applied after the ReLU activation function to prevent overfitting by randomly ignoring neurons during training.

[0026] (6) (7) in and is the weight matrix, and is the bias vector, is the input action data matrix.

[0027] ③Output layer The output layer has | | units, of which A collection of dimensions representing missing data. The predicted value is given by the following formula: (8) in and are the weight matrix and bias vector of the output layer respectively. The function ensures that the output value is within the valid range.

[0028] This architecture can effectively model the correlation between dimensions, thereby achieving accurate sample interior filling and improving performance in subsequent reinforcement learning downstream tasks.

[0029] (2) Implementation details The specific implementation details of model training are as follows Figure 2 shown.

[0030] First, load the dataset and filter out samples without missing values .

[0031] Secondly, divide the data set into training set, validation set and test set, with the proportions of 70%, 15% and 15% respectively. And use DataLoader to generate an iterable data loader.

[0032] Then, the model is trained. The Adam optimizer with a learning rate of 0.0001 is used during training to minimize the Huber loss function, which balances the benefits of mean squared error (MSE) and mean absolute error (MAE). The Huber loss function is defined as: (9) in is the true value, is the predicted value, is the threshold for switching from quadratic loss to linear loss.

[0033] In each training round, training is performed first, then validation is performed, and the loss value is calculated. To prevent overfitting, early stopping is applied to terminate the training early when the validation loss has not improved within a specified number of rounds (called patience value). This ensures that the model maintains optimal performance while avoiding unnecessary additional training rounds.

[0034] Finally, save the best model to the specified path.

[0035] Step 3: Apply the model First, load the dataset containing the missing action data , read the action data and convert it into a PyTorch tensor. Secondly, load the pre-trained model, the input dimension of which is the number of non-missing columns| |, the output dimension is the number of missing columns| |. Then, the code performs data filling processing column by column. For each column with missing values, the code first determines which samples have missing values ​​in the column, and uses the values ​​of the non-missing columns as input, and passes them to the model for prediction in batches. The results of each batch of predictions (the data filled with missing values) are replaced in turn to the corresponding positions in the cloned dataset. In order to improve efficiency, batch processing (batch_size = 512) is used to perform prediction and filling in batches. Finally, the filled dataset is obtained .

[0036] Step 4: Downstream tasks Finally, the dataset filled in the previous step Used to train offline reinforcement learning (RL) agents. Training reinforcement learning agents ensures that they can learn from comprehensive and accurate information, which is critical for developing effective strategies and enhancing decision-making capabilities in complex environments.

[0037] Although the present invention has been described above with reference to the embodiments, various modifications may be made thereto and parts thereof may be replaced by equivalents without departing from the scope of the present invention. In particular, as long as there is no structural conflict, the various features in the embodiments disclosed in the present invention may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A correlation-based offline reinforcement learning method for filling missing action data in dimensions, characterized in that: The steps include: S1. After the agent interacts with the environment, the data set is obtained , for the dataset Conduct analysis to identify missing dimensions; S2, build and train the model; S3. Load the dataset containing the missing action data, read the action data, convert it into a PyTorch tensor, and then load the pre-trained model to train the dataset. Interpolate the missing values ​​in to obtain the interpolated data set ; S4. The obtained data set Used for training offline reinforcement learning agents.

2. The correlation-based offline reinforcement learning dimension missing action data filling method according to claim 1 is characterized in that: In step S1, the data set The specific steps to analyze and identify missing dimensions are as follows: The dataset The action dataset in is defined as a matrix ,in represents the total number of samples in the dataset, Represents the dimension size of the action space, matrix Each entity in Corresponding to The sample number Dimensions: ； Using a binary mask matrix Identify missing data where express There are missing express is complete: ； Based on this mask matrix, all complete action data samples are defined as , which only contains samples with complete dimensions, It is expressed as: ； in, Representation Matrix No. samples, which have no missing values ​​in any dimension; In addition, using the matrix to classify which dimensions are missing or complete, if at least one sample ,That , then the dimension is considered is missing, otherwise, if all samples of , then the dimension is considered is complete: ； 。 3. The correlation-based offline reinforcement learning dimension missing action data filling method according to claim 1 is characterized in that: The model includes an input layer, a hidden layer and an output layer; The input layer consists of | | units, of which Represents the set of dimensions containing available data, it processes observed features and uses the captured correlations to predict missing values; The hidden layers introduce nonlinearity to capture complex relationships in the data, and Dropout is applied after the ReLU activation function to prevent overfitting by randomly ignoring neurons during training. ； ； in and is the weight matrix, and is the bias vector, is the input action data matrix; The output layer has | | units, of which The set of dimensions representing missing data, the predicted value is given by the following formula: ； in and are the weight matrix and bias vector of the output layer, respectively. The function ensures that the output value is within the valid range.

4. The correlation-based offline reinforcement learning dimension missing action data filling method according to claim 1 is characterized in that: The specific training steps of the model are as follows: First, load the dataset and filter out samples without missing values ; Secondly, divide the data set into training set, validation set and test set, with the proportions of 70%, 15% and 15% respectively, and use DataLoader to generate an iterable data loader; Then, the model is trained, and the Adam optimizer with a learning rate of 0.0001 is used during the training process to minimize the Huber loss function, which is defined as: ； in is the true value, is the predicted value, is the threshold for switching from quadratic loss to linear loss, In each training round, training is performed first, then validation is performed, and the loss value is calculated. Finally, the best model is saved to the specified path.

5. The correlation-based offline reinforcement learning dimension missing action data filling method according to claim 1 is characterized in that: In step S3, the data set Interpolate the missing values ​​in to obtain the interpolated data set The specific steps are as follows: The input dimension of the model is the number of non-missing columns | |, the output dimension is the number of missing columns| | Then, the code performs data filling processing column by column. For each column with missing values, the code first determines which samples have missing values ​​in the column, and uses the values ​​of non-missing columns as input, and passes them to the model for prediction in batches. The results of each batch of predictions are replaced in turn to the corresponding positions in the cloned dataset. Finally, the filled dataset is obtained. .

Images