Constrained reinforcement learning

The method enhances offline RL by training neural networks to output Q functions with safety constraints, addressing distribution shifts and optimizing rewards, suitable for robotic and autonomous driving applications.

JP7870832B2Active Publication Date: 2026-06-05INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 4 Cites 0 Cited by

Patent Information

Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
INTERNATIONAL BUSINESS MACHINE CORPORATION
Filing Date
2022-09-05
Publication Date
2026-06-05

AI Technical Summary

Technical Problem

Conventional offline reinforcement learning (RL) methods struggle with handling multiple constraints and distribution shifts, and risk-aware RL faces challenges in incorporating side constraints effectively.

Method used

A method for stepwise uncertainty-aware offline reinforcement learning that involves training a neural network to output individual Q functions for rewards and safety constraints, using a feature mapping function to fine-tune weights and subtract uncertainty from the expected value, employing uncertainty-aware algorithms like LSPI and LSTDQ to optimize policies.

Benefits of technology

This approach allows for conservative estimation of Q functions, addressing distribution shifts and ensuring compliance with safety constraints while optimizing rewards, applicable in robotic and autonomous driving scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007870832000009
    Figure 0007870832000009
  • Figure 0007870832000010
    Figure 0007870832000010
  • Figure 0007870832000011
    Figure 0007870832000011
Patent Text Reader

Abstract

A computer-implemented method for offline reinforcement learning using a dataset is provided. The method includes training a neural network that inputs state-action pairs and outputs an individual Q-function for each of the reward and one or more safety constraints. The neural network has a linear output layer and remaining nonlinear layers that are represented by a feature mapping function. The training step includes obtaining the feature mapping function by constructing a Q-function based on the dataset according to an offline reinforcement algorithm. The training step further includes using the feature mapping function to fine-tune weights between the reward and the one or more safety constraints, and during the obtaining and fine-tuning steps, an estimate of the Q-function is given by subtracting an uncertainty from an expectation of the Q-function. The uncertainty is a function that maps state-action pairs to an error size.
Need to check novelty before this filing date? Find Prior Art