Reinforcement learning shared control method and device for teleoperated mobile robot

Abstract

The application relates to a reinforcement learning shared control method and device for teleoperation mobile robots. The method comprises the following steps: acquiring remote robot state information and receiving operator driving instructions; generating autonomous control instructions by using a rolling horizon reinforcement learning; constructing a driver model to predict the steering intention to calculate a human-machine conflict index, and constructing an operation capability index based on a time delay, a speed and a curvature; optimizing the human-machine control weight by using reinforcement learning; and weighting and fusing the driver and the autonomous instructions to obtain a final control instruction. By using the method, adaptive dynamic allocation of human-machine authority can be realized, teleoperation safety can be ensured, and collaborative efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a reinforcement learning shared control method and apparatus for teleoperated mobile robots. Background Technology

[0002] Teleoperated mobile robots have broad application prospects in fields such as hazardous environment detection, disaster relief, and remote inspection. In a typical teleoperation system, the operator sends control commands to the remote robot through the human-machine interface and control equipment of the local control station, while simultaneously receiving sensing information about the remote environment through a communication network, thereby achieving real-time control of the remote robot.

[0003] However, one of the core challenges facing teleoperation systems is communication latency. Due to factors such as signal transmission distance, network congestion, and protocol overhead, there is a delay in the action of the operator's control commands reaching the remote robot, and there is also an observation delay in the transmission of environmental perception information from the remote location back to the local control station. This two-way communication latency causes the environmental information received by the operator to lag behind the actual situation at the remote location, and the control commands issued by the operator cannot be immediately applied to the robot. This significantly reduces the operator's situational awareness and control accuracy, while increasing the operator's cognitive load. In scenarios such as high-speed travel, complex curvature paths, and dense obstacles, it can easily lead to safety accidents such as collisions or deviations from the trajectory.

[0004] Existing technologies offer several solutions to the teleoperation latency problem: The first category is prediction-based methods, which compensate for observation latency by constructing predictions of the remote environment locally using virtual image overlay or augmented reality. However, this method relies on accurate environmental modeling and has limited applicability in highly dynamic environments. The second category is latency prediction compensation methods, which directly address the impact of latency on closed-loop control stability from a system theory perspective by introducing latency compensation mechanisms such as wave variable transformation, scattering transformation, or Smith predictors. However, this approach sacrifices system transparency and cannot fundamentally alleviate operator workload. The third category is teleoperation shared control methods, which mitigate the impact of latency on maneuverability by introducing an autonomous controller to collaboratively control the robot with the operator. However, existing teleoperation shared control methods still have shortcomings in the following aspects: First, they lack accurate modeling and recognition of the operator's driving intentions, easily leading to human-machine conflict; second, they fail to fully consider the impact of communication latency and complex remote environments on the operator's actual control capabilities; third, permission allocation strategies often adopt rule-based or static optimization methods, lacking online learning and dynamic optimization capabilities, making it difficult to adapt to time-varying communication environments and complex task requirements.

[0005] Therefore, in view of the shortcomings of the prior art, this application provides a reinforcement learning shared control method for teleoperated mobile robots, which aims to solve the technical problem of how to achieve real-time estimation of the operator's driving intention and driving ability in teleoperation scenarios with communication delays, and to use a rolling temporal reinforcement learning method to adaptively and dynamically allocate human-machine driving authority, so as to give full play to the advantages of human-machine collaboration while ensuring the safety of operation. Summary of the Invention

[0006] Therefore, it is necessary to provide a reinforcement learning-based shared control method and apparatus for teleoperated mobile robots that can adaptively and dynamically allocate human-machine driving permissions, improve teleoperation safety and collaborative efficiency, and address the aforementioned technical problems.

[0007] A reinforcement learning-based shared control method for teleoperated mobile robots, comprising: Acquire motion status information of the remote mobile robot and transmit it to the local control station; receive driving input commands generated by the operator through the control device. Based on motion state information, an online autonomous control strategy is solved using a rolling time-domain reinforcement learning algorithm to generate autonomous control commands. A driver model is constructed to predict the operator's control intentions. Based on the predicted control intentions and autonomous control commands, human-machine conflict indicators are calculated. Furthermore, operational capability indicators are constructed based on communication latency, driving speed, and path curvature in motion state information. The value of the operational capability indicator decreases as communication latency, driving speed, and path curvature increase. Based on human-machine conflict indicators and operational capability indicators, human-machine control weights are optimized through rolling time-domain reinforcement learning. Based on the human-machine control weights, the driver's input commands and autonomous control commands are weighted and fused to obtain the final control commands, which are then sent to the remote mobile robot for execution.

[0008] In one embodiment, the method further includes: extracting the input control sequence predicted by the driver model and the optimal control sequence generated by the autonomous control strategy; The human-machine conflict index is calculated based on the input control sequence and the optimal control sequence as follows: (Modify the position of the square in the formula) in, Indicates the length of the prediction time domain. , express The maximum value, This represents the control sequence input by the driver. This represents the optimal control sequence generated by the autonomous controller. It is a discount factor. It is the amplitude coefficient. It is the penalty coefficient for human-machine conflict.

[0009] In one embodiment, the method further includes: constructing operational capability indicators based on communication latency, driving speed, and path curvature in the motion state information.

[0010]

[0011]

[0012] in, Indicators representing operational capability. , , These represent driving speed, communication latency, and path curvature, respectively. , , These represent the maximum values ​​of driving speed, communication latency, and path curvature, respectively. , , , , All are adjustment coefficients. and All are the rates at which driving ability decreases with speed.

[0013] In one embodiment, the method further includes: constructing a reward function based on the human-machine conflict index and the operational capability index; the reward function parameters... The value of the parameter is negatively correlated with the human-machine conflict index and the operational capability index. The value is positively correlated with the human-machine conflict index and the operational capability index; The human-machine control weights are solved online with the goal of minimizing the cumulative reward. The human-machine control weights The range of values ​​is ,in =1 indicates that it is completely under operator control. =0 indicates that it is completely controlled by an autonomous system; The reward function is:

[0014] The optimization objective is:

[0015] in, Represents the tracking error vector. Indicates control weight, This indicates the steering control command applied to the mobile robot. This indicates the driver's steering control input. , ,and Indicates the weighting coefficient. The barrier function adjusts the parameters. and This represents the obstacle function corresponding to the state and input constraints. Indicates the length of the prediction time domain. This represents the terminal cost function.

[0016] In one embodiment, the method further includes: weighted fusion of the driver input command and the autonomous control command to calculate the final control command.

[0017] in, This indicates that the driver inputs a command. This indicates the autonomous control command. The human-machine control weights are defined as follows.

[0018] In one embodiment, the method further includes: at each control moment, performing rolling optimization within a finite time domain based on the robot's real-time state information to solve for the optimal control sequence, and outputting the control action at the first moment of the optimal control sequence as the autonomous control instruction.

[0019] In one embodiment, the driver model is constructed based on the operator's historical driving input data to predict the operator's future driving intentions, which correspond to the input control sequence in the prediction time domain.

[0020] A reinforcement learning-based shared control device for teleoperated mobile robots, the device comprising: The driver operation instruction generation module is used to acquire the motion status information of the remote mobile robot and transmit it to the local control station, and to receive the driving input instructions generated by the operator through the control device; The autonomous control command generation module is used to solve the autonomous control strategy online using a rolling time-domain reinforcement learning algorithm based on the motion state information, and generate autonomous control commands. The indicator calculation module is used to construct a driver model to predict the operator's operating intentions, calculate human-machine conflict indicators based on the predicted operating intentions and the autonomous control commands, and construct an operational capability indicator based on communication latency, driving speed and path curvature in the motion state information; wherein, the value of the operational capability indicator decreases as the communication latency increases, the driving speed increases and the path curvature increases. The weight optimization module is used to optimize the human-machine control weights through rolling time-domain reinforcement learning based on the human-machine conflict index and the operational capability index. The execution module is used to perform weighted fusion of the driver input command and the autonomous control command according to the human-machine control weight to obtain the final control command, and send the final control command to the remote mobile robot for execution.

[0021] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps: Acquire motion status information of the remote mobile robot and transmit it to the local control station; receive driving input commands generated by the operator through the control device. Based on the motion state information, the autonomous control strategy is solved online using a rolling time-domain reinforcement learning algorithm to generate autonomous control commands. A driver model is constructed to predict the operator's operating intentions. Based on the predicted operating intentions and the autonomous control commands, a human-machine conflict index is calculated. An operational capability index is constructed based on communication latency, driving speed, and path curvature in the motion state information. The value of the operational capability index decreases as the communication latency, driving speed, and path curvature increase. Based on the human-machine conflict index and the operational capability index, the human-machine control weights are optimized through rolling temporal reinforcement learning. Based on the human-machine control weights, the driver's input instructions and the autonomous control instructions are weighted and fused to obtain the final control instructions, which are then sent to the remote mobile robot for execution.

[0022] A computer-readable storage medium having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire motion status information of the remote mobile robot and transmit it to the local control station; receive driving input commands generated by the operator through the control device. Based on the motion state information, the autonomous control strategy is solved online using a rolling time-domain reinforcement learning algorithm to generate autonomous control commands. A driver model is constructed to predict the operator's operating intentions. Based on the predicted operating intentions and the autonomous control commands, a human-machine conflict index is calculated. An operational capability index is constructed based on communication latency, driving speed, and path curvature in the motion state information. The value of the operational capability index decreases as the communication latency, driving speed, and path curvature increase. Based on the human-machine conflict index and the operational capability index, the human-machine control weights are optimized through rolling temporal reinforcement learning. Based on the human-machine control weights, the driver's input instructions and the autonomous control instructions are weighted and fused to obtain the final control instructions, which are then sent to the remote mobile robot for execution.

[0023] The aforementioned reinforcement learning-based shared control method and apparatus for teleoperated mobile robots acquires the motion state information of the remote robot and combines it with real-time driving input commands generated by the operator. Then, it uses a rolling temporal reinforcement learning algorithm to solve the autonomous control strategy online and simultaneously constructs a driver model to predict the operator's maneuvering intentions, thereby calculating a human-robot conflict index. Simultaneously, it considers the impact of communication latency, driving speed, and path curvature on the operator's control capabilities and constructs an operational capability index. Based on these two dynamic indices, rolling temporal reinforcement learning is used again to optimize the human-robot control weights online, ultimately achieving an adaptive weighted fusion of driver input commands and autonomous control commands. This method can perceive environmental changes and operator status in real time, dynamically adjust human-robot permission allocation, effectively alleviate human-robot conflicts caused by communication latency in complex time-varying environments, reduce the operator's cognitive load, and significantly improve the operational safety and task execution efficiency of the teleoperated mobile robot. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a reinforcement learning shared control method for a teleoperated mobile robot in one embodiment. Figure 2 This is a structural block diagram of a reinforcement learning shared control device for a teleoperated mobile robot in one embodiment. Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0026] The reinforcement learning-based shared control method for teleoperated mobile robots provided in this application can be applied in the following environment: The method is applied to a teleoperation system consisting of a local control station, a communication network, and a remote mobile robot-environment. The local control station is equipped with a human-machine interface and control devices, through which the operator generates driving input commands. The remote mobile robot is equipped with a multi-source sensor array, such as LiDAR, cameras, GPS, and inertial measurement units, to collect motion state information in real time. The local control station communicates with the computing unit of the remote mobile robot via a network. The robot's computing unit receives operator commands and its own state information, and runs the shared control method provided in this application to generate final control commands to drive the robot to perform teleoperation tasks. The local control station can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, while the computing unit of the remote mobile robot can be a standalone industrial computer or an embedded processor.

[0027] In one embodiment, such as Figure 1 As shown, a reinforcement learning-based shared control method for teleoperated mobile robots is provided. The method is illustrated using an example of its application to a computing unit mounted on a remote mobile robot, and includes the following: Step 102: Obtain the motion status information of the remote mobile robot and transmit it to the local control station, and receive driving input commands generated by the operator through the control device.

[0028] During teleoperation, the remote mobile robot uses its onboard multi-source sensor array to collect real-time motion status information in the remote environment. This information includes, but is not limited to, environmental perception data generated by the fusion of LiDAR and cameras, and robot pose, velocity, and acceleration acquired by GPS and inertial measurement unit. A portion of the collected motion status information is transmitted to the local control station via a communication network, allowing the operator to assess the remote environment through a human-machine interface. Based on the received perception information, the operator generates corresponding driving input commands using control devices such as the steering wheel and pedals. These driving input commands are then transmitted back to the remote robot's computing unit via the communication network.

[0029] Step 104: Based on the motion state information, solve the autonomous control strategy online using the rolling time-domain reinforcement learning algorithm to generate autonomous control commands.

[0030] Specifically, at each control moment, the rolling temporal reinforcement learning algorithm performs rolling optimization within a pre-defined finite time domain based on the robot's current state to solve for an optimal control sequence. This sequence represents a set of optimal control actions for future moments. The rolling temporal reinforcement learning algorithm only outputs the control action of the first moment in this sequence as the current autonomous control command. When entering the next control moment, after the robot's state is updated, the above optimization process is repeated, thereby achieving online rolling temporal decision-making in uncertain environments.

[0031] Step 106: Construct a driver model to predict the operator's operating intentions, calculate human-machine conflict indicators based on the predicted operating intentions and autonomous control commands, and construct operating capability indicators based on communication delay, driving speed and path curvature in motion state information.

[0032] To adaptively integrate operator driving input commands and autonomous control commands, this method constructs a shared control decision framework. First, a driver model is built to model the operator's driving behavior and predict their intentions. This model can be pre-trained based on the operator's historical driving input data. Using this model, the operator's maneuvering intentions within a future prediction time domain can be predicted and quantified as an input control sequence. The predicted input control sequence is compared with the optimal control sequence generated by the autonomous control strategy to calculate a human-machine conflict index. This index quantifies the deviation between the operator's desired control behavior and the robot's autonomous controller's control behavior. When the operator's intention aligns with the autonomous controller's, the conflict index approaches 1; when there is a significant discrepancy, the conflict index approaches 0.

[0033] Secondly, a latency-speed-curvature coupled operational capability index is constructed to comprehensively evaluate the operator's actual control capability under current communication latency, travel speed, and path curvature conditions. The core design principle of this index is: the greater the communication latency, the slower the operator's perception and response to changes in the remote environment, and the weaker the control capability; the higher the travel speed, the longer the distance the robot covers per unit latency, requiring a higher operator reaction speed and increasing control difficulty; the greater the path curvature, the higher the requirement for operator directional control accuracy, further increasing control difficulty under latency conditions. Therefore, the value of the operational capability index is designed to decrease with increasing communication latency, travel speed, and path curvature. A lower value indicates a more limited operator control capability and a greater need for control assistance from the autonomous system.

[0034] Step 108: Based on the human-machine conflict index and operational capability index, optimize the human-machine control weights through rolling temporal reinforcement learning.

[0035] The weight parameter ρ takes the value of a closed interval between 0 and 1. ρ=1 indicates complete operator control, ρ=0 indicates complete autonomous system control, and intermediate values ​​represent a proportional sharing of control authority between humans and machines. The optimization process aims to minimize the cumulative reward. The reward function design here considers both conflict and capability indicators, and its parameters... The values ​​are negatively correlated with the human-machine conflict index and the operational ability index. The values ​​are positively correlated with the human-machine conflict index and the operational capability index. This means that when human-machine conflict occurs... Figure 1 When the operator is highly skilled, the system tends to assign the operator a higher control weight; conversely, when there are severe conflicts or the operator's ability is limited due to factors such as high latency, high speed, and large curvature, the system tends to assign a higher weight to the autonomous controller to ensure operational safety.

[0036] Step 110: Based on the human-machine control weights, the driver's input instructions and autonomous control instructions are weighted and fused to obtain the final control instructions, which are then sent to the remote mobile robot for execution.

[0037] After obtaining the optimized human-machine control weights ρ, the operator's driving input commands and autonomous control commands are weighted and fused to calculate the final control command. The calculated final control command is sent to the actuator of the remote mobile robot to drive the robot to complete the teleoperation task.

[0038] The reinforcement learning-based shared control method for teleoperated mobile robots described above constructs two dynamic evaluation factors—human-machine conflict index and operational capability index—and applies rolling temporal reinforcement learning to achieve online optimization and adaptive allocation of control weights. This solves the problems of poor adaptability and susceptibility to human-machine conflict inherent in traditional shared control methods in complex, time-varying teleoperation environments. Unlike static or rule-based weight allocation methods, this method can dynamically and continuously adjust human-machine permissions based on real-time communication latency, working environment complexity, and operator intent. This maximizes the decision-making advantages of human operators and the precise execution capabilities of the autonomous machine controller while ensuring safety, significantly improving the overall system performance and robustness.

[0039] In one embodiment, the human-machine conflict index is calculated based on the predicted manipulation intention and autonomous control commands, including: extracting the input control sequence predicted by the driver model and the optimal control sequence generated by the autonomous control strategy; and calculating the human-machine conflict index based on the input control sequence and the optimal control sequence. (Modify the position of the square in the formula) in, Indicates the length of the prediction time domain. , express The maximum value, This represents the control sequence input by the driver. This represents the optimal control sequence generated by the autonomous controller. It is a discount factor. It is the amplitude coefficient. It is the penalty coefficient for human-machine conflict.

[0040] The aforementioned human-robot conflict index is used to quantify the degree of deviation between the operator's expected control behavior and the robot's autonomous controller's control behavior. When the operator's intention is consistent with the autonomous controller, the conflict index is close to 1, indicating good human-robot cooperation; when there is a significant discrepancy, the conflict index is close to 0.

[0041] In one embodiment, operational capability indicators are constructed based on communication latency, driving speed, and path curvature from motion state information, including: Based on communication latency, driving speed, and path curvature from motion status information, the operational capability index is constructed as follows:

[0042]

[0043]

[0044] in, Indicators representing operational capability. , , These represent driving speed, communication latency, and path curvature, respectively. , , These represent the maximum values ​​of driving speed, communication latency, and path curvature, respectively. , , , , All are adjustment coefficients. and All are the rates at which driving ability decreases with speed.

[0045] The capability index is designed based on the following core ideas: the greater the communication latency, the slower the operator's perception and response to changes in the remote environment, and the weaker the control capability; the higher the travel speed, the longer the distance the robot covers per unit latency, the higher the requirements for the operator's reaction speed, and the greater the control difficulty; the greater the path curvature, the higher the requirements for the operator's directional control accuracy, further increasing the control difficulty under latency conditions. The capability index is quantitatively calculated by establishing a coupled mathematical model of latency, speed, and curvature. Its output value reflects the operator's ability to safely and effectively control the robot under current conditions. The lower the value, the more limited the operator's control capability, requiring more control assistance from the autonomous system.

[0046] Based on the calculated human-machine conflict index and operational capability index, a rolling temporal reinforcement learning method is used to optimize the human-machine control weights online. Specifically: A reward function is constructed based on human-machine conflict indicators and operational capability indicators; the parameters of the reward function are... The value of the parameter is negatively correlated with the human-machine conflict index and the operational capability index. The value is positively correlated with the human-machine conflict index and the operational capability index; The human-machine control weights are solved online with the goal of minimizing the cumulative reward. Human-machine control weights The range of values ​​is ,in =1 indicates that it is completely under operator control. =0 indicates that it is completely controlled by an autonomous system; The reward function is:

[0047] The optimization objective is:

[0048] in, Represents the tracking error vector. Indicates control weight, This indicates the steering control command applied to the mobile robot. This indicates the driver's steering control input. , ,and Indicates the weighting coefficient. The barrier function adjusts the parameters. and This represents the obstacle function corresponding to the state and input constraints. Indicates the length of the prediction time domain. This represents the terminal cost function.

[0049] In this embodiment, the specific calculation method of each core indicator and the mathematical expression of the optimization target are clarified through the above formula description, making the implementation of the scheme clearer and more certain, and able to be accurately reproduced and applied. Furthermore, the defined mathematical model is closely related to the physical process of human-machine shared control, which improves the interpretability and optimality of control decisions.

[0050] In one embodiment, the driver input command and the autonomous control command are weighted and fused to obtain the final control command as follows: the driver input command and the autonomous control command are weighted and fused to calculate the final control command.

[0051] in, This indicates that the driver has entered a command. Indicates autonomous control command, Human-machine control weights.

[0052] In one embodiment, the process of solving the autonomous control strategy online using a rolling temporal reinforcement learning algorithm is performed in a discrete state and action space. The robot's state information is obtained by fusing data from an onboard multi-source sensor array, forming a state vector. In the prediction time domain, the algorithm, using the concept of model predictive control, combines the robot's dynamics model and the optimization objective function to simulate the future trajectory and uses reinforcement learning to solve for the optimal action sequence in the current state. After each solution is completed, only the first item in the sequence is executed to react using the latest sensor information, forming a closed-loop control. This mechanism endows the autonomous controller with the ability to cope with dynamic environments, enabling it to quickly generate obstacle avoidance or stabilization control commands in situations such as the sudden appearance of obstacles.

[0053] In one embodiment, the driver model used to predict the operator's future maneuvering intentions can be implemented using a sequence model such as a Long Short-Term Memory (LSTM) network. This model takes a sequence of states and operator control commands within a past time window as input and outputs predictions of control commands for the next N time points. Before the actual teleoperation task is executed, the model is trained offline using the operator's historical driving data in real-world environments to learn its steering control characteristics under different environmental perception outcomes. During task execution, the predictive model can provide an approximate estimate of the current intention to calculate local conflict indices, enabling the shared control framework to make relatively reasonable decisions even when uplink commands arrive late.

[0054] In one embodiment, before acquiring the motion state information of the remote mobile robot and receiving operator instructions, an initialization step is included: loading and calibrating the parameters of the driver model, and initializing the value network or policy network in the rolling temporal reinforcement learning. This initialization process can utilize pre-collected data from various typical teleoperation scenarios, enabling the core algorithm to possess basic environmental understanding and decision-making capabilities from the outset, greatly shortening the adaptation period after online startup and improving the system's practicality.

[0055] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0056] In one embodiment, such as Figure 2 As shown, a reinforcement learning shared control device for teleoperated mobile robots is provided, including: a driver operation instruction generation module 202, an autonomous control instruction generation module 204, an index calculation module 206, a weight optimization module 208, and an execution module 210.

[0057] The driver operation instruction generation module 202 is used to acquire the motion status information of the remote mobile robot and transmit it to the local control station. It is also responsible for receiving the driving input instructions generated by the operator through the control device.

[0058] The autonomous control command generation module 204 is used to solve the autonomous control strategy online using a rolling time-domain reinforcement learning algorithm based on motion state information, thereby generating autonomous control commands.

[0059] The index calculation module 206 is used to construct a driver model to predict the operator's control intentions and calculate human-machine conflict indices based on the predicted control intentions and autonomous control commands. This module is also responsible for constructing operational capability indices based on communication latency, driving speed, and path curvature from motion state information. The values ​​of the constructed operational capability indices are designed to decrease as communication latency, driving speed, and path curvature increase.

[0060] The weight optimization module 208 is used to optimize the human-machine control weights online through rolling time-domain reinforcement learning based on the human-machine conflict index and operational capability index output by the index calculation module.

[0061] The execution module 210 is used to perform weighted fusion of the driver input command received by the driver operation command generation module and the autonomous control command generated by the autonomous control command generation module according to the human-machine control weight output by the weight optimization module, to obtain the final control command, and send the final control command to the remote mobile robot for execution.

[0062] For specific limitations regarding the reinforcement learning shared control device for teleoperated mobile robots described above, please refer to the limitations of the reinforcement learning shared control method for teleoperated mobile robots mentioned earlier, which will not be repeated here. Each module in the above device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0063] In one embodiment, a computer device is provided, which may be a computing unit deployed on a remote mobile robot, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program (which implements the method of this application), and a database. This database stores driver model parameters, reinforcement learning training model parameters, and historical teleoperation data. The network interface is used for data communication with a local control station in a communication network. When the computer program is executed by the processor, it implements a reinforcement learning shared control method for teleoperated mobile robots.

[0064] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0066] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0067] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0068] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0069] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A reinforcement learning-based shared control method for teleoperated mobile robots, characterized in that, The method includes: Acquire motion status information of the remote mobile robot and transmit it to the local control station; receive driving input commands generated by the operator through the control device. Based on the motion state information, the autonomous control strategy is solved online using a rolling time-domain reinforcement learning algorithm to generate autonomous control commands. A driver model is constructed to predict the operator's operating intentions. Based on the predicted operating intentions and the autonomous control commands, a human-machine conflict index is calculated. An operational capability index is constructed based on communication latency, driving speed, and path curvature in the motion state information. The value of the operational capability index decreases as the communication latency, driving speed, and path curvature increase. Based on the human-machine conflict index and the operational capability index, the human-machine control weights are optimized through rolling temporal reinforcement learning. Based on the human-machine control weights, the driver's input instructions and the autonomous control instructions are weighted and fused to obtain the final control instructions, which are then sent to the remote mobile robot for execution.

2. The method according to claim 1, characterized in that, Human-machine conflict indicators are calculated based on the predicted manipulation intent and the autonomous control commands, including: Extract the input control sequence predicted by the driver model and the optimal control sequence generated by the autonomous control strategy; The human-machine conflict index is calculated based on the input control sequence and the optimal control sequence as follows: (Modify the position of the square in the formula) in, Indicates the length of the prediction time domain. , express The maximum value, This represents the control sequence input by the driver. This represents the optimal control sequence generated by the autonomous controller. It is a discount factor. It is the amplitude coefficient. It is the penalty coefficient for human-machine conflict.

3. The method according to claim 1, characterized in that, Based on communication latency, driving speed, and path curvature in the motion state information, operational capability indicators are constructed, including: Based on communication latency, driving speed, and path curvature in the motion state information, the operational capability index is constructed as follows: in, Indicators representing operational capability. , , These represent driving speed, communication latency, and path curvature, respectively. , , These represent the maximum values ​​of driving speed, communication latency, and path curvature, respectively. , , , , All are adjustment coefficients. and All are the rates at which driving ability decreases with speed.

4. The method according to claim 1, characterized in that, Optimizing human-machine control weights through rolling temporal reinforcement learning includes: A reward function is constructed based on the human-machine conflict index and the operational capability index; the parameters of the reward function are... The value of the parameter is negatively correlated with the human-machine conflict index and the operational capability index. The value is positively correlated with the human-machine conflict index and the operational capability index; The human-machine control weights are solved online with the goal of minimizing the cumulative reward. The human-machine control weights The range of values ​​is ,in =1 indicates that it is completely under operator control. =0 indicates that it is completely controlled by an autonomous system; The reward function is: The optimization objective is: in, Represents the tracking error vector. Indicates control weight, This indicates the steering control command applied to the mobile robot. This indicates the driver's steering control input. , ,and Indicates the weighting coefficient. The barrier function adjusts the parameters. and This represents the obstacle function corresponding to the state and input constraints. Indicates the length of the prediction time domain. This represents the terminal cost function.

5. The method according to claim 1, characterized in that, The method for weighted fusion of the driver input command and the autonomous control command to obtain the final control command is as follows: The driver input command and the autonomous control command are weighted and fused to calculate the final control command: in, This indicates that the driver inputs a command. This indicates the autonomous control command. The human-machine control weights are defined as follows.

6. The method according to claim 1, characterized in that, Online solutions for autonomous control policies using rolling temporal reinforcement learning algorithms include: At each control moment, based on the robot's real-time state information, rolling optimization is performed within a finite time domain to solve for the optimal control sequence, and the control action of the first moment in the optimal control sequence is output as the autonomous control instruction.

7. The method according to any one of claims 1 to 6, characterized in that, The driver model is constructed based on the operator's historical driving input data and is used to predict the operator's future driving intentions, which correspond to the input control sequence in the prediction time domain.

8. A reinforcement learning shared control device for teleoperated mobile robots, characterized in that, The device includes: The driver operation instruction generation module is used to acquire the motion status information of the remote mobile robot and transmit it to the local control station, and to receive the driving input instructions generated by the operator through the control device; The autonomous control command generation module is used to solve the autonomous control strategy online using a rolling time-domain reinforcement learning algorithm based on the motion state information, and generate autonomous control commands. The indicator calculation module is used to construct a driver model to predict the operator's operating intentions, calculate human-machine conflict indicators based on the predicted operating intentions and the autonomous control commands, and construct an operational capability indicator based on communication latency, driving speed and path curvature in the motion state information; wherein, the value of the operational capability indicator decreases as the communication latency increases, the driving speed increases and the path curvature increases. The weight optimization module is used to optimize the human-machine control weights through rolling time-domain reinforcement learning based on the human-machine conflict index and the operational capability index. The execution module is used to perform weighted fusion of the driver input command and the autonomous control command according to the human-machine control weight to obtain the final control command, and send the final control command to the remote mobile robot for execution.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images