A sim2real model construction method and device based on reinforcement learning

Abstract

The application provides a Sim2Real model construction method and device based on reinforcement learning, and relates to the technical field of data processing.The method comprises the following steps: obtaining evaluation indexes of a simulation environment and a real environment; using a linear weighting method to quantize the weighted difference between the simulation environment and the real environment indexes according to the evaluation indexes; building a Sim2Real model for mutual conversion of data between the simulation environment and the real environment; and using a reinforcement learning algorithm to perform field self-adaptive training on the Sim2Real model with the minimum weighted difference between the simulation environment and the real environment indexes as the target, so as to obtain a final Sim2Real model.In the application, the Sim2Real model for data conversion between the simulation environment and the real environment is built, data layer adjustment is performed, and the migration error from the simulation environment to the real environment is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a Sim2Real model construction method and device based on reinforcement learning. BACKGROUND

[0002] In modern manufacturing, service and healthcare industries, intelligent robots are playing an increasingly important role, driving the development of automation and intelligence. In these fields, mobile robots are widely used in logistics transportation, warehouse management, medical assistance and other tasks. In order to achieve these tasks, mobile robots must have the ability of autonomous navigation, which means they can move autonomously in unknown or partially unknown environments, plan paths, avoid obstacles, and finally reach the target location. The core of active navigation technology lies in the robot's perception of the environment, path planning and decision control.

[0003] In recent years, with the development of artificial intelligence technology, especially the introduction of machine learning and reinforcement learning methods, active navigation technology has been further improved. Reinforcement learning (RL) guides robots to learn how to act in the environment through a reward mechanism, gradually forming the best behavior strategy. Reinforcement learning methods do not require an explicit environment model and can learn to adapt to complex and variable environments through a large number of trial-and-error processes, making it very promising in dynamic and complex environments.

[0004] However, due to the differences between simulation and reality, the strategy trained in the simulation environment often fails to fully reflect the complexity and dynamic changes of the real environment in the real environment. At the same time, existing reinforcement learning methods, although excellent in specific environments, often lack sufficient generalization ability, resulting in inadequate adaptability of the strategy when the robot faces new or sudden environmental changes. SUMMARY

[0005] In order to solve the technical problems that due to the differences between simulation and reality, the strategy trained in the simulation environment often fails to fully reflect the complexity and dynamic changes of the real environment in the real environment, and at the same time, existing reinforcement learning methods, although excellent in specific environments, often lack sufficient generalization ability, resulting in inadequate adaptability of the strategy when the robot faces new or sudden environmental changes, the present application provides a Sim2Real model construction method and device based on reinforcement learning.

[0006] The technical solutions provided by the embodiments of the present application are as follows:

[0007] First aspect:

[0008] An embodiment of the present invention provides a Sim2Real model construction method based on reinforcement learning, comprising:

[0009] S1: Obtain evaluation indicators of simulation environment and real environment;

[0010] S2: using a linear weighted method, according to the evaluation index, quantifying the weighted difference between the simulation environment and the real environment index;

[0011] S3: Build a Sim2Real model to convert data between the simulation environment and the real environment;

[0012] S4: Using a reinforcement learning algorithm, with the goal of minimizing the weighted difference between the simulated environment and the real environment indicators, the Sim2Real model is subjected to domain adaptive training to obtain a final Sim2Real model.

[0013] Second aspect:

[0014] An embodiment of the present invention provides a Sim2Real model construction device based on reinforcement learning, comprising:

[0015] The acquisition module is used to obtain the evaluation indicators of the simulation environment and the real environment;

[0016] A quantification module, configured to quantify the weighted difference between the simulation environment and the real environment indicators according to the evaluation indicators using a linear weighting method;

[0017] Building a module for building a Sim2Real model that converts data between the simulation environment and the real environment;

[0018] The training module is used to perform domain adaptive training on the Sim2Real model using a reinforcement learning algorithm with the goal of minimizing the weighted difference between the simulated environment and the real environment indicators to obtain a final Sim2Real model.

[0019] The third aspect:

[0020] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for constructing a Sim2Real model based on reinforcement learning as described in the first aspect is implemented.

[0021] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0022] (1) In the present application, by obtaining the evaluation indexes of the simulation environment and the real environment, the differences between the two are clarified, the weighted differences between these evaluation indexes are quantified by using the linear weighting method, a Sim2Real model capable of converting data between the simulation environment and the real environment is built, the data level adjustment is made for the differences between the two, the migration error from the simulation environment to the real environment is reduced, and the complexity and dynamic changes of the real environment are effectively reflected.

[0023] (2) In the present application, the reinforcement learning algorithm is used to train the Sim2Real model in the field to adapt to the minimum weighted difference between the simulation environment and the real environment, and the final Sim2Real model is obtained. By continuously optimizing the model, it can perform well in a wider environment, enhances the adaptability of the model when facing various changes in reality, and improves the generalization ability of the strategy. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A flowchart of a Sim2Real model construction method based on reinforcement learning provided in an embodiment of the present application is shown.

[0026] Figure 2 A structural diagram of a Sim2Real model construction system based on reinforcement learning provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] The technical solutions in the present application will be described below with reference to the drawings.

[0028] In the embodiments of the present application, the words such as "example", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0029] In the embodiments of the present application, the terms "image" and "picture" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized. The terms "of", "corresponding" and "relevant" can be used interchangeably, and it should be pointed out that the meanings expressed are consistent when the distinction is not emphasized.

[0030] In the embodiments of the present application, sometimes the subscript such as W1 can be mistakenly used as a non-subscript such as W1, and the meanings expressed are consistent when the distinction is not emphasized.

[0031] To make the technical problems, technical solutions and advantages to be solved by the present application clearer, the following will be described in detail with reference to the drawings and specific embodiments.

[0032] Reference is made to the accompanying drawings and specific embodiments of the present application Figure 1 , a flowchart of a Sim2Real model construction method based on reinforcement learning provided by an embodiment of the present application is shown.

[0033] The embodiment of the present application provides a Sim2Real model construction method based on reinforcement learning, which can be implemented by a Sim2Real model construction device based on reinforcement learning. The Sim2Real model construction device based on reinforcement learning can be a terminal or a server.

[0034] The Sim2Real (Simulation to Reality) model is a technical method for solving the migration problem from a simulation environment to a real environment in the fields of robotics, autonomous driving, reinforcement learning, etc.

[0035] The processing flow of the Sim2Real model construction method based on reinforcement learning can include the following steps:

[0036] S1: Obtain evaluation indexes of the simulation environment and the real environment.

[0037] In one possible implementation, the evaluation indexes include a success weighted path length and a navigation success rate.

[0038] The success weighted path length is specifically:

[0039]

[0040] Wherein, SPL represents the success weighted path length, N represents the number of training nodes, S b represents the navigation success or failure at the current node number, L b represents the optimal shortest path length to the target point at the current node number, and P brepresents the path length walked by the robot in the actual test.

[0041] The navigation success rate is specifically:

[0042]

[0043] wherein Success represents the success rate, S represents the number of successful navigation times, and T represents the ratio of the total number of navigation tests.

[0044] In the present application, by comparing the path walked by the robot when successfully navigating with the optimal path, the efficiency of the robot navigation path can be evaluated. A higher SPL value means that the robot has chosen a path closer to the optimal path when completing the navigation task. At the same time, PL emphasizes the efficiency of the path, while Success emphasizes the completion rate of the task. The combination of the two makes the evaluation balance efficiency and success rate, avoiding the one-sidedness of single indicator evaluation.

[0045] S2: Using a linear weighting method, the weighted difference between the simulation environment and the real environment indicators is quantified according to the evaluation indicators.

[0046] The linear weighting method is a commonly used multi-indicator decision-making method, which is used to make decisions or evaluations by combining multiple indicators or factors. By assigning a weight to each indicator, the weighted values of each indicator are summed to obtain a comprehensive score or evaluation value.

[0047] In one possible implementation, the weighted difference between the simulation environment and the real environment indicators in S2 is specifically:

[0048] Sim2Real Gap = ω1|SPL sim -SPL real |+ ω2|Success sim -Success real |

[0049] wherein Sim2Real Gap represents the weighted difference between the simulation environment and the real environment indicators, ω1 represents the weight coefficient of the success weighted path length, ω2 represents the weight coefficient of the success rate, SPL sim represents the success weighted path length in the simulation environment, SPL real represents the success weighted path length in the real environment, Success represents the success rate, Success sim represents the navigation success rate in the simulation environment, Success real represents the navigation success rate in the real environment.

[0050] Optionally, the weight coefficient of the success weighted path length is 0.5, and the weight coefficient of the success rate is 0.5.

[0051] In the present application, by weighted sum of two key indicators, a comprehensive score value is generated, which can effectively reflect the overall difference between the simulation environment and the real environment, so as to help the developer to understand more clearly how the model in the real environment deviates from the simulation result. At the same time, giving the same weight to the two key indicators can avoid the distortion of the evaluation result caused by the overweight of a certain indicator.

[0052] S3: Build a Sim2Real model for mutual conversion of data between simulation environment and real environment.

[0053] It should be noted that the purpose of building the Sim2Real platform is to realize the mutual conversion of data between the simulation environment and the real environment, reduce the difference of data format, so as to achieve the goal of environment data fusion. A large amount of state, action, reward and other information will be generated in the process of interaction with high-fidelity simulation environment and real environment, which will be stored in the buffer pool for analysis and evaluation, and the effective data will be reused.

[0054] In one possible implementation, S3 specifically includes:

[0055] S301: Install Gazebo and Rviz in the Sim2Real model, and start the simulation environment of TurtleBot3 robot.

[0056] Gazebo is a powerful open source robot simulation tool, widely used in robot research and development. It provides a highly configurable simulation environment that can simulate complex physical properties and sensor data. It helps users test and optimize robot systems in a virtual environment.

[0057] Rviz is a visualization tool of ROS (Robot Operating System), which is used to display and analyze sensor data, robot state and environment information in the robot operating system.

[0058] In the present application, Gazebo provides a simulation platform to simulate robot operation, and Rviz provides a tool to visualize and debug data and behavior in simulation. The two are used together to achieve efficient robot system development and verification.

[0059] S302: Develop a bridge node from the simulation environment to the Gazebo-like real environment to determine the mutual conversion relationship between the simulation environment and the real environment, and the bridge node is used to create Ros node and subscribe and publish topics.

[0060] In ROS (Robot Operating System), a node is the basic unit of communication, and each node is an execution unit responsible for handling specific functions or tasks. Nodes can be independent processes or run on the same machine, executing different parts of the robot system.

[0061] In the present application, by bridging the nodes and data conversion mechanism, the compatibility and consistency of data format between the simulation environment and the real environment are ensured. This consistency is the basis for ensuring that the model behaves consistently in both environments, allowing the model to seamlessly apply the knowledge obtained in the simulation environment to the real environment.

[0062] S303: The simulation-to-reality node is responsible for converting the discrete actions output in the simulation environment into continuous actions executable in the real environment.

[0063] S304: The reality-to-simulation node converts sensor-related topic data in the real environment into topic data recognizable by the robot in the simulation environment.

[0064] S305: Create a Ros node in the simulation environment to receive topic data.

[0065] S306: Fuse topic data and sensor topic data in the simulation environment, update the strategy, and output action instructions.

[0066] S307: Input the action instructions into the simulation environment and the real environment respectively for training.

[0067] In the present application, by converting data between the simulation environment and the real environment, the Sim2Real model can be exposed to more diverse data during training, reducing the overfitting of the model to a single environment, thereby enhancing its adaptability and robustness in different environments. At the same time, in actual operation, errors of robots or autonomous driving systems may cause danger or damage. By testing and optimizing in the simulation environment first, the possibility of errors in the real environment can be minimized, improving the overall safety of the system.

[0068] Further, by storing data in the simulation and real environments into the buffer pool, the system can effectively reuse these data for subsequent analysis and evaluation. This approach not only improves the utilization of data, but also improves the overall performance of the system by further analyzing and improving the model or strategy.

[0069] S4: Use reinforcement learning algorithm to train Sim2Real model for field adaptation, with the goal of minimizing the weighted difference between simulation environment and real environment indicators, to obtain the final Sim2Real model.

[0070] Among them, reinforcement learning (RL) is a method of machine learning that aims to learn how to choose actions to maximize cumulative rewards through interaction with the environment. The core of reinforcement learning is to continuously optimize the policy through exploration and exploitation, so that the agent can make optimal decisions in different environments.

[0071] It should be noted that the method based on domain adaptation collects relevant data of the real environment, such as sensor readings, environmental layout and similar dynamic elements. These data are combined with the data in the simulation environment to form a more comprehensive training set, which is jointly added to the training process of the model, so that the model can be exposed to more diverse environmental features. Adjust the ROS topics sent by the real environment and the action data calculated by the policy network to make them compatible with the simulation environment data. Adjust the training logic so that the agent can receive both simulation and real environment data and update the policy model.

[0072] In one possible implementation, S4 specifically includes:

[0073] S401: Obtain the original ROS image message sent by the real environment.

[0074] Among them, ROS image (ROS Image) refers to a message format used in the Robot Operating System (ROS) to represent and transmit image data. ROS image is commonly used in robot vision systems, and image data captured by cameras or other sensors can be published, subscribed and processed in the ROS network.

[0075] S402: Convert the original ROS image message to OpenCV image format.

[0076] Among them, OpenCV (Open Source Computer Vision Library) is an open source computer vision library widely used in real-time image processing and computer vision tasks. It provides rich functions and tools to support image and video processing, analysis, recognition and other applications.

[0077] S403: Specify the data type of the converted image as 32-bit floating point number, and adjust the image size to 256 pixels in height and width.

[0078] S404: Store the original ROS image message in the observation data queue through the callback function.

[0079] Among them, the callback function (Callback Function) is a programming technique that executes a predefined function when an event occurs. Callback functions are a mechanism used for handling asynchronous operations, event-driven programming, and reactive programming. They are widely used in many programming languages and frameworks

[0080] S405: Determine whether the observation data queue is empty. If yes, continue the training process of the simulation environment data. Otherwise, take the real environment data from the observation data queue and use it together with the simulation data as a dataset to complete the update of the policy.

[0081] In the present application, the integration of real environment data into the training process helps to narrow the gap between simulation and reality. By including sensor readings and environmental layout information from the real environment in the training set, the model can learn the actual situation in the real environment and better cope with real challenges that cannot be fully simulated in the simulation environment.

[0082] S406: Obtain the integer action data through the calculation of the Actor-Critic network.

[0083] In the present application, through the calculation of the Actor-Critic network, the action data is transferred from the simulation environment to the real environment, which can ensure that the robot's action execution in the real environment is consistent with that in the simulation. This ensures that the policy learned by the model can be accurately implemented in the real environment, thereby reducing errors caused by action differences.

[0084] S407: Send the action data to the real environment to make the robot in the real environment execute the action data and update the environment state.

[0085] In the Sim2Real model construction method based on reinforcement learning, a joint training strategy is adopted. Specifically, the simulation environment can quickly provide a large amount of data for efficient training of reinforcement learning, while the real environment provides key correction data. By training the model on a large scale in the simulation environment and combining it with small-scale fine-tuning in the real environment, the model can be optimized in both areas, thereby better performing in the real world. We introduce the method of field adaptation to align the simulation and real state distributions, gradually approaching the real state, thereby reducing the difference in distribution.

[0086] The state space represents the current state of the system, which is used to quantify the difference between the simulation environment and the real environment. In this process, the state space includes multiple evaluation indicators that reflect the degree of difference between simulation and reality. Set the state space as S, which can be represented as:

[0087] S={E S ,E r}

[0088] where S denotes the state space, E S is the evaluation metric of the simulation environment, E r is the evaluation metric of the real environment.

[0089] The state space consists of the robot's velocity v s , angle θ s , sensor data d s in the simulation environment and the corresponding data (v r , θ r , d r ) in the real environment. The state space not only contains the properties of the robot in the simulation environment, but also includes the comparison results with the real world, quantifying the accuracy of the simulation model on different indicators.

[0090] The action space is designed to adjust the parameters in the simulation environment to narrow the gap between simulation and reality. Set the action space as A, denoted as:

[0091] A = {a1, a2, …, a m}

[0092] where A denotes the action space, a j denotes the jth action in the action space, j = 1, 2, …, m, and m denotes the total number of actions.

[0093] Each a j represents a specific parameter that can be adjusted, and by changing these parameters, the performance of the simulation environment is affected, thereby affecting the adaptability of the Sim2Real model to the real scene.

[0094] In the training of Sim2Real, the design of the reward function is related to the goal of minimizing the weighted difference between the simulation environment and the real environment. By introducing the time dimension to measure the difference between simulation and real environment at different times, it helps the model to adapt to the environment in the long-term dynamic changes, and the reward function is set as:

[0095]

[0096] where R denotes the reward function, T denotes the time span of the entire task, ω i denotes the weight of the ith indicator, n denotes the total number of indicators, E s (i, t) denotes the value of the ith indicator at time t in the simulation environment, E r (i, t) denotes the value of the ith indicator at time t in the real environment.

[0097] By introducing the dimension of time, the reward function can be measured based on the state difference at each time, and the cumulative value of the reward value through the reduction of the time difference makes the performance of the simulation environment gradually approach the real environment over time.

[0098] In the present application, the Sim2Real model is trained by reinforcement learning algorithm for domain adaptation, combined with real environment data and simulation environment data, which helps to improve the performance of the model in the real environment, reduce the gap between simulation and reality, enhance the generalization ability of the strategy, optimize the training efficiency and accuracy. At the same time, such training process can improve the robustness of the model and improve the effectiveness of the strategy model, providing a more reliable and efficient solution for the agent in practical application.

[0099] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0100] (1) In the present application, by obtaining the evaluation indexes of the simulation environment and the real environment, the differences between the two are determined, and the linear weighting method is used to quantify the weighted differences between these evaluation indexes. By building a Sim2Real model that can convert data between simulation environment and real environment, data-level adjustment is made for the differences between the two, reducing the transfer error from simulation environment to real environment, and effectively reflecting the complexity and dynamic changes of the real environment.

[0101] (2) In the present application, the reinforcement learning algorithm is used to train the Sim2Real model for domain adaptation with the minimum weighted difference between the simulation environment and the real environment indexes as the target, and the final Sim2Real model is obtained. By continuously optimizing the model to perform well in a wider range of environments, the adaptability of the model to various changes in reality is enhanced, and the generalization ability of the strategy is improved.

[0102] Reference is made to the accompanying drawings Figure 2 , which shows a structural schematic diagram of a Sim2Real model construction device based on reinforcement learning provided by the present application.

[0103] The present application also provides a Sim2Real model construction device 20 based on reinforcement learning, comprising:

[0104] The acquisition module 201 is configured to acquire evaluation indexes of the simulation environment and the real environment.

[0105] In one possible implementation, the evaluation indexes include a success weighted path length and a navigation success rate.

[0106] The success weighted path length is specifically:

[0107]

[0108] wherein SPL represents a success weighted path length, N represents a number of training nodes, S b represents a success or failure of navigation at a current node, L b represents an optimal shortest path length to a target point at a current node, P b represents a path length actually traveled by the robot in a test;

[0109] The navigation success rate is specifically:

[0110]

[0111] wherein Success represents a success rate, S represents a number of successful navigations, and T represents a ratio of a total number of navigation tests.

[0112] The quantification module 202 is configured to utilize a linear weighting method to quantify a weighted difference between the simulation environment and the real environment indicators according to the evaluation indicators.

[0113] In one possible implementation, the quantification of the weighted difference between the simulation environment and the real environment indicators is specifically:

[0114] Sim2Real Gap = ω1|SPL sim -SPL real |+ ω2|Success sim -Success real |

[0115] wherein Sim2Real Gap represents the weighted difference between the simulation environment and the real environment indicators, ω1 represents a weight coefficient of the success weighted path length, ω2 represents a weight coefficient of the success rate, SPL sim represents a success weighted path length in a simulation environment, SPL real represents a success weighted path length in a real environment, Success sim represents a navigation success rate in a simulation environment, Success real represents a navigation success rate in a real environment.

[0116] The building module 203 is configured to build a Sim2Real model for mutual conversion of data between the simulation environment and the real environment.

[0117] In one possible implementation, the building module 203 is configured to:

[0118] install Gazebo and Rviz in the Sim2Real model, and start a simulation environment of a TurtleBot3 robot.

[0119] a bridge node of the simulation environment to the Gazebo real environment is developed, and a data mutual conversion relationship between the simulation environment and the real environment is determined, the bridge node being configured to create a Ros node and subscribe and publish a topic;

[0120] the simulation-to-reality node is configured to convert discrete actions output in the simulation environment into continuous actions executable in the real environment;

[0121] the reality-to-simulation node is configured to convert sensor-related topic data in the real environment into topic data recognizable by the robot in the simulation environment;

[0122] a Ros node of the simulation environment is created to receive the topic data;

[0123] the topic data and sensor topic data in the simulation environment are fused, and a policy is updated to output an action instruction;

[0124] the action instruction is input into the simulation environment and the real environment, respectively, for training.

[0125] The training module 204 is configured to perform field self-adaptive training on the Sim2Real model by using a reinforcement learning algorithm, so that a weighted difference between the simulation environment and the real environment is minimized, and finally obtain the Sim2Real model.

[0126] In a possible implementation, the training module 204 is configured to:

[0127] obtain an original ROS image message sent by the real environment;

[0128] convert the original ROS image message into an image format of OpenCV;

[0129] specify a data type of the converted image as a 32-bit floating point number, and adjust an image size to be 256 pixels in height and width;

[0130] store the original ROS image message in an observation data queue through a callback function;

[0131] determine whether the observation data queue is empty; if yes, continue the training process of the simulation environment data; otherwise, take real environment data from the observation data queue and use the real environment data and the simulation data together as a data set to complete updating of the policy;

[0132] obtain integer action data through calculation of an Actor-Critic network;

[0133] send the action data to the real environment to cause a robot in the real environment to perform the action data and update the environment state.

[0134] The Sim2Real model construction device 20 based on reinforcement learning provided by the application can execute the Sim2Real model construction method based on reinforcement learning described above, and achieve the same or similar technical effects. To avoid repetition, the application will not be described again.

[0135] The technical scheme provided by the embodiments of the application has at least the following beneficial effects:

[0136] (1) In the application, by obtaining evaluation indexes of the simulation environment and the real environment, the differences between the two are determined, the weighted differences between the evaluation indexes are quantified by using a linear weighting method, a Sim2Real model capable of converting data between the simulation environment and the real environment is built, data-level adjustment is made for the differences between the two, the migration error from the simulation environment to the real environment is reduced, and the complexity and dynamic changes of the real environment are effectively reflected.

[0137] (2) In the application, the reinforcement learning algorithm is used to train the Sim2Real model in the field to adapt to the environment, and the final Sim2Real model is obtained. By continuously optimizing the model, it can perform well in a wider environment, and the adaptability of the model to various changes in reality is enhanced, and the generalization ability of the strategy is improved.

[0138] It should be understood that the processor in the embodiments of the application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0139] It should also be understood that the memory in the embodiments of the present application can be volatile or nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory can be random access memory (RAM) used as external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0140] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0141] It should be understood that the term "and / or" herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0142] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0143] It should be understood that in various embodiments of the present application, the size of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0144] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0146] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0147] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0148] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0149] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0150] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the Sim2Real model construction method based on reinforcement learning.

[0151] The computer readable storage medium provided by the present application can realize the steps and effects of the Sim2Real model construction method based on reinforcement learning of the above-mentioned method embodiment, and the present application will not be repeated here.

[0152] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0153] (1) In the present application, by obtaining the evaluation indexes of the simulation environment and the real environment, the differences between the two are determined, the weighted differences between the evaluation indexes are quantified by using the linear weighting method, a Sim2Real model capable of converting data between the simulation environment and the real environment is built, the data level is adjusted according to the differences between the two, the migration error from the simulation environment to the real environment is reduced, and the complexity and dynamic changes of the real environment are effectively reflected.

[0154] (2) In the present application, the reinforcement learning algorithm is used to train the Sim2Real model in the field to obtain the final Sim2Real model, with the minimum weighted difference between the simulation environment and the real environment as the target. The model is continuously optimized to perform well in a wider environment, enhancing the adaptability of the model to various changes in reality and improving the generalization ability of the strategy.

[0155] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0156] The following points need to be explained:

[0157] (1) The attached drawings of the embodiments of the present application only involve the structures involved in the embodiments of the present application, and other structures can be referred to the general design.

[0158] (2) In order to be clear, the thickness of the layer or region is enlarged or reduced in the drawings used for describing the embodiments of the present application, that is, the drawings are not drawn according to the actual proportion. It can be understood that when an element such as a layer, a film, a region or a substrate is referred to as being located "on" or "under" another element, the element can be "directly" located on or under another element or there can be an intermediate element.

[0159] (3) In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other to obtain new embodiments.

[0160] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A Sim2Real model construction method based on reinforcement learning, characterized in that, Comprise: S1: obtaining evaluation indexes of simulation environment and real environment; S2: using linear weighting method, quantifying the weighted difference between simulation environment and real environment indexes according to the evaluation indexes; S3: building Sim2Real model for mutual conversion of data between simulation environment and real environment; S4: using reinforcement learning algorithm, taking the minimum weighted difference between simulation environment and real environment indexes as the goal, training the Sim2Real model in the field to obtain the final Sim2Real model; Wherein, the weighted difference between simulation environment and real environment indexes quantified in S2 is: ； wherein, represents a weighted difference between the simulation environment and the real environment indicator, ω 1 represents a weight coefficient of the success weighted path length, ω 2 represents a weight coefficient of the success rate, represents a success weighted path length in the simulation environment, represents a success weighted path length in the real environment, represents a success rate, represents a navigation success rate in the simulation environment, represents a navigation success rate in the real environment; Wherein, in the training of the Sim2Real model, the reward function is set as: ； wherein, R represents a reward function, T represents a time span of the entire task, ω i represents a weight of the i th metric, n represents a total number of metrics, represents a value of the t th metric at the i th time point in the simulation environment, represents a value of the t th metric at the i th time point in the real environment.

2. The reinforcement learning based Sim2Real model building method according to claim 1, wherein, The evaluation indexes include: successful weighted path length and navigation success rate; The successful weighted path length is specifically: ； wherein, SPL represents the successful weighted path length, N represents the number of trained nodes, S b represents the success or failure of navigation at the current node number, L b represents the optimal shortest path length to reach the target point at the current node number, P b represents the path length actually traveled by the robot in the test; The navigation success rate is specifically: ； wherein, represents the success rate, S represents the number of navigation successes, T represents the ratio of the total number of navigation tests. 3.The reinforcement learning based Sim2Real model construction method of claim 1, wherein, The S3 specifically includes: S301: install Gazebo and Rviz in the Sim2Real model, and start the simulation environment of TurtleBot3 robot; S302: develop the bridge node of the simulation environment to Gazebo real environment, determine the data mutual conversion relationship between simulation environment and real environment, and the bridge node is used to create Ros node and subscribe and publish topics; S303: the node from simulation to reality is responsible for converting the discrete action output in the simulation environment into continuous action executable in the real environment; S304: the node from reality to simulation converts the sensor related topic data in the real environment into topic data recognizable by the robot in the simulation environment; S305: create simulation environment Ros node to receive the topic data; S306: fuse the topic data and the topic data of the sensor in the simulation environment, update the strategy, and output action instruction; S307: input the action instruction into the simulation environment and the real environment respectively for training. 4.The reinforcement learning based Sim2Real model construction method of claim 1, wherein, S4 specifically includes: S401: obtaining the original ROS image message sent by the real environment; S402: converting the original ROS image message into OpenCV image format; S403: specifying the data type of the converted image as 32-bit floating point number, and adjusting the image size to 256 pixels in height and width; S404: storing the original ROS image message in the observation data queue through callback function; S405: judging whether the observation data queue is empty; if yes, continue the training process of simulation environment data; otherwise, taking out the real environment data from the observation data queue and using it together with the simulation data as data set to complete the update of strategy; S406: obtaining integer action data through Actor-Critic network calculation; S407: sending the action data to the real environment to make the robot in the real environment execute the action data and update the environment state. 5.A Sim2Real model construction apparatus based on reinforcement learning, characterized in that, Comprise: An acquisition module is configured to acquire evaluation indexes of simulation environment and real environment; a quantification module configured to quantize the weighted difference between the simulation environment and the real environment according to the evaluation index by using a linear weighting method; a building module configured to build a Sim2Real model for converting data between the simulation environment and the real environment; a training module configured to perform field self-adaptive training on the Sim2Real model by using a reinforcement learning algorithm, so as to minimize the weighted difference between the simulation environment and the real environment, and obtain a final Sim2Real model; wherein the quantification of the weighted difference between the simulation environment and the real environment in the quantification module is specifically as follows: ； wherein, represents a weighted difference between the simulation environment and the real environment indicator, ω 1 represents a weight coefficient of the success weighted path length, ω 2 represents a weight coefficient of the success rate, represents a success weighted path length in the simulation environment, represents a success weighted path length in the real environment, represents a success rate, represents a navigation success rate in the simulation environment, represents a navigation success rate in the real environment; wherein, in the training of the Sim2Real model, a reward function is set as follows: ； in, R represents the reward function, T Indicates the time span of the entire task, ω i Indicates the i The weight of the indicator, n Indicates the total number of indicators, In the simulation environment t Moment i The value of the indicator, In real environment t Moment i The value of an indicator.

6. The reinforcement learning based Sim2Real model building apparatus according to claim 5, wherein, The evaluation index includes a successful weighted path length and a navigation success rate. The successful weighted path length is specifically as follows: ； in, SPL represents the success weighted path length, N Indicates the number of training sessions, S b Indicates whether navigation succeeded or failed under the current section number. L b Indicates the optimal shortest path length to the target point under the current number of nodes, P b Indicates the length of the path traveled by the robot in the actual test; The navigation success rate is specifically as follows: ； wherein, represents the success rate, S represents the number of navigation successes, T represents the ratio of the total number of navigation tests.

7. The reinforcement learning based Sim2Real model building apparatus according to claim 5, wherein, The building module is configured to: install Gazebo and Rviz in the Sim2Real model, and start a simulation environment of a TurtleBot3 robot; develop a bridge node of the simulation environment to a Gazebo real environment, determine a data mutual conversion relationship between the simulation environment and the real environment, and create a Ros node and subscribe and publish a topic by using the bridge node; a simulation-to-reality node is responsible for converting discrete actions output in the simulation environment into continuous actions executable in the real environment; a reality-to-simulation node converts sensor-related topic data in the real environment into topic data recognizable by a robot in the simulation environment; create a simulation environment Ros node to receive the topic data; fuse the topic data and sensor topic data in the simulation environment, update a strategy, and output an action instruction; input the action instruction into the simulation environment and the real environment respectively for training.

8. The reinforcement learning based Sim2Real model building apparatus according to claim 5, wherein, The training module is configured to: obtain an original ROS image message sent by the real environment; convert the original ROS image message into an image format of OpenCV; specify a data type of the converted image as a 32-bit floating point number, and adjust an image size to be 256 pixels in height and width; store the original ROS image message in an observation data queue through a callback function; determine whether the observation data queue is empty; if yes, continue the training process of the simulation environment data; otherwise, take out real environment data from the observation data queue, and update a strategy together with simulation data as a data set; obtain integer action data through calculation of an Actor-Critic network; send the action data to the real environment, so that a robot in the real environment executes the action data, and updates an environment state.

Images